Python爬⾍简单实例——⾖瓣电影评论数据的爬取
⼀、前⾔
⾖瓣⽹是⼀家基于⽤户对于图书、电影和⾳乐兴趣⽽搭建的社交⽹站，由杨勃创⽴于2005年。⾖瓣⽹推崇算法，根据⽤户对⾳乐、书、电影等进⾏的操作，⾃动给出同类趣味和友邻推荐。基于记录和分享⽽
⽣成的“⽤户价值”是⾖瓣的核⼼竞争⼒。⾖瓣⽹的⼀⼤特⾊是不做运营，⽤户⾃发组建的⼩组是⾖瓣特有的⽂化和社区产物。
爬取⾖瓣⽹上⾯的电影评论数据具有很重要的作⽤。电影评论数据是NLP（⾃然语⾔处理）重要的数据。通过电影评论数据集可以进⼀步做中⽂分词、命名实体识别、关键词提取、句法分析、⽂本向量化、情感分析、舆情分析等进⼀步的数据处理和应⽤。既然数据这么重要，废话不多说，紧接着就开始电影数据的爬取。
⼆、代码及结果分析展⽰
这次选取的实例电影是《我不是药神》：它是由⽂牧野执导，宁浩、徐峥共同监制的剧情⽚，徐峥、周⼀围、王传君、谭卓、章宇、杨新鸣等主演 。该⽚于2018年7⽉5⽇在中国上映。影⽚讲述了神油店⽼板程勇从⼀个交不起房租的男性保健品商贩，⼀跃成为印度仿制药“格列宁”独家代理商的故事，这⾥先放⼀张海报。
紧接着查看《我不是药神》的⾖瓣评论。还是⽼样⼦F12—>NetWork—>XHR，我们发现这⾥⾯竟然没有xhr⽂件，谢天谢地，这说明这些评论数据都不是通过Ajax来传送数据的。这样⼯作就简单太多了---------直接通过寻URL的规律爬取每页的数据并保存。
那就直接来吧：
这样规律就到了，只需要通过⼀个变量改变start的值就可以。
接下来直接给出代码：
# -*-coding:utf-8-*-
quest
from bs4 import BeautifulSoup
import random
import time
import jieba
import wordcloud
def getHtml(url):
"""获取url页⾯"""
user_agents = [
'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36 OPR/26.0.1656.60',
'Opera/8.0 (Windows NT 5.1; U; en)',
'Mozilla/5.0 (Windows NT 5.1; U; en; rv:1.8.1) Gecko/20061208 Firefox/2.0.0 Opera 9.50',
'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; en) Opera 9.50',
'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:34.0) Gecko/20100101 Firefox/34.0',
'Mozilla/5.0 (X11; U; Linux x86_64; zh-CN; rv:1.9.2.10) Gecko/20100922 Ubuntu/10.10 (maverick) Firefox/3.6.10',
'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2 ',
'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36',
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11',
'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.133 Safari/534.16',
'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36',
'Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko',
'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.11 TaoBrowser/2.0 Safari/536.11',
'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.71 Safari/537.1 LBBROWSER',
'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E)',
'Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.84 Safari/535.11 SE 2.X MetaSr 1.0',
'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; SV1; QQDownload 732; .NET4.0C; .NET4.0E; SE 2.X MetaSr 1.0) ',
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; AcooBrowser; .NET CLR 1.1.4322; .NET CLR 2.0.50727)",
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Acoo Browser; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)",
"Mozilla/4.0 (compatible; MSIE 7.0; AOL 9.5; AOLBuild 4337.35; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)",
"Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)",
"Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 2.0.50727; Media Center PC "Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729 "Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 5.2; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.2; .NET CLR 3.0.04506.30)",
"Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN) AppleWebKit/523.15 (KHTML, like Gecko, Safari/419.3) Arora/0.3 (Change: 287 c9dfb30)",
"Mozilla/5.0 (X11; U; Linux; en-US) AppleWebKit/527+ (KHTML, like Gecko, Safari/419.3) Arora/0.6",
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.2pre) Gecko/20070215 K-Ninja/2.1.1",
"Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9) Gecko/20080705 Firefox/3.0 Kapiko/3.0",
"Mozilla/5.0 (X11; Linux i686; U;) Gecko/20070322 Kazehakase/0.4.5",
"Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko Fedora/1.9.0.8-1.fc10 Kazehakase/0.5.6",
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/535.20 (KHTML, like Gecko) Chrome/19.0.1036.7 Safari/535.20",
"Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; fr) Presto/2.9.168 Version/11.52" ]
python 爬虫教学headers = {
'Cookie': 你的cookie,
'User-Agent': str(random.choice(user_agents)),
'Referer': 'https: // movie.douban / subject / 26752088 / comments?status = P', 'Connection': 'keep-alive'
}
req = quest.Request(url,headers=headers)
req = quest.urlopen(req)
content = ad().decode('utf-8')
return content
def getComment(url):
"""解析HTML页⾯"""
html = getHtml(url)
soupComment = BeautifulSoup(html, 'html.parser')
comments = soupComment.findAll('span', 'short')
onePageComments = []
for comment in comments:
# Text()+'\n')
onePageComments.Text()+'\n')
return onePageComments
def wordAnalysis():
f = open('
C:/Users/Administrator/PycharmProjects/practice1/我不是药神.txt','r',encoding = 'utf-8')
content = f.read()
f.close()
ls = jieba.lcut(content)
txt = ' '.join(ls)
w = wordcloud.WordCloud(font_path='c:\windows\Fonts\STZHONGS.TTF', width=1000, height=700, background_color='white') w.generate(txt)
<_file('Movie.png')
if __name__ == '__main__':
f = open('我不是药神.txt', 'a', encoding='utf-8')
j = 0
for page in range(15): # ⾖瓣爬取多页评论需要验证。
url = 'movie.douban/subject/26752088/comments?start=' + str(20*page) + '&limit=20&sort=new_score&status=P' print('第%s页的评论:' % (page))
print(url + '\n')
for i in getComment(url):
f.write(str(j))
f.write(i)
print(j,i)
j += 1
time.sleep(10)
print('\n')
wordAnalysis()
词云结果如下图所⽰：