用Python實(shí)現(xiàn)網(wǎng)絡(luò)爬蟲、蜘蛛

《用Python實(shí)現(xiàn)網(wǎng)絡(luò)爬蟲、蜘蛛》由會(huì)員分享，可在線閱讀，更多相關(guān)《用Python實(shí)現(xiàn)網(wǎng)絡(luò)爬蟲、蜘蛛（115頁珍藏版）》請(qǐng)?jiān)谘b配圖網(wǎng)上搜索。

1、python 中如何提取網(wǎng)頁正文啊 謝謝 import urllib.request ? url=" ? response=urllib.request.urlopen(url) ? page=response.read() ? python提取網(wǎng)頁中的文本 1. import os,sys,datetime??? 2. import httplib,urllib, re??? 3. from sgmllib import SGMLParser??? 4. ?? 5. import types??? 6. ?? 7. class Html2txt(

2、SGMLParser):??? 8. ????def reset(self):??? 9. ????????self.text = ''?? 10. ????????self.inbody = True?? 11. ???????? SGMLParser.reset(self)??? 12. ????def handle_data(self,text):??? 13. ????????if self.inbody:??? 14. ????????????self.text += text??? 15. ?? 16. ????def start_head(se

3、lf,text):??? 17. ????????self.inbody = False?? 18. ????def end_head(self):??? 19. ????????self.inbody = True?? 20. ?? 21. ?? 22. if __name__ == "__main__":??? 23. ???? parser = Html2txt()??? 24. ???? parser.feed(urllib.urlopen("").read())??? 25. ???? parser.close()??? 26. ????pri

4、nt parser.text.strip()?? python 下載網(wǎng)頁 import httplib?? conn=httplib.HTTPConnection("") conn.request("GET","/index.html") r1=conn.getresponse() print r1.status,r1.reason data=r1.read() print data conn.close 用python下載網(wǎng)頁，超級(jí)簡(jiǎn)單！ from urllib import urlopen webdata = urlopen("").read() print w

5、ebdata 深入python里面有 python?下載網(wǎng)頁內(nèi)容,用python的pycurl模塊實(shí)現(xiàn) 1. 用python 下載網(wǎng)頁內(nèi)容還是很不錯(cuò)的，之前是使用urllib模塊實(shí)驗(yàn)的，但聽說有pycurl這個(gè)模塊，而且比urllib好，所以嘗試下，廢話不說，以下是代碼 2. 3. 4. #!/usr/bin/env python 5. # -*- coding: utf-8 -*- 6. import StringIO 7. import pycurl 8. 9. def writefile(fstr,xfilename): ? f=open(xfilename,'w

6、') ? f.write(fstr) ? f.close 10. 1. html = StringIO.StringIO() 2. c = pycurl.Curl() 3. myurl='' 4. ? 5. c.setopt(pycurl.URL, myurl) 6. ? 7. #寫的回調(diào) 8. c.setopt(pycurl.WRITEFUNCTION, html.write) 9. ? 10. c.setopt(pycurl.FOLLOWLOCATION, 1) 11. ? 12. #最大重定向次數(shù),可以預(yù)防重定向陷阱 13. c.setopt(pycurl

7、.MAXREDIRS, 5) 14. ? 15. #連接超時(shí)設(shè)置 16. c.setopt(pycurl.CONNECTTIMEOUT, 60) 17. c.setopt(pycurl.TIMEOUT, 300) 18. ? 19. #模擬瀏覽器 20. c.setopt(pycurl.USERAGENT, "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)") 21. ? 22. ? 23. ? 24. #訪問,阻塞到訪問結(jié)束 25. c.perform() 26.

8、? 27. #打印出 200(HTTP狀態(tài)碼，可以不需要) 28. print c.getinfo(pycurl.HTTP_CODE) 29. ? 30. #輸出網(wǎng)頁的內(nèi)容 31. print html.getvalue() 32. #保存成down.txt文件 33. writefile(html.getvalue(),"down.txt") python的pycurl模塊的安裝可以到 不同系統(tǒng)使用不同版本，自己看看 總結(jié)下，Python 下載網(wǎng)頁的幾種方法 1 fd = urllib2.urlopen(url_link) data = fd.r

9、ead() 這是最簡(jiǎn)潔的一種，當(dāng)然也是Get的方法 2 通過GET的方法 def GetHtmlSource(url): try: htmSource = '' req = urllib2.Request(url) fd = urllib2.urlopen(req,"") while 1: data = fd.read(1024) if not len(data)

10、: break htmSource += data fd.close() del fd del req htmSource = htmSource.decode('cp936') htmSource = formatStr(htmSource) return htmSource except socket.er

11、ror, err: str_err = "%s" % err return "" 3 通過GET的方法 def GetHtmlSource_Get(htmurl): htmSource = "" try: urlx = httplib.urlsplit(htmurl) conn = httplib.HTTPConnection(loc) conn.connect() conn.putrequ

12、est("GET", htmurl, None) conn.putheader("Content-Length", 0) conn.putheader("Connection", "close") conn.endheaders() res = conn.getresponse() htmSource = res.read() except Exception(), err: trackback.print_exec(

13、) conn.close() return htmSource 通過POST的方法 def GetHtmlSource_Post(getString): htmSource = "" try: url = httplib.urlsplit(":8080") conn = httplib.HTTPConnection(loc) conn.connect() conn.putrequest("POST

14、", "/sipo/zljs/hyjs-jieguo.jsp") conn.putheader("Content-Length", len(getString)) conn.putheader("Content-Type", "application/x-www-form-urlencoded") conn.putheader("Connection", " Keep-Alive") conn.endheaders() conn.send(getString) f = conn.ge

15、tresponse() if not f: raise socket.error, "timed out" htmSource = f.read() f.close() conn.close() return htmSource except Exception(), err: trackback.print_exec() conn.close(

16、) return htmSource 本文來自CSDN博客，轉(zhuǎn)載請(qǐng)標(biāo)明出處： Django+python+BeautifulSoup組合的垂直搜索爬蟲 使用python+BeautifulSoup完成爬蟲抓取特定數(shù)據(jù)的工作，并使用Django搭建一個(gè)管理平臺(tái)，用來協(xié)調(diào)抓取工作。 因?yàn)樽约汉芟矚gDjango admin后臺(tái)，所以這次用這個(gè)后臺(tái)對(duì)抓取到的鏈接進(jìn)行管理，使我的爬蟲可以應(yīng)對(duì)各種后期的需求。比如分時(shí)段抓取，定期的對(duì)已經(jīng)抓取的地址重新抓取。數(shù)據(jù)庫是用python自帶的sqlite3，所以很方便。 這幾天正好在做

17、一個(gè)電影推薦系統(tǒng)，需要些電影數(shù)據(jù)。本文的例子是對(duì)豆瓣電影抓取特定的數(shù)據(jù)。 第一步：建立Django模型 模仿nutch的爬蟲思路，這里簡(jiǎn)化了。每次抓取任務(wù)開始先從數(shù)據(jù)庫里找到未保存的(is_save = False)的鏈接，放到抓取鏈表里。你也可以根據(jù)自己的需求去過濾鏈接。 python代碼： view plaincopy to clipboardprint? 01.class Crawl_URL(models.Model): 02. url = models.URLField('抓取地址',max_length=100, unique=Tr

18、ue) 03. weight = models.SmallIntegerField('抓取深度',default = 0)#抓取深度起始1 04. is_save = models.BooleanField('是否已保存',default= False)# 05. date = models.DateTimeField('保存時(shí)間',auto_now_add=True,blank=True,null=True) 06. def __unicode__(self): 07. return self.url cla

19、ss Crawl_URL(models.Model): url = models.URLField('抓取地址',max_length=100, unique=True) weight = models.SmallIntegerField('抓取深度',default = 0)#抓取深度起始1 is_save = models.BooleanField('是否已保存',default= False)# date = models.DateTimeField('保存時(shí)間',auto_now_add=True,blank=True,null=True)

20、 def __unicode__(self): return self.url 然后生成相應(yīng)的表。 還需要一個(gè)admin管理后臺(tái) view plaincopy to clipboardprint? 01.class Crawl_URLAdmin(admin.ModelAdmin): 02. list_display = ('url','weight','is_save','date',) 03. ordering = ('-id',) 04. list_filter = ('is_save','weight'

21、,'date',) 05. fields = ('url','weight','is_save',) 06.admin.site.register(Crawl_URL, Crawl_URLAdmin) class Crawl_URLAdmin(admin.ModelAdmin): list_display = ('url','weight','is_save','date',) ordering = ('-id',) list_filter = ('is_save','weight','date',) fields = ('u

22、rl','weight','is_save',) admin.site.register(Crawl_URL, Crawl_URLAdmin) 第二步，編寫爬蟲代碼 爬蟲是單線程，并且每次抓取后都有相應(yīng)的暫定，豆瓣網(wǎng)會(huì)禁止一定強(qiáng)度抓取的爬蟲 爬蟲根據(jù)深度來控制，每次都是先生成鏈接，然后抓取，并解析出更多的鏈接，最后將抓取過的鏈接is_save=true，并把新鏈接存入數(shù)據(jù)庫中。每次一個(gè)深度抓取完后都需要花比較長(zhǎng)的時(shí)候把鏈接導(dǎo)入數(shù)據(jù)庫。因?yàn)樾枰袛噫溄邮欠褚汛嫒霐?shù)據(jù)庫。 這個(gè)只對(duì)滿足正則表達(dá)式 的地址進(jìn)行數(shù)據(jù)解析。并且直接忽略掉不是電影模塊的鏈接。 第一次抓取

23、需要在后臺(tái)加個(gè)鏈接，比如 python代碼： #這段代碼不能格式化發(fā) # coding=UTF-8 import urllib2 from BeautifulSoup import * from urlparse import urljoin from pysqlite2 import dbapi2 as sqlite from movie.models import * from django.contrib.auth.models import User from time import sleep image_path = 'C:/Users/s

24、oul/djcodetest/picture/' user = User.objects.get(id=1) def crawl(depth=10): for i in range(1,depth): print '開始抓取 for %d....'%i pages = Crawl_URL.objects.filter(is_save=False) newurls={} for crawl_page in pages: page = crawl_page.url

25、 try: c=urllib2.urlopen(page) except: continue try: #解析元數(shù)據(jù)和url soup=BeautifulSoup(c.read()) #解析電影頁面 if re.search(r'^ read_html(soup)

26、 #解析出有效的鏈接，放入newurls links=soup('a') for link in links: if 'href' in dict(link.attrs): url=urljoin(page,link['href']) if url.find("'")!=-1: continue

27、 if len(url) > 60: continue url=url.split('#')[0] # removie location portion if re.search(r'^', url): newurls[url]= crawl_page.weight + 1 #連接有效。存入字典中 try: print 'add url :

28、' except: pass except Exception.args: try: print "Could not parse : %s" % args except: pass #newurls存入數(shù)據(jù)庫 is_save=False weight=i

29、 crawl_page.is_save = True crawl_page.save() #休眠2.5秒 sleep(2.5) save_url(newurls) #保存url，放到數(shù)據(jù)庫里 def save_url(newurls): for (url,weight) in newurls.items(): url = Crawl_URL(url=url,weight=weight) try:

30、 url.save() except: try: print 'url重復(fù):' except: pass return True 第三步，用BeautifulSoup解析頁面 抽取出電影標(biāo)題，圖片，劇情介紹，主演，標(biāo)簽，地區(qū)。關(guān)于BeautifulSoup的使用可以看這里BeautifulSoup技術(shù)文檔 view plaincopy to clipboardprint?01.#抓取數(shù)據(jù) 02.def re

31、ad_html(soup): 03. #解析出標(biāo)題 04. html_title = soup.html.head.title.string 05. title = html_title[:len(html_title)-5] 06. #解析出電影介紹 07. try: 08. intro = soup.find('span',attrs={'class':'all hidden'}).text 09. except: 10. try: 11. node = soup.fi

32、nd('div',attrs={'class':'blank20'}).previousSibling 12. intro = node.contents[0]+node.contents[2] 13. except: 14. try: 15. contents = soup.find('div',attrs={'class':'blank20'}).previousSibling.previousSibling.text 16. intro = conte

33、nts[:len(contents)-22] 17. except: 18. intro = u'暫無' 19. 20. #取得圖片 21. html_image = soup('a',href=pile(' 22. data = urllib2.urlopen(html_image).read() 23. image = '201003/'+html_image[html_image.rfind('/')+1:] 24. f = file(image_path+image,

34、'wb') 25. f.write(data) 26. f.close() 27. 28. 29. #解析出地區(qū) 30. try: 31. soup_obmo = soup.find('div',attrs={'class':'obmo'}).findAll('span') 32. html_area = soup_obmo[0].nextSibling.split('/') 33. area = html_area[0].lstrip() 34. excep

35、t: 35. area = '' 36. 37. #time = soup_obmo[1].nextSibling.split(' ')[1] 38. #time = time.strptime(html_time,'%Y-%m-%d') 39. 40. #生成電影對(duì)象 41. new_movie = Movie(title=title,intro=intro,area=area,version='暫無',upload_user=user,image=image) 42. new_movie.save()

36、 43. try: 44. actors = soup.find('div',attrs={'id':'info'}).findAll('span')[5].nextSibling.nextSibling.string.split(' ')[0] 45. actors_list = Actor.objects.filter(name = actors) 46. if len(actors_list) == 1: 47. actor = actors_list[0] 48. new

37、_movie.actors.add(actor) 49. else: 50. actor = Actor(name=actors) 51. actor.save() 52. new_movie.actors.add(actor) 53. except: 54. pass 55. 56. #tag 57. tags = soup.find('div',attrs={'class':'blank20'}).findAll('a') 5

38、8. for tag_html in tags: 59. tag_str = tag_html.string 60. if len(tag_str) > 4: 61. continue 62. tag_list = Tag.objects.filter(name = tag_str) 63. if len(tag_list) == 1: 64. tag = tag_list[0] 65. 66. new_movie

39、.tags.add(tag) 67. else: 68. tag = Tag(name=tag_str) 69. tag.save() 70. new_movie.tags.add(tag) 71. #try: 72. 73. #except Exception.args: 74. # print "Could not download : %s" % args 75. print r'download success' 76.

40、 #抓取數(shù)據(jù) def read_html(soup): #解析出標(biāo)題 title = html_title[:len(html_title)-5] #解析出電影介紹 try: intro = soup.find('span',attrs={'class':'all hidden'}).text except: try: node = soup.find('div',attrs={'class':'blank20'}).previousSibling

41、 intro = node.contents[0]+node.contents[2] except: try: intro = contents[:len(contents)-22] except: intro = u'暫無' #取得圖片 html_image = soup('a',href=pile(' data = urllib2.urlopen(html_image).read() image

42、 = '201003/'+html_image[html_image.rfind('/')+1:] f = file(image_path+image,'wb') f.write(data) f.close() #解析出地區(qū) try: soup_obmo = soup.find('div',attrs={'class':'obmo'}).findAll('span') html_area = soup_obmo[0].nextSibling.split('/')

43、 area = html_area[0].lstrip() except: area = '' #time = soup_obmo[1].nextSibling.split(' ')[1] #time = time.strptime(html_time,'%Y-%m-%d') #生成電影對(duì)象 new_movie = Movie(title=title,intro=intro,area=area,version='暫無',upload_user=user,image=image) new_mo

44、vie.save() try: actors = soup.find('div',attrs={'id':'info'}).findAll('span')[5].nextSibling.nextSibling.string.split(' ')[0] actors_list = Actor.objects.filter(name = actors) if len(actors_list) == 1: actor = actors_list[0] new_movie.actors.

45、add(actor) else: actor = Actor(name=actors) actor.save() new_movie.actors.add(actor) except: pass #tag tags = soup.find('div',attrs={'class':'blank20'}).findAll('a') for tag_html in tags: tag_str = tag_ht

46、ml.string if len(tag_str) > 4: continue tag_list = Tag.objects.filter(name = tag_str) if len(tag_list) == 1: tag = tag_list[0] new_movie.tags.add(tag) else: tag = Tag(name=tag_str) tag.

47、save() new_movie.tags.add(tag) #try: #except Exception.args: # print "Could not download : %s" % args print r'download success' 豆瓣的電影頁面并不是很對(duì)稱，所以有時(shí)候抓取的結(jié)果可能會(huì)有點(diǎn)出入 本文來自CSDN博客，轉(zhuǎn)載請(qǐng)標(biāo)明出處： 爬Google的查詢頁 最近沒有沒有Google API key了，因此只能自己將查詢對(duì)應(yīng)的URL準(zhǔn)備好，然

48、后通過腳本將該鏈接對(duì)應(yīng)的網(wǎng)頁爬下來。我們假定，我們要爬這樣一個(gè)頁面： 我們可以直接在瀏覽器輸入上面的URL，可以看到，是Google對(duì)應(yīng)IBM這個(gè)查詢的返回頁。我們現(xiàn)在的目的是通過python程序把這個(gè)返回頁download下來，存在本地，為后面的工作準(zhǔn)備數(shù)據(jù)。 一下就是我的代碼： import urllib2 , urlparse , gzip from StringIO import StringIO USER_AGENT = 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CL

49、R 2.0.50727; CIBA; InfoPath.2; 360SE)' class SmartRedirectHandler(urllib2.HTTPRedirectHandler): def http_error_301(self , req , fp , code , msg , headers): result = urllib2.HTTPRedirectHandler.http_error_301( self , req , fp , code , msg , headers) result.s

50、tatus = code return result def http_error_302(self , req , fp , code , msg , headers): result = urllib2.HTTPRedirectHandler.http_error_302( self , req , fp , code , msg , headers) result.status = code return result class DefaultErrorHandler(ur

51、llib2.HTTPDefaultErrorHandler): def http_error_default(self , req , fp , code , msg , headers): result = urllib2.HTTPError(req.get_full_url() , code , msg , headers , fp) result.status = code return result def openAnything(source , etag = None , lastmodified = N

52、one , agent= USER_AGENT): if hasattr(source , 'read'): return source if source == '-' : return sys.stdin if urlparse.urlparse(source)[0] == 'http' : request = urllib2.Request(source) request.add_header('USER-AGENT', agent) requ

53、est.add_header('Cookie', 'PREF=ID=5e8d1d15fe369be8:U=95d9eb627acc6c79:LD=en:NW=1:CR=2:TM=1270616770:LM=1270616914:S=FAiBW5rEW2azKJXk; NID=33=rXimIAwO_TvEyFlE4lBRkxr1x3TTVh36maim2Cn0gk3b3SAbtn79qkAtgIli18d382TnTCFMOXjzgqxQFCEWLHEbnyf-MtVwfa4-pYmXSMkUMPYqDi61ZmmqyPcBbwzP') if etag :

54、 request.add_header('If-None-Match', etag) if lastmodified : request.add_header('If-Modified-Since', lastmodified) # request.add_header('Accept-encoding', 'gzip') # request.add_header('Connection', 'Keep-Alive') opener = urllib2.build_opener(SmartRe

55、directHandler() , DefaultErrorHandler()) return opener.open(request) try : return open(source) except (IOError , OSError) : pass return StringIO(str(sources)) def fetch(source , etag = None , last_modified = None , agent =USER_AGENT): re

56、sult = {} f = openAnything(source , etag , last_modified , agent) result['data'] = f.read() if hasattr(f , 'headers'): result['etag'] = f.headers.get('ETag') result['lastmodified'] = f.headers.get('Last-Modified') if f.headers.get('content-encoding' , '')=='

57、gzip' : result['date']=gzip.GzipFile(fileobj = StringIO(result['data']) ).read() if hasattr(f , 'url') : result['url'] = f.url result['status'] = 200 if hasattr(f , 'status') : result['status'] = f.status f.close() return result if __na

58、me__=='__main__' : result = fetch(" print result['data'] print result['url'] print result['status'] 要注意一下幾點(diǎn)： 1、要加User-agent，否則GOOGLE會(huì)返回403的forbidden信息，因?yàn)镻ython默認(rèn)的User-agent是python........ 2、如果不想被重定向到Google 中國，哦，NO，是Google 香港，那么要記得在自己的cookie中找到對(duì)應(yīng)的cookie加上去 3、這種東西要多

59、抓包，一個(gè)字節(jié)一個(gè)字節(jié)的比對(duì)，要相信，瀏覽器也是程序?qū)懙?，瀏覽器能做到的程序也能做到 本文來自CSDN博客，轉(zhuǎn)載請(qǐng)標(biāo)明出處： 下載糗事百科的內(nèi)容_python版 1. #coding:utf-82.3.import urllib.request 5.import sqlite3 6.import threading 7.import time 8.9.class logger(object): 10. def log(self,*msg): 11. for i in msg: 12. print(i) 13.14.

60、Log = logger() 15.Log.log('測(cè)試下') 16.17.class downloader(object): 18. 19. def __init__(self,url): 20. self.url = url 21. 22. def download(self): 23. Log.log('開始下載',self.url) 24. try: 25. content = urllib.request.urlopen(self.url).read() 26.

61、 #req = urllib.request.Request(url)27. #response = urllib.request.urlopen(req)28. #content = response.read()29. Log.log('下載完畢') 30. return(content) 31. except: 32. Log.log('下載出錯(cuò)') 33. return(None) 34. 35.

62、 36.class parser(object): 37. 38. def __init__(self,content): 39. #獲得根節(jié)點(diǎn)40. self.html = xml.dom.minidom.parseString(content) 41. 42. def parse(self): 43. Log.log('開始提取數(shù)據(jù)') 44. contents = {'content':'','url':[]} 45. #獲得div節(jié)點(diǎn)46.

63、 divs = self.html.getElementsByTagName('div') 47. #獲得content節(jié)點(diǎn)48. for div in divs: 49. if div.hasAttribute('class') and \ 50. div.getAttribute('class') == 'content': 51. #獲得糗事百科的內(nèi)容52. textNode = div.childNodes[0] 53.

64、 qContent = textNode.data 54. #數(shù)據(jù)填充55. contents['content'] = qContent 56. 57. #獲得上一糗事、下一糗事節(jié)點(diǎn)58. spans = self.html.getElementsByTagName('span') 59. for span in spans: 60. pspan = span.parentNode 61. if pspan.tag

65、Name == 'a': 62. #pspan為對(duì)應(yīng)的鏈接,此時(shí)需要將對(duì)應(yīng)的地址加入數(shù)據(jù)庫63. url = pspan.getAttribute('href') 64. qid = url[10:][:-4] 65. #數(shù)據(jù)填充66. contents['url'].append(qid) 67. Log.log('提取數(shù)據(jù)完畢') 68. return(contents) 69.70.def downloadPage(qid,db): 71. url = ' content = downloader(url).