网络爬虫抓取心得体会（网络爬虫之用户名密码及验证码登陆）

使用requests包来爬取首先尝试用用户名密码自动登陆，如果失败，则需要采用cookie登陆，今天小编就来说说关于网络爬虫抓取心得体会?下面更多详细答案一起来看看吧!

网络爬虫抓取心得体会

• 使用requests包来爬取。首先尝试用用户名密码自动登陆，如果失败，则需要采用cookie登陆。

• 配置文件config.ini，其中包括用户名密码信息，如果有验证码情况，需要手动登陆一次网站获取cookie信息。

• 判断登陆成功与否，看生成的html文件中有没有用户信息。

• [info] email = xxxx@163.com password = xxxx [cookies] q_c1 = cap_id = _za = __utmt = __utma = __utmb = __utmc = __utmz = __utmv = z_c0 = unlock_ticket =

# -*- coding: utf-8 -*- ''' 网络爬虫之用户名密码及验证码登陆：爬取知乎网站 ''' import requests import ConfigParser def create_session(): cf = ConfigParser.ConfigParser() cf.read('config.ini') cookies = cf.items('cookies') cookies = dict(cookies) from pprint import pprint pprint(cookies) email = cf.get('info', 'email') password = cf.get('info', 'password') session = requests.session() login_data = {'email': email, 'password': password} header = { 'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.124 Safari/537.36', 'Host': 'www.zhihu.com', 'Referer': 'http://www.zhihu.com/' } r = session.post('http://www.zhihu.com/login/email', data=login_data, headers=header) if r.json()['r'] == 1: print 'Login Failed, reason is:', for m in r.json()['data']: print r.json()['data'][m] print 'So we use cookies to login in...' has_cookies = False for key in cookies: if key != '__name__' and cookies[key] != '': has_cookies = True break if has_cookies is False: raise ValueError('请填写config.ini文件中的cookies项.') else: r = session.get('http://www.zhihu.com/login/email', cookies=cookies) # 实现验证码登陆 with open('login.html', 'w') as fp: fp.write(r.content) return session, cookies if __name__ == '__main__': requests_session, requests_cookies = create_session() url = 'http://www.zhihu.com/topic/19552832' # content = requests_session.get(url).content # 未登陆 content = requests_session.get(url, cookies=requests_cookies).content # 已登陆 with open('url.html', 'w') as fp: fp.write(content)