拉勾网的反爬策略

更新于2019-01-23 ************
（请注意日期，因为不保证往后的日子里面反爬策略还有效）
 
1. 封IP，这个没的说，肯定要使用代理IP
2. scrapy里面的需要添加headers，而headers中一定要加上Cookies的数据。 之前要做Request中的cookies参数添加cookies，现在发现失效了，只能在headers中添加cookies数据。
 

   headers = {'Accept': 'application/json,text/javascript,*/*;q=0.01', 'Accept-Encoding':

        'gzip,deflate,br',

               'Accept-Language': 'zh-CN,zh;q=0.9,en;q=0.8', 'Cache-Control': 'no-cache',

               # 'Connection': 'keep-alive',

               'Content-Type': 'application/x-www-form-urlencoded;charset=UTF-8',

               'Cookie': 'JSESSIONID=ABAAABAABEEAAJAACF8F22F99AFA35F9EEC28F2D0E46A41;_ga=GA1.2.331323650.1548204973;_gat=1;Hm_lvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1548204973;user_trace_token=20190123085612-adf35b62-1ea9-11e9-b744-5254005c3644;LGSID=20190123085612-adf35c69-1ea9-11e9-b744-5254005c3644;PRE_UTM=;PRE_HOST=;PRE_SITE=;PRE_LAND=https%3A%2F%2Fwww.lagou.com%2F;LGUID=20190123085612-adf35ed5-1ea9-11e9-b744-5254005c3644;_gid=GA1.2.1809874038.1548204973;index_location_city=%E6%B7%B1%E5%9C%B3;TG-TRACK-CODE=index_search;SEARCH_ID=169bf76c08b548f8830967a1968d10ca;Hm_lpvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1548204985;LGRID=20190123085624-b52a0555-1ea9-11e9-b744-5254005c3644',

               'Host': 'www.lagou.com', 'Origin': 'https://www.lagou.com', 'Pragma': 'no-cache',

               'Referer': 'https://www.lagou.com/jobs/list_%E7%88%AC%E8%99%AB?labelWords=&fromSearch=true&suginput=',

               'User-Agent': 'Mozilla/5.0(WindowsNT6.3;WOW64)AppleWebKit/537.36(KHTML,likeGecko)Chrome/71.0.3578.98Safari/537.36',

               'X-Anit-Forge-Code': '0',

               'X-Anit-Forge-Token': 'None',

               'X-Requested-With': 'XMLHttpRequest'

               }