网络安全
集思录这个站点的技术人员挺菜的，连cdn都不用
网络 • 李魔佛 发表了文章 • 0 个评论 • 1476 次浏览 • 2022-05-09 23:21
用稍微大点的ddos攻击下，就容易瘫痪大了的。。。。。。。
心真大。
用稍微大点的ddos攻击下，就容易瘫痪大了的。。。。。。。
心真大。
网络爬虫的分类
网络 • wanbainip 发表了文章 • 0 个评论 • 2740 次浏览 • 2020-10-23 17:51
网络爬虫应该选择 Nutch、Crawler4j、WebMagic、scrapy、WebCollector 还是选择其他？
网络爬虫我们可以分为三大类型：
（1）分布式爬虫：Nutch
（2）JAVA爬虫：Crawler4j、WebMagic、WebCollector
（3）非JAVA爬虫：scrapy（基于Python语言开发）
网络爬虫采集数据过程中，经常会遇到被反爬虫机制限制，最常见的就是ip被限制，网站根据ip的访问频率来识别
判断爬虫，当相同的ip频繁的对目标服务器进行访问，那么就会触发服务器的反爬虫机制，这个时候ip就会被限制
或者无法进行访问，爬虫工作将无法继续进行，因此需要借用到万变ip代理来突破ip的限制。
查看全部
网络爬虫我们可以分为三大类型：
（1）分布式爬虫：Nutch
（2）JAVA爬虫：Crawler4j、WebMagic、WebCollector
（3）非JAVA爬虫：scrapy（基于Python语言开发）
网络爬虫采集数据过程中，经常会遇到被反爬虫机制限制，最常见的就是ip被限制，网站根据ip的访问频率来识别
判断爬虫，当相同的ip频繁的对目标服务器进行访问，那么就会触发服务器的反爬虫机制，这个时候ip就会被限制
或者无法进行访问，爬虫工作将无法继续进行，因此需要借用到万变ip代理来突破ip的限制。
查看全部
网络爬虫应该选择 Nutch、Crawler4j、WebMagic、scrapy、WebCollector 还是选择其他？
网络爬虫我们可以分为三大类型：
（1）分布式爬虫：Nutch
（2）JAVA爬虫：Crawler4j、WebMagic、WebCollector
（3）非JAVA爬虫：scrapy（基于Python语言开发）
网络爬虫采集数据过程中，经常会遇到被反爬虫机制限制，最常见的就是ip被限制，网站根据ip的访问频率来识别
判断爬虫，当相同的ip频繁的对目标服务器进行访问，那么就会触发服务器的反爬虫机制，这个时候ip就会被限制
或者无法进行访问，爬虫工作将无法继续进行，因此需要借用到万变ip代理来突破ip的限制。
网络爬虫我们可以分为三大类型：
（1）分布式爬虫：Nutch
（2）JAVA爬虫：Crawler4j、WebMagic、WebCollector
（3）非JAVA爬虫：scrapy（基于Python语言开发）
网络爬虫采集数据过程中，经常会遇到被反爬虫机制限制，最常见的就是ip被限制，网站根据ip的访问频率来识别
判断爬虫，当相同的ip频繁的对目标服务器进行访问，那么就会触发服务器的反爬虫机制，这个时候ip就会被限制
或者无法进行访问，爬虫工作将无法继续进行，因此需要借用到万变ip代理来突破ip的限制。
薅“疫情公益”羊毛，黑产恶意爬取各大出版社电子书上万册
python爬虫 • Magiccc 发表了文章 • 0 个评论 • 3237 次浏览 • 2020-02-26 13:17
疫情以来，所有企业都上班延期选择在线复工，在我们居家自我隔离期间，极验观察爬虫却没有消停，反而爬虫行为更加活跃且更胜往常。本周五，我们和无糖信息一起聊聊线上爬虫的“疫情”。
疫情以来，所有企业都上班延期选择在线复工，在我们居家自我隔离期间，极验观察爬虫却没有消停，反而爬虫行为更加活跃且更胜往常。本周五，我们和无糖信息一起聊聊线上爬虫的“疫情”。
修改win7 hosts文件IP域名映射问题的一个坑
网络安全 • 李魔佛 发表了文章 • 0 个评论 • 4408 次浏览 • 2019-04-13 20:33
因为讨厌一些网站的弹窗，还有想要高效地学习，把一些娱乐网站给屏蔽了，比如QQ。
所以在hosts文件中添加了如下内容：
路径：C:\Windows\System32\drivers\etc
127.0.0.1 qq.com
127.0.0.1 cnbeta.com
10.4.16.75 windows10.microdone.cn
50.112.79.106 mukodik
0.0.0.0 account.jetbrains.com
结果发现一直无法生效。
后来在ping下发现，ping qq.com 返回的是127.0.0.1是生效的。
但是ping www.qq.com是能够ping到腾讯的服务器的，原来问题在于，你在浏览器输入qq.com，浏览器默认帮你加了www.前缀。
所以你要hosts文件生效，就需要把www.的前缀也加上去。
然后重新刷新下DNS就可以了
ipconfig /flushdns
还有就是，如果你用了代理软件，比如lantern之类的，这个hosts文件也不会起作用。 查看全部
所以在hosts文件中添加了如下内容：
路径：C:\Windows\System32\drivers\etc
127.0.0.1 qq.com
127.0.0.1 cnbeta.com
10.4.16.75 windows10.microdone.cn
50.112.79.106 mukodik
0.0.0.0 account.jetbrains.com
结果发现一直无法生效。
后来在ping下发现，ping qq.com 返回的是127.0.0.1是生效的。
但是ping www.qq.com是能够ping到腾讯的服务器的，原来问题在于，你在浏览器输入qq.com，浏览器默认帮你加了www.前缀。
所以你要hosts文件生效，就需要把www.的前缀也加上去。
然后重新刷新下DNS就可以了
ipconfig /flushdns
还有就是，如果你用了代理软件，比如lantern之类的，这个hosts文件也不会起作用。 查看全部
因为讨厌一些网站的弹窗，还有想要高效地学习，把一些娱乐网站给屏蔽了，比如QQ。
所以在hosts文件中添加了如下内容：
路径：C:\Windows\System32\drivers\etc
结果发现一直无法生效。
后来在ping下发现，ping qq.com 返回的是127.0.0.1是生效的。
但是ping www.qq.com是能够ping到腾讯的服务器的，原来问题在于，你在浏览器输入qq.com，浏览器默认帮你加了www.前缀。
所以你要hosts文件生效，就需要把www.的前缀也加上去。
然后重新刷新下DNS就可以了
ipconfig /flushdns
还有就是，如果你用了代理软件，比如lantern之类的，这个hosts文件也不会起作用。
所以在hosts文件中添加了如下内容：
路径：C:\Windows\System32\drivers\etc
127.0.0.1 qq.com
127.0.0.1 cnbeta.com
10.4.16.75 windows10.microdone.cn
50.112.79.106 mukodik
0.0.0.0 account.jetbrains.com
结果发现一直无法生效。
后来在ping下发现，ping qq.com 返回的是127.0.0.1是生效的。
但是ping www.qq.com是能够ping到腾讯的服务器的，原来问题在于，你在浏览器输入qq.com，浏览器默认帮你加了www.前缀。
所以你要hosts文件生效，就需要把www.的前缀也加上去。
然后重新刷新下DNS就可以了
ipconfig /flushdns
还有就是，如果你用了代理软件，比如lantern之类的，这个hosts文件也不会起作用。
阻挡黑客的最原始最暴力的方法 ---用胶带粘住你的摄像头
网络安全 • 李魔佛 发表了文章 • 0 个评论 • 4360 次浏览 • 2016-06-23 22:15
Facebook 的 CEO 扎克伯格经常喜欢发发照片跟网友互动什么的。最近他在 Facebook 个人账号上面发了一条消息，照片中的他扛着一个镜框庆祝 Instagram 的月活用户数突破了 5 亿大关。但是眼尖的用户却注意到了其他的细节—似乎扎克的笔记本摄像头和麦克风插口都是用胶带蒙住的（题图红色箭头所指）。
扎克为什么要把摄像头蒙住呢？这得先说一下 Ratting（Remote Access Trojan）这种行为。所谓的 Ratting，是指黑客通过植入木马远程控制受害用户设备的行为，而进行这种行为的黑客一般叫做 ratter。除了窃取设备的敏感数据以外，激活用户摄像头和麦克风偷拍视频也是 ratter 常干的事情。
不过扎克伯格这么专业的人也把摄像头蒙起来究竟是偏执狂还是好做法呢？安全专家认为是后者，原因有三：
一是扎克伯格是一个高价值的攻击目标。无论是情报机构还是为了罪犯无疑都会对扎克伯格的资料虎视眈眈，而对于那些为了证明自己黑客功力的人来说，扎克也是一个很理想的目标。所以采取预防措施是很自然的事情。
二是把采集音视频的入口盖住是一种成本低廉且基本的安全防护办法。如果想窃听安全会议，有经验的黑客一般都会先数数哪些设备没有遮住摄像头然后再确定下手的目标。
三是扎克未必就不会被攻破。事实上本月初就爆出了黑客袭击扎克伯格部分社交网络，盗取了他的 Twitter、Pinterest、LinkedIn 帐号的消息。把这两件事联系在一起，更容易解释他的那台 Macbook 上面的胶带。
事实上，采取这种做法的人并不止扎克一个。就连 FBI 局长 James Comey 也把自己的笔记本摄像头蒙上了胶带—原因很简单，因为他看到一个比他更聪明的人也这么干。所以你要不要也蒙上呢？ 查看全部
扎克为什么要把摄像头蒙住呢？这得先说一下 Ratting（Remote Access Trojan）这种行为。所谓的 Ratting，是指黑客通过植入木马远程控制受害用户设备的行为，而进行这种行为的黑客一般叫做 ratter。除了窃取设备的敏感数据以外，激活用户摄像头和麦克风偷拍视频也是 ratter 常干的事情。
不过扎克伯格这么专业的人也把摄像头蒙起来究竟是偏执狂还是好做法呢？安全专家认为是后者，原因有三：
一是扎克伯格是一个高价值的攻击目标。无论是情报机构还是为了罪犯无疑都会对扎克伯格的资料虎视眈眈，而对于那些为了证明自己黑客功力的人来说，扎克也是一个很理想的目标。所以采取预防措施是很自然的事情。
二是把采集音视频的入口盖住是一种成本低廉且基本的安全防护办法。如果想窃听安全会议，有经验的黑客一般都会先数数哪些设备没有遮住摄像头然后再确定下手的目标。
三是扎克未必就不会被攻破。事实上本月初就爆出了黑客袭击扎克伯格部分社交网络，盗取了他的 Twitter、Pinterest、LinkedIn 帐号的消息。把这两件事联系在一起，更容易解释他的那台 Macbook 上面的胶带。
事实上，采取这种做法的人并不止扎克一个。就连 FBI 局长 James Comey 也把自己的笔记本摄像头蒙上了胶带—原因很简单，因为他看到一个比他更聪明的人也这么干。所以你要不要也蒙上呢？ 查看全部
Facebook 的 CEO 扎克伯格经常喜欢发发照片跟网友互动什么的。最近他在 Facebook 个人账号上面发了一条消息，照片中的他扛着一个镜框庆祝 Instagram 的月活用户数突破了 5 亿大关。但是眼尖的用户却注意到了其他的细节—似乎扎克的笔记本摄像头和麦克风插口都是用胶带蒙住的（题图红色箭头所指）。
扎克为什么要把摄像头蒙住呢？这得先说一下 Ratting（Remote Access Trojan）这种行为。所谓的 Ratting，是指黑客通过植入木马远程控制受害用户设备的行为，而进行这种行为的黑客一般叫做 ratter。除了窃取设备的敏感数据以外，激活用户摄像头和麦克风偷拍视频也是 ratter 常干的事情。
不过扎克伯格这么专业的人也把摄像头蒙起来究竟是偏执狂还是好做法呢？安全专家认为是后者，原因有三：
一是扎克伯格是一个高价值的攻击目标。无论是情报机构还是为了罪犯无疑都会对扎克伯格的资料虎视眈眈，而对于那些为了证明自己黑客功力的人来说，扎克也是一个很理想的目标。所以采取预防措施是很自然的事情。
二是把采集音视频的入口盖住是一种成本低廉且基本的安全防护办法。如果想窃听安全会议，有经验的黑客一般都会先数数哪些设备没有遮住摄像头然后再确定下手的目标。
三是扎克未必就不会被攻破。事实上本月初就爆出了黑客袭击扎克伯格部分社交网络，盗取了他的 Twitter、Pinterest、LinkedIn 帐号的消息。把这两件事联系在一起，更容易解释他的那台 Macbook 上面的胶带。
事实上，采取这种做法的人并不止扎克一个。就连 FBI 局长 James Comey 也把自己的笔记本摄像头蒙上了胶带—原因很简单，因为他看到一个比他更聪明的人也这么干。所以你要不要也蒙上呢？
扎克为什么要把摄像头蒙住呢？这得先说一下 Ratting（Remote Access Trojan）这种行为。所谓的 Ratting，是指黑客通过植入木马远程控制受害用户设备的行为，而进行这种行为的黑客一般叫做 ratter。除了窃取设备的敏感数据以外，激活用户摄像头和麦克风偷拍视频也是 ratter 常干的事情。
不过扎克伯格这么专业的人也把摄像头蒙起来究竟是偏执狂还是好做法呢？安全专家认为是后者，原因有三：
一是扎克伯格是一个高价值的攻击目标。无论是情报机构还是为了罪犯无疑都会对扎克伯格的资料虎视眈眈，而对于那些为了证明自己黑客功力的人来说，扎克也是一个很理想的目标。所以采取预防措施是很自然的事情。
二是把采集音视频的入口盖住是一种成本低廉且基本的安全防护办法。如果想窃听安全会议，有经验的黑客一般都会先数数哪些设备没有遮住摄像头然后再确定下手的目标。
三是扎克未必就不会被攻破。事实上本月初就爆出了黑客袭击扎克伯格部分社交网络，盗取了他的 Twitter、Pinterest、LinkedIn 帐号的消息。把这两件事联系在一起，更容易解释他的那台 Macbook 上面的胶带。
事实上，采取这种做法的人并不止扎克一个。就连 FBI 局长 James Comey 也把自己的笔记本摄像头蒙上了胶带—原因很简单，因为他看到一个比他更聪明的人也这么干。所以你要不要也蒙上呢？
联通手机 无信号 原来被他人补卡 把银行卡支付宝的钱全部转走 如何做到？
回复网络 • 低调的哥哥 回复了问题 • 1 人关注 • 1 个回复 • 7712 次浏览 • 2016-06-05 20:16
Firefox抓包分析 （拉勾网抓包分析）
python爬虫 • 低调的哥哥 发表了文章 • 16 个评论 • 19992 次浏览 • 2016-05-09 18:30
针对一些JS网页，动态网页在源码中无法看到它的内容，可以通过抓包分析出其JSON格式的数据。 网页通过通过这些JSON数据对网页的内容进行填充，然后就看到网页里显示相关的内容了。
使用过chrome，firefox，wireshark来抓过包，比较方便的是chrome，不需要安装第三方的其它插件，不过打开新页面的时候又要重新开一个捕捉页面，会错过一些实时的数据。
wireshark需要专门掌握它自己的过滤规则，学习成本摆在那里。
最好用的还是firefox＋firebug第三方插件。
接下来以拉勾网为例。
打开firebug功能
www.lagou.com 在左侧栏随便点击一个岗位，以android为例
在firebug中，需要点击“网络”选项卡，然后选择XHR。
Post的信息就是我们需要关注的，点击post的链接
点击了Android之后 我们从浏览器上传了几个参数到拉勾的服务器
一个是 first ＝true， 一个是kd ＝ android， （关键字） 一个是pn ＝1 （page number 页码）
所以我们就可以模仿这一个步骤来构造一个数据包来模拟用户的点击动作。post_data = {'first':'true','kd':'Android','pn':'1'}
然后使用python中库中最简单的requests库来提交数据。 而这些数据 正是抓包里看到的数据。import requests
url = "http://www.lagou.com/jobs/posi ... ot%3B
return_data=requests.post(url,data=post_data)
print return_data.text
呐，打印出来的数据就是返回来的json数据{"code":0,"success":true,"requestId":null,"resubmitToken":null,"msg":null,"content":{"pageNo":1,"pageSize":15,"positionResult":{"totalCount":5000,"pageSize":15,"locationInfo":{"city":null,"district":null,"businessZone":null},"result":[{"createTime":"2016-05-05 17:27:50","companyId":50889,"positionName":"Android","positionType":"移动开发","workYear":"3-5年","education":"本科","jobNature":"全职","companyShortName":"和创（北京）科技股份有限公司","city":"北京","salary":"20k-35k","financeStage":"上市公司","positionId":1455217,"companyLogo":"i/image/M00/03/44/Cgp3O1ax7JWAOSzUAABS3OF0A7w289.jpg","positionFirstType":"技术","companyName":"和创科技（红圈营销）","positionAdvantage":"上市公司，持续股权激励政策，技术极客云集","industryField":"移动互联网 · 企业服务","score":1372,"district":"西城区","companyLabelList":["弹性工作","敏捷研发","股票期权","年底双薪"],"deliverCount":13,"leaderName":"刘学臣","companySize":"2000人以上","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":48,"imstate":"today","createTimeSort":1462440470000,"positonTypesMap":null,"hrScore":77,"flowScore":148,"showCount":6627,"pvScore":73.2258060280967,"plus":"是","businessZones":["新街口","德胜门","小西天"],"publisherId":994817,"loginTime":1462876049000,"appShow":3141,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":99,"adWord":1,"formatCreateTime":"2016-05-05","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-05 18:30:16","companyId":50889,"positionName":"Android","positionType":"移动开发","workYear":"3-5年","education":"本科","jobNature":"全职","companyShortName":"和创（北京）科技股份有限公司","city":"北京","salary":"20k-35k","financeStage":"上市公司","positionId":1440576,"companyLogo":"i/image/M00/03/44/Cgp3O1ax7JWAOSzUAABS3OF0A7w289.jpg","positionFirstType":"技术","companyName":"和创科技（红圈营销）","positionAdvantage":"上市公司，持续股权激励政策，技术爆棚！","industryField":"移动互联网 · 企业服务","score":1372,"district":"海淀区","companyLabelList":["弹性工作","敏捷研发","股票期权","年底双薪"],"deliverCount":6,"leaderName":"刘学臣","companySize":"2000人以上","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":48,"imstate":"today","createTimeSort":1462444216000,"positonTypesMap":null,"hrScore":77,"flowScore":148,"showCount":3214,"pvScore":73.37271526202157,"plus":"是","businessZones":["双榆树","中关村","大钟寺"],"publisherId":994817,"loginTime":1462876049000,"appShow":1782,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":99,"adWord":1,"formatCreateTime":"2016-05-05","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-10 18:41:29","companyId":94307,"positionName":"Android","positionType":"移动开发","workYear":"3-5年","education":"本科","jobNature":"全职","companyShortName":"宁波海大物联科技有限公司","city":"宁波","salary":"8k-15k","financeStage":"成长型(A轮)","positionId":1070249,"companyLogo":"image2/M00/03/32/CgqLKVXtWiuAUbXgAAB1g_5FW3Y484.png?cc=0.6152940313331783","positionFirstType":"技术","companyName":"海大物联","positionAdvantage":"一流的技术团队，丰厚的薪资回报。","industryField":"移动互联网 · 企业服务","score":1353,"district":"鄞州区","companyLabelList":["节日礼物","年底双薪","带薪年假","年度旅游"],"deliverCount":0,"leaderName":"暂没有填写","companySize":"50-150人","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":48,"imstate":"today","createTimeSort":1462876889000,"positonTypesMap":null,"hrScore":75,"flowScore":167,"showCount":1031,"pvScore":47.6349840620252,"plus":"是","businessZones":null,"publisherId":2494230,"loginTime":1462885305000,"appShow":184,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":63,"adWord":0,"formatCreateTime":"18:41发布","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-10 17:57:43","companyId":89004,"positionName":"Android","positionType":"移动开发","workYear":"1-3年","education":"学历不限","jobNature":"全职","companyShortName":"温州康宁医院股份有限公司","city":"杭州","salary":"10k-20k","financeStage":"上市公司","positionId":1387825,"companyLogo":"i/image/M00/02/C2/CgqKkVabp--APWTjAACHHJJxyPc207.png","positionFirstType":"技术","companyName":"的的心理","positionAdvantage":"上市公司内部创业项目。优质福利待遇+期权","industryField":"移动互联网 · 医疗健康","score":1344,"district":"江干区","companyLabelList":["年底双薪","股票期权","带薪年假","招募合伙人"],"deliverCount":5,"leaderName":"杨怡","companySize":"500-2000人","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":48,"imstate":"today","createTimeSort":1462874263000,"positonTypesMap":null,"hrScore":74,"flowScore":153,"showCount":1312,"pvScore":66.87818057124453,"plus":"是","businessZones":["四季青","景芳"],"publisherId":3655492,"loginTime":1462873104000,"appShow":573,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":69,"adWord":0,"formatCreateTime":"17:57发布","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-10 13:49:47","companyId":15071,"positionName":"Android","positionType":"移动开发","workYear":"3-5年","education":"本科","jobNature":"全职","companyShortName":"杭州短趣网络传媒技术有限公司","city":"杭州","salary":"10k-20k","financeStage":"成长型(A轮)","positionId":1803257,"companyLogo":"image2/M00/0B/80/CgpzWlYYse2AJgc0AABG9iSEWAE052.jpg","positionFirstType":"技术","companyName":"短趣网","positionAdvantage":"高额项目奖金，行业内有竞争力的薪资水平","industryField":"移动互联网 · 社交网络","score":1343,"district":null,"companyLabelList":["绩效奖金","年终分红","五险一金","通讯津贴"],"deliverCount":1,"leaderName":"王强宇","companySize":"50-150人","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":28,"imstate":"today","createTimeSort":1462859387000,"positonTypesMap":null,"hrScore":68,"flowScore":178,"showCount":652,"pvScore":32.82081357576065,"plus":"否","businessZones":null,"publisherId":4362468,"loginTime":1462870318000,"appShow":0,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":69,"adWord":0,"formatCreateTime":"13:49发布","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-10 13:55:08","companyId":28422,"positionName":"Android","positionType":"移动开发","workYear":"3-5年","education":"本科","jobNature":"全职","companyShortName":"成都品果科技有限公司","city":"北京","salary":"18k-30k","financeStage":"成长型(B轮)","positionId":290875,"companyLogo":"i/image/M00/02/F3/Cgp3O1ah7FuAbSnkAACMlcPiWXk393.png","positionFirstType":"技术","companyName":"camera360","positionAdvantage":"高大上的福利待遇、发展前景等着你哦！","industryField":"移动互联网","score":1339,"district":"海淀区","companyLabelList":["年终分红","绩效奖金","年底双薪","五险一金"],"deliverCount":6,"leaderName":"徐灏","companySize":"15-50人","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":0,"imstate":"disabled","createTimeSort":1462859708000,"positonTypesMap":null,"hrScore":80,"flowScore":188,"showCount":1199,"pvScore":19.745453118211834,"plus":"是","businessZones":["中关村","北京大学","苏州街"],"publisherId":389753,"loginTime":1462866640000,"appShow":0,"calcScore":false,"showOrder":1433137697136,"haveDeliver":false,"orderBy":71,"adWord":0,"formatCreateTime":"13:55发布","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-10 17:57:55","companyId":89004,"positionName":"Android","positionType":"移动开发","workYear":"不限","education":"学历不限","jobNature":"全职","companyShortName":"温州康宁医院股份有限公司","city":"杭州","salary":"10k-20k","financeStage":"上市公司","positionId":1410975,"companyLogo":"i/image/M00/02/C2/CgqKkVabp--APWTjAACHHJJxyPc207.png","positionFirstType":"技术","companyName":"的的心理","positionAdvantage":"上市公司内部创业团队","industryField":"移动互联网 · 医疗健康","score":1335,"district":null,"companyLabelList":["年底双薪","股票期权","带薪年假","招募合伙人"],"deliverCount":9,"leaderName":"杨怡","companySize":"500-2000人","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":48,"imstate":"today","createTimeSort":1462874275000,"positonTypesMap":null,"hrScore":74,"flowScore":144,"showCount":2085,"pvScore":77.9570832081189,"plus":"是","businessZones":null,"publisherId":3655492,"loginTime":1462873104000,"appShow":711,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":69,"adWord":0,"formatCreateTime":"17:57发布","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-09 09:46:32","companyId":113895,"positionName":"Android开发工程师","positionType":"移动开发","workYear":"3-5年","education":"本科","jobNature":"全职","companyShortName":"北京互动金服科技有限公司","city":"北京","salary":"15k-25k","financeStage":"成长型(不需要融资)","positionId":1473342,"companyLogo":"i/image/M00/03/D8/CgqKkVbEA_uAe1k4AAHTfy3RxPY812.jpg","positionFirstType":"技术","companyName":"互动科技","positionAdvantage":"五险一金 补充医疗 年终奖 福利津贴","industryField":"移动互联网 · O2O","score":1326,"district":"海淀区","companyLabelList":,"deliverCount":32,"leaderName":"暂没有填写","companySize":"50-150人","randomScore":0,"countAdjusted":false,"relScore":980,"adjustScore":48,"imstate":"today","createTimeSort":1462758392000,"positonTypesMap":null,"hrScore":82,"flowScore":153,"showCount":3741,"pvScore":67.01698353391613,"plus":"是","businessZones":["白石桥","魏公村","万寿寺","白石桥","魏公村","万寿寺"],"publisherId":3814477,"loginTime":1462874842000,"appShow":0,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":63,"adWord":0,"formatCreateTime":"1天前发布","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-10 16:50:51","companyId":23999,"positionName":"Android","positionType":"移动开发","workYear":"1-3年","education":"本科","jobNature":"全职","companyShortName":"南京智鹤电子科技有限公司","city":"长沙","salary":"8k-12k","financeStage":"成长型(A轮)","positionId":1804917,"companyLogo":"image1/M00/35/EB/CgYXBlWc5KGAVeL8AAAOi4lPhWU502.jpg","positionFirstType":"技术","companyName":"智鹤科技","positionAdvantage":"弹性工作制 技术氛围浓厚","industryField":"移动互联网","score":1322,"district":null,"companyLabelList":["股票期权","绩效奖金","专项奖金","年终分红"],"deliverCount":1,"leaderName":"暂没有填写","companySize":"50-150人","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":0,"imstate":"disabled","createTimeSort":1462870251000,"positonTypesMap":null,"hrScore":62,"flowScore":191,"showCount":283,"pvScore":15.939035855045429,"plus":"否","businessZones":null,"publisherId":282621,"loginTime":1462869967000,"appShow":0,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":69,"adWord":0,"formatCreateTime":"16:50发布","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-08 23:12:56","companyId":24287,"positionName":"android","positionType":"移动开发","workYear":"3-5年","education":"本科","jobNature":"全职","companyShortName":"杭州腾展科技有限公司","city":"杭州","salary":"15k-22k","financeStage":"成熟型(D轮及以上)","positionId":1197868,"companyLogo":"image1/M00/0B/7D/Cgo8PFTzIBOAEd2dAACMq9tQoMA797.png","positionFirstType":"技术","companyName":"腾展叮咚（Dingtone）","positionAdvantage":"每半年调整薪资，今年上市！","industryField":"移动互联网 · 社交网络","score":1322,"district":"西湖区","companyLabelList":["出国旅游","股票期权","精英团队","强悍的创始人"],"deliverCount":9,"leaderName":"魏松祥（Steve Wei)","companySize":"50-150人","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":48,"imstate":"disabled","createTimeSort":1462720376000,"positonTypesMap":null,"hrScore":71,"flowScore":137,"showCount":3786,"pvScore":87.51582865460942,"plus":"是","businessZones":["文三路","古荡","高新文教区"],"publisherId":2946659,"loginTime":1462891920000,"appShow":940,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":66,"adWord":0,"formatCreateTime":"2天前发布","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-10 09:38:43","companyId":19875,"positionName":"Android","positionType":"移动开发","workYear":"3-5年","education":"本科","jobNature":"全职","companyShortName":"维沃移动通信有限公司","city":"南京","salary":"12k-24k","financeStage":"初创型(未融资)","positionId":938099,"companyLogo":"image1/M00/00/25/Cgo8PFTUWH-Ab57wAABKOdLbNuw116.png","positionFirstType":"技术","companyName":"vivo","positionAdvantage":"vivo，追求极致","industryField":"移动互联网","score":1320,"district":"建邺区","companyLabelList":["年终分红","五险一金","带薪年假","年度旅游"],"deliverCount":4,"leaderName":"暂没有填写","companySize":"2000人以上","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":48,"imstate":"today","createTimeSort":1462844323000,"positonTypesMap":null,"hrScore":57,"flowScore":149,"showCount":981,"pvScore":72.14107985481958,"plus":"是","businessZones":["沙洲","小行","赛虹桥"],"publisherId":302876,"loginTime":1462871424000,"appShow":353,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":66,"adWord":0,"formatCreateTime":"09:38发布","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-10 09:49:59","companyId":20473,"positionName":"安卓开发工程师","positionType":"移动开发","workYear":"3-5年","education":"大专","jobNature":"全职","companyShortName":"广州棒谷网络科技有限公司","city":"广州","salary":"8k-15k","financeStage":"成长型(A轮)","positionId":1733545,"companyLogo":"image1/M00/0F/36/Cgo8PFT9AgGAciySAAA1THfEIAE433.jpg","positionFirstType":"技术","companyName":"广州棒谷网络科技有限公司","positionAdvantage":"五险一金 大平台 带薪休假","industryField":"电子商务","score":1320,"district":null,"companyLabelList":["项目奖金","绩效奖金","年终奖","五险一金"],"deliverCount":15,"leaderName":"大邹","companySize":"500-2000人","randomScore":0,"countAdjusted":false,"relScore":980,"adjustScore":48,"imstate":"today","createTimeSort":1462844999000,"positonTypesMap":null,"hrScore":79,"flowScore":144,"showCount":3943,"pvScore":77.78928844199473,"plus":"是","businessZones":null,"publisherId":235413,"loginTime":1462878251000,"appShow":1562,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":69,"adWord":0,"formatCreateTime":"09:49发布","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-04 11:41:59","companyId":87117,"positionName":"Android","positionType":"移动开发","workYear":"1-3年","education":"本科","jobNature":"全职","companyShortName":"南京信通科技有限责任公司","city":"南京","salary":"10k-12k","financeStage":"成长型(不需要融资)","positionId":966059,"companyLogo":"image1/M00/3F/51/CgYXBlXASfuADyOsAAA_na14zho635.jpg?cc=0.6724986131303012","positionFirstType":"技术","companyName":"联创集团信通科技","positionAdvantage":"提供完善的福利和薪酬晋升制度","industryField":"移动互联网 · 教育","score":1320,"district":"鼓楼区","companyLabelList":["节日礼物","带薪年假","补充医保","补充子女医保"],"deliverCount":4,"leaderName":"暂没有填写","companySize":"150-500人","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":48,"imstate":"today","createTimeSort":1462333319000,"positonTypesMap":null,"hrScore":88,"flowScore":143,"showCount":3161,"pvScore":79.41660570401937,"plus":"是","businessZones":["虎踞路","龙江","西桥"],"publisherId":2230973,"loginTime":1462871511000,"appShow":711,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":41,"adWord":0,"formatCreateTime":"2016-05-04","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-10 09:26:58","companyId":103051,"positionName":"Android","positionType":"移动开发","workYear":"1-3年","education":"大专","jobNature":"全职","companyShortName":"浙江米果网络股份有限公司","city":"杭州","salary":"12k-22k","financeStage":"成长型(A轮)","positionId":1233873,"companyLogo":"image2/M00/10/DF/CgqLKVYwKQSAR2p4AAJAM590SJM137.png?cc=0.17502541467547417","positionFirstType":"技术","companyName":"米果小站","positionAdvantage":"充分的发展成长空间","industryField":"移动互联网","score":1320,"district":"滨江区","companyLabelList":["年底双薪","股票期权","午餐补助","五险一金"],"deliverCount":5,"leaderName":"暂没有填写","companySize":"150-500人","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":48,"imstate":"disabled","createTimeSort":1462843618000,"positonTypesMap":null,"hrScore":72,"flowScore":137,"showCount":1582,"pvScore":87.93158026917071,"plus":"是","businessZones":["江南","长河","西兴"],"publisherId":2992735,"loginTime":1462889479000,"appShow":312,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":63,"adWord":0,"formatCreateTime":"09:26发布","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-10 10:01:39","companyId":70044,"positionName":"Android","positionType":"移动开发","workYear":"1-3年","education":"大专","jobNature":"全职","companyShortName":"武汉平行世界网络科技有限公司","city":"武汉","salary":"9k-13k","financeStage":"初创型(天使轮)","positionId":664813,"companyLogo":"image2/M00/00/3E/CgqLKVXdccSAQE91AACv-6V33Vo860.jpg","positionFirstType":"技术","companyName":"平行世界","positionAdvantage":"弹性工作、带薪年假、待遇优厚、3号线直达","industryField":"电子商务 · 文化娱乐","score":1320,"district":"蔡甸区","companyLabelList":["年底双薪","待遇优厚","专项奖金","带薪年假"],"deliverCount":18,"leaderName":"暂没有填写","companySize":"50-150人","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":48,"imstate":"today","createTimeSort":1462845699000,"positonTypesMap":null,"hrScore":76,"flowScore":128,"showCount":1934,"pvScore":99.6001810598155,"plus":"是","businessZones":["沌口"],"publisherId":1694134,"loginTime":1462892151000,"appShow":759,"calcScore":false,"showOrder":1433141412915,"haveDeliver":false,"orderBy":68,"adWord":0,"formatCreateTime":"10:01发布","totalCount":0,"searchScore":0.0}]}}}
在XHR中点击JSON，就可以看到浏览器返回来的数据了。 是不是跟上面使用python程序抓取的一样呢？
是不是很简单？
如果想获得第2页，第3页的数据呢？
只需要修改pn＝x 中的值就可以了。post_data = {'first':'true','kd':'Android','pn':'2'} ＃获取第2页的内容
如果想要获取全部内容，写一个循环语句就可以了。
版权所有，转载请说明出处：www.30daydo.com
查看全部
使用过chrome，firefox，wireshark来抓过包，比较方便的是chrome，不需要安装第三方的其它插件，不过打开新页面的时候又要重新开一个捕捉页面，会错过一些实时的数据。
wireshark需要专门掌握它自己的过滤规则，学习成本摆在那里。
最好用的还是firefox＋firebug第三方插件。
接下来以拉勾网为例。
打开firebug功能
www.lagou.com 在左侧栏随便点击一个岗位，以android为例
在firebug中，需要点击“网络”选项卡，然后选择XHR。
Post的信息就是我们需要关注的，点击post的链接
点击了Android之后 我们从浏览器上传了几个参数到拉勾的服务器
一个是 first ＝true， 一个是kd ＝ android， （关键字） 一个是pn ＝1 （page number 页码）
所以我们就可以模仿这一个步骤来构造一个数据包来模拟用户的点击动作。post_data = {'first':'true','kd':'Android','pn':'1'}
然后使用python中库中最简单的requests库来提交数据。 而这些数据 正是抓包里看到的数据。import requests
url = "http://www.lagou.com/jobs/posi ... ot%3B
return_data=requests.post(url,data=post_data)
print return_data.text
呐，打印出来的数据就是返回来的json数据{"code":0,"success":true,"requestId":null,"resubmitToken":null,"msg":null,"content":{"pageNo":1,"pageSize":15,"positionResult":{"totalCount":5000,"pageSize":15,"locationInfo":{"city":null,"district":null,"businessZone":null},"result":[{"createTime":"2016-05-05 17:27:50","companyId":50889,"positionName":"Android","positionType":"移动开发","workYear":"3-5年","education":"本科","jobNature":"全职","companyShortName":"和创（北京）科技股份有限公司","city":"北京","salary":"20k-35k","financeStage":"上市公司","positionId":1455217,"companyLogo":"i/image/M00/03/44/Cgp3O1ax7JWAOSzUAABS3OF0A7w289.jpg","positionFirstType":"技术","companyName":"和创科技（红圈营销）","positionAdvantage":"上市公司，持续股权激励政策，技术极客云集","industryField":"移动互联网 · 企业服务","score":1372,"district":"西城区","companyLabelList":["弹性工作","敏捷研发","股票期权","年底双薪"],"deliverCount":13,"leaderName":"刘学臣","companySize":"2000人以上","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":48,"imstate":"today","createTimeSort":1462440470000,"positonTypesMap":null,"hrScore":77,"flowScore":148,"showCount":6627,"pvScore":73.2258060280967,"plus":"是","businessZones":["新街口","德胜门","小西天"],"publisherId":994817,"loginTime":1462876049000,"appShow":3141,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":99,"adWord":1,"formatCreateTime":"2016-05-05","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-05 18:30:16","companyId":50889,"positionName":"Android","positionType":"移动开发","workYear":"3-5年","education":"本科","jobNature":"全职","companyShortName":"和创（北京）科技股份有限公司","city":"北京","salary":"20k-35k","financeStage":"上市公司","positionId":1440576,"companyLogo":"i/image/M00/03/44/Cgp3O1ax7JWAOSzUAABS3OF0A7w289.jpg","positionFirstType":"技术","companyName":"和创科技（红圈营销）","positionAdvantage":"上市公司，持续股权激励政策，技术爆棚！","industryField":"移动互联网 · 企业服务","score":1372,"district":"海淀区","companyLabelList":["弹性工作","敏捷研发","股票期权","年底双薪"],"deliverCount":6,"leaderName":"刘学臣","companySize":"2000人以上","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":48,"imstate":"today","createTimeSort":1462444216000,"positonTypesMap":null,"hrScore":77,"flowScore":148,"showCount":3214,"pvScore":73.37271526202157,"plus":"是","businessZones":["双榆树","中关村","大钟寺"],"publisherId":994817,"loginTime":1462876049000,"appShow":1782,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":99,"adWord":1,"formatCreateTime":"2016-05-05","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-10 18:41:29","companyId":94307,"positionName":"Android","positionType":"移动开发","workYear":"3-5年","education":"本科","jobNature":"全职","companyShortName":"宁波海大物联科技有限公司","city":"宁波","salary":"8k-15k","financeStage":"成长型(A轮)","positionId":1070249,"companyLogo":"image2/M00/03/32/CgqLKVXtWiuAUbXgAAB1g_5FW3Y484.png?cc=0.6152940313331783","positionFirstType":"技术","companyName":"海大物联","positionAdvantage":"一流的技术团队，丰厚的薪资回报。","industryField":"移动互联网 · 企业服务","score":1353,"district":"鄞州区","companyLabelList":["节日礼物","年底双薪","带薪年假","年度旅游"],"deliverCount":0,"leaderName":"暂没有填写","companySize":"50-150人","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":48,"imstate":"today","createTimeSort":1462876889000,"positonTypesMap":null,"hrScore":75,"flowScore":167,"showCount":1031,"pvScore":47.6349840620252,"plus":"是","businessZones":null,"publisherId":2494230,"loginTime":1462885305000,"appShow":184,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":63,"adWord":0,"formatCreateTime":"18:41发布","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-10 17:57:43","companyId":89004,"positionName":"Android","positionType":"移动开发","workYear":"1-3年","education":"学历不限","jobNature":"全职","companyShortName":"温州康宁医院股份有限公司","city":"杭州","salary":"10k-20k","financeStage":"上市公司","positionId":1387825,"companyLogo":"i/image/M00/02/C2/CgqKkVabp--APWTjAACHHJJxyPc207.png","positionFirstType":"技术","companyName":"的的心理","positionAdvantage":"上市公司内部创业项目。优质福利待遇+期权","industryField":"移动互联网 · 医疗健康","score":1344,"district":"江干区","companyLabelList":["年底双薪","股票期权","带薪年假","招募合伙人"],"deliverCount":5,"leaderName":"杨怡","companySize":"500-2000人","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":48,"imstate":"today","createTimeSort":1462874263000,"positonTypesMap":null,"hrScore":74,"flowScore":153,"showCount":1312,"pvScore":66.87818057124453,"plus":"是","businessZones":["四季青","景芳"],"publisherId":3655492,"loginTime":1462873104000,"appShow":573,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":69,"adWord":0,"formatCreateTime":"17:57发布","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-10 13:49:47","companyId":15071,"positionName":"Android","positionType":"移动开发","workYear":"3-5年","education":"本科","jobNature":"全职","companyShortName":"杭州短趣网络传媒技术有限公司","city":"杭州","salary":"10k-20k","financeStage":"成长型(A轮)","positionId":1803257,"companyLogo":"image2/M00/0B/80/CgpzWlYYse2AJgc0AABG9iSEWAE052.jpg","positionFirstType":"技术","companyName":"短趣网","positionAdvantage":"高额项目奖金，行业内有竞争力的薪资水平","industryField":"移动互联网 · 社交网络","score":1343,"district":null,"companyLabelList":["绩效奖金","年终分红","五险一金","通讯津贴"],"deliverCount":1,"leaderName":"王强宇","companySize":"50-150人","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":28,"imstate":"today","createTimeSort":1462859387000,"positonTypesMap":null,"hrScore":68,"flowScore":178,"showCount":652,"pvScore":32.82081357576065,"plus":"否","businessZones":null,"publisherId":4362468,"loginTime":1462870318000,"appShow":0,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":69,"adWord":0,"formatCreateTime":"13:49发布","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-10 13:55:08","companyId":28422,"positionName":"Android","positionType":"移动开发","workYear":"3-5年","education":"本科","jobNature":"全职","companyShortName":"成都品果科技有限公司","city":"北京","salary":"18k-30k","financeStage":"成长型(B轮)","positionId":290875,"companyLogo":"i/image/M00/02/F3/Cgp3O1ah7FuAbSnkAACMlcPiWXk393.png","positionFirstType":"技术","companyName":"camera360","positionAdvantage":"高大上的福利待遇、发展前景等着你哦！","industryField":"移动互联网","score":1339,"district":"海淀区","companyLabelList":["年终分红","绩效奖金","年底双薪","五险一金"],"deliverCount":6,"leaderName":"徐灏","companySize":"15-50人","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":0,"imstate":"disabled","createTimeSort":1462859708000,"positonTypesMap":null,"hrScore":80,"flowScore":188,"showCount":1199,"pvScore":19.745453118211834,"plus":"是","businessZones":["中关村","北京大学","苏州街"],"publisherId":389753,"loginTime":1462866640000,"appShow":0,"calcScore":false,"showOrder":1433137697136,"haveDeliver":false,"orderBy":71,"adWord":0,"formatCreateTime":"13:55发布","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-10 17:57:55","companyId":89004,"positionName":"Android","positionType":"移动开发","workYear":"不限","education":"学历不限","jobNature":"全职","companyShortName":"温州康宁医院股份有限公司","city":"杭州","salary":"10k-20k","financeStage":"上市公司","positionId":1410975,"companyLogo":"i/image/M00/02/C2/CgqKkVabp--APWTjAACHHJJxyPc207.png","positionFirstType":"技术","companyName":"的的心理","positionAdvantage":"上市公司内部创业团队","industryField":"移动互联网 · 医疗健康","score":1335,"district":null,"companyLabelList":["年底双薪","股票期权","带薪年假","招募合伙人"],"deliverCount":9,"leaderName":"杨怡","companySize":"500-2000人","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":48,"imstate":"today","createTimeSort":1462874275000,"positonTypesMap":null,"hrScore":74,"flowScore":144,"showCount":2085,"pvScore":77.9570832081189,"plus":"是","businessZones":null,"publisherId":3655492,"loginTime":1462873104000,"appShow":711,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":69,"adWord":0,"formatCreateTime":"17:57发布","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-09 09:46:32","companyId":113895,"positionName":"Android开发工程师","positionType":"移动开发","workYear":"3-5年","education":"本科","jobNature":"全职","companyShortName":"北京互动金服科技有限公司","city":"北京","salary":"15k-25k","financeStage":"成长型(不需要融资)","positionId":1473342,"companyLogo":"i/image/M00/03/D8/CgqKkVbEA_uAe1k4AAHTfy3RxPY812.jpg","positionFirstType":"技术","companyName":"互动科技","positionAdvantage":"五险一金 补充医疗 年终奖 福利津贴","industryField":"移动互联网 · O2O","score":1326,"district":"海淀区","companyLabelList":,"deliverCount":32,"leaderName":"暂没有填写","companySize":"50-150人","randomScore":0,"countAdjusted":false,"relScore":980,"adjustScore":48,"imstate":"today","createTimeSort":1462758392000,"positonTypesMap":null,"hrScore":82,"flowScore":153,"showCount":3741,"pvScore":67.01698353391613,"plus":"是","businessZones":["白石桥","魏公村","万寿寺","白石桥","魏公村","万寿寺"],"publisherId":3814477,"loginTime":1462874842000,"appShow":0,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":63,"adWord":0,"formatCreateTime":"1天前发布","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-10 16:50:51","companyId":23999,"positionName":"Android","positionType":"移动开发","workYear":"1-3年","education":"本科","jobNature":"全职","companyShortName":"南京智鹤电子科技有限公司","city":"长沙","salary":"8k-12k","financeStage":"成长型(A轮)","positionId":1804917,"companyLogo":"image1/M00/35/EB/CgYXBlWc5KGAVeL8AAAOi4lPhWU502.jpg","positionFirstType":"技术","companyName":"智鹤科技","positionAdvantage":"弹性工作制 技术氛围浓厚","industryField":"移动互联网","score":1322,"district":null,"companyLabelList":["股票期权","绩效奖金","专项奖金","年终分红"],"deliverCount":1,"leaderName":"暂没有填写","companySize":"50-150人","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":0,"imstate":"disabled","createTimeSort":1462870251000,"positonTypesMap":null,"hrScore":62,"flowScore":191,"showCount":283,"pvScore":15.939035855045429,"plus":"否","businessZones":null,"publisherId":282621,"loginTime":1462869967000,"appShow":0,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":69,"adWord":0,"formatCreateTime":"16:50发布","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-08 23:12:56","companyId":24287,"positionName":"android","positionType":"移动开发","workYear":"3-5年","education":"本科","jobNature":"全职","companyShortName":"杭州腾展科技有限公司","city":"杭州","salary":"15k-22k","financeStage":"成熟型(D轮及以上)","positionId":1197868,"companyLogo":"image1/M00/0B/7D/Cgo8PFTzIBOAEd2dAACMq9tQoMA797.png","positionFirstType":"技术","companyName":"腾展叮咚（Dingtone）","positionAdvantage":"每半年调整薪资，今年上市！","industryField":"移动互联网 · 社交网络","score":1322,"district":"西湖区","companyLabelList":["出国旅游","股票期权","精英团队","强悍的创始人"],"deliverCount":9,"leaderName":"魏松祥（Steve Wei)","companySize":"50-150人","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":48,"imstate":"disabled","createTimeSort":1462720376000,"positonTypesMap":null,"hrScore":71,"flowScore":137,"showCount":3786,"pvScore":87.51582865460942,"plus":"是","businessZones":["文三路","古荡","高新文教区"],"publisherId":2946659,"loginTime":1462891920000,"appShow":940,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":66,"adWord":0,"formatCreateTime":"2天前发布","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-10 09:38:43","companyId":19875,"positionName":"Android","positionType":"移动开发","workYear":"3-5年","education":"本科","jobNature":"全职","companyShortName":"维沃移动通信有限公司","city":"南京","salary":"12k-24k","financeStage":"初创型(未融资)","positionId":938099,"companyLogo":"image1/M00/00/25/Cgo8PFTUWH-Ab57wAABKOdLbNuw116.png","positionFirstType":"技术","companyName":"vivo","positionAdvantage":"vivo，追求极致","industryField":"移动互联网","score":1320,"district":"建邺区","companyLabelList":["年终分红","五险一金","带薪年假","年度旅游"],"deliverCount":4,"leaderName":"暂没有填写","companySize":"2000人以上","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":48,"imstate":"today","createTimeSort":1462844323000,"positonTypesMap":null,"hrScore":57,"flowScore":149,"showCount":981,"pvScore":72.14107985481958,"plus":"是","businessZones":["沙洲","小行","赛虹桥"],"publisherId":302876,"loginTime":1462871424000,"appShow":353,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":66,"adWord":0,"formatCreateTime":"09:38发布","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-10 09:49:59","companyId":20473,"positionName":"安卓开发工程师","positionType":"移动开发","workYear":"3-5年","education":"大专","jobNature":"全职","companyShortName":"广州棒谷网络科技有限公司","city":"广州","salary":"8k-15k","financeStage":"成长型(A轮)","positionId":1733545,"companyLogo":"image1/M00/0F/36/Cgo8PFT9AgGAciySAAA1THfEIAE433.jpg","positionFirstType":"技术","companyName":"广州棒谷网络科技有限公司","positionAdvantage":"五险一金 大平台 带薪休假","industryField":"电子商务","score":1320,"district":null,"companyLabelList":["项目奖金","绩效奖金","年终奖","五险一金"],"deliverCount":15,"leaderName":"大邹","companySize":"500-2000人","randomScore":0,"countAdjusted":false,"relScore":980,"adjustScore":48,"imstate":"today","createTimeSort":1462844999000,"positonTypesMap":null,"hrScore":79,"flowScore":144,"showCount":3943,"pvScore":77.78928844199473,"plus":"是","businessZones":null,"publisherId":235413,"loginTime":1462878251000,"appShow":1562,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":69,"adWord":0,"formatCreateTime":"09:49发布","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-04 11:41:59","companyId":87117,"positionName":"Android","positionType":"移动开发","workYear":"1-3年","education":"本科","jobNature":"全职","companyShortName":"南京信通科技有限责任公司","city":"南京","salary":"10k-12k","financeStage":"成长型(不需要融资)","positionId":966059,"companyLogo":"image1/M00/3F/51/CgYXBlXASfuADyOsAAA_na14zho635.jpg?cc=0.6724986131303012","positionFirstType":"技术","companyName":"联创集团信通科技","positionAdvantage":"提供完善的福利和薪酬晋升制度","industryField":"移动互联网 · 教育","score":1320,"district":"鼓楼区","companyLabelList":["节日礼物","带薪年假","补充医保","补充子女医保"],"deliverCount":4,"leaderName":"暂没有填写","companySize":"150-500人","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":48,"imstate":"today","createTimeSort":1462333319000,"positonTypesMap":null,"hrScore":88,"flowScore":143,"showCount":3161,"pvScore":79.41660570401937,"plus":"是","businessZones":["虎踞路","龙江","西桥"],"publisherId":2230973,"loginTime":1462871511000,"appShow":711,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":41,"adWord":0,"formatCreateTime":"2016-05-04","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-10 09:26:58","companyId":103051,"positionName":"Android","positionType":"移动开发","workYear":"1-3年","education":"大专","jobNature":"全职","companyShortName":"浙江米果网络股份有限公司","city":"杭州","salary":"12k-22k","financeStage":"成长型(A轮)","positionId":1233873,"companyLogo":"image2/M00/10/DF/CgqLKVYwKQSAR2p4AAJAM590SJM137.png?cc=0.17502541467547417","positionFirstType":"技术","companyName":"米果小站","positionAdvantage":"充分的发展成长空间","industryField":"移动互联网","score":1320,"district":"滨江区","companyLabelList":["年底双薪","股票期权","午餐补助","五险一金"],"deliverCount":5,"leaderName":"暂没有填写","companySize":"150-500人","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":48,"imstate":"disabled","createTimeSort":1462843618000,"positonTypesMap":null,"hrScore":72,"flowScore":137,"showCount":1582,"pvScore":87.93158026917071,"plus":"是","businessZones":["江南","长河","西兴"],"publisherId":2992735,"loginTime":1462889479000,"appShow":312,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":63,"adWord":0,"formatCreateTime":"09:26发布","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-10 10:01:39","companyId":70044,"positionName":"Android","positionType":"移动开发","workYear":"1-3年","education":"大专","jobNature":"全职","companyShortName":"武汉平行世界网络科技有限公司","city":"武汉","salary":"9k-13k","financeStage":"初创型(天使轮)","positionId":664813,"companyLogo":"image2/M00/00/3E/CgqLKVXdccSAQE91AACv-6V33Vo860.jpg","positionFirstType":"技术","companyName":"平行世界","positionAdvantage":"弹性工作、带薪年假、待遇优厚、3号线直达","industryField":"电子商务 · 文化娱乐","score":1320,"district":"蔡甸区","companyLabelList":["年底双薪","待遇优厚","专项奖金","带薪年假"],"deliverCount":18,"leaderName":"暂没有填写","companySize":"50-150人","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":48,"imstate":"today","createTimeSort":1462845699000,"positonTypesMap":null,"hrScore":76,"flowScore":128,"showCount":1934,"pvScore":99.6001810598155,"plus":"是","businessZones":["沌口"],"publisherId":1694134,"loginTime":1462892151000,"appShow":759,"calcScore":false,"showOrder":1433141412915,"haveDeliver":false,"orderBy":68,"adWord":0,"formatCreateTime":"10:01发布","totalCount":0,"searchScore":0.0}]}}}
在XHR中点击JSON，就可以看到浏览器返回来的数据了。 是不是跟上面使用python程序抓取的一样呢？
是不是很简单？
如果想获得第2页，第3页的数据呢？
只需要修改pn＝x 中的值就可以了。post_data = {'first':'true','kd':'Android','pn':'2'} ＃获取第2页的内容
如果想要获取全部内容，写一个循环语句就可以了。
版权所有，转载请说明出处：www.30daydo.com
查看全部
针对一些JS网页，动态网页在源码中无法看到它的内容，可以通过抓包分析出其JSON格式的数据。 网页通过通过这些JSON数据对网页的内容进行填充，然后就看到网页里显示相关的内容了。
使用过chrome，firefox，wireshark来抓过包，比较方便的是chrome，不需要安装第三方的其它插件，不过打开新页面的时候又要重新开一个捕捉页面，会错过一些实时的数据。
wireshark需要专门掌握它自己的过滤规则，学习成本摆在那里。
最好用的还是firefox＋firebug第三方插件。
接下来以拉勾网为例。
打开firebug功能
www.lagou.com 在左侧栏随便点击一个岗位，以android为例
在firebug中，需要点击“网络”选项卡，然后选择XHR。
Post的信息就是我们需要关注的，点击post的链接
点击了Android之后 我们从浏览器上传了几个参数到拉勾的服务器
一个是 first ＝true， 一个是kd ＝ android， （关键字） 一个是pn ＝1 （page number 页码）
所以我们就可以模仿这一个步骤来构造一个数据包来模拟用户的点击动作。
然后使用python中库中最简单的requests库来提交数据。 而这些数据 正是抓包里看到的数据。
呐，打印出来的数据就是返回来的json数据
在XHR中点击JSON，就可以看到浏览器返回来的数据了。 是不是跟上面使用python程序抓取的一样呢？
是不是很简单？
如果想获得第2页，第3页的数据呢？
只需要修改pn＝x 中的值就可以了。
如果想要获取全部内容，写一个循环语句就可以了。
版权所有，转载请说明出处：www.30daydo.com
使用过chrome，firefox，wireshark来抓过包，比较方便的是chrome，不需要安装第三方的其它插件，不过打开新页面的时候又要重新开一个捕捉页面，会错过一些实时的数据。
wireshark需要专门掌握它自己的过滤规则，学习成本摆在那里。
最好用的还是firefox＋firebug第三方插件。
接下来以拉勾网为例。
打开firebug功能
www.lagou.com 在左侧栏随便点击一个岗位，以android为例
在firebug中，需要点击“网络”选项卡，然后选择XHR。
Post的信息就是我们需要关注的，点击post的链接
点击了Android之后 我们从浏览器上传了几个参数到拉勾的服务器
一个是 first ＝true， 一个是kd ＝ android， （关键字） 一个是pn ＝1 （page number 页码）
所以我们就可以模仿这一个步骤来构造一个数据包来模拟用户的点击动作。
post_data = {'first':'true','kd':'Android','pn':'1'}然后使用python中库中最简单的requests库来提交数据。 而这些数据 正是抓包里看到的数据。
import requests
url = "http://www.lagou.com/jobs/posi ... ot%3B
return_data=requests.post(url,data=post_data)
print return_data.text
呐，打印出来的数据就是返回来的json数据
{"code":0,"success":true,"requestId":null,"resubmitToken":null,"msg":null,"content":{"pageNo":1,"pageSize":15,"positionResult":{"totalCount":5000,"pageSize":15,"locationInfo":{"city":null,"district":null,"businessZone":null},"result":[{"createTime":"2016-05-05 17:27:50","companyId":50889,"positionName":"Android","positionType":"移动开发","workYear":"3-5年","education":"本科","jobNature":"全职","companyShortName":"和创（北京）科技股份有限公司","city":"北京","salary":"20k-35k","financeStage":"上市公司","positionId":1455217,"companyLogo":"i/image/M00/03/44/Cgp3O1ax7JWAOSzUAABS3OF0A7w289.jpg","positionFirstType":"技术","companyName":"和创科技（红圈营销）","positionAdvantage":"上市公司，持续股权激励政策，技术极客云集","industryField":"移动互联网 · 企业服务","score":1372,"district":"西城区","companyLabelList":["弹性工作","敏捷研发","股票期权","年底双薪"],"deliverCount":13,"leaderName":"刘学臣","companySize":"2000人以上","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":48,"imstate":"today","createTimeSort":1462440470000,"positonTypesMap":null,"hrScore":77,"flowScore":148,"showCount":6627,"pvScore":73.2258060280967,"plus":"是","businessZones":["新街口","德胜门","小西天"],"publisherId":994817,"loginTime":1462876049000,"appShow":3141,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":99,"adWord":1,"formatCreateTime":"2016-05-05","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-05 18:30:16","companyId":50889,"positionName":"Android","positionType":"移动开发","workYear":"3-5年","education":"本科","jobNature":"全职","companyShortName":"和创（北京）科技股份有限公司","city":"北京","salary":"20k-35k","financeStage":"上市公司","positionId":1440576,"companyLogo":"i/image/M00/03/44/Cgp3O1ax7JWAOSzUAABS3OF0A7w289.jpg","positionFirstType":"技术","companyName":"和创科技（红圈营销）","positionAdvantage":"上市公司，持续股权激励政策，技术爆棚！","industryField":"移动互联网 · 企业服务","score":1372,"district":"海淀区","companyLabelList":["弹性工作","敏捷研发","股票期权","年底双薪"],"deliverCount":6,"leaderName":"刘学臣","companySize":"2000人以上","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":48,"imstate":"today","createTimeSort":1462444216000,"positonTypesMap":null,"hrScore":77,"flowScore":148,"showCount":3214,"pvScore":73.37271526202157,"plus":"是","businessZones":["双榆树","中关村","大钟寺"],"publisherId":994817,"loginTime":1462876049000,"appShow":1782,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":99,"adWord":1,"formatCreateTime":"2016-05-05","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-10 18:41:29","companyId":94307,"positionName":"Android","positionType":"移动开发","workYear":"3-5年","education":"本科","jobNature":"全职","companyShortName":"宁波海大物联科技有限公司","city":"宁波","salary":"8k-15k","financeStage":"成长型(A轮)","positionId":1070249,"companyLogo":"image2/M00/03/32/CgqLKVXtWiuAUbXgAAB1g_5FW3Y484.png?cc=0.6152940313331783","positionFirstType":"技术","companyName":"海大物联","positionAdvantage":"一流的技术团队，丰厚的薪资回报。","industryField":"移动互联网 · 企业服务","score":1353,"district":"鄞州区","companyLabelList":["节日礼物","年底双薪","带薪年假","年度旅游"],"deliverCount":0,"leaderName":"暂没有填写","companySize":"50-150人","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":48,"imstate":"today","createTimeSort":1462876889000,"positonTypesMap":null,"hrScore":75,"flowScore":167,"showCount":1031,"pvScore":47.6349840620252,"plus":"是","businessZones":null,"publisherId":2494230,"loginTime":1462885305000,"appShow":184,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":63,"adWord":0,"formatCreateTime":"18:41发布","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-10 17:57:43","companyId":89004,"positionName":"Android","positionType":"移动开发","workYear":"1-3年","education":"学历不限","jobNature":"全职","companyShortName":"温州康宁医院股份有限公司","city":"杭州","salary":"10k-20k","financeStage":"上市公司","positionId":1387825,"companyLogo":"i/image/M00/02/C2/CgqKkVabp--APWTjAACHHJJxyPc207.png","positionFirstType":"技术","companyName":"的的心理","positionAdvantage":"上市公司内部创业项目。优质福利待遇+期权","industryField":"移动互联网 · 医疗健康","score":1344,"district":"江干区","companyLabelList":["年底双薪","股票期权","带薪年假","招募合伙人"],"deliverCount":5,"leaderName":"杨怡","companySize":"500-2000人","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":48,"imstate":"today","createTimeSort":1462874263000,"positonTypesMap":null,"hrScore":74,"flowScore":153,"showCount":1312,"pvScore":66.87818057124453,"plus":"是","businessZones":["四季青","景芳"],"publisherId":3655492,"loginTime":1462873104000,"appShow":573,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":69,"adWord":0,"formatCreateTime":"17:57发布","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-10 13:49:47","companyId":15071,"positionName":"Android","positionType":"移动开发","workYear":"3-5年","education":"本科","jobNature":"全职","companyShortName":"杭州短趣网络传媒技术有限公司","city":"杭州","salary":"10k-20k","financeStage":"成长型(A轮)","positionId":1803257,"companyLogo":"image2/M00/0B/80/CgpzWlYYse2AJgc0AABG9iSEWAE052.jpg","positionFirstType":"技术","companyName":"短趣网","positionAdvantage":"高额项目奖金，行业内有竞争力的薪资水平","industryField":"移动互联网 · 社交网络","score":1343,"district":null,"companyLabelList":["绩效奖金","年终分红","五险一金","通讯津贴"],"deliverCount":1,"leaderName":"王强宇","companySize":"50-150人","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":28,"imstate":"today","createTimeSort":1462859387000,"positonTypesMap":null,"hrScore":68,"flowScore":178,"showCount":652,"pvScore":32.82081357576065,"plus":"否","businessZones":null,"publisherId":4362468,"loginTime":1462870318000,"appShow":0,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":69,"adWord":0,"formatCreateTime":"13:49发布","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-10 13:55:08","companyId":28422,"positionName":"Android","positionType":"移动开发","workYear":"3-5年","education":"本科","jobNature":"全职","companyShortName":"成都品果科技有限公司","city":"北京","salary":"18k-30k","financeStage":"成长型(B轮)","positionId":290875,"companyLogo":"i/image/M00/02/F3/Cgp3O1ah7FuAbSnkAACMlcPiWXk393.png","positionFirstType":"技术","companyName":"camera360","positionAdvantage":"高大上的福利待遇、发展前景等着你哦！","industryField":"移动互联网","score":1339,"district":"海淀区","companyLabelList":["年终分红","绩效奖金","年底双薪","五险一金"],"deliverCount":6,"leaderName":"徐灏","companySize":"15-50人","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":0,"imstate":"disabled","createTimeSort":1462859708000,"positonTypesMap":null,"hrScore":80,"flowScore":188,"showCount":1199,"pvScore":19.745453118211834,"plus":"是","businessZones":["中关村","北京大学","苏州街"],"publisherId":389753,"loginTime":1462866640000,"appShow":0,"calcScore":false,"showOrder":1433137697136,"haveDeliver":false,"orderBy":71,"adWord":0,"formatCreateTime":"13:55发布","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-10 17:57:55","companyId":89004,"positionName":"Android","positionType":"移动开发","workYear":"不限","education":"学历不限","jobNature":"全职","companyShortName":"温州康宁医院股份有限公司","city":"杭州","salary":"10k-20k","financeStage":"上市公司","positionId":1410975,"companyLogo":"i/image/M00/02/C2/CgqKkVabp--APWTjAACHHJJxyPc207.png","positionFirstType":"技术","companyName":"的的心理","positionAdvantage":"上市公司内部创业团队","industryField":"移动互联网 · 医疗健康","score":1335,"district":null,"companyLabelList":["年底双薪","股票期权","带薪年假","招募合伙人"],"deliverCount":9,"leaderName":"杨怡","companySize":"500-2000人","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":48,"imstate":"today","createTimeSort":1462874275000,"positonTypesMap":null,"hrScore":74,"flowScore":144,"showCount":2085,"pvScore":77.9570832081189,"plus":"是","businessZones":null,"publisherId":3655492,"loginTime":1462873104000,"appShow":711,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":69,"adWord":0,"formatCreateTime":"17:57发布","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-09 09:46:32","companyId":113895,"positionName":"Android开发工程师","positionType":"移动开发","workYear":"3-5年","education":"本科","jobNature":"全职","companyShortName":"北京互动金服科技有限公司","city":"北京","salary":"15k-25k","financeStage":"成长型(不需要融资)","positionId":1473342,"companyLogo":"i/image/M00/03/D8/CgqKkVbEA_uAe1k4AAHTfy3RxPY812.jpg","positionFirstType":"技术","companyName":"互动科技","positionAdvantage":"五险一金 补充医疗 年终奖 福利津贴","industryField":"移动互联网 · O2O","score":1326,"district":"海淀区","companyLabelList":,"deliverCount":32,"leaderName":"暂没有填写","companySize":"50-150人","randomScore":0,"countAdjusted":false,"relScore":980,"adjustScore":48,"imstate":"today","createTimeSort":1462758392000,"positonTypesMap":null,"hrScore":82,"flowScore":153,"showCount":3741,"pvScore":67.01698353391613,"plus":"是","businessZones":["白石桥","魏公村","万寿寺","白石桥","魏公村","万寿寺"],"publisherId":3814477,"loginTime":1462874842000,"appShow":0,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":63,"adWord":0,"formatCreateTime":"1天前发布","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-10 16:50:51","companyId":23999,"positionName":"Android","positionType":"移动开发","workYear":"1-3年","education":"本科","jobNature":"全职","companyShortName":"南京智鹤电子科技有限公司","city":"长沙","salary":"8k-12k","financeStage":"成长型(A轮)","positionId":1804917,"companyLogo":"image1/M00/35/EB/CgYXBlWc5KGAVeL8AAAOi4lPhWU502.jpg","positionFirstType":"技术","companyName":"智鹤科技","positionAdvantage":"弹性工作制 技术氛围浓厚","industryField":"移动互联网","score":1322,"district":null,"companyLabelList":["股票期权","绩效奖金","专项奖金","年终分红"],"deliverCount":1,"leaderName":"暂没有填写","companySize":"50-150人","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":0,"imstate":"disabled","createTimeSort":1462870251000,"positonTypesMap":null,"hrScore":62,"flowScore":191,"showCount":283,"pvScore":15.939035855045429,"plus":"否","businessZones":null,"publisherId":282621,"loginTime":1462869967000,"appShow":0,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":69,"adWord":0,"formatCreateTime":"16:50发布","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-08 23:12:56","companyId":24287,"positionName":"android","positionType":"移动开发","workYear":"3-5年","education":"本科","jobNature":"全职","companyShortName":"杭州腾展科技有限公司","city":"杭州","salary":"15k-22k","financeStage":"成熟型(D轮及以上)","positionId":1197868,"companyLogo":"image1/M00/0B/7D/Cgo8PFTzIBOAEd2dAACMq9tQoMA797.png","positionFirstType":"技术","companyName":"腾展叮咚（Dingtone）","positionAdvantage":"每半年调整薪资，今年上市！","industryField":"移动互联网 · 社交网络","score":1322,"district":"西湖区","companyLabelList":["出国旅游","股票期权","精英团队","强悍的创始人"],"deliverCount":9,"leaderName":"魏松祥（Steve Wei)","companySize":"50-150人","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":48,"imstate":"disabled","createTimeSort":1462720376000,"positonTypesMap":null,"hrScore":71,"flowScore":137,"showCount":3786,"pvScore":87.51582865460942,"plus":"是","businessZones":["文三路","古荡","高新文教区"],"publisherId":2946659,"loginTime":1462891920000,"appShow":940,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":66,"adWord":0,"formatCreateTime":"2天前发布","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-10 09:38:43","companyId":19875,"positionName":"Android","positionType":"移动开发","workYear":"3-5年","education":"本科","jobNature":"全职","companyShortName":"维沃移动通信有限公司","city":"南京","salary":"12k-24k","financeStage":"初创型(未融资)","positionId":938099,"companyLogo":"image1/M00/00/25/Cgo8PFTUWH-Ab57wAABKOdLbNuw116.png","positionFirstType":"技术","companyName":"vivo","positionAdvantage":"vivo，追求极致","industryField":"移动互联网","score":1320,"district":"建邺区","companyLabelList":["年终分红","五险一金","带薪年假","年度旅游"],"deliverCount":4,"leaderName":"暂没有填写","companySize":"2000人以上","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":48,"imstate":"today","createTimeSort":1462844323000,"positonTypesMap":null,"hrScore":57,"flowScore":149,"showCount":981,"pvScore":72.14107985481958,"plus":"是","businessZones":["沙洲","小行","赛虹桥"],"publisherId":302876,"loginTime":1462871424000,"appShow":353,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":66,"adWord":0,"formatCreateTime":"09:38发布","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-10 09:49:59","companyId":20473,"positionName":"安卓开发工程师","positionType":"移动开发","workYear":"3-5年","education":"大专","jobNature":"全职","companyShortName":"广州棒谷网络科技有限公司","city":"广州","salary":"8k-15k","financeStage":"成长型(A轮)","positionId":1733545,"companyLogo":"image1/M00/0F/36/Cgo8PFT9AgGAciySAAA1THfEIAE433.jpg","positionFirstType":"技术","companyName":"广州棒谷网络科技有限公司","positionAdvantage":"五险一金 大平台 带薪休假","industryField":"电子商务","score":1320,"district":null,"companyLabelList":["项目奖金","绩效奖金","年终奖","五险一金"],"deliverCount":15,"leaderName":"大邹","companySize":"500-2000人","randomScore":0,"countAdjusted":false,"relScore":980,"adjustScore":48,"imstate":"today","createTimeSort":1462844999000,"positonTypesMap":null,"hrScore":79,"flowScore":144,"showCount":3943,"pvScore":77.78928844199473,"plus":"是","businessZones":null,"publisherId":235413,"loginTime":1462878251000,"appShow":1562,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":69,"adWord":0,"formatCreateTime":"09:49发布","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-04 11:41:59","companyId":87117,"positionName":"Android","positionType":"移动开发","workYear":"1-3年","education":"本科","jobNature":"全职","companyShortName":"南京信通科技有限责任公司","city":"南京","salary":"10k-12k","financeStage":"成长型(不需要融资)","positionId":966059,"companyLogo":"image1/M00/3F/51/CgYXBlXASfuADyOsAAA_na14zho635.jpg?cc=0.6724986131303012","positionFirstType":"技术","companyName":"联创集团信通科技","positionAdvantage":"提供完善的福利和薪酬晋升制度","industryField":"移动互联网 · 教育","score":1320,"district":"鼓楼区","companyLabelList":["节日礼物","带薪年假","补充医保","补充子女医保"],"deliverCount":4,"leaderName":"暂没有填写","companySize":"150-500人","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":48,"imstate":"today","createTimeSort":1462333319000,"positonTypesMap":null,"hrScore":88,"flowScore":143,"showCount":3161,"pvScore":79.41660570401937,"plus":"是","businessZones":["虎踞路","龙江","西桥"],"publisherId":2230973,"loginTime":1462871511000,"appShow":711,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":41,"adWord":0,"formatCreateTime":"2016-05-04","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-10 09:26:58","companyId":103051,"positionName":"Android","positionType":"移动开发","workYear":"1-3年","education":"大专","jobNature":"全职","companyShortName":"浙江米果网络股份有限公司","city":"杭州","salary":"12k-22k","financeStage":"成长型(A轮)","positionId":1233873,"companyLogo":"image2/M00/10/DF/CgqLKVYwKQSAR2p4AAJAM590SJM137.png?cc=0.17502541467547417","positionFirstType":"技术","companyName":"米果小站","positionAdvantage":"充分的发展成长空间","industryField":"移动互联网","score":1320,"district":"滨江区","companyLabelList":["年底双薪","股票期权","午餐补助","五险一金"],"deliverCount":5,"leaderName":"暂没有填写","companySize":"150-500人","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":48,"imstate":"disabled","createTimeSort":1462843618000,"positonTypesMap":null,"hrScore":72,"flowScore":137,"showCount":1582,"pvScore":87.93158026917071,"plus":"是","businessZones":["江南","长河","西兴"],"publisherId":2992735,"loginTime":1462889479000,"appShow":312,"calcScore":false,"showOrder":0,"haveDeliver":false,"orderBy":63,"adWord":0,"formatCreateTime":"09:26发布","totalCount":0,"searchScore":0.0},{"createTime":"2016-05-10 10:01:39","companyId":70044,"positionName":"Android","positionType":"移动开发","workYear":"1-3年","education":"大专","jobNature":"全职","companyShortName":"武汉平行世界网络科技有限公司","city":"武汉","salary":"9k-13k","financeStage":"初创型(天使轮)","positionId":664813,"companyLogo":"image2/M00/00/3E/CgqLKVXdccSAQE91AACv-6V33Vo860.jpg","positionFirstType":"技术","companyName":"平行世界","positionAdvantage":"弹性工作、带薪年假、待遇优厚、3号线直达","industryField":"电子商务 · 文化娱乐","score":1320,"district":"蔡甸区","companyLabelList":["年底双薪","待遇优厚","专项奖金","带薪年假"],"deliverCount":18,"leaderName":"暂没有填写","companySize":"50-150人","randomScore":0,"countAdjusted":false,"relScore":1000,"adjustScore":48,"imstate":"today","createTimeSort":1462845699000,"positonTypesMap":null,"hrScore":76,"flowScore":128,"showCount":1934,"pvScore":99.6001810598155,"plus":"是","businessZones":["沌口"],"publisherId":1694134,"loginTime":1462892151000,"appShow":759,"calcScore":false,"showOrder":1433141412915,"haveDeliver":false,"orderBy":68,"adWord":0,"formatCreateTime":"10:01发布","totalCount":0,"searchScore":0.0}]}}}
在XHR中点击JSON，就可以看到浏览器返回来的数据了。 是不是跟上面使用python程序抓取的一样呢？
是不是很简单？
如果想获得第2页，第3页的数据呢？
只需要修改pn＝x 中的值就可以了。
post_data = {'first':'true','kd':'Android','pn':'2'} ＃获取第2页的内容如果想要获取全部内容，写一个循环语句就可以了。
版权所有，转载请说明出处：www.30daydo.com
集思录这个站点的技术人员挺菜的，连cdn都不用
网络 • 李魔佛 发表了文章 • 0 个评论 • 1476 次浏览 • 2022-05-09 23:21
用稍微大点的ddos攻击下，就容易瘫痪大了的。。。。。。。
心真大。
用稍微大点的ddos攻击下，就容易瘫痪大了的。。。。。。。
心真大。
薅“疫情公益”羊毛，黑产恶意爬取各大出版社电子书上万册
python爬虫 • Magiccc 发表了文章 • 0 个评论 • 3237 次浏览 • 2020-02-26 13:17
疫情以来，所有企业都上班延期选择在线复工，在我们居家自我隔离期间，极验观察爬虫却没有消停，反而爬虫行为更加活跃且更胜往常。本周五，我们和无糖信息一起聊聊线上爬虫的“疫情”。
疫情以来，所有企业都上班延期选择在线复工，在我们居家自我隔离期间，极验观察爬虫却没有消停，反而爬虫行为更加活跃且更胜往常。本周五，我们和无糖信息一起聊聊线上爬虫的“疫情”。
修改win7 hosts文件IP域名映射问题的一个坑
网络安全 • 李魔佛 发表了文章 • 0 个评论 • 4408 次浏览 • 2019-04-13 20:33
因为讨厌一些网站的弹窗，还有想要高效地学习，把一些娱乐网站给屏蔽了，比如QQ。
所以在hosts文件中添加了如下内容：
路径：C:\Windows\System32\drivers\etc
127.0.0.1 qq.com
127.0.0.1 cnbeta.com
10.4.16.75 windows10.microdone.cn
50.112.79.106 mukodik
0.0.0.0 account.jetbrains.com
结果发现一直无法生效。
后来在ping下发现，ping qq.com 返回的是127.0.0.1是生效的。
但是ping www.qq.com是能够ping到腾讯的服务器的，原来问题在于，你在浏览器输入qq.com，浏览器默认帮你加了www.前缀。
所以你要hosts文件生效，就需要把www.的前缀也加上去。
然后重新刷新下DNS就可以了
ipconfig /flushdns
还有就是，如果你用了代理软件，比如lantern之类的，这个hosts文件也不会起作用。 查看全部
所以在hosts文件中添加了如下内容：
路径：C:\Windows\System32\drivers\etc
127.0.0.1 qq.com
127.0.0.1 cnbeta.com
10.4.16.75 windows10.microdone.cn
50.112.79.106 mukodik
0.0.0.0 account.jetbrains.com
结果发现一直无法生效。
后来在ping下发现，ping qq.com 返回的是127.0.0.1是生效的。
但是ping www.qq.com是能够ping到腾讯的服务器的，原来问题在于，你在浏览器输入qq.com，浏览器默认帮你加了www.前缀。
所以你要hosts文件生效，就需要把www.的前缀也加上去。
然后重新刷新下DNS就可以了
ipconfig /flushdns
还有就是，如果你用了代理软件，比如lantern之类的，这个hosts文件也不会起作用。 查看全部
因为讨厌一些网站的弹窗，还有想要高效地学习，把一些娱乐网站给屏蔽了，比如QQ。
所以在hosts文件中添加了如下内容：
路径：C:\Windows\System32\drivers\etc
结果发现一直无法生效。
后来在ping下发现，ping qq.com 返回的是127.0.0.1是生效的。
但是ping www.qq.com是能够ping到腾讯的服务器的，原来问题在于，你在浏览器输入qq.com，浏览器默认帮你加了www.前缀。
所以你要hosts文件生效，就需要把www.的前缀也加上去。
然后重新刷新下DNS就可以了
ipconfig /flushdns
还有就是，如果你用了代理软件，比如lantern之类的，这个hosts文件也不会起作用。
所以在hosts文件中添加了如下内容：
路径：C:\Windows\System32\drivers\etc
127.0.0.1 qq.com
127.0.0.1 cnbeta.com
10.4.16.75 windows10.microdone.cn
50.112.79.106 mukodik
0.0.0.0 account.jetbrains.com
结果发现一直无法生效。
后来在ping下发现，ping qq.com 返回的是127.0.0.1是生效的。
但是ping www.qq.com是能够ping到腾讯的服务器的，原来问题在于，你在浏览器输入qq.com，浏览器默认帮你加了www.前缀。
所以你要hosts文件生效，就需要把www.的前缀也加上去。
然后重新刷新下DNS就可以了
ipconfig /flushdns
还有就是，如果你用了代理软件，比如lantern之类的，这个hosts文件也不会起作用。
阻挡黑客的最原始最暴力的方法 ---用胶带粘住你的摄像头
网络安全 • 李魔佛 发表了文章 • 0 个评论 • 4360 次浏览 • 2016-06-23 22:15
Facebook 的 CEO 扎克伯格经常喜欢发发照片跟网友互动什么的。最近他在 Facebook 个人账号上面发了一条消息，照片中的他扛着一个镜框庆祝 Instagram 的月活用户数突破了 5 亿大关。但是眼尖的用户却注意到了其他的细节—似乎扎克的笔记本摄像头和麦克风插口都是用胶带蒙住的（题图红色箭头所指）。
扎克为什么要把摄像头蒙住呢？这得先说一下 Ratting（Remote Access Trojan）这种行为。所谓的 Ratting，是指黑客通过植入木马远程控制受害用户设备的行为，而进行这种行为的黑客一般叫做 ratter。除了窃取设备的敏感数据以外，激活用户摄像头和麦克风偷拍视频也是 ratter 常干的事情。
不过扎克伯格这么专业的人也把摄像头蒙起来究竟是偏执狂还是好做法呢？安全专家认为是后者，原因有三：
一是扎克伯格是一个高价值的攻击目标。无论是情报机构还是为了罪犯无疑都会对扎克伯格的资料虎视眈眈，而对于那些为了证明自己黑客功力的人来说，扎克也是一个很理想的目标。所以采取预防措施是很自然的事情。
二是把采集音视频的入口盖住是一种成本低廉且基本的安全防护办法。如果想窃听安全会议，有经验的黑客一般都会先数数哪些设备没有遮住摄像头然后再确定下手的目标。
三是扎克未必就不会被攻破。事实上本月初就爆出了黑客袭击扎克伯格部分社交网络，盗取了他的 Twitter、Pinterest、LinkedIn 帐号的消息。把这两件事联系在一起，更容易解释他的那台 Macbook 上面的胶带。
事实上，采取这种做法的人并不止扎克一个。就连 FBI 局长 James Comey 也把自己的笔记本摄像头蒙上了胶带—原因很简单，因为他看到一个比他更聪明的人也这么干。所以你要不要也蒙上呢？ 查看全部
扎克为什么要把摄像头蒙住呢？这得先说一下 Ratting（Remote Access Trojan）这种行为。所谓的 Ratting，是指黑客通过植入木马远程控制受害用户设备的行为，而进行这种行为的黑客一般叫做 ratter。除了窃取设备的敏感数据以外，激活用户摄像头和麦克风偷拍视频也是 ratter 常干的事情。
不过扎克伯格这么专业的人也把摄像头蒙起来究竟是偏执狂还是好做法呢？安全专家认为是后者，原因有三：
一是扎克伯格是一个高价值的攻击目标。无论是情报机构还是为了罪犯无疑都会对扎克伯格的资料虎视眈眈，而对于那些为了证明自己黑客功力的人来说，扎克也是一个很理想的目标。所以采取预防措施是很自然的事情。
二是把采集音视频的入口盖住是一种成本低廉且基本的安全防护办法。如果想窃听安全会议，有经验的黑客一般都会先数数哪些设备没有遮住摄像头然后再确定下手的目标。
三是扎克未必就不会被攻破。事实上本月初就爆出了黑客袭击扎克伯格部分社交网络，盗取了他的 Twitter、Pinterest、LinkedIn 帐号的消息。把这两件事联系在一起，更容易解释他的那台 Macbook 上面的胶带。
事实上，采取这种做法的人并不止扎克一个。就连 FBI 局长 James Comey 也把自己的笔记本摄像头蒙上了胶带—原因很简单，因为他看到一个比他更聪明的人也这么干。所以你要不要也蒙上呢？ 查看全部
Facebook 的 CEO 扎克伯格经常喜欢发发照片跟网友互动什么的。最近他在 Facebook 个人账号上面发了一条消息，照片中的他扛着一个镜框庆祝 Instagram 的月活用户数突破了 5 亿大关。但是眼尖的用户却注意到了其他的细节—似乎扎克的笔记本摄像头和麦克风插口都是用胶带蒙住的（题图红色箭头所指）。
扎克为什么要把摄像头蒙住呢？这得先说一下 Ratting（Remote Access Trojan）这种行为。所谓的 Ratting，是指黑客通过植入木马远程控制受害用户设备的行为，而进行这种行为的黑客一般叫做 ratter。除了窃取设备的敏感数据以外，激活用户摄像头和麦克风偷拍视频也是 ratter 常干的事情。
不过扎克伯格这么专业的人也把摄像头蒙起来究竟是偏执狂还是好做法呢？安全专家认为是后者，原因有三：
一是扎克伯格是一个高价值的攻击目标。无论是情报机构还是为了罪犯无疑都会对扎克伯格的资料虎视眈眈，而对于那些为了证明自己黑客功力的人来说，扎克也是一个很理想的目标。所以采取预防措施是很自然的事情。
二是把采集音视频的入口盖住是一种成本低廉且基本的安全防护办法。如果想窃听安全会议，有经验的黑客一般都会先数数哪些设备没有遮住摄像头然后再确定下手的目标。
三是扎克未必就不会被攻破。事实上本月初就爆出了黑客袭击扎克伯格部分社交网络，盗取了他的 Twitter、Pinterest、LinkedIn 帐号的消息。把这两件事联系在一起，更容易解释他的那台 Macbook 上面的胶带。
事实上，采取这种做法的人并不止扎克一个。就连 FBI 局长 James Comey 也把自己的笔记本摄像头蒙上了胶带—原因很简单，因为他看到一个比他更聪明的人也这么干。所以你要不要也蒙上呢？
扎克为什么要把摄像头蒙住呢？这得先说一下 Ratting（Remote Access Trojan）这种行为。所谓的 Ratting，是指黑客通过植入木马远程控制受害用户设备的行为，而进行这种行为的黑客一般叫做 ratter。除了窃取设备的敏感数据以外，激活用户摄像头和麦克风偷拍视频也是 ratter 常干的事情。
不过扎克伯格这么专业的人也把摄像头蒙起来究竟是偏执狂还是好做法呢？安全专家认为是后者，原因有三：
一是扎克伯格是一个高价值的攻击目标。无论是情报机构还是为了罪犯无疑都会对扎克伯格的资料虎视眈眈，而对于那些为了证明自己黑客功力的人来说，扎克也是一个很理想的目标。所以采取预防措施是很自然的事情。
二是把采集音视频的入口盖住是一种成本低廉且基本的安全防护办法。如果想窃听安全会议，有经验的黑客一般都会先数数哪些设备没有遮住摄像头然后再确定下手的目标。
三是扎克未必就不会被攻破。事实上本月初就爆出了黑客袭击扎克伯格部分社交网络，盗取了他的 Twitter、Pinterest、LinkedIn 帐号的消息。把这两件事联系在一起，更容易解释他的那台 Macbook 上面的胶带。
事实上，采取这种做法的人并不止扎克一个。就连 FBI 局长 James Comey 也把自己的笔记本摄像头蒙上了胶带—原因很简单，因为他看到一个比他更聪明的人也这么干。所以你要不要也蒙上呢？
