忘记的QQ密码
曾经用了十多年的QQ忘记了密码,只有选择找回,找回密码问题是我的梦想是什么,过了这么多年,完全忘了当时是怎么回答的,就开始挨个慢慢试,房子,钱,车,工作,都不对,直到那时才意识到,弄丢的并QQ密码,而是梦想
windows下logstash6.5安装配置 网上的教程是linux,有坑
https://www.cnblogs.com/jstarseven/p/7704893.html
百度里搜到的logstash安装配置教程 千篇一律都是这样子。
启动服务测试一下是否安装成功:
上面的是linux运行的。
windows下把logstash改为logstash.bat 然后运行。
报错:
后来搜了下国外的网站。原来问题出在那个单引号上,把单引号改为双引号就可以了。
收起阅读 »
百度里搜到的logstash安装配置教程 千篇一律都是这样子。
启动服务测试一下是否安装成功:
cd bin
./logstash -e 'input { stdin { } } output { stdout {} }'
上面的是linux运行的。
windows下把logstash改为logstash.bat 然后运行。
报错:
ERROR: Unknown command '{'
See: 'bin/logstash --help'
[ERROR] 2020-02-28 10:04:29.307 [main] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
后来搜了下国外的网站。原来问题出在那个单引号上,把单引号改为双引号就可以了。
收起阅读 »
薅“疫情公益”羊毛,黑产恶意爬取各大出版社电子书上万册
疫情以来,所有企业都上班延期选择在线复工,在我们居家自我隔离期间,极验观察爬虫却没有消停,反而爬虫行为更加活跃且更胜往常。本周五,我们和无糖信息一起聊聊线上爬虫的“疫情”。
个人的知识星球
最近开通了知识星球,尝试在里面分享一些量化投资套利的学习教程。在里面你可以对不懂的问题进行提问,球主会一一的耐心解答。
微信扫一扫加入我的知识星球
星球的第一篇文章
python获取全市场LOF基金折溢价数据并进行套利
市场是总共的LOF基金有301只(上图右下角的圈圈是所有基金的条数),而集思录上只有120只左右,所以有些溢价厉害(大于10%)的LOF基金并没有在集思录的网站上显示,这对于专注于套利的投资者来说,会损失很多潜在的套利机会。
点击查看大图
我回复了该贴后,有大量的人私信我,问我能否提供一份这个数据,或者教对方如何获取这些数据。 因为人数众多,也没有那么多精力来一一回答。毕竟不同人的水平背景不一样,逐个回答起来也很累,所以就回答了几个朋友的问题后就一一婉拒了。
然后在几个投资群里,居然也有人提到这个数据,在咨询如何才能获取到这个完整的数据,并且可以实时更新显示。 因为我的微信群昵称和集思录是一样的,所以不少人@我,我也都简单的回复了下,是使用python抓取的数据,数据保存到Mysql和MongoDB。 代码行数不多,100行都不到。
具体实现在星球会有完整代码。 收起阅读 »
微信扫一扫加入我的知识星球
星球的第一篇文章
python获取全市场LOF基金折溢价数据并进行套利
市场是总共的LOF基金有301只(上图右下角的圈圈是所有基金的条数),而集思录上只有120只左右,所以有些溢价厉害(大于10%)的LOF基金并没有在集思录的网站上显示,这对于专注于套利的投资者来说,会损失很多潜在的套利机会。
点击查看大图
我回复了该贴后,有大量的人私信我,问我能否提供一份这个数据,或者教对方如何获取这些数据。 因为人数众多,也没有那么多精力来一一回答。毕竟不同人的水平背景不一样,逐个回答起来也很累,所以就回答了几个朋友的问题后就一一婉拒了。
然后在几个投资群里,居然也有人提到这个数据,在咨询如何才能获取到这个完整的数据,并且可以实时更新显示。 因为我的微信群昵称和集思录是一样的,所以不少人@我,我也都简单的回复了下,是使用python抓取的数据,数据保存到Mysql和MongoDB。 代码行数不多,100行都不到。
具体实现在星球会有完整代码。 收起阅读 »
希望能够找到一个可以一起对彪的人
看了老男孩的某个视频教程,觉得这个挺有道理的。
目前貌似身边缺少一个这样的人,希望在这个网站上能够遇到一个这样的朋友。
有意的可以私聊吧。 非诚勿扰。
目前貌似身边缺少一个这样的人,希望在这个网站上能够遇到一个这样的朋友。
有意的可以私聊吧。 非诚勿扰。
证券etf和券商etf的区别
证券etf是:国泰中证全指证券公司ETF(512880)LOF/ETF
而券商etf是:
华宝中证全指证券公司ETF (512000)LOF/ETF
二者都是指数/lof基金,而且持仓差不多都一样的。
证券etf的持仓:
券商etf的持仓:
不同的是规模,证券ETF的规模要比券商etf的规模要大得多。目前是2倍左右的差距【2019年的数据】。 所以如果你关心的是流动性,那么可以买证券ETF。
最新的规模【2020-06】其实二者在缩小,具体数据见文末。
证券ETF的规模是190亿
而券商ETF规模为160亿。(最近增长比较多,笔者在1年前记录的数据为30亿,可以到http://fundf10.eastmoney.com/gmbd_512000.html 查看历史规模。现在券商ETF的规模翻了一番。
更多量化分析,关注公众号:可转债量化分析
券商开户股票万一免5,基金申购一折,拖拉机6+1,关注公众号留言。
可转债手续费 百万分之二,一万块手续费才2分钱,没有最低限制(没有最低收1元,1毛这种)
(教你使用python进行量化分析股票,可转债数据)
收起阅读 »
而券商etf是:
华宝中证全指证券公司ETF (512000)LOF/ETF
二者都是指数/lof基金,而且持仓差不多都一样的。
证券etf的持仓:
券商etf的持仓:
不同的是规模,证券ETF的规模要比券商etf的规模要大得多。目前是2倍左右的差距【2019年的数据】。 所以如果你关心的是流动性,那么可以买证券ETF。
最新的规模【2020-06】其实二者在缩小,具体数据见文末。
证券ETF的规模是190亿
而券商ETF规模为160亿。(最近增长比较多,笔者在1年前记录的数据为30亿,可以到http://fundf10.eastmoney.com/gmbd_512000.html 查看历史规模。现在券商ETF的规模翻了一番。
更多量化分析,关注公众号:可转债量化分析
券商开户股票万一免5,基金申购一折,拖拉机6+1,关注公众号留言。
可转债手续费 百万分之二,一万块手续费才2分钱,没有最低限制(没有最低收1元,1毛这种)
(教你使用python进行量化分析股票,可转债数据)
收起阅读 »
socketio中client的sio wait用法
用于阻塞当前的线程,后面的操作不会进行,直到服务端断开。
比如上述代码中,如果调用了sio.wait() , 那么next是不会被打印的。
如果注释掉后,那么next就可以正常被打印。 收起阅读 »
import time
import socketio
sio = socketio.Client()
start_timer = None
def send_ping():
global start_timer
start_timer = time.time()
sio.emit('ping_from_client')
@sio.event
def connect():
print('connected to server')
send_ping()
@sio.event
def pong_from_server(data):
global start_timer
latency = time.time() - start_timer
print('latency is {0:.2f} ms'.format(latency * 1000))
sio.sleep(1)
send_ping()
if __name__ == '__main__':
sio.connect('http://localhost:5000')
sio.wait()
print('next')
比如上述代码中,如果调用了sio.wait() , 那么next是不会被打印的。
如果注释掉后,那么next就可以正常被打印。 收起阅读 »
dataframe 根据日期重采样 计算个数
按照日期重新采样,计算每天的个数new_df = df.resample('D').count()
失落
记住今天,铭记。
bandcamp移动开发更简单
bandcamp移动开发更简单http://ydkfpgjd.bandcamp.com/
http://TalkingData.bandcamp.com/
http://Bugly.bandcamp.com/
http://Box2D.bandcamp.com/
http://aineice.bandcamp.com/
http://wyyp.bandcamp.com/
http://Prepo.bandcamp.com/
http://Chipmunk.bandcamp.com/
http://openinstall.bandcamp.com/
http://MobileInsight.bandcamp.com/
http://zhugelo.bandcamp.com/
http://CobubRazor.bandcamp.com/
http://Testin.bandcamp.com/
http://crashlytics.bandcamp.com/
http://APKProtect.bandcamp.com/
http://Ucloud.bandcamp.com/
http://ydkfpgj.bandcamp.com/releases
http://TalkingData.bandcamp.com/releases
http://Bugly.bandcamp.com/releases
http://Box2D.bandcamp.com/releases
http://aineice.bandcamp.com/releases
http://wyyp.bandcamp.com/releases
http://Prepo.bandcamp.com/releases
http://Chipmunk.bandcamp.com/releases
http://openinstall.bandcamp.com/releases
http://MobileInsight.bandcamp.com/releases
http://zhugelo.bandcamp.com/releases
http://CobubRazor.bandcamp.com/releases
http://Testin.bandcamp.com/releases
http://crashlytics.bandcamp.com/releases
http://APKProtect.bandcamp.com/releases
http://Ucloud.bandcamp.com/releases 收起阅读 »
http://TalkingData.bandcamp.com/
http://Bugly.bandcamp.com/
http://Box2D.bandcamp.com/
http://aineice.bandcamp.com/
http://wyyp.bandcamp.com/
http://Prepo.bandcamp.com/
http://Chipmunk.bandcamp.com/
http://openinstall.bandcamp.com/
http://MobileInsight.bandcamp.com/
http://zhugelo.bandcamp.com/
http://CobubRazor.bandcamp.com/
http://Testin.bandcamp.com/
http://crashlytics.bandcamp.com/
http://APKProtect.bandcamp.com/
http://Ucloud.bandcamp.com/
http://ydkfpgj.bandcamp.com/releases
http://TalkingData.bandcamp.com/releases
http://Bugly.bandcamp.com/releases
http://Box2D.bandcamp.com/releases
http://aineice.bandcamp.com/releases
http://wyyp.bandcamp.com/releases
http://Prepo.bandcamp.com/releases
http://Chipmunk.bandcamp.com/releases
http://openinstall.bandcamp.com/releases
http://MobileInsight.bandcamp.com/releases
http://zhugelo.bandcamp.com/releases
http://CobubRazor.bandcamp.com/releases
http://Testin.bandcamp.com/releases
http://crashlytics.bandcamp.com/releases
http://APKProtect.bandcamp.com/releases
http://Ucloud.bandcamp.com/releases 收起阅读 »
mongodb日期条件查找
有时候用new Date() 不管用
多线程调用yolo模型会出错(包括使用cv2载入yolo模型)
占坑。
多线程调用yolo模型会出错(包括使用cv2载入yolo模型)
多线程调用yolo模型会出错(包括使用cv2载入yolo模型)
RuntimeError: `get_session` is not available when using TensorFlow 2.0.
这个问题是TensorFlow版本问题,在2.0以上get_session是被移除了。需要做一些修改,或者把tf降级。可以安装1.15版本
pip install tensorflow==1.15 --upgrade
pip install tensorflow==1.15 --upgrade
Here, we will see how we can upgrade our code to work with tensorflow 2.0.收起阅读 »
This error is usually faced when we are loading pre-trained model with tensorflow session/graph or we are building flask api over a pre-trained model and loading model in tensorflow graph to avoid collision of sessions while application is getting multiple requests at once or say in case of multi-threadinng
After tensorflow 2.0 upgrade, i also started facing above error in one of my project when i had built api of pre-trained model with flask. So i looked around in tensorflow 2.0 documents to find a workaround, to avoid this runtime error and upgrade my code to work with tensorflow 2.0 as well rather than downgrading it to tensorflow 1.x .
I had a project on which i had written tutorial as well on how to build Flask api on trained keras model of text classification and then use it in production
But this project was not working after tensorflow upgrade and was facing runtime error.
Stacktrace of error was something like below:
File "/Users/Upasana/Documents/playground/deploy-keras-model-in-production/src/main.py", line 37, in model_predict
with backend.get_session().graph.as_default() as g:
File "/Users/Upasana/Documents/playground/deploy-keras-model-in-production/venv-tf2/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 379, in get_session
'`get_session` is not available '
RuntimeError: `get_session` is not available when using TensorFlow 2.0.
Related code to get model
with backend.get_session().graph.as_default() as g:
model = SentimentService.get_model1()
Related code to load model
def load_deep_model(self, model):
json_file = open('./src/mood-saved-models/' + model + '.json', 'r')
loaded_model_json = json_file.read()
loaded_model = model_from_json(loaded_model_json)
loaded_model.load_weights("./src/mood-saved-models/" + model + ".h5")
loaded_model._make_predict_function()
return loaded_model
get_session is removed in tensorflow 2.0 and hence not available.
so, in order to load saved model we switched methods. Rather than using keras’s load_model, we used tensorflow to load model so that we can load model using distribution strategy.
Note
The tf.distribute.Strategy API provides an abstraction for distributing your training across multiple processing units.
New code to get model
another_strategy = tf.distribute.MirroredStrategy()
with another_strategy.scope():
model = SentimentService.get_model1()
New code to load model
def load_deep_model(self, model):
loaded_model = tf.keras.models.load_model("./src/mood-saved-models/"model + ".h5")
return loaded_model
This worked and solved the problem with runtime error of get_session not available in tensorflow 2.0 . You can refer to Tensorflow 2.0 upgraded article too
Hope, this will solve your problem too. Thanks for following this article.
yolo voc_label 源码分析
作用读取每个xml文件,把坐标转化为相对坐标,对应文件名保存起来
收起阅读 »
import xml.etree.ElementTree as ET
import pickle
import os
from os import listdir, getcwd
from os.path import join
sets = [('2012', 'train'), ('2012', 'val'), ('2007', 'train'), ('2007', 'val'), ('2007', 'test')]
# 20类
classes = ["aeroplane", "bicycle", "bird", "boat", "bottle", "bus", "car", "cat", "chair", "cow", "diningtable", "dog",
"horse", "motorbike", "person", "pottedplant", "sheep", "sofa", "train", "tvmonitor"]
# size w,h
# box x-min,x-max,y-min,y-max
def convert(size, box):
dw = 1. / size[0]
dh = 1. / size[1]
x = (box[0] + box[1]) / 2.0 # 中心点位置
y = (box[2] + box[3]) / 2.0
w = box[1] - box[0]
h = box[3] - box[2]
x = x * dw
w = w * dw
y = y * dh
h = h * dh # 全部转化为相对坐标
return (x, y, w, h)
def convert_annotation(year, image_id):
# 找到2个同样的文件
in_file = open('VOCdevkit/VOC%s/Annotations/%s.xml' % (year, image_id))
out_file = open('VOCdevkit/VOC%s/labels/%s.txt' % (year, image_id), 'w')
tree = ET.parse(in_file)
root = tree.getroot()
size = root.find('size')
w = int(size.find('width').text)
h = int(size.find('height').text)
for obj in root.iter('object'):
difficult = obj.find('difficult').text
cls = obj.find('name').text
if cls not in classes or int(difficult) == 1: # difficult ==1 的不要了
continue
cls_id = classes.index(cls) # 排在第几位
xmlbox = obj.find('bndbox')
b = (float(xmlbox.find('xmin').text), float(xmlbox.find('xmax').text), float(xmlbox.find('ymin').text),
float(xmlbox.find('ymax').text))
# 传入的是w,h 与框框的周边
bb = convert((w, h), b)
out_file.write(str(cls_id) + " " + " ".join([str(a) for a in bb]) + '\n')
wd = getcwd()
for year, image_set in sets:
# ('2012', 'train') 循环5次
# 创建目录 一次性
if not os.path.exists('VOCdevkit/VOC%s/labels/' % (year)):
os.makedirs('VOCdevkit/VOC%s/labels/' % (year))
# 图片的id数据
image_ids = open('VOCdevkit/VOC%s/ImageSets/Main/%s.txt' % (year, image_set)).read().strip().split()
# 结果写入这个文件
list_file = open('%s_%s.txt' % (year, image_set), 'w')
for image_id in image_ids:
list_file.write('%s/VOCdevkit/VOC%s/JPEGImages/%s.jpg\n' % (wd, year, image_id)) # 补全路径
convert_annotation(year, image_id)
list_file.close()
收起阅读 »
用docker编译go代码
如果偶尔需要编译go代码,但是又不想要安装一堆乱七八糟的依赖和Go编译器,可以利用docker来实现。 应该是解决起来话费时间最小的。
1. 用文本编辑你的go代码,现在以hello world为例:
package main import "fmt" func main() {
/* 这是我的第一个简单的程序 */
fmt.Println("Hello, World!")
}
2. 然后直接使用docker执行编译。docker首先会自动去下载go的编译器,顺便把所有的依赖给解决掉
docker run --rm -v "$(pwd)":/usr/src/hello -w /usr/src/hello golang:1.3 go build -v
最后会在本地生成一个编译好的hello静态文件。
上述docker命令的具体含义就是把当前路径挂在到docker容器里头,然后切换到改到改路径下,然后进行编译。
收起阅读 »
docker实战 代码勘误
作者在官网刊登的勘误记录:
看这本书的作者一定要看,不然坑挺多的。
一路采坑过来的哭着说。
看这本书的作者一定要看,不然坑挺多的。
一路采坑过来的哭着说。
Last updated August 21, 2016收起阅读 »
In an effort to offer continued support beyond publication, we have listed many updates to code due to version updates.
[code - omission] Page 18
The command to start the "mailer" is missing a line. Where the book reads:
docker run -d \
--name mailer \
the proper command should read:
docker run -d \
--name mailer \
dockerinaction/ch2_mailer
[code - regression] Page 68
Newer versions of Docker have changed the structure of the JSON returned by the docker inspect subcommand. If the following command does not work then use the replacement. Original:
docker inspect --format "{{json .Volumes}}" reader
Replacement:
docker inspect --format "{{json .Mounts}}" reader
[code - regression] Page 69
Newer versions of Docker have changed the structure of the JSON returned by the docker inspect subcommand. If the following command does not work then use the replacement. Original:
docker inspect --format "{{json .Volumes}}" student
Replacement:
docker inspect --format "{{json .Mounts}}" student
[code - regression] Page 74
The alpine image entrypoint has changed since original publication and has been unset. The last command on the page should now read:
docker run --rm \
--volumes-from tools \
--entrypoint /bin/sh \
alpine:latest \
-c 'ls /operations/*'
[code - regression] Page 75
The docker exec example on the top of page 75 was printed with the wrong tool name. The correct command is:
docker exec important_application /operations/tools/diagnostics
[code - regression] Page 86
It appears that nslookup behavior in the alpine image has changed. To run the example use the busybox:1 image.
docker run --rm \
--hostname barker \
busybox:1 \
nslookup barker
[code - regression] Page 87 (top)
It appears that nslookup behavior in the alpine image has changed. To run the example use the busybox:1 image.
docker run --rm \
--dns 8.8.8.8 \
busybox:1 \
nslookup docker.com
[code - regression] Page 87 (bottom)
It appears that nslookup behavior in the alpine image has changed. To run the example use the busybox:1 image.
docker run --rm \
--dns-search docker.com \
busybox:1 \
nslookup registry.hub
[code - regression] Page 88 (bottom)
It appears that nslookup behavior in the alpine image has changed. To run the example use the busybox:1 image.
docker run --rm \
--add-host test:10.10.10.255 \
busybox:1 \
nslookup test
[code - regression] Page 106
There are a few new problems with this example. First, the named repository (dockerfile/mariadb) no longer exists. You can use mariadb:5.5 as replacement. However, the second problem is that containers created from the mariadb image perform certain initialization at startup. That initialization work requires certain capabilities and to be started with the default user. The system should instead drop permissions after the initialization work is complete. Note that the real value of this example is in demonstrating different resource isolation mechanisms. It is not so important that you get it working. You can start the database with the following command:
docker run -d --name ch6_mariadb \
--memory 256m \
--cpu-shares 1024 \
--cap-drop net_raw \
-e MYSQL_ROOT_PASSWORD=test \
mariadb:5.5
[code - regression] Page 107
Containers created from the wordpress:4.1 image perform certain initialization at startup and expect certain environment variables. That initialization work requires certain capabilities and to be started with the default user. The system should instead drop permissions after the initialization work is complete. Note that the real value of this example is in demonstrating different resource isolation mechanisms. It is not so important that you get it working. You can start wordpress with the following command:
docker run -d -P --name ch6_wordpress \
--memory 512m \
--cpu-shares 512 \
--cap-drop net_raw \
-e WORDPRESS_DB_PASSWORD=test \
mariadb:5.5
[code - typo] Page 109
The device access example is missing the "run" subcommand. The command listed as:
docker -it --rm \
--device /dev/video0:/dev/video0 \
ubuntu:latest ls -al /dev
should have been written as:
docker run -it --rm \
--device /dev/video0:/dev/video0 \
ubuntu:latest ls -al /dev
[code - typo] Page 110 - 111
Several commands are missing the "run" subcommand. In each case the command begins with
docker -d ...
and should have been written as:
docker run -d ...
[code - regression] Page 115 (bottom)
The busybox and alpine images have been updated to fix the problem described in the paragraph below. The 'su' command does not have the SUID bit set and will not provide any automatic privilege escalation.
[command correction] Page 116
Boot2Docker has been discontinued and rolled into a newer project called Docker Machine. Because a reader is unlikely to have the boot2docker command installed, the command at the top of this page should be changed from:
boot2docker ssh
to the Docker Machine equivalent:
docker-machine ssh default
where default is the name of the machine you created.
[code - regression] Page 119
The ifconfig command has since been removed from ubuntu:latest. Instead of using the ubuntu:latest image for these examples use ubuntu:trusty. The example using ifconfig should look like:
docker run --rm \
--privileged \
ubuntu:trusty ifconfig
[Illustration Mistake] Page 136
Image layer ced2 on the left side of the illustration is listed at c3d2 on the right side. These two layers should represent the same item.
[code - typo] Page 140
Containers need not be in a running state in order to export their file system. The first command on page 140 uses the "run" subcommand but the command listed will never be able to start. Replace "run" with "create." The command should appear as follows:
docker create --name export-test \
dockerinaction/ch7_packed:latest ./echo For Export
[code - missing line] Page 146
In the example Dockerfile near the top of the page the line with the RUN directive is missing part of the instruction. That line should read:
RUN apt-get update && apt-get install -y git
[code - evolution] Page 215
The registry:2 configuration file now requires the population of additional fields under "maintenance > uploadpurging." The example should currently look like:
# Filename: s3-config.yml
version: 0.1
log:
level: debug
fields:
service: registry
environment: development
storage:
cache:
layerinfo: inmemory
s3:
accesskey:
secretkey:
region:
bucket:
encrypt: true
secure: true
v4auth: true
chunksize: 5242880
rootdirectory: /s3/object/name/prefix
maintenance:
uploadpurging:
enabled: true
age: 168h
interval: 24h
dryrun: false
readonly:
enabled: false
http:
addr: :5000
secret: asecretforlocaldevelopment
debug:
addr: localhost:5001
[code - evolution] Page 216
The registry:2 configuration file now requires the population of additional fields under "maintenance > uploadpurging." The example should currently look like:
# Filename: rados-config.yml
version: 0.1
log:
level: debug
fields:
service: registry
environment: development
storage:
cache:
layerinfo: inmemory
rados:
poolname: radospool
username: radosuser
chunksize: 4194304
maintenance:
uploadpurging:
enabled: false
age: 168h
interval: 24h
dryrun: false
readonly:
enabled: false
http:
addr: :5000
secret: asecretforlocaldevelopment
debug:
addr: localhost:5001
[code - evolution] Page 218
The registry:2 configuration file now requires the population of additional fields under "maintenance > uploadpurging." The example should currently look like:
# Filename: redis-config.yml
version: 0.1
log:
level: debug
fields:
service: registry
environment: development
http:
addr: :5000
secret: asecretforlocaldevelopment
debug:
addr: localhost:5001
storage:
cache:
blobdescriptor: redis
s3:
accesskey:
secretkey:
region:
bucket:
encrypt: true
secure: true
v4auth: true
chunksize: 5242880
rootdirectory: /s3/object/name/prefix
maintenance:
uploadpurging:
enabled: true
age: 168h
interval: 24h
dryrun: false
readonly:
enabled: false
redis:
addr: redis-host:6379
password: asecret
dialtimeout: 10ms
readtimeout: 10ms
writetimeout: 10ms
pool:
maxidle: 16
maxactive: 64
idletimeout: 300s
[code - typo] Page 220
The name of the file shown should be scalable-config.yml as in previous examples. This example also requires the addition of the newer uploadpurging attributes. The mainenance section of the file should be as follows:
maintenance:
uploadpurging:
enabled: true
age: 168h
interval: 24h
dryrun: false
readonly:
enabled: false
[text - typo] Page 240
In the second paragraph the reader is instructed to, "Open ./coffee/api/api.py" this is not the correct location of the file. The correct file location is at, "./coffee/app/api.py."
[text - typo] Page 262
The refere nce to "flock.json" in the first sentence of the third paragraph should be "flock.yml."
[code - typo] Page 270
The git clone command uses the ssh protocol instead of https. The command should read as follows:
git clone https://github.com/dockerinact ... i.git
jieba.posseg TypeError: cannot unpack non-iterable pair object 词性分析报错
词性标注的例子出现错误 'pair' object is not iterable
例子:
例子:
import jieba.posseg as pseg
seg_list = pseg.cut("我爱北京天安门")
for word,flag in seg_list:
print(word)
print(flag)
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-5-f105f6980f88> in <module>()
1 import jieba.posseg as pseg
2 seg_list = pseg.cut("我爱北京天安门")
----> 3 for word,flag in seg_list:
4 print(word)
5 print(flag)
TypeError: cannot unpack non-iterable pair object
原因是新版本中seg_list是一个生成器,所以只能 for win seg_list
然后从word中解包出来
print(w.word)
print(w.flag)
这样问题就解决了。收起阅读 »
scrapy在settings中定义变量不能包含小写!
如果变量名包含小写字母,那么你的变量会被过滤掉,在scrapy编码的其他地方都会无法被识别。
比如定义了一个叫 Redis_host = '192.168.1.1',的值
然后在spider中,如果你调用self.settings.get('Redis_host')
那么返回值是 None。
如果用REDIS_HOST定义,那么就可以正确返回它的值。
如果你一定要用小写,也有其他方法可正常调用。
先导入settings文件
fromt xxxx import setttings # xxx为项目名
host = settings.Redis_host # 直接导入一个文件的形式来调用是可以的 收起阅读 »
比如定义了一个叫 Redis_host = '192.168.1.1',的值
然后在spider中,如果你调用self.settings.get('Redis_host')
那么返回值是 None。
如果用REDIS_HOST定义,那么就可以正确返回它的值。
如果你一定要用小写,也有其他方法可正常调用。
先导入settings文件
fromt xxxx import setttings # xxx为项目名
host = settings.Redis_host # 直接导入一个文件的形式来调用是可以的 收起阅读 »
etree.strip_tags的用法
直接从官方文档那里拿过来,发现这个函数功能还挺不错的。
它把参数中的标签从源htmlelement中删除,并且把里面的标签文本给合并进来。
举个例子:
因为上述操作直接应用于test_element上的,所以test_element的值已经被修改了。
所以现在test_element 的值是
b'<p>helloworld</p>'
原创文章,转载请注明出处
http://30daydo.com/article/553
收起阅读 »
它把参数中的标签从源htmlelement中删除,并且把里面的标签文本给合并进来。
举个例子:
from lxml.html import etree
from lxml.html import fromstring, HtmlElement
test_html = '''<p><span>hello</span><span>world</span></p>'''
test_element = fromstring(test_html)
etree.strip_tags(test_element,'span') # 清除span标签
etree.tostring(test_element)
因为上述操作直接应用于test_element上的,所以test_element的值已经被修改了。
所以现在test_element 的值是
b'<p>helloworld</p>'
原创文章,转载请注明出处
http://30daydo.com/article/553
收起阅读 »
pycharm自带的版本控制软件挺好用的
pycharm自带的git,svn版本控制工具已经很好用的了,所以以后可以直接不用sourcetree这种专业的GUI管理软件了
最近是忙的飞起,没有更新文章
最近家里事多,加上换公司交接工作,新公司需要提供一些资料,跑银行,体检等。
真是希望时间能够停顿下来让我歇歇。
真是希望时间能够停顿下来让我歇歇。
aiohttp异步下载图片
保存图片的时候不能用自带的open函数打开文件,需要用到异步io库 aiofiles来打开
原创文章,
转载请注明出处:
http://30daydo.com/article/537
收起阅读 »
url = 'http://xyhz.huizhou.gov.cn/static/js/common/jigsaw/images/{}.jpg'
headers={'User-Agent':'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'}
async def getPage(num):
async with aiohttp.ClientSession() as session:
async with session.get(url.format(num),headers=headers) as resp:
if resp.status==200:
f= await aiofiles.open('{}.jpg'.format(num),mode='wb')
await f.write(await resp.read())
await f.close()
loop = asyncio.get_event_loop()
tasks = [getPage(i) for i in range(5)]
loop.run_until_complete(asyncio.wait(tasks))
原创文章,
转载请注明出处:
http://30daydo.com/article/537
收起阅读 »
基于文本及符号密度的网页正文提取方法 python实现
基于文本及符号密度的网页正文提取方法 python实现
项目路径https://github.com/Rockyzsu/CodePool/tree/master/GeneralNewsExtractor
完成后在本文详细介绍,
请密切关注。 收起阅读 »
项目路径https://github.com/Rockyzsu/CodePool/tree/master/GeneralNewsExtractor
完成后在本文详细介绍,
请密切关注。 收起阅读 »
根据东财股吧爬虫数据进行自然语言分析,展示股市热度
根据东财股吧爬虫数据进行自然语言分析,展示股市热度
项目开展中.....
https://github.com/Rockyzsu/StockPredict
完工后会把代码搬上来并加注释。
### 2019-11-17 更新 ######
股市舆情情感分类可视化系统
此Web基于Django+Bootstrap+Echarts等框架,个股交易行情数据调用了Tushare接口。对于舆情文本数据采取先爬取东方财富网股吧论坛标题词语设置机器学习训练集,在此基础上运用scikit-learn机器学习朴素贝叶斯方法构建文本分类器。通过Django Web框架,将所得数据传递到前端经过Bootstrap渲染过的html,对数据使用Echarts进行图表可视化处理
不足之处或交流学习欢迎通过邮箱联系我
目前的功能:
个股历史交易行情
个股相关词云展示
情感字典舆情预测
朴素贝叶斯舆情预测
Quick Start
在项目当前目录下: $ python manage.py runserver
浏览器打开127.0.0.1:8000
收起阅读 »
项目开展中.....
https://github.com/Rockyzsu/StockPredict
完工后会把代码搬上来并加注释。
### 2019-11-17 更新 ######
股市舆情情感分类可视化系统
此Web基于Django+Bootstrap+Echarts等框架,个股交易行情数据调用了Tushare接口。对于舆情文本数据采取先爬取东方财富网股吧论坛标题词语设置机器学习训练集,在此基础上运用scikit-learn机器学习朴素贝叶斯方法构建文本分类器。通过Django Web框架,将所得数据传递到前端经过Bootstrap渲染过的html,对数据使用Echarts进行图表可视化处理
不足之处或交流学习欢迎通过邮箱联系我
目前的功能:
个股历史交易行情
个股相关词云展示
情感字典舆情预测
朴素贝叶斯舆情预测
Quick Start
在项目当前目录下: $ python manage.py runserver
浏览器打开127.0.0.1:8000
收起阅读 »
python exchange保存备份邮件
python exchange保存备份邮件
方便自己平时备份邮件。
原创文章,
转载请注明出处
http://30daydo.com/article/534
收起阅读 »
方便自己平时备份邮件。
# -*-coding=utf-8-*-
# @Time : 2019/9/9 9:25
# @File : mail_backup.py
# @Author :
import codecs
import re
import config
import os
from exchangelib import DELEGATE, Account, Credentials, Configuration, NTLM, Message, Mailbox, HTMLBody,FileAttachment,ItemAttachment
from exchangelib.protocol import BaseProtocol, NoVerifyHTTPAdapter
#此句用来消除ssl证书错误,exchange使用自签证书需加上
BaseProtocol.HTTP_ADAPTER_CLS = NoVerifyHTTPAdapter
# 输入你的域账号如example\xxx
cred = Credentials(r'example\xxx', 你的邮箱密码)
configx = Configuration(server='mail.credlink.com', credentials=cred, auth_type=NTLM)
a = Account(
primary_smtp_address='你的邮箱地址', config=configx, autodiscover=False, access_type=DELEGATE
)
for item in a.inbox.all().order_by('-datetime_received')[:100]:
print(item.subject, item.sender, item.unique_body,item.datetime_received)
name = item.subject
name = re.sub('[\/:*?"<>|]', '-', name)
local_path = os.path.join('inbox', name+'.html')
with codecs.open(local_path, 'w','utf-8') as f:
f.write(item.unique_body)
for attachment in item.attachments:
if isinstance(attachment, FileAttachment):
name = attachment.name
name = re.sub('[\/:*?"<>|]','-',name)
local_path = os.path.join('inbox', attachment.name)
with codecs.open(local_path, 'wb') as f:
f.write(attachment.content)
print('Saved attachment to', local_path)
elif isinstance(attachment, ItemAttachment):
if isinstance(attachment.item, Message):
name=attachment.item.subject
name = re.sub('[\/:*?"<>|]', '-', name)
local_path = os.path.join('inbox', 'attachment')
with codecs.open(local_path, 'w') as f:
f.write(attachment.item.body)
原创文章,
转载请注明出处
http://30daydo.com/article/534
收起阅读 »