mongodb motor 异步操作比同步操作的时间要慢?

数据库量化投机者 回复了问题 • 2 人关注 • 1 个回复 • 664 次浏览 • 2019-08-03 09:01 • 来自相关话题

random.randint的用法

python李魔佛 发表了文章 • 0 个评论 • 260 次浏览 • 2019-08-01 16:31 • 来自相关话题

random.randint的用法:
from random import randint

randint(0,1)
Out[25]: 1

randint(0,1)
Out[26]: 1

randint(0,1)
Out[27]: 1

randint(0,1)
Out[28]: 1

randint(0,1)
Out[29]: 0

randint(0,1)
Out[30]: 1
random.randint(a,b)
 
输出的整数范围包含a和b,和之间的整数
  查看全部
random.randint的用法:
from random import randint

randint(0,1)
Out[25]: 1

randint(0,1)
Out[26]: 1

randint(0,1)
Out[27]: 1

randint(0,1)
Out[28]: 1

randint(0,1)
Out[29]: 0

randint(0,1)
Out[30]: 1

random.randint(a,b)
 
输出的整数范围包含a和b,和之间的整数
 

python执行shell命令时报错: -/bin/sh: 命令:not found的解决办法

Linux李魔佛 发表了文章 • 0 个评论 • 528 次浏览 • 2019-07-29 15:13 • 来自相关话题

file='test.txt'
cmd = f'rsync -av {file} root@10.18.6.46:/home/cjw/'

p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE,executable="/bin/bash")
output, error = p.communicate()
if p.returncode != 0:
print("Error while running - %s" % cmd)
print(error)
print(output) 
用sublime3 运行的时候一直报错。
后来发现,这个是sublime3的运行环境问题, 直接用shell执行 python main.py 执行上面的代码,命令可以正常运行。
/bin/sh: 1: rsync: not found 查看全部
     file='test.txt'
cmd = f'rsync -av {file} root@10.18.6.46:/home/cjw/'

p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE,executable="/bin/bash")
output, error = p.communicate()
if p.returncode != 0:
print("Error while running - %s" % cmd)
print(error)
print(output)
 
用sublime3 运行的时候一直报错。
后来发现,这个是sublime3的运行环境问题, 直接用shell执行 python main.py 执行上面的代码,命令可以正常运行。
/bin/sh: 1: rsync: not found

python并行编程手册 勘误

书籍李魔佛 发表了文章 • 0 个评论 • 223 次浏览 • 2019-07-28 17:06 • 来自相关话题

python并行编程手册中文版
 
65页的进程创建, p.join() 不能写到循环里面,不然的话会阻塞下一次进程的创建,因为下一次进程要卡在join这里。
 
可以改成这样的 p0 = multiprocessing.Process(name=str(0), target=foo, args=(0,))
p0.start()
p1 = multiprocessing.Process(name=str(1), target=foo, args=(1,))
p1.start()
p2 = multiprocessing.Process(name=str(2), target=foo, args=(2,))
p2.start()
p3 = multiprocessing.Process(name=str(3), target=foo, args=(3,))
p3.start()
p4 = multiprocessing.Process(name=str(4), target=foo, args=(4,))
p4.start()

p5 = multiprocessing.Process(name=str(5), target=foo, args=(5,))
p5.start()

p0.join()
p1.join()
p2.join()
p3.join()
p4.join()
p5.join() 
而且后面发现,整本书都是有这个问题的。 查看全部
python并行编程手册中文版
 
65页的进程创建, p.join() 不能写到循环里面,不然的话会阻塞下一次进程的创建,因为下一次进程要卡在join这里。
 
可以改成这样的
 p0 = multiprocessing.Process(name=str(0), target=foo, args=(0,))
p0.start()
p1 = multiprocessing.Process(name=str(1), target=foo, args=(1,))
p1.start()
p2 = multiprocessing.Process(name=str(2), target=foo, args=(2,))
p2.start()
p3 = multiprocessing.Process(name=str(3), target=foo, args=(3,))
p3.start()
p4 = multiprocessing.Process(name=str(4), target=foo, args=(4,))
p4.start()

p5 = multiprocessing.Process(name=str(5), target=foo, args=(5,))
p5.start()

p0.join()
p1.join()
p2.join()
p3.join()
p4.join()
p5.join()
 
而且后面发现,整本书都是有这个问题的。

mongodb find得到的数据顺序每次都是一样的

数据库李魔佛 发表了文章 • 0 个评论 • 208 次浏览 • 2019-07-26 09:00 • 来自相关话题

只要用的find内容不变,那么返回的内容顺序也就都一样的。
只要用的find内容不变,那么返回的内容顺序也就都一样的。

kindle使用率低

书籍liwanqiang 回复了问题 • 4 人关注 • 4 个回复 • 1007 次浏览 • 2019-07-25 17:49 • 来自相关话题

[Articles to save]

闲聊李魔佛 发表了文章 • 0 个评论 • 208 次浏览 • 2019-07-21 15:31 • 来自相关话题

Since on Raspberrypi and can't launch note application , using this web page to save articles link to store later.
 
https://www.jisilu.cn/question/321759 -Done
https://www.80shihua.com/archives/1590 -Done
  查看全部
Since on Raspberrypi and can't launch note application , using this web page to save articles link to store later.
 
https://www.jisilu.cn/question/321759 -Done
https://www.80shihua.com/archives/1590 -Done
 

Raspberrypi 2 Install or upgrade Python3.6

树莓派李魔佛 发表了文章 • 0 个评论 • 174 次浏览 • 2019-07-21 14:55 • 来自相关话题

Since no chinese input method in my raspberrypi, i can only write with English.
 
Raspberrypi has python2. 7 and python3.4, but i want to upgrade to python3.6+.
 
Python3.6 support some new feature such as print(f'{name}') and x=1_000_242_200 expression.
 
How to upgrade ?
 

$ wget https://www.python.org/ftp/pyt ... 1.tgz $ tar zxvf Python-3.6.1.tgz $ cd Python-3.6.1

then run command:

$ sudo ./configure && sudo make && sudo make install

wait for about 20mins (low perf of raspberrypi :( )
 
then you run command:
python3
 
it will using the new python3.6 version:
 

Python 3.6.1 (default, Jul 21 2019, 14:26:28) 
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
 

 
Enjoy it ! 查看全部
Since no chinese input method in my raspberrypi, i can only write with English.
 
Raspberrypi has python2. 7 and python3.4, but i want to upgrade to python3.6+.
 
Python3.6 support some new feature such as print(f'{name}') and x=1_000_242_200 expression.
 
How to upgrade ?
 

$ wget https://www.python.org/ftp/pyt ... 1.tgz 
$ tar zxvf Python-3.6.1.tgz $ cd Python-3.6.1

then run command:

$ sudo ./configure && sudo make && sudo make install

wait for about 20mins (low perf of raspberrypi :( )
 
then you run command:
python3
 
it will using the new python3.6 version:
 


Python 3.6.1 (default, Jul 21 2019, 14:26:28) 
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
 


 
Enjoy it !

frontera运行link_follower.py 报错:doesn't define any object named 'FIFO'

python爬虫李魔佛 发表了文章 • 0 个评论 • 240 次浏览 • 2019-07-18 11:29 • 来自相关话题

代码如下:
from __future__ import print_function

import re

import requests

from frontera.contrib.requests.manager import RequestsFrontierManager
# from frontera.contrib.requests.manager import RequestsFrontierManager
from frontera import Settings

from six.moves.urllib.parse import urljoin


SETTINGS = Settings()
SETTINGS.BACKEND = 'frontera.contrib.backends.memory.FIFO'
# SETTINGS.BACKEND = 'frontera.contrib.backends.memory.MemoryDistributedBackend'

SETTINGS.LOGGING_MANAGER_ENABLED = True
SETTINGS.LOGGING_BACKEND_ENABLED = True
SETTINGS.MAX_REQUESTS = 100
SETTINGS.MAX_NEXT_REQUESTS = 10

SEEDS = [
'http://www.imdb.com',
]

LINK_RE = re.compile(r'<a.+?href="(.*?)".?>', re.I)


def extract_page_links(response):
return [urljoin(response.url, link) for link in LINK_RE.findall(response.text)]

if __name__ == '__main__':

frontier = RequestsFrontierManager(SETTINGS)
frontier.add_seeds([requests.Request(url=url) for url in SEEDS])
while True:
next_requests = frontier.get_next_requests()
if not next_requests:
break
for request in next_requests:
try:
response = requests.get(request.url)
links = [
requests.Request(url=url)
for url in extract_page_links(response)
]
frontier.page_crawled(response)
print('Crawled', response.url, '(found', len(links), 'urls)')

if links:
frontier.links_extracted(request, links)
except requests.RequestException as e:
error_code = type(e).__name__
frontier.request_error(request, error_code)
print('Failed to process request', request.url, 'Error:', e)

 无论用的py2或者py3,都会报以下的错误。raise NameError("Module '%s' doesn't define any object named '%s'" % (module, name))
NameError: Module 'frontera.contrib.backends.memory' doesn't define any object named 'FIFO' 查看全部
代码如下:
from __future__ import print_function

import re

import requests

from frontera.contrib.requests.manager import RequestsFrontierManager
# from frontera.contrib.requests.manager import RequestsFrontierManager
from frontera import Settings

from six.moves.urllib.parse import urljoin


SETTINGS = Settings()
SETTINGS.BACKEND = 'frontera.contrib.backends.memory.FIFO'
# SETTINGS.BACKEND = 'frontera.contrib.backends.memory.MemoryDistributedBackend'

SETTINGS.LOGGING_MANAGER_ENABLED = True
SETTINGS.LOGGING_BACKEND_ENABLED = True
SETTINGS.MAX_REQUESTS = 100
SETTINGS.MAX_NEXT_REQUESTS = 10

SEEDS = [
'http://www.imdb.com',
]

LINK_RE = re.compile(r'<a.+?href="(.*?)".?>', re.I)


def extract_page_links(response):
return [urljoin(response.url, link) for link in LINK_RE.findall(response.text)]

if __name__ == '__main__':

frontier = RequestsFrontierManager(SETTINGS)
frontier.add_seeds([requests.Request(url=url) for url in SEEDS])
while True:
next_requests = frontier.get_next_requests()
if not next_requests:
break
for request in next_requests:
try:
response = requests.get(request.url)
links = [
requests.Request(url=url)
for url in extract_page_links(response)
]
frontier.page_crawled(response)
print('Crawled', response.url, '(found', len(links), 'urls)')

if links:
frontier.links_extracted(request, links)
except requests.RequestException as e:
error_code = type(e).__name__
frontier.request_error(request, error_code)
print('Failed to process request', request.url, 'Error:', e)

 无论用的py2或者py3,都会报以下的错误。
raise NameError("Module '%s' doesn't define any object named '%s'" % (module, name))
NameError: Module 'frontera.contrib.backends.memory' doesn't define any object named 'FIFO'

scrapy-rabbitmq 不支持python3 [修改源码使它支持]

python爬虫李魔佛 发表了文章 • 0 个评论 • 239 次浏览 • 2019-07-17 17:24 • 来自相关话题

官方版本在2015年就没有更新了。
在python3上运行的收会报错。
 
需要修改以下地方:
 
待续。。
官方版本在2015年就没有更新了。
在python3上运行的收会报错。
 
需要修改以下地方:
 
待续。。