Suppose you have a matrix M whose (i,j)-th element equals



M_ij = 2*i + 3*j

One way to define this matrix would be



i, j = np.indices((2,3))

M = 2*i + 3*j

which yields



array([[0, 3, 6],

       [2, 5, 8]])

In other words, np.indices returns arrays which can be used as indices. The elements in i indicate the row index:



In [12]: i

Out[12]: 

array([[0, 0, 0],

       [1, 1, 1]])

The elements in j indicate the column index:



In [13]: j

Out[13]: 

array([[0, 1, 2],

       [0, 1, 2]])

distanceTransform

Calculates the distance to the closest zero pixel for each pixel of the source image.





Python: cv2.distanceTransform(src, distanceType, maskSize[, dst]) → dst



Python: cv.DistTransform(src, dst, distance_type=CV_DIST_L2, mask_size=3, mask=None, labels=None) → None

Parameters:	

src – 8-bit, single-channel (binary) source image.

dst – Output image with calculated distances. It is a 32-bit floating-point, single-channel image of the same size as src .

distanceType – Type of distance. It can be CV_DIST_L1, CV_DIST_L2 , or CV_DIST_C .

maskSize – Size of the distance transform mask. It can be 3, 5, or CV_DIST_MASK_PRECISE (the latter option is only supported by the first function). In case of the CV_DIST_L1 or CV_DIST_C distance type, the parameter is forced to 3 because a  3\times 3 mask gives the same result as  5\times 5 or any larger aperture.

labels – Optional output 2D array of labels (the discrete Voronoi diagram). It has the type CV_32SC1 and the same size as src . See the details below.

labelType – Type of the label array to build. If labelType==DIST_LABEL_CCOMP then each connected component of zeros in src (as well as all the non-zero pixels closest to the connected component) will be assigned the same label. If labelType==DIST_LABEL_PIXEL then each zero pixel (and all the non-zero pixels closest to it) gets its own label.

The functions distanceTransform calculate the approximate or precise distance from every binary image pixel to the nearest zero pixel. For zero image pixels, the distance will obviously be zero.

When maskSize == CV_DIST_MASK_PRECISE and distanceType == CV_DIST_L2 , the function runs the algorithm described in [Felzenszwalb04]. This algorithm is parallelized with the TBB library.



In other cases, the algorithm [Borgefors86] is used. This means that for a pixel the function finds the shortest path to the nearest zero pixel consisting of basic shifts: horizontal, vertical, diagonal, or knight’s move (the latest is available for a 5\times 5 mask). The overall distance is calculated as a sum of these basic distances. Since the distance function should be symmetric, all of the horizontal and vertical shifts must have the same cost (denoted as a ), all the diagonal shifts must have the same cost (denoted as b ), and all knight’s moves must have the same cost (denoted as c ). For the CV_DIST_C and CV_DIST_L1 types, the distance is calculated precisely, whereas for CV_DIST_L2 (Euclidean distance) the distance can be calculated only with a relative error (a 5\times 5 mask gives more accurate results). For a,``b`` , and c , OpenCV uses the values suggested in the original paper:



CV_DIST_C	(3\times 3)	a = 1, b = 1

CV_DIST_L1	(3\times 3)	a = 1, b = 2

CV_DIST_L2	(3\times 3)	a=0.955, b=1.3693

CV_DIST_L2	(5\times 5)	a=1, b=1.4, c=2.1969

Typically, for a fast, coarse distance estimation CV_DIST_L2, a 3\times 3 mask is used. For a more accurate distance estimation CV_DIST_L2 , a 5\times 5 mask or the precise algorithm is used. Note that both the precise and the approximate algorithms are linear on the number of pixels.



The second variant of the function does not only compute the minimum distance for each pixel (x, y) but also identifies the nearest connected component consisting of zero pixels (labelType==DIST_LABEL_CCOMP) or the nearest zero pixel (labelType==DIST_LABEL_PIXEL). Index of the component/pixel is stored in \texttt{labels}(x, y) . When labelType==DIST_LABEL_CCOMP, the function automatically finds connected components of zero pixels in the input image and marks them with distinct labels. When labelType==DIST_LABEL_CCOMP, the function scans through the input image and marks all the zero pixels with distinct labels.



In this mode, the complexity is still linear. That is, the function provides a very fast way to compute the Voronoi diagram for a binary image. Currently, the second variant can use only the approximate distance transform algorithm, i.e. maskSize=CV_DIST_MASK_PRECISE is not supported yet.



Note

An example on using the distance transform can be found at opencv_source_code/samples/cpp/distrans.cpp

(Python) An example on using the distance transform can be found at opencv_source/samples/python2/distrans.py

ERRORS:

?: (admin.E408) 'django.contrib.auth.middleware.AuthenticationMiddleware' must be in MIDDLEWARE in order to use the admin application.

# 测试一下那个方差

x=[1,2,3,4,5,6,7,8,9,10]

X = np.array(x)

def my_fangca(X):

    l=len(X)

    mean=X.mean()

    sum_ = 0

    sum_std=0

    for i in X:

        sum_+=(i-mean)**2

    var_=sum_/l

    std_=(sum_/(l))**0.5

    return var_,std_

# -*- coding: utf-8 -*-

# website: http://30daydo.com

# @Time : 2019/6/30 12:03

# @File : main.py



import requests

import re

import os



url = 'http://180.153.255.6/mobile/v1/album/track/ts-1571294887744?albumId=23057324&device=android&isAsc=true&isQueryInvitationBrand=true&pageId={}&pageSize=20&pre_page=0'

headers = {'User-Agent': 'Xiaomi'}



def download():

    for i in range(1, 3):

        r = requests.get(url=url.format(i), headers=headers)

        js_data = r.json()

        data_list = js_data.get('data', {}).get('list', [])

        for item in data_list:

            trackName = item.get('title')

            trackName = re.sub('[\/\\\:\*\?\"\<\>\|]', '_', trackName)

            # trackName=re.sub(':','',trackName)

            src_url = item.get('playUrl64')

            filename = '{}.mp3'.format(trackName)

            if not os.path.exists(filename):



                try:

                    r0 = requests.get(src_url, headers=headers)

                except Exception as e:

                    print(e)

                    print(trackName)

                    r0 = requests.get(src_url, headers=headers)





                else:

                    with open(filename, 'wb') as f:

                        f.write(r0.content)



                    print('{} downloaded'.format(trackName))



            else:

                print(f'{filename}已经下载过了')



import shutil



def rename_():

    for i in range(1, 3):

        r = requests.get(url=url.format(i), headers=headers)

        js_data = r.json()

        data_list = js_data.get('data', {}).get('list', [])

        for item in data_list:

            trackName = item.get('title')

            trackName = re.sub('[\/\\\:\*\?\"\<\>\|]', '_', trackName)

            src_url = item.get('playUrl64')



            orderNo=item.get('orderNo')



            filename = '{}.mp3'.format(trackName)

            try:



                if os.path.exists(filename):

                    new_file='{}_{}.mp3'.format(orderNo,trackName)

                    shutil.move(filename,new_file)

            except Exception as e:

                print(e)











if __name__=='__main__':

    rename_()

# -*- coding: utf-8 -*-

# website: http://30daydo.com

# @Time : 2019/6/30 12:03

# @File : main.py



import requests

import re

url = 'https://www.ximalaya.com/revision/play/album?albumId=23057324&pageNum=1&sort=1&pageSize=60'

headers={'User-Agent':'Xiaomi'}



r = requests.get(url=url,headers=headers)

js_data = r.json()

data_list = js_data.get('data',{}).get('tracksAudioPlay',)

for item in data_list:

    trackName=item.get('trackName')

    trackName=re.sub(':','',trackName)

    src_url = item.get('src')

    try:

        r0=requests.get(src_url,headers=headers)

    except Exception as e:

        print(e)

        print(trackName)

    else:

        with open('{}.m4a'.format(trackName),'wb') as f:

            f.write(r0.content)

        print('{} downloaded'.format(trackName))

def iter_demo():



    class DefineIter(object):



        def __init__(self,length):

            self.length = length

            self.data = range(self.length)

            self.index=0



        def __iter__(self):

            return self





        def __next__(self):



            if self.index >=self.length:

                # return None

                raise StopIteration



            d = self.data[self.index]*50

            self.index =self.index + 1



            return d



    a = DefineIter(10)

    print(type(a))

    for i in a:

        print(i)

方式

通过 Live Template 快速添加时间



步骤

1、添加一个 Template Group 命名为 Common

2、添加一个 Live Template 设置如下

Abbreviation： time

Description ： current time

Template Text:  $time$



Edit Variables ->  Expresssion : date("yyyy-MM-dd HH:mm:ss")







3、让设置生效

Define->Everywhere



4、使用

输入 time 后 按下tab键 就能转换为当前时间了

pywinauto.application.AppStartError: Could not create the process "C:\全能行证券交易终端\xiadan.exe"
Error returned by CreateProcess: (2, 'CreateProcess', '系统找不到指定的文件。')

class GJ(CommonConfig):

    DEFAULT_EXE_PATH = "C:\\Tool\\xiadan.exe"

  File "C:\ProgramData\Anaconda3\lib\site-packages\pymongo\helpers.py", line 155, in _check_command_response

    raise OperationFailure(msg % errmsg, code, response)

pymongo.errors.OperationFailure: Query for sharded findAndModify must contain the shard key

2019-06-10 16:14:32 [scrapy.core.engine] INFO: Closing spider (finished)

2019-06-10 16:14:32 [scrapy.statscollectors] INFO: Dumping Scrapy stats:

Targeted Operations vs. Broadcast Operations

Generally, the fastest queries in a sharded environment are those that mongos route to a single shard, using the shard key and the cluster meta data from the config server. These targeted operations use the shard key value to locate the shard or subset of shards that satisfy the query document.

For queries that don’t include the shard key, mongos must query all shards, wait for their responses and then return the result to the application. These “scatter/gather” queries can be long running operations.

Broadcast Operations

mongos instances broadcast queries to all shards for the collection unless the mongos can determine which shard or subset of shards stores this data.



After the mongos receives responses from all shards, it merges the data and returns the result document. The performance of a broadcast operation depends on the overall load of the cluster, as well as variables like network latency, individual shard load, and number of documents returned per shard. Whenever possible, favor operations that result in targeted operation over those that result in a broadcast operation.

Multi-update operations are always broadcast operations.

The updateMany() and deleteMany() methods are broadcast operations, unless the query document specifies the shard key in full.

Targeted Operations

mongos can route queries that include the shard key or the prefix of a compound shard key a specific shard or set of shards. mongos uses the shard key value to locate the chunk whose range includes the shard key value and directs the query at the shard containing that chunk.



For example, if the shard key is:

copy

{ a: 1, b: 1, c: 1 }



The mongos program can route queries that include the full shard key or either of the following shard key prefixes at a specific shard or set of shards:

copy

{ a: 1 }

{ a: 1, b: 1 }



All insertOne() operations target to one shard. Each document in the insertMany() array targets to a single shard, but there is no guarantee all documents in the array insert into a single shard.

All updateOne(), replaceOne() and deleteOne() operations must include the shard key or _id in the query document. MongoDB returns an error if these methods are used without the shard key or _id.

Depending on the distribution of data in the cluster and the selectivity of the query, mongos may still perform a broadcast operation to fulfill these queries.

Index Use

If the query does not include the shard key, the mongos must send the query to all shards as a “scatter/gather” operation. Each shard will, in turn, use either the shard key index or another more efficient index to fulfill the query.

If the query includes multiple sub-expressions that reference the fields indexed by the shard key and the secondary index, the mongos can route the queries to a specific shard and the shard will use the index that will allow it to fulfill most efficiently.

Sharded Cluster Security

Use Internal Authentication to enforce intra-cluster security and prevent unauthorized cluster components from accessing the cluster. You must start each mongod or mongos in the cluster with the appropriate security settings in order to enforce internal authentication.

See Deploy Sharded Cluster with Keyfile Access Control for a tutorial on deploying a secured shardedcluster.

Cluster Users

Sharded clusters support Role-Based Access Control (RBAC) for restricting unauthorized access to cluster data and operations. You must start each mongod in the cluster, including the config servers, with the --auth option in order to enforce RBAC. Alternatively, enforcing Internal Authentication for inter-cluster security also enables user access controls via RBAC.

With RBAC enforced, clients must specify a --username, --password, and --authenticationDatabase when connecting to the mongos in order to access cluster resources.

Each cluster has its own cluster users. These users cannot be used to access individual shards.

See Enable Access Control for a tutorial on enabling adding users to an RBAC-enabled MongoDB deployment.

# 拯救损坏的jupyter 文件

import re

import codecs



pattern = re.compile('"source": \[(.*?)\]\s+\},',re.S)

filename = 'tushare_usage.ipynb'

with codecs.open(filename,encoding='utf8') as f:

    content = f.read()

    

source = pattern.findall(content)

for s in source:

    t=s.replace('\\n','')

    t=re.sub('"','',t)

    t=re.sub('(,$)','',t)

    print(t)

import pymongo

db = pymongo.MongoClient('mongodb://10.18.6.46,10.18.6.26,10.18.6.102')

db = pymongo.MongoClient('mongodb://10.18.6.46:8888,10.18.6.26:9999,10.18.6.102:7777')

coll=db['testdb']['testcollection'].find()

for i in coll:

    print(i)

{'_id': ObjectId('5cf4c7981ee9edff72e5c503'), 'username': 'hello'}

{'_id': ObjectId('5cf4c7991ee9edff72e5c504'), 'username': 'hello'}

{'_id': ObjectId('5cf4c7991ee9edff72e5c505'), 'username': 'hello'}

{'_id': ObjectId('5cf4c79a1ee9edff72e5c506'), 'username': 'hello'}

{'_id': ObjectId('5cf4c7b21ee9edff72e5c507'), 'username': 'hello world'}

collection = db['testdb']['testcollection']



for i in range(10):

    collection.insert({'username':'huston{}'.format(i)})

{

  "took" : 3,

  "timed_out" : false,

  "_shards" : {

    "total" : 5,

    "successful" : 5,

    "skipped" : 0,

    "failed" : 0

  },

  "hits" : {

    "total" : 5,

    "max_score" : 1.0,

    "hits" : [

      {

        "_index" : "test",

        "_type" : "mydoc",

        "_id" : "XxyrM2kBVzdNcvl_GHv2",

        "_score" : 1.0,

        "_source" : {

          "name" : "Shiled",

          "twitter" : "Sonny sql is awesome",

          "date" : "2018-12-27",

          "id" : 1240,

          "tags" : [

            "red",

            "shit"

          ]

        }

      },

      {

        "_index" : "test",

        "_type" : "mydoc",

        "_id" : "YByrM2kBVzdNcvl_tnvm",

        "_score" : 1.0,

        "_source" : {

          "name" : "YYerk",

          "twitter" : "sql is awesome",

          "date" : "2008-12-27",

          "id" : 12357,

          "tags" : [

            "red"

          ]

        }

      },

      {

        "_index" : "test",

        "_type" : "mydoc",

        "_id" : "7777",

        "_score" : 1.0,

        "_source" : {

          "name" : "Rocky Chen",

          "twitter" : "sql is awesome",

          "date" : "2008-12-27",

          "id" : 9999

        }

      },

      {

        "_index" : "test",

        "_type" : "mydoc",

        "_id" : "YhzDN2kBVzdNcvl_enuT",

        "_score" : 1.0,

        "_source" : {

          "name" : "YYerk",

          "twitter" : "sql is awesome",

          "date" : "2008-12-27",

          "id" : 888888,

          "tags" : [

            "red",

            "green"

          ]

        }

      },

      {

        "_index" : "test",

        "_type" : "mydoc",

        "_id" : "YxzDN2kBVzdNcvl_u3th",

        "_score" : 1.0,

        "_source" : {

          "name" : "YYerk",

          "twitter" : "sql is awesome",

          "date" : "2008-12-27",

          "id" : 888888,

          "tags" : [

            "red",

            "green"

          ]

        }

      }

    ]

  }

}

 

路由一个文档到一个分片中编辑
当索引一个文档的时候，文档会被存储到一个主分片中。 Elasticsearch 如何知道一个文档应该存放到哪个分片中呢？当我们创建文档时，它如何决定这个文档应当被存储在分片 1 还是分片 2 中呢？
首先这肯定不会是随机的，否则将来要获取文档的时候我们就不知道从何处寻找了。实际上，这个过程是根据下面这个公式决定的：
shard = hash(routing) % number_of_primary_shards
routing 是一个可变值，默认是文档的 _id ，也可以设置成一个自定义的值。 routing 通过 hash 函数生成一个数字，然后这个数字再除以 number_of_primary_shards （主分片的数量）后得到 余数 。这个分布在 0 到 number_of_primary_shards-1 之间的余数，就是我们所寻求的文档所在分片的位置。
这就解释了为什么我们要在创建索引的时候就确定好主分片的数量 并且永远不会改变这个数量：因为如果数量变化了，那么所有之前路由的值都会无效，文档也再也找不到了。

{

  "nodes" : {

    "yl-qYmh1SXqzJsfI4d1ddw" : {

      "name" : "node-3",

      "ephemeral_id" : "UsJ9rFELTiCW07oHE9YMdg",

      "transport_address" : "10.18.6.26:9300",

      "attributes" : {

        "ml.machine_memory" : "6088101888",

        "rack" : "r1",

        "ml.max_open_jobs" : "20",

        "xpack.installed" : "true",

        "ml.enabled" : "true"

      }

    },

    "wT7wUd3iTkujYUsbVNVv-w" : {

      "name" : "node-1",

      "ephemeral_id" : "fP-vgSb0SdemnHDyaJUsWw",

      "transport_address" : "10.18.6.102:9300",

      "attributes" : {

        "ml.machine_memory" : "8256720896",

        "rack" : "r1",

        "xpack.installed" : "true",

        "ml.max_open_jobs" : "20",

        "ml.enabled" : "true"

      }

    }

  },

  "indices" : {

    "test" : { }

  },

  "shards" : [

    [

      {

        "state" : "STARTED",

        "primary" : true,

        "node" : "wT7wUd3iTkujYUsbVNVv-w",

        "relocating_node" : null,

        "shard" : 1,

        "index" : "test",

        "allocation_id" : {

          "id" : "k-8E4dL7QmGgwcsNsUCP6Q"

        }

      },

      {

        "state" : "STARTED",

        "primary" : false,

        "node" : "yl-qYmh1SXqzJsfI4d1ddw",

        "relocating_node" : null,

        "shard" : 1,

        "index" : "test",

        "allocation_id" : {

          "id" : "lvOpQIKgRUibkulr3nRfEw"

        }

      }

    ]

  ]

}

POST get-together/_search

{

  "query":

  {

    "match": {

      "name": {

     "type":"phrase", 

      "query":"enterprise london",

      "slop":1

    }}

  },

  "_source": "name"

}

{

  "error": {

    "root_cause": [

      {

        "type": "parsing_exception",

        "reason": "[match] query does not support [type]",

        "line": 6,

        "col": 13

      }

    ],

    "type": "parsing_exception",

    "reason": "[match] query does not support [type]",

    "line": 6,

    "col": 13

  },

  "status": 400

}

POST get-together/_search

{

  "query":

  {

    "match_phrase": {

      "name": {

     

      "query":"enterprise london",

      "slop":1

    }}

  },

  "_source": "name"

}

{

  "took" : 2,

  "timed_out" : false,

  "_shards" : {

    "total" : 2,

    "successful" : 2,

    "skipped" : 0,

    "failed" : 0

  },

  "hits" : {

    "total" : 1,

    "max_score" : 1.3243701,

    "hits" : [

      {

        "_index" : "get-together",

        "_type" : "_doc",

        "_id" : "5",

        "_score" : 1.3243701,

        "_source" : {

          "name" : "Enterprise search London get-together"

        }

      }

    ]

  }

}

POST cnbeta/doc/cUxO42oB9O-zF2ru-rs-/_update

{

  "doc":{

  "title":"中国操作系统"

  }

}

    file_path=r'9927_15562445086485238.png'

    file=open(file_path, 'rb').read()

    r=requests.post(url=code_url,data=file)

    print(r.text)

public abstract class Vehicle {

}

 

public interface Flyable {

    public void fly();

}

 

public class FlyableImpl implements Flyable {

    public void fly() {

        System.out.println("I am flying");

    }

} 

 

public class Airplane extends Vehicle implements Flyable {

    private flyable;

 

    public Airplane() {

        flyable = new FlyableImpl();

    }

 

    public void fly() {

        flyable.fly();

    }

}

class Vehicle(object):

    pass

 

class PlaneMixin(object):

    def fly(self):

        print 'I am flying'

 

class Airplane(Vehicle, PlaneMixin):

    pass

• 首先它必须表示某一种功能，而不是某个物品，如同Java中的Runnable，Callable等

• 其次它必须责任单一，如果有多个功能，那就写多个Mixin类
• 然后，它不依赖于子类的实现
• 最后，子类即便没有继承这个Mixin类，也照样可以工作，就是缺少了某个功能。（比如飞机照样可以载客，就是不能飞了^_^）

	ticker	secShortName	secFullName	y_chgPct

31	601288	农业银行	中国农业银行股份有限公司	-0.341178

11	600015	华夏银行	华夏银行股份有限公司	1.856174

45	601988	中国银行	中国银行股份有限公司	2.248533

32	601328	交通银行	交通银行股份有限公司	3.532657

30	601229	上海银行	上海银行股份有限公司	3.725781

35	601398	工商银行	中国工商银行股份有限公司	4.771403

40	601818	光大银行	中国光大银行股份有限公司	5.643119

27	601169	北京银行	北京银行股份有限公司	6.205580

14	600016	民生银行	中国民生银行股份有限公司	7.092815

5	002936	郑州银行	郑州银行股份有限公司	7.551112

49	601998	中信银行	中信银行股份有限公司	8.181956

43	601939	建设银行	中国建设银行股份有限公司	9.402651

41	601838	成都银行	成都银行股份有限公司	9.424554

52	603323	苏农银行	江苏苏州农村商业银行股份有限公司	12.375732

20	600926	杭州银行	杭州银行股份有限公司	12.933645

8	600000	浦发银行	上海浦东发展银行股份有限公司	14.752244

39	601577	长沙银行	长沙银行股份有限公司	14.792683

18	600908	无锡银行	无锡农村商业银行股份有限公司	16.181704

3	002807	江阴银行	江苏江阴农村商业银行股份有限公司	19.274586

48	601997	贵阳银行	贵阳银行股份有限公司	20.489563

4	002839	张家港行	江苏张家港农村商业银行股份有限公司	20.599511

25	601166	兴业银行	兴业银行股份有限公司	21.206503

24	601128	常熟银行	江苏常熟农村商业银行股份有限公司	21.571187

19	600919	江苏银行	江苏银行股份有限公司	23.218299

22	601009	南京银行	南京银行股份有限公司	26.297500

16	600036	招商银行	招商银行股份有限公司	27.518708

0	000001	平安银行	平安银行股份有限公司	31.624747

2	002142	宁波银行	宁波银行股份有限公司	31.729062

6	002948	青岛银行	青岛银行股份有限公司	48.602573

7	002958	青农商行	青岛农村商业银行股份有限公司	108.983776

42	601860	紫金银行	江苏紫金农村商业银行股份有限公司	115.147347

21	600928	西安银行	西安银行股份有限公司	128.496683

js=re.sub('\r\n','',js)

async def fetch(session,url, data):

    async with session.post(url=url, data=data, headers=headers) as response:

        return await response.json()

import aiohttp

import asyncio



page = 30



post_data = {

    'page': 1,

    'pageSize': 10,

    'keyWord': '',

    'dpIds': '',

}



headers = {

    

    "Accept-Encoding": "gzip, deflate",

    "Accept-Language": "en-US,en;q=0.9",

    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.108 Safari/537.36",

    "X-Requested-With": "XMLHttpRequest",

}



result=





async def fetch(session,url, data):

    async with session.post(url=url, data=data, headers=headers) as response:

        return await response.json()



async def parse(html):

    xzcf_list = html.get('newtxzcfList')

    if xzcf_list is None:

        return

    for i in xzcf_list:

        result.append(i)



async def downlod(page):

    data=post_data.copy()

    data['page']=page

    url = 'http://credit.chaozhou.gov.cn/tfieldTypeActionJson!initXzcfListnew.do'

    async with aiohttp.ClientSession() as session:

            html=await fetch(session,url,data)

            await parse(html)



loop = asyncio.get_event_loop()

tasks=[asyncio.ensure_future(downlod(i)) for i in range(1,page)]

tasks=asyncio.gather(*tasks)

# print(tasks)

loop.run_until_complete(tasks)

# loop.close()

# print(result)

count=0

for i in result:

    print(i.get('cfXdrMc'))

    count+=1

print(f'total {count}')

import requests

from lxml import etree

import asyncio

import aiohttp

import pandas

import re

import math

import time



loction_info = '''    1→杭州

    2→武汉

    3→北京

    按ENTER确认：'''

loction_select = input(loction_info)

loction_dic = {'1': 'hz',

               '2': 'wh',

               '3': 'bj'}

city_url = 'https://{}.lianjia.com/ershoufang/'.format(loction_dic[loction_select])

down = input('请输入价格下限（万）:')

up = input('请输入价格上限（万）:')



inter_list = [(int(down), int(up))]





def half_inter(inter):

    lower = inter[0]

    upper = inter[1]

    delta = int((upper - lower) / 2)

    inter_list.remove(inter)

    print('已经缩小价格区间', inter)

    inter_list.append((lower, lower + delta))

    inter_list.append((lower + delta, upper))





pagenum = {}





def get_num(inter):

    url = city_url + 'bp{}ep{}/'.format(inter[0], inter[1])

    r = requests.get(url).text

    print(r)

    num = int(etree.HTML(r).xpath("//h2[@class='total fl']/span/text()")[0].strip())

    pagenum[(inter[0], inter[1])] = num

    return num





totalnum = get_num(inter_list[0])



judge = True

while judge:

    a = [get_num(x) > 3000 for x in inter_list]

    if True in a:

        judge = True

    else:

        judge = False

    for i in inter_list:

        if get_num(i) > 3000:

            half_inter(i)

print('价格区间缩小完毕！')



url_lst = []

url_lst_failed = []

url_lst_successed = []

url_lst_duplicated = []



for i in inter_list:

    totalpage = math.ceil(pagenum[i] / 30)

    for j in range(1, totalpage + 1):

        url = city_url + 'pg{}bp{}ep{}/'.format(j, i[0], i[1])

        url_lst.append(url)

print('url列表获取完毕！')



info_lst = []





async def get_info(url):

    async with aiohttp.ClientSession() as session:

        async with session.get(url, timeout=5) as resp:

            if resp.status != 200:

                url_lst_failed.append(url)

            else:

                url_lst_successed.append(url)

            r = await resp.text()

            nodelist = etree.HTML(r).xpath("//ul[@class='sellListContent']/li")

            # print('-------------------------------------------------------------')

            # print('开始抓取第{}个页面的数据,共计{}个页面'.format(url_lst.index(url),len(url_lst)))

            # print('开始抓取第{}个页面的数据,共计{}个页面'.format(url_lst.index(url), len(url_lst)))

            # print('开始抓取第{}个页面的数据,共计{}个页面'.format(url_lst.index(url), len(url_lst)))

            # print('-------------------------------------------------------------')

            info_dic = {}

            index = 1

            print('开始抓取{}'.format(resp.url))

            print('开始抓取{}'.format(resp.url))

            print('开始抓取{}'.format(resp.url))

            for node in nodelist:

                try:

                    info_dic['title'] = node.xpath(".//div[@class='title']/a/text()")[0]

                except:

                    info_dic['title'] = '/'

                try:

                    info_dic['href'] = node.xpath(".//div[@class='title']/a/@href")[0]

                except:

                    info_dic['href'] = '/'

                try:

                    info_dic['xiaoqu'] = \

                    node.xpath(".//div[@class='houseInfo']")[0].xpath('string(.)').replace(' ', '').split('|')[0]

                except:

                    info_dic['xiaoqu'] = '/'

                try:

                    info_dic['huxing'] = \

                    node.xpath(".//div[@class='houseInfo']")[0].xpath('string(.)').replace(' ', '').split('|')[1]

                except:

                    info_dic['huxing'] = '/'

                try:

                    info_dic['area'] = \

                    node.xpath(".//div[@class='houseInfo']")[0].xpath('string(.)').replace(' ', '').split('|')[2]

                except:

                    info_dic['area'] = '/'

                try:

                    info_dic['chaoxiang'] = \

                    node.xpath(".//div[@class='houseInfo']")[0].xpath('string(.)').replace(' ', '').split('|')[3]

                except:

                    info_dic['chaoxiang'] = '/'

                try:

                    info_dic['zhuangxiu'] = \

                    node.xpath(".//div[@class='houseInfo']")[0].xpath('string(.)').replace(' ', '').split('|')[4]

                except:

                    info_dic['zhuangxiu'] = '/'

                try:

                    info_dic['dianti'] = \

                    node.xpath(".//div[@class='houseInfo']")[0].xpath('string(.)').replace(' ', '').split('|')[5]

                except:

                    info_dic['dianti'] = '/'

                try:

                    info_dic['louceng'] = re.findall('\((.*)\)', node.xpath(".//div[@class='positionInfo']/text()")[0])

                except:

                    info_dic['louceng'] = '/'

                try:

                    info_dic['nianxian'] = re.findall('\)(.*?)年', node.xpath(".//div[@class='positionInfo']/text()")[0])

                except:

                    info_dic['nianxian'] = '/'

                try:

                    info_dic['guanzhu'] = ''.join(re.findall('[0-9]', node.xpath(".//div[@class='followInfo']/text()")[

                        0].replace(' ', '').split('/')[0]))

                except:

                    info_dic['guanzhu'] = '/'

                try:

                    info_dic['daikan'] = ''.join(re.findall('[0-9]',

                                                            node.xpath(".//div[@class='followInfo']/text()")[0].replace(

                                                                ' ', '').split('/')[1]))

                except:

                    info_dic['daikan'] = '/'

                try:

                    info_dic['fabu'] = node.xpath(".//div[@class='followInfo']/text()")[0].replace(' ', '').split('/')[

                        2]

                except:

                    info_dic['fabu'] = '/'

                try:

                    info_dic['totalprice'] = node.xpath(".//div[@class='totalPrice']/span/text()")[0]

                except:

                    info_dic['totalprice'] = '/'

                try:

                    info_dic['unitprice'] = node.xpath(".//div[@class='unitPrice']/span/text()")[0].replace('单价', '')

                except:

                    info_dic['unitprice'] = '/'

                if True in [info_dic['href'] in dic.values() for dic in info_lst]:

                    url_lst_duplicated.append(info_dic)

                else:

                    info_lst.append(info_dic)

                print('第{}条:    {}→房屋信息抓取完毕！'.format(index, info_dic['title']))

                index += 1

                info_dic = {}





start = time.time()



# 首次抓取url_lst中的信息，部分url没有对其发起请求，不知道为什么

tasks = [asyncio.ensure_future(get_info(url)) for url in url_lst]

loop = asyncio.get_event_loop()

loop.run_until_complete(asyncio.wait(tasks))



# 将没有发起请求的url放入一个列表，对其进行循环抓取，直到所有url都被发起请求

url_lst_unrequested = []

for url in url_lst:

    if url not in url_lst_successed or url_lst_failed:

        url_lst_unrequested.append(url)

while len(url_lst_unrequested) > 0:

    tasks_unrequested = [asyncio.ensure_future(get_info(url)) for url in url_lst_unrequested]

    loop.run_until_complete(asyncio.wait(tasks_unrequested))

    url_lst_unrequested = []

    for url in url_lst:

        if url not in url_lst_successed:

            url_lst_unrequested.append(url)

end = time.time()

print('当前价格区间段内共有{}套二手房源\(包含{}条重复房源\),实际获得{}条房源信息。'.format(totalnum, len(url_lst_duplicated), len(info_lst)))

print('总共耗时{}秒'.format(end - start))



df = pandas.DataFrame(info_lst)

df.to_csv("ljwh.csv", encoding='gbk')

import matplotlib.pyplot as plt

import numpy as np

import time

from collections import OrderedDict



def softmax(a):

    a = a - np.max(a)

    exp_a = np.exp(a)

    exp_a_sum = np.sum(exp_a)

    return exp_a / exp_a_sum





def cross_entropy_error(t, y):

    delta = 1e-7

    s = -1 * np.sum(t * np.log(y + delta))

    #     print('cross entropy ',s)

    return s





class simpleNet:

    def __init__(self):

        self.W = np.random.randn(2, 3)



    def predict(self, x):

        print('current w',self.W)

        return np.dot(x, self.W)



    def loss(self, x, t):

        z = self.predict(x)

        #         print(z)

        #         print(z.ndim)

        y = softmax(z)

        #         print('y',y)

        loss = cross_entropy_error(y, t)  # y为预测的值

        return loss







def numerical_gradient_(f, x):  # 针对2维的情况 甚至是多维

    h = 1e-4  # 0.0001

    grad = np.zeros_like(x)



    it = np.nditer(x, flags=['multi_index'], op_flags=['readwrite'])

    while not it.finished:

        idx = it.multi_index

        print('idx', idx)

        tmp_val = x[idx]

        x[idx] = float(tmp_val) + h

        fxh1 = f(x)  # f(x+h)

        print('fxh1 ', fxh1)

        # print('current W', net.W)

        x[idx] = tmp_val - h

        fxh2 = f(x)  # f(x-h)

        print('fxh2 ', fxh2)

        # print('next currnet W ', net.W)

        grad[idx] = (fxh1 - fxh2) / (2 * h)



        x[idx] = tmp_val  # 还原值

        it.iternext()



    return grad







net = simpleNet()

x=np.array([0.6,0.9])

t = np.array([0.0,0.0,1.0])



def f(W):

    return net.loss(x,t)



grads =numerical_gradient_(f,net.W)

print(grads)

MTW881U3Z5-eyJsaWNlbnNlSWQiOiJNVFc4ODFVM1o1IiwibGljZW5zZWVOYW1lIjoiTnNzIEltIiwiYXNzaWduZWVOYW1lIjoiIiwiYXNzaWduZWVFbWFpbCI6IiIsImxpY2Vuc2VSZXN0cmljdGlvbiI6IkZvciBlZHVjYXRpb25hbCB1c2Ugb25seSIsImNoZWNrQ29uY3VycmVudFVzZSI6ZmFsc2UsInByb2R1Y3RzIjpbeyJjb2RlIjoiSUkiLCJwYWlkVXBUbyI6IjIwMTktMTEtMDYifSx7ImNvZGUiOiJBQyIsInBhaWRVcFRvIjoiMjAxOS0xMS0wNiJ9LHsiY29kZSI6IkRQTiIsInBhaWRVcFRvIjoiMjAxOS0xMS0wNiJ9LHsiY29kZSI6IlBTIiwicGFpZFVwVG8iOiIyMDE5LTExLTA2In0seyJjb2RlIjoiR08iLCJwYWlkVXBUbyI6IjIwMTktMTEtMDYifSx7ImNvZGUiOiJETSIsInBhaWRVcFRvIjoiMjAxOS0xMS0wNiJ9LHsiY29kZSI6IkNMIiwicGFpZFVwVG8iOiIyMDE5LTExLTA2In0seyJjb2RlIjoiUlMwIiwicGFpZFVwVG8iOiIyMDE5LTExLTA2In0seyJjb2RlIjoiUkMiLCJwYWlkVXBUbyI6IjIwMTktMTEtMDYifSx7ImNvZGUiOiJSRCIsInBhaWRVcFRvIjoiMjAxOS0xMS0wNiJ9LHsiY29kZSI6IlBDIiwicGFpZFVwVG8iOiIyMDE5LTExLTA2In0seyJjb2RlIjoiUk0iLCJwYWlkVXBUbyI6IjIwMTktMTEtMDYifSx7ImNvZGUiOiJXUyIsInBhaWRVcFRvIjoiMjAxOS0xMS0wNiJ9LHsiY29kZSI6IkRCIiwicGFpZFVwVG8iOiIyMDE5LTExLTA2In0seyJjb2RlIjoiREMiLCJwYWlkVXBUbyI6IjIwMTktMTEtMDYifSx7ImNvZGUiOiJSU1UiLCJwYWlkVXBUbyI6IjIwMTktMTEtMDYifV0sImhhc2giOiIxMDgyODE0Ni8wIiwiZ3JhY2VQZXJpb2REYXlzIjowLCJhdXRvUHJvbG9uZ2F0ZWQiOmZhbHNlLCJpc0F1dG9Qcm9sb25nYXRlZCI6ZmFsc2V9-aKyalfjUfiV5UXfhaMGgOqrMzTYy2rnsmobL47k8tTpR/jvG6HeL3FxxleetI+W+Anw3ZSe8QAMsSxqVS4podwlQgIe7f+3w7zyAT1j8HMVlfl2h96KzygdGpDSbwTbwOkJ6/5TQOPgAP86mkaSiM97KgvkZV/2nXQHRz1yhm+MT+OsioTwxDhd/22sSGq6KuIztZ03UvSciEmyrPdl2ueJw1WuT9YmFjdtTm9G7LuXvCM6eav+BgCRm+wwtUeDfoQqigbp0t6FQgkdQrcjoWvLSB0IUgp/f4qGf254fA7lXskT2VCFdDvi0jgxLyMVct1cKnPdM6fkHnbdSXKYDWw==-MIIElTCCAn2gAwIBAgIBCTANBgkqhkiG9w0BAQsFADAYMRYwFAYDVQQDDA1KZXRQcm9maWxlIENBMB4XDTE4MTEwMTEyMjk0NloXDTIwMTEwMjEyMjk0NlowaDELMAkGA1UEBhMCQ1oxDjAMBgNVBAgMBU51c2xlMQ8wDQYDVQQHDAZQcmFndWUxGTAXBgNVBAoMEEpldEJyYWlucyBzLnIuby4xHTAbBgNVBAMMFHByb2QzeS1mcm9tLTIwMTgxMTAxMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxcQkq+zdxlR2mmRYBPzGbUNdMN6OaXiXzxIWtMEkrJMO/5oUfQJbLLuMSMK0QHFmaI37WShyxZcfRCidwXjot4zmNBKnlyHodDij/78TmVqFl8nOeD5+07B8VEaIu7c3E1N+e1doC6wht4I4+IEmtsPAdoaj5WCQVQbrI8KeT8M9VcBIWX7fD0fhexfg3ZRt0xqwMcXGNp3DdJHiO0rCdU+Itv7EmtnSVq9jBG1usMSFvMowR25mju2JcPFp1+I4ZI+FqgR8gyG8oiNDyNEoAbsR3lOpI7grUYSvkB/xVy/VoklPCK2h0f0GJxFjnye8NT1PAywoyl7RmiAVRE/EKwIDAQABo4GZMIGWMAkGA1UdEwQCMAAwHQYDVR0OBBYEFGEpG9oZGcfLMGNBkY7SgHiMGgTcMEgGA1UdIwRBMD+AFKOetkhnQhI2Qb1t4Lm0oFKLl/GzoRykGjAYMRYwFAYDVQQDDA1KZXRQcm9maWxlIENBggkA0myxg7KDeeEwEwYDVR0lBAwwCgYIKwYBBQUHAwEwCwYDVR0PBAQDAgWgMA0GCSqGSIb3DQEBCwUAA4ICAQAF8uc+YJOHHwOFcPzmbjcxNDuGoOUIP+2h1R75Lecswb7ru2LWWSUMtXVKQzChLNPn/72W0k+oI056tgiwuG7M49LXp4zQVlQnFmWU1wwGvVhq5R63Rpjx1zjGUhcXgayu7+9zMUW596Lbomsg8qVve6euqsrFicYkIIuUu4zYPndJwfe0YkS5nY72SHnNdbPhEnN8wcB2Kz+OIG0lih3yz5EqFhld03bGp222ZQCIghCTVL6QBNadGsiN/lWLl4JdR3lJkZzlpFdiHijoVRdWeSWqM4y0t23c92HXKrgppoSV18XMxrWVdoSM3nuMHwxGhFyde05OdDtLpCv+jlWf5REAHHA201pAU6bJSZINyHDUTB+Beo28rRXSwSh3OUIvYwKNVeoBY+KwOJ7WnuTCUq1meE6GkKc4D/cXmgpOyW/1SmBz3XjVIi/zprZ0zf3qH5mkphtg6ksjKgKjmx1cXfZAAX6wcDBNaCL+Ortep1Dh8xDUbqbBVNBL4jbiL3i3xsfNiyJgaZ5sX7i8tmStEpLbPwvHcByuf59qJhV/bZOl8KqJBETCDJcY6O2aqhTUy+9x93ThKs1GKrRPePrWPluud7ttlgtRveit/pcBrnQcXOl1rHq7ByB8CFAxNotRUYL9IF5n3wJOgkPojMy6jetQA5Ogc8Sm7RG6vg1yow==

x=np.array([[1,2,3,4],[5,6,7,8]])

array([[1, 2, 3, 4],

       [5, 6, 7, 8]])

x.flatten()

array([1, 2, 3, 4, 5, 6, 7, 8])