cd bin

 ./logstash -e 'input { stdin { } } output { stdout {} }'

ERROR: Unknown command '{'



See: 'bin/logstash --help'

[ERROR] 2020-02-28 10:04:29.307 [main] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

redis://[:password]@localhost:6379/0

rediss://[:password]@localhost:6379/0

unix://[:password]@/path/to/socket.sock?db=0

db.test_tab.insert({array:[]})

db.test_tab.insert({array:[]})

db.test_tab.insert({array:[]})

db.test_tab.insert({array:[1,2,3,4,5]})

db.test_tab.insert({array:[1,2,3,4,5,6]})

db.getCollection("example").find({array:{$exists:true,$ne:[]}}); # 字段不为0

 # helper

    class RedisHelp(object):



        def __init__(self,channel):

            # self.pool = redis.ConnectionPool('10.18.6.46',port=6379)



            # self.conn = redis.Redis(connection_pool=self.pool)

            # 上面的方式无法使用订阅者 发布者模式



            self.conn = redis.Redis(host='10.18.6.46')

            self.publish_channel = channel

            self.subscribe_channel = channel





        def publish(self,msg):

            self.conn.publish(self.publish_channel,msg) # 1. 渠道名 ,2 信息



        def subscribe(self):

            self.pub = self.conn.pubsub()

            self.pub.subscribe(self.subscribe_channel)

            self.pub.parse_response()

            print('initial')

            return self.pub





    helper = RedisHelp('cuiqingcai')



    # 订阅者

    if sys.argv[1]=='s':

        print('in subscribe mode')

        pub = helper.subscribe()

        while 1:

            print('waiting for publish')

            pubsub.check_health()

            msg = pub.parse_response()



            s=str(msg[2],encoding='utf-8')

            print(s)

            if s=='exit':

                break





    # 发布者

    elif sys.argv[1]=='p':

        print('in publish mode')

        msg = sys.argv[2]

        print(f'msg -> {msg}')

        helper.publish(msg)

db.test.update({},{$rename:{'旧字段':'新字段'}},true,true)

db.getCollection('example').update({},{$rename:{'corp':'企业'}})

db.getCollection('example').update({},{$rename:{'corp.address':'corp.地址'}})

ERRORS:

?: (admin.E408) 'django.contrib.auth.middleware.AuthenticationMiddleware' must be in MIDDLEWARE in order to use the admin application.

  File "C:\ProgramData\Anaconda3\lib\site-packages\pymongo\helpers.py", line 155, in _check_command_response

    raise OperationFailure(msg % errmsg, code, response)

pymongo.errors.OperationFailure: Query for sharded findAndModify must contain the shard key

2019-06-10 16:14:32 [scrapy.core.engine] INFO: Closing spider (finished)

2019-06-10 16:14:32 [scrapy.statscollectors] INFO: Dumping Scrapy stats:

Targeted Operations vs. Broadcast Operations

Generally, the fastest queries in a sharded environment are those that mongos route to a single shard, using the shard key and the cluster meta data from the config server. These targeted operations use the shard key value to locate the shard or subset of shards that satisfy the query document.

For queries that don’t include the shard key, mongos must query all shards, wait for their responses and then return the result to the application. These “scatter/gather” queries can be long running operations.

Broadcast Operations

mongos instances broadcast queries to all shards for the collection unless the mongos can determine which shard or subset of shards stores this data.



After the mongos receives responses from all shards, it merges the data and returns the result document. The performance of a broadcast operation depends on the overall load of the cluster, as well as variables like network latency, individual shard load, and number of documents returned per shard. Whenever possible, favor operations that result in targeted operation over those that result in a broadcast operation.

Multi-update operations are always broadcast operations.

The updateMany() and deleteMany() methods are broadcast operations, unless the query document specifies the shard key in full.

Targeted Operations

mongos can route queries that include the shard key or the prefix of a compound shard key a specific shard or set of shards. mongos uses the shard key value to locate the chunk whose range includes the shard key value and directs the query at the shard containing that chunk.



For example, if the shard key is:

copy

{ a: 1, b: 1, c: 1 }



The mongos program can route queries that include the full shard key or either of the following shard key prefixes at a specific shard or set of shards:

copy

{ a: 1 }

{ a: 1, b: 1 }



All insertOne() operations target to one shard. Each document in the insertMany() array targets to a single shard, but there is no guarantee all documents in the array insert into a single shard.

All updateOne(), replaceOne() and deleteOne() operations must include the shard key or _id in the query document. MongoDB returns an error if these methods are used without the shard key or _id.

Depending on the distribution of data in the cluster and the selectivity of the query, mongos may still perform a broadcast operation to fulfill these queries.

Index Use

If the query does not include the shard key, the mongos must send the query to all shards as a “scatter/gather” operation. Each shard will, in turn, use either the shard key index or another more efficient index to fulfill the query.

If the query includes multiple sub-expressions that reference the fields indexed by the shard key and the secondary index, the mongos can route the queries to a specific shard and the shard will use the index that will allow it to fulfill most efficiently.

Sharded Cluster Security

Use Internal Authentication to enforce intra-cluster security and prevent unauthorized cluster components from accessing the cluster. You must start each mongod or mongos in the cluster with the appropriate security settings in order to enforce internal authentication.

See Deploy Sharded Cluster with Keyfile Access Control for a tutorial on deploying a secured shardedcluster.

Cluster Users

Sharded clusters support Role-Based Access Control (RBAC) for restricting unauthorized access to cluster data and operations. You must start each mongod in the cluster, including the config servers, with the --auth option in order to enforce RBAC. Alternatively, enforcing Internal Authentication for inter-cluster security also enables user access controls via RBAC.

With RBAC enforced, clients must specify a --username, --password, and --authenticationDatabase when connecting to the mongos in order to access cluster resources.

Each cluster has its own cluster users. These users cannot be used to access individual shards.

See Enable Access Control for a tutorial on enabling adding users to an RBAC-enabled MongoDB deployment.

import pymongo

db = pymongo.MongoClient('mongodb://10.18.6.46,10.18.6.26,10.18.6.102')

db = pymongo.MongoClient('mongodb://10.18.6.46:8888,10.18.6.26:9999,10.18.6.102:7777')

coll=db['testdb']['testcollection'].find()

for i in coll:

    print(i)

{'_id': ObjectId('5cf4c7981ee9edff72e5c503'), 'username': 'hello'}

{'_id': ObjectId('5cf4c7991ee9edff72e5c504'), 'username': 'hello'}

{'_id': ObjectId('5cf4c7991ee9edff72e5c505'), 'username': 'hello'}

{'_id': ObjectId('5cf4c79a1ee9edff72e5c506'), 'username': 'hello'}

{'_id': ObjectId('5cf4c7b21ee9edff72e5c507'), 'username': 'hello world'}

collection = db['testdb']['testcollection']



for i in range(10):

    collection.insert({'username':'huston{}'.format(i)})

{

  "took" : 3,

  "timed_out" : false,

  "_shards" : {

    "total" : 5,

    "successful" : 5,

    "skipped" : 0,

    "failed" : 0

  },

  "hits" : {

    "total" : 5,

    "max_score" : 1.0,

    "hits" : [

      {

        "_index" : "test",

        "_type" : "mydoc",

        "_id" : "XxyrM2kBVzdNcvl_GHv2",

        "_score" : 1.0,

        "_source" : {

          "name" : "Shiled",

          "twitter" : "Sonny sql is awesome",

          "date" : "2018-12-27",

          "id" : 1240,

          "tags" : [

            "red",

            "shit"

          ]

        }

      },

      {

        "_index" : "test",

        "_type" : "mydoc",

        "_id" : "YByrM2kBVzdNcvl_tnvm",

        "_score" : 1.0,

        "_source" : {

          "name" : "YYerk",

          "twitter" : "sql is awesome",

          "date" : "2008-12-27",

          "id" : 12357,

          "tags" : [

            "red"

          ]

        }

      },

      {

        "_index" : "test",

        "_type" : "mydoc",

        "_id" : "7777",

        "_score" : 1.0,

        "_source" : {

          "name" : "Rocky Chen",

          "twitter" : "sql is awesome",

          "date" : "2008-12-27",

          "id" : 9999

        }

      },

      {

        "_index" : "test",

        "_type" : "mydoc",

        "_id" : "YhzDN2kBVzdNcvl_enuT",

        "_score" : 1.0,

        "_source" : {

          "name" : "YYerk",

          "twitter" : "sql is awesome",

          "date" : "2008-12-27",

          "id" : 888888,

          "tags" : [

            "red",

            "green"

          ]

        }

      },

      {

        "_index" : "test",

        "_type" : "mydoc",

        "_id" : "YxzDN2kBVzdNcvl_u3th",

        "_score" : 1.0,

        "_source" : {

          "name" : "YYerk",

          "twitter" : "sql is awesome",

          "date" : "2008-12-27",

          "id" : 888888,

          "tags" : [

            "red",

            "green"

          ]

        }

      }

    ]

  }

}

 

路由一个文档到一个分片中编辑
当索引一个文档的时候，文档会被存储到一个主分片中。 Elasticsearch 如何知道一个文档应该存放到哪个分片中呢？当我们创建文档时，它如何决定这个文档应当被存储在分片 1 还是分片 2 中呢？
首先这肯定不会是随机的，否则将来要获取文档的时候我们就不知道从何处寻找了。实际上，这个过程是根据下面这个公式决定的：
shard = hash(routing) % number_of_primary_shards
routing 是一个可变值，默认是文档的 _id ，也可以设置成一个自定义的值。 routing 通过 hash 函数生成一个数字，然后这个数字再除以 number_of_primary_shards （主分片的数量）后得到 余数 。这个分布在 0 到 number_of_primary_shards-1 之间的余数，就是我们所寻求的文档所在分片的位置。
这就解释了为什么我们要在创建索引的时候就确定好主分片的数量 并且永远不会改变这个数量：因为如果数量变化了，那么所有之前路由的值都会无效，文档也再也找不到了。

{

  "nodes" : {

    "yl-qYmh1SXqzJsfI4d1ddw" : {

      "name" : "node-3",

      "ephemeral_id" : "UsJ9rFELTiCW07oHE9YMdg",

      "transport_address" : "10.18.6.26:9300",

      "attributes" : {

        "ml.machine_memory" : "6088101888",

        "rack" : "r1",

        "ml.max_open_jobs" : "20",

        "xpack.installed" : "true",

        "ml.enabled" : "true"

      }

    },

    "wT7wUd3iTkujYUsbVNVv-w" : {

      "name" : "node-1",

      "ephemeral_id" : "fP-vgSb0SdemnHDyaJUsWw",

      "transport_address" : "10.18.6.102:9300",

      "attributes" : {

        "ml.machine_memory" : "8256720896",

        "rack" : "r1",

        "xpack.installed" : "true",

        "ml.max_open_jobs" : "20",

        "ml.enabled" : "true"

      }

    }

  },

  "indices" : {

    "test" : { }

  },

  "shards" : [

    [

      {

        "state" : "STARTED",

        "primary" : true,

        "node" : "wT7wUd3iTkujYUsbVNVv-w",

        "relocating_node" : null,

        "shard" : 1,

        "index" : "test",

        "allocation_id" : {

          "id" : "k-8E4dL7QmGgwcsNsUCP6Q"

        }

      },

      {

        "state" : "STARTED",

        "primary" : false,

        "node" : "yl-qYmh1SXqzJsfI4d1ddw",

        "relocating_node" : null,

        "shard" : 1,

        "index" : "test",

        "allocation_id" : {

          "id" : "lvOpQIKgRUibkulr3nRfEw"

        }

      }

    ]

  ]

}