MongoDB: 9. Sharding (2)

MongoDB Auto-Sharding 解决了海量存储和动态扩容的问题，但离实际生产环境所需的高可靠(high reliability)、高可用(high availability)还有些距离。

解决方案:
Shard: 使用 Replica Sets，确保每个数据节点都具有备份、自动容错转移、自动恢复能力。
Config: 使用 3 个配置服务器，确保元数据完整性(two-phase commit)。
Route: 配合 LVS，实现负载平衡，提高接入性能(high performance)。
以下我们配置一个 Replica Sets + Sharding 测试环境。
装个配置过程建议都用 IP 地址，以免出错。

(1) 首先建好所有的数据库目录。

$ sudo mkdir -p /var/mongodb/10001
$ sudo mkdir -p /var/mongodb/10002
$ sudo mkdir -p /var/mongodb/10003

$ sudo mkdir -p /var/mongodb/10011
$ sudo mkdir -p /var/mongodb/10012
$ sudo mkdir -p /var/mongodb/10013

$ sudo mkdir -p /var/mongodb/config1
$ sudo mkdir -p /var/mongodb/config2
$ sudo mkdir -p /var/mongodb/config3
(2) 配置 Shard Replica Sets。

$ sudo ./mongod --shardsvr --fork --logpath /dev/null --dbpath /var/mongodb/10001 --port 10001 --nohttpinterface --replSet set1
forked process: 4974
all output going to: /dev/null

$ sudo ./mongod --shardsvr --fork --logpath /dev/null --dbpath /var/mongodb/10002 --port 10002 --nohttpinterface --replSet set1
forked process: 4988
all output going to: /dev/null

$ sudo ./mongod --shardsvr --fork --logpath /dev/null --dbpath /var/mongodb/10003 --port 10003 --nohttpinterface --replSet set1
forked process: 5000
all output going to: /dev/null$ ./mongo --port 10001

MongoDB shell version: 1.6.2
connecting to: 127.0.0.1:10001/test

> cfg = { _id:'set1', members:[
... { _id:0, host:'192.168.1.202:10001' },
... { _id:1, host:'192.168.1.202:10002' },
... { _id:2, host:'192.168.1.202:10003' }
... ]};

> rs.initiate(cfg)
{
"info" : "Config now saved locally. Should come online in about a minute.",
"ok" : 1
}

> rs.status()
{
"set" : "set1",
"date" : "Tue Sep 07 2010 10:25:28 GMT+0800 (CST)",
"myState" : 5,
"members" : [
{
"_id" : 0,
"name" : "yuhen-server64:10001",
"health" : 1,
"state" : 5,
"self" : true
},
{
"_id" : 1,
"name" : "192.168.1.202:10002",
"health" : -1,
"state" : 6,
"uptime" : 0,
"lastHeartbeat" : "Thu Jan 01 1970 08:00:00 GMT+0800 (CST)"
},
{
"_id" : 2,
"name" : "192.168.1.202:10003",
"health" : -1,
"state" : 6,
"uptime" : 0,
"lastHeartbeat" : "Thu Jan 01 1970 08:00:00 GMT+0800 (CST)"
}
],
"ok" : 1
}
配置第二组 Shard Replica Sets。

$ sudo ./mongod --shardsvr --fork --logpath /dev/null --dbpath /var/mongodb/10011 --port 10011 --nohttpinterface --replSet set2
forked process: 5086
all output going to: /dev/null

$ sudo ./mongod --shardsvr --fork --logpath /dev/null --dbpath /var/mongodb/10012 --port 10012 --nohttpinterface --replSet set2
forked process: 5098
all output going to: /dev/null

$ sudo ./mongod --shardsvr --fork --logpath /dev/null --dbpath /var/mongodb/10013 --port 10013 --nohttpinterface --replSet set2
forked process: 5112
all output going to: /dev/null$ ./mongo --port 10011

MongoDB shell version: 1.6.2
connecting to: 127.0.0.1:10011/test

> cfg = { _id:'set2', members:[
... { _id:0, host:'192.168.1.202:10011' },
... { _id:1, host:'192.168.1.202:10012' },
... { _id:2, host:'192.168.1.202:10013' }
... ]}

> rs.initiate(cfg)
{
"info" : "Config now saved locally. Should come online in about a minute.",
"ok" : 1
}

> rs.status()
{
"set" : "set2",
"date" : "Tue Sep 07 2010 10:28:37 GMT+0800 (CST)",
"myState" : 1,
"members" : [
{
"_id" : 0,
"name" : "yuhen-server64:10011",
"health" : 1,
"state" : 1,
"self" : true
},
{
"_id" : 1,
"name" : "192.168.1.202:10012",
"health" : 0,
"state" : 6,
"uptime" : 0,
"lastHeartbeat" : "Tue Sep 07 2010 10:28:36 GMT+0800 (CST)",
"errmsg" : "still initializing"
},
{
"_id" : 2,
"name" : "192.168.1.202:10013",
"health" : 1,
"state" : 5,
"uptime" : 1,
"lastHeartbeat" : "Tue Sep 07 2010 10:28:36 GMT+0800 (CST)",
"errmsg" : "."
}
],
"ok" : 1
}
(3) 启动 Config Server。

我们可以只使用 1 个 Config Server，但 3 个理论上更有保障性。

Chunk information is the main data stored by the config servers. Each config server has a complete copy of all chunk information. A two-phase
commit is used to ensure the consistency of the configuration data among the config servers.

If any of the config servers is down, the cluster's meta-data goes read only. However, even in such a failure state, the MongoDB cluster can still
be read from and written to.
注意！这个不是 Replica Sets，不需要 --replSet 参数。

$ sudo ./mongod --configsvr --fork --logpath /dev/null --dbpath /var/mongodb/config1 --port 20000 --nohttpinterface
forked process: 5177
all output going to: /dev/null

$ sudo ./mongod --configsvr --fork --logpath /dev/null --dbpath /var/mongodb/config2 --port 20001 --nohttpinterface
forked process: 5186
all output going to: /dev/null

$ sudo ./mongod --configsvr --fork --logpath /dev/null --dbpath /var/mongodb/config3 --port 20002 --nohttpinterface
forked process: 5195
all output going to: /dev/null$ ps aux | grep configsvr | grep -v grep
root ./mongod --configsvr --fork --logpath /dev/null --dbpath /var/mongodb/config1 --port 20000 --nohttpinterface
root ./mongod --configsvr --fork --logpath /dev/null --dbpath /var/mongodb/config2 --port 20001 --nohttpinterface
root ./mongod --configsvr --fork --logpath /dev/null --dbpath /var/mongodb/config3 --port 20002 --nohttpinterface
(4) 启动 Route Server。

注意 --configdb 参数。

$ sudo ./mongos --fork --logpath /dev/null --configdb "192.168.1.202:20000,192.168.1.202:20001,192.168.1.202:20002"
forked process: 5209
all output going to: /dev/null$ ps aux | grep mongos | grep -v grep
root ./mongos --fork --logpath /dev/null --configdb 192.168.1.202:20000,192.168.1.202:20001,192.168.1.202:20002
(5) 开始配置 Sharding。

注意 addshard 添加 Replica Sets 的格式。

$ ./mongo

MongoDB shell version: 1.6.2
connecting to: test

> use admin
switched to db admin

> db.runCommand({ addshard:'set1/192.168.1.202:10001,192.168.1.202:10002,192.168.1.202:10003' })
{ "shardAdded" : "set1", "ok" : 1 }

> db.runCommand({ addshard:'set2/192.168.1.202:10011,192.168.1.202:10012,192.168.1.202:10013' })
{ "shardAdded" : "set2", "ok" : 1 }

> db.runCommand({ enablesharding:'test' })
{ "ok" : 1 }

> db.runCommand({ shardcollection:'test.data', key:{_id:1} })
{ "collectionsharded" : "test.data", "ok" : 1 }

> db.runCommand({ listshards:1 })
{
"shards" : [
{
"_id" : "set1",
"host" : "set1/192.168.1.202:10001,192.168.1.202:10002,192.168.1.202:10003"
},
{
"_id" : "set2",
"host" : "set2/192.168.1.202:10011,192.168.1.202:10012,192.168.1.202:10013"
}
],
"ok" : 1
}

> printShardingStatus()

--- Sharding Status ---
sharding version: { "_id" : 1, "version" : 3 }
shards:
{
"_id" : "set1",
"host" : "set1/192.168.1.202:10001,192.168.1.202:10002,192.168.1.202:10003"
}
{
"_id" : "set2",
"host" : "set2/192.168.1.202:10011,192.168.1.202:10012,192.168.1.202:10013"
}
databases:
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
{ "_id" : "test", "partitioned" : true, "primary" : "set1" }
test.data chunks:
{ "_id" : { $minKey : 1 } } -->> { "_id" : { $maxKey : 1 } } on : set1 { "t" : 1000, "i" : 0 }

---- 配置结束 ------

OK! 基本搞定，可以测试一下。

> use test
switched to db test

> db.data.insert({name:1})
> db.data.insert({name:2})
> db.data.insert({name:3})

> db.data.find()
{ "_id" : ObjectId("4c85a6d9ce93b9b1b302ebe7"), "name" : 1 }
{ "_id" : ObjectId("4c85a6dbce93b9b1b302ebe8"), "name" : 2 }
{ "_id" : ObjectId("4c85a6ddce93b9b1b302ebe9"), "name" : 3 }
至于为 Route 配置 LVS，可参考 《LVS 负载均衡》。