[Linux] 架設 MongoDB Sharded Cluster 與驗證資料分散性 @ Ubuntu 14.04

今年年初時，把玩過 MongoDB Replica Set ，也試過一些 Map-Reduce 計算等，最近終於又開始抽空實驗了 :P 當時看到不少 Sharded Cluster 搭建在 MongoDB Replica Set 身上，深感 process (server) 的數量不低，就遲遲沒跳下去玩，最近才發現就算只是單一個 MongoDB Server 也可以當作 shared Cluster 一員，就方便啦！

此次 MongoDB Sharded Cluster 架構：

• 跑兩個 MongoDB Server，分別在 port 10001, 10002
• 跑一個 MongoDB Config Server - port 20001
• 跑一個 MongoDB Routing Server - port 30001

以下操作全部都在一台 ubuntu 14.04 server：

$ rm -rf /tmp/mongo && mkdir -p /tmp/mongo/db/1 /tmp/mongo/db/2 /tmp/mongo/db/c /tmp/mongo/log

$ mongod --port 10001 --dbpath /tmp/mongo/db/1 --logpath /tmp/mongo/log/1 &
$ mongod --port 10002 --dbpath /tmp/mongo/db/2 --logpath /tmp/mongo/log/2 &
$ mongod --port 20001 --configsvr --dbpath /tmp/mongo/db/c --logpath /tmp/mongo/log/c &

$ mongos --port 30001 --configdb 127.0.0.1:20001 &

設定 partition 用法：

$ mongo --port 30001
mongos> sh.status()
--- Sharding Status ---
  sharding version: {
        "_id" : 1,
        "version" : 4,
        "minCompatibleVersion" : 4,
        "currentVersion" : 5,
        "clusterId" : ObjectId("###############")
}
  shards:
  databases:
        {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
mongos> sh.addShard('127.0.0.1:10001')
{ "shardAdded" : "shard0000", "ok" : 1 }
mongos> sh.addShard('127.0.0.1:10002')
{ "shardAdded" : "shard0001", "ok" : 1 }
mongos> sh.status()
--- Sharding Status ---
  sharding version: {
        "_id" : 1,
        "version" : 4,
        "minCompatibleVersion" : 4,
        "currentVersion" : 5,
        "clusterId" : ObjectId("#####################")
}
  shards:
        {  "_id" : "shard0000",  "host" : "127.0.0.1:10001" }
        {  "_id" : "shard0001",  "host" : "127.0.0.1:10002" }
  databases:
        {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
mongos> sh.enableSharding('ydb')
{ "ok" : 1 }
mongos> sh.status()
--- Sharding Status ---
  sharding version: {
        "_id" : 1,
        "version" : 4,
        "minCompatibleVersion" : 4,
        "currentVersion" : 5,
        "clusterId" : ObjectId("######################")
}
  shards:
        {  "_id" : "shard0000",  "host" : "127.0.0.1:10001" }
        {  "_id" : "shard0001",  "host" : "127.0.0.1:10002" }
  databases:
        {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
        {  "_id" : "test",  "partitioned" : false,  "primary" : "shard0000" }
        {  "_id" : "ydb",  "partitioned" : true,  "primary" : "shard0000" }
mongos> use ydb
switched to db ydb
mongos> db.ycollection.ensureIndex( { _id : "hashed" } )
{
        "raw" : {
                "127.0.0.1:10001" : {
                        "createdCollectionAutomatically" : true,
                        "numIndexesBefore" : 1,
                        "numIndexesAfter" : 2,
                        "ok" : 1
                }
        },
        "ok" : 1
}
mongos> sh.shardCollection("ydb.ycollection", { "_id": "hashed" } )
{ "collectionsharded" : "ydb.ycollection", "ok" : 1 }
mongos> sh.status()
--- Sharding Status ---
  sharding version: {
        "_id" : 1,
        "version" : 4,
        "minCompatibleVersion" : 4,
        "currentVersion" : 5,
        "clusterId" : ObjectId("#####################")
}
  shards:
        {  "_id" : "shard0000",  "host" : "127.0.0.1:10001" }
        {  "_id" : "shard0001",  "host" : "127.0.0.1:10002" }
  databases:
        {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
        {  "_id" : "test",  "partitioned" : false,  "primary" : "shard0000" }
        {  "_id" : "ydb",  "partitioned" : true,  "primary" : "shard0000" }
                ydb.ycollection
                        shard key: { "_id" : "hashed" }
                        chunks:
                                shard0000       2
                                shard0001       2
                        { "_id" : { "$minKey" : 1 } } -->> { "_id" : NumberLong("-4611686018427387902") } on : shard0000 Timestamp(2, 2)
                        { "_id" : NumberLong("-4611686018427387902") } -->> { "_id" : NumberLong(0) } on : shard0000 Timestamp(2, 3)
                        { "_id" : NumberLong(0) } -->> { "_id" : NumberLong("4611686018427387902") } on : shard0001 Timestamp(2, 4)
                        { "_id" : NumberLong("4611686018427387902") } -->> { "_id" : { "$maxKey" : 1 } } on : shard0001 Timestamp(2, 5)

簡易驗證資料分散性：

mongos>

db.ycollection.insert({user:"a", item: "1", rate: 0.3})
db.ycollection.insert({user:"a", item: "3", rate: 0.5})
db.ycollection.insert({user:"a", item: "4", rate: 0.9})
db.ycollection.insert({user:"b", item: "2", rate: 0.1})
db.ycollection.insert({user:"b", item: "3", rate: 0.6})
db.ycollection.insert({user:"b", item: "5", rate: 0.2})
db.ycollection.insert({user:"c", item: "3", rate: 0.2})
db.ycollection.insert({user:"c", item: "4", rate: 0.7})

mongos> db.ycollection.find()
{ "_id" : ObjectId("#################"), "user" : "a", "item" : "3", "rate" : 0.5 }
{ "_id" : ObjectId("#################"), "user" : "a", "item" : "1", "rate" : 0.3 }
{ "_id" : ObjectId("#################"), "user" : "a", "item" : "4", "rate" : 0.9 }
{ "_id" : ObjectId("#################"), "user" : "b", "item" : "2", "rate" : 0.1 }
{ "_id" : ObjectId("#################"), "user" : "b", "item" : "5", "rate" : 0.2 }
{ "_id" : ObjectId("#################"), "user" : "b", "item" : "3", "rate" : 0.6 }
{ "_id" : ObjectId("#################"), "user" : "c", "item" : "3", "rate" : 0.2 }
{ "_id" : ObjectId("#################"), "user" : "c", "item" : "4", "rate" : 0.7 }

接著，想要驗證資料的確是儲存在不同台 mongodb server ，作法就是把資料透過 mongodump 出來(bson)，再用 bsondump 印出來驗證：

$ cd /tmp
$ mongodump --port 10001 -d ydb -c ycollection
$ bsondump dump/ydb/ycollection.bson
{ "_id" : ObjectId( "#############" ), "user" : "a", "item" : "1", "rate" : 0.3 }
{ "_id" : ObjectId( "#############" ), "user" : "b", "item" : "2", "rate" : 0.1 }
{ "_id" : ObjectId( "#############" ), "user" : "b", "item" : "3", "rate" : 0.6 }
{ "_id" : ObjectId( "#############" ), "user" : "c", "item" : "4", "rate" : 0.7 }
4 objects found

$ mongodump --port 10002 -d ydb -c ycollection
$ bsondump dump/ydb/ycollection.bson
{ "_id" : ObjectId( "#############" ), "user" : "a", "item" : "3", "rate" : 0.5 }
{ "_id" : ObjectId( "#############" ), "user" : "a", "item" : "4", "rate" : 0.9 }
{ "_id" : ObjectId( "#############" ), "user" : "b", "item" : "5", "rate" : 0.2 }
{ "_id" : ObjectId( "#############" ), "user" : "c", "item" : "3", "rate" : 0.2 }
4 objects found

因此可完成驗證，資料的確分在不同區！有興趣還可以用 MongoDB Aggregate (Map-Reduce) 實作共現矩陣(Co-Occurrence Matrix)及簡易的推薦系統 來驗證 partition 演算法的正確性。

[MySQL] Got fatal error 1236 from master when reading data from binary log: 'Could not find first log file name in binary log index file'

些前在阿里雲、AWS RDS 架設 MySQL Master-Master Replication 出現此問題 Orz 將兩邊設定為各是對方的 Master 如此達到資料跨國同步。然而，卻發現在阿里雲作為 Replication Slave 角色的出現這個問題 Orz 由於在阿里雲的 DB Server 偏向 Cache 角色，確認後資料屬性，決定解決的方案是將阿里雲機器清空再重 AWS RDS 匯入，筆記一下解決步驟：

1. 兩邊都停下 slave 角色動作
• ALiYun-DB> STOP SLAVE
• AWS-RDB> CALL mysql.rds_stop_replication
2. 從 AWS RDS 匯出資料、並查看 MASTER_LOG_FILE 和 MASTER_LOG_POS 資訊
• 在 AWS RDS 標準做法是 Using Replication to Export Amazon RDS MySQL 5.6 Data，若 DB 資料更新沒很頻繁或是重要料更動低，可以直接考慮用 mysqldump
• AWS-RDB> SHOW MASTER STATUS \G
3. 匯入阿里雲 DB Server，並設定 Slave 角色資訊，啟動 Slave 工作
• $ mysql -h AliYunDB < AWSRDBExport.sql
• ALiYun-DB> CHANGE MASTER TO MASTER_LOG_FILE='aws-rdb-mysql-bin-changelog.#######', MASTER_LOG_POS=#####;
• ALiYun-DB> START SLAVE
• ALiYun-DB> SHOW SLAVE STATUS \G
4. 查看阿里雲 MASTER_LOG_FILE 和 MASTER_LOG_POS，更新 AWS RDS Slave 角色資訊，重新啟動 AWS RDS Slave 工作
• ALiYun-DB> SHOW MASTER STATUS \G
• MASTER_LOG_FILE
• MASTER_LOG_POS
• AWS-RDB> CALL mysql.rds_reset_external_master;
• AWS-RDB> CALL mysql.rds_set_external_master (
      'aliyun-db-server-ip',
      3306,
      'repl_account',
      'repl_password',
      'aliyun-db-MASTER_LOG_FILE',
      aliyun-db-MASTER_LOG_POS
  ,   0 )
• AWS-RDB > CALL mysql.rds_start_replication;
• AWS-RDS > CALL mysql.rds_set_configuration('binlog retention hours', 24*14);

[Linux] 使用 MongoDB Aggregate (Map-Reduce) 實作共現矩陣(Co-Occurrence Matrix)及簡易的推薦系統 @ Ubuntu 14.04

這陣子查閱了一些推薦系統的作法，發現早在 2012 年網路上就有非常多 Apache Mahout 的使用心得，這一套已經內含許多常見的演算法精華，故許多人都是採用 Apache Mahout 搭建推薦系統的。至於為何還要自己刻一個出來？單純練功啦，在此就練練 Partition、Sort 和 Merge (Map-Reduce) 的分析方式，但採用 MongoDB Aggregate 實作。

此例挑選 user, item 的關聯紀錄，例如 user1 買了 item1, item2, item5，而 user2 買了 item3, item4 等。在推薦系統中，大多就是將 input 前置處理，在挑恰當的演算法加以計算，接著處理 output。在此採用共現矩陣(Co-Occurrence Matrix) 的用法，以 user, item 的關聯紀錄，可以統計出一個 Item-based Co-Occurrence Matrix，簡言之就是各個 item 之間的關係，例如透過 item 1 跟 item 2 一起出現現象，整理成一個計算方式。

假設共有 a, b, c 三個使用者，而有 1, 2, 3, 4, 5，共有五種物品，其中 a, b, c 分別購買部分用品，並對各物品的評價(或價錢等意義)如下：

a, 1, 0.3
a, 3, 0.5
a, 4, 0.9

b, 2, 0.1
b, 3, 0.6
b, 5, 0.2

c, 3, 0.2
c, 4, 0.7

其中, 共現矩陣(Co-Occurrence Matrix) 如下：

\  1, 2, 3, 4, 5
1: 1, 0, 1, 1, 0
2: 0, 1, 1, 0, 1
3: 1, 1, 3, 2, 1
4: 1, 0, 2, 2, 0
5: 0, 1, 1, 0, 1

(1, 1) = 1, 代表 1 僅出現在 1 個使用者的清單
(4, 4) = 2, 代表 4 出現在 2 個使用者的清單
(3, 3) = 3, 代表 3 出現在 3 個使用者的清單
(4, 3) = 2, 代表 4 跟 3 同時出現的次數有兩次

接著，把使用者的購物清單也向量化(或矩陣):

\    a,   b,   c
1: 0.3,   0,   0,
2:   0, 0.1,   0,
3: 0.5, 0.6, 0.2,
4: 0.9,   0, 0.7,
5:   0, 0.2,   0,

當這兩個矩陣相成的結果，即為使用者與物品的關聯成果，且每個物品的分數即可當作使用者感興趣的分數。


以 MongoDB 為例，先將資料一筆筆新增至 db.rec 中：

db.rec.drop()
db.rec.insert({user:"a", item: "1", rate: 0.3})
db.rec.insert({user:"a", item: "3", rate: 0.5})
db.rec.insert({user:"a", item: "4", rate: 0.9})
db.rec.insert({user:"b", item: "2", rate: 0.1})
db.rec.insert({user:"b", item: "3", rate: 0.6})
db.rec.insert({user:"b", item: "5", rate: 0.2})
db.rec.insert({user:"c", item: "3", rate: 0.2})
db.rec.insert({user:"c", item: "4", rate: 0.7})

> db.rec.find()
{ "_id" : ObjectId("5450dedfbdfb5bad287c953c"), "user" : "a", "item" : "1", "rate" : 0.3 }
{ "_id" : ObjectId("5450dedfbdfb5bad287c953d"), "user" : "a", "item" : "3", "rate" : 0.5 }
{ "_id" : ObjectId("5450dedfbdfb5bad287c953e"), "user" : "a", "item" : "4", "rate" : 0.9 }
{ "_id" : ObjectId("5450dedfbdfb5bad287c953f"), "user" : "b", "item" : "2", "rate" : 0.1 }
{ "_id" : ObjectId("5450dedfbdfb5bad287c9540"), "user" : "b", "item" : "3", "rate" : 0.6 }
{ "_id" : ObjectId("5450dedfbdfb5bad287c9541"), "user" : "b", "item" : "5", "rate" : 0.2 }
{ "_id" : ObjectId("5450dedfbdfb5bad287c9542"), "user" : "c", "item" : "3", "rate" : 0.2 }
{ "_id" : ObjectId("5450dee0bdfb5bad287c9543"), "user" : "c", "item" : "4", "rate" : 0.7 }

接著分析 user 跟 item 的個數，分別儲存在 db.user 跟 db.item 中：

> db.item.drop()
true
> db.rec.aggregate([{$group: {_id: "$item"}}, {$sort: {_id: 1}}]).forEach(function(input) {
db.item.insert(input);
})
> db.item.find()
{ "_id" : "1" }
{ "_id" : "2" }
{ "_id" : "3" }
{ "_id" : "4" }
{ "_id" : "5" }

> db.user.drop()
> db.rec.aggregate([{$group: {_id: "$user"}}, {$sort: {_id: 1}}]).forEach(function(input) {
db.user.insert(input);
})
> db.user.find()
{ "_id" : "a" }
{ "_id" : "b" }
{ "_id" : "c" }

建立 Item-based Co-Occurrence Matrix（在此採用sparse matrix）：

> db.user_item_list.drop();
> db.rec.aggregate({ $group: {_id: "$user", items: {$push: "$item"} }}).forEach(function(input){
db.user_item_list.insert(input);
})

> db.comatrix_input.drop()
> db.user_item_list.find().forEach(function(input){
for (var i in input.items){
var key = input.items[i]+"-"+input.items[i];
db.comatrix_input.insert( { key: key, value: 1} );
for (var j in input.items){
if (j>i) {
var key = input.items[i]+"-"+input.items[j];
db.comatrix_input.insert( { key: key, value: 1} );
key = input.items[j]+"-"+input.items[i];
db.comatrix_input.insert( { key: key, value: 1} );
}
}
}
});
> db.comatrix.drop()
> db.comatrix_input.aggregate({ $group: {_id: "$key", value: {$sum: "$value"}}}).forEach(function(input) {
db.comatrix.insert(input)
});
> db.comatrix.find()
{ "_id" : "1-3", "value" : 1 }
{ "_id" : "3-5", "value" : 1 }
{ "_id" : "5-2", "value" : 1 }
{ "_id" : "1-4", "value" : 1 }
{ "_id" : "3-1", "value" : 1 }
{ "_id" : "3-2", "value" : 1 }
{ "_id" : "2-5", "value" : 1 }
{ "_id" : "2-3", "value" : 1 }
{ "_id" : "1-1", "value" : 1 }
{ "_id" : "3-3", "value" : 3 }
{ "_id" : "5-3", "value" : 1 }
{ "_id" : "4-1", "value" : 1 }
{ "_id" : "4-3", "value" : 2 }
{ "_id" : "3-4", "value" : 2 }
{ "_id" : "5-5", "value" : 1 }
{ "_id" : "4-4", "value" : 2 }
{ "_id" : "2-2", "value" : 1 }

透過共現矩陣推算 user 對 item 的喜好程度：

> db.user_prefer.drop()
> db.item.find().forEach(function(in1) {
var value = 0;
db.item.find().forEach(function(in2) {
var key = in1._id+"-"+in2._id;
var factorInfo = db.comatrix.findOne( {_id : key} )
if (factorInfo && factorInfo.value) {
db.user.find().forEach(function(in3){
var user_id = in3._id
var get = db.rec.findOne({user: in3._id, item: in2._id})
if (get)
db.user_prefer.insert( {user: in3._id, item: in1._id, value: factorInfo.value * get.rate} )
})
}
})
})

> db.recommendation.drop()
> db.user_prefer.aggregate([{$group: {_id: { user: "$user", item:"$item"}, value: {$sum: "$value"}}}, {$sort: {_id: 1}}]).forEach(function(input){
db.recommendation.insert({_id: input._id, user: input._id.user, item: input._id.item, value: input.value})
})
> db.recommendation.find()
{ "_id" : { "user" : "a", "item" : "1" }, "user" : "a", "item" : "1", "value" : 1.7000000000000002 }
{ "_id" : { "user" : "a", "item" : "2" }, "user" : "a", "item" : "2", "value" : 0.5 }
{ "_id" : { "user" : "a", "item" : "3" }, "user" : "a", "item" : "3", "value" : 3.6 }
{ "_id" : { "user" : "a", "item" : "4" }, "user" : "a", "item" : "4", "value" : 3.1 }
{ "_id" : { "user" : "a", "item" : "5" }, "user" : "a", "item" : "5", "value" : 0.5 }
{ "_id" : { "user" : "b", "item" : "1" }, "user" : "b", "item" : "1", "value" : 0.6 }
{ "_id" : { "user" : "b", "item" : "2" }, "user" : "b", "item" : "2", "value" : 0.8999999999999999 }
{ "_id" : { "user" : "b", "item" : "3" }, "user" : "b", "item" : "3", "value" : 2.1 }
{ "_id" : { "user" : "b", "item" : "4" }, "user" : "b", "item" : "4", "value" : 1.2 }
{ "_id" : { "user" : "b", "item" : "5" }, "user" : "b", "item" : "5", "value" : 0.8999999999999999 }
{ "_id" : { "user" : "c", "item" : "1" }, "user" : "c", "item" : "1", "value" : 0.8999999999999999 }
{ "_id" : { "user" : "c", "item" : "2" }, "user" : "c", "item" : "2", "value" : 0.2 }
{ "_id" : { "user" : "c", "item" : "3" }, "user" : "c", "item" : "3", "value" : 2 }
{ "_id" : { "user" : "c", "item" : "4" }, "user" : "c", "item" : "4", "value" : 1.7999999999999998 }
{ "_id" : { "user" : "c", "item" : "5" }, "user" : "c", "item" : "5", "value" : 0.2 }

清掉 user 已擁有 items：

> db.rec.find().forEach(function(input){
var key = { "user": input.user, "item": input.item }
db.recommendation.remove( { _id : { $eq: key } } )
})

> db.recommendation.find()
{ "_id" : { "user" : "a", "item" : "2" }, "user" : "a", "item" : "2", "value" : 0.5 }
{ "_id" : { "user" : "a", "item" : "5" }, "user" : "a", "item" : "5", "value" : 0.5 }
{ "_id" : { "user" : "b", "item" : "1" }, "user" : "b", "item" : "1", "value" : 0.6 }
{ "_id" : { "user" : "b", "item" : "4" }, "user" : "b", "item" : "4", "value" : 1.2 }
{ "_id" : { "user" : "c", "item" : "1" }, "user" : "c", "item" : "1", "value" : 0.8999999999999999 }
{ "_id" : { "user" : "c", "item" : "2" }, "user" : "c", "item" : "2", "value" : 0.2 }
{ "_id" : { "user" : "c", "item" : "5" }, "user" : "c", "item" : "5", "value" : 0.2 }

收工！

[Linux] Docker 使用筆記 - 常用指令 @ Ubuntu 14.04

搜尋可用 Images (此例關鍵字為 Ubuntu):

$ sudo docker search ubuntu

下載或更新指定 Image (此例為 ubuntu 14.04):

$ sudo docker pull ubuntu:14.04

列出所有 Images:

$ sudo docker images

刪除特定 Image:

$ sudo docker rmi IMAGE_ID

得知 container 的資訊或指定欄位(IPAddress、HostPort為例):

$ sudo docker inspect CONTAINER_ID_OR_NAME
$ sudo docker inspect --format '{{.NetworkSettings.IPAddress}}' CONTAINER_ID_OR_NAME
$ sudo docker inspect CONTAINER_ID_OR_NAME | grep "HostPort" | uniq | awk -F'"' '{print $4}'

停止所有 container:

$ sudo docker stop $(sudo docker ps -q)

刪除所有 container:

$ sudo docker rm $(sudo docker ps -a -q)

刪除所有 image:

$ sudo docker rmi $(sudo docker images -q)

刪除 untagged images:

$ sudo docker rmi $(sudo docker images -qf "dangling=true")

使用 nscenter 進入 container:

$ sudo docker inspect --format '{{.State.Pid}}' CONTAINER_ID
PID_NUMBER
$ sudo nsenter --target PID_NUMBER --mount --uts --ipc --net --pid

[Linux] Docker 使用筆記 - 透過 nsenter 進入當前 container 操作環境 @ Ubuntu 14.04

早期的 Docker 可以直接用 lxc-attach 取得 container 的 terminal 環境：

$ sudo docker ps --no-trunc
$ sudo lxc-attach -n CONTAINER_ID

但新版的只會噴 lxc-attach: failed to get the init pid 訊息，所以可以改用 nsenter 來操作吧！

$ sudo docker version
Client version: 1.0.1
Client API version: 1.12
Go version (client): go1.2.1
Git commit (client): 990021a
Server version: 1.0.1
Server API version: 1.12
Go version (server): go1.2.1
Git commit (server): 990021a

$ curl https://www.kernel.org/pub/linux/utils/util-linux/v2.24/util-linux-2.24.tar.gz | tar -zxf-
$ cd util-linux-2.24
$ ./configure --without-ncurses
$ make nsenter
$ sudo cp nsenter /usr/local/bin

$ sudo docker inspect --format "{{ .State.Pid }}" CONTAINER_ID_OR_NAME
$ sudo nsenter --target $PID --mount --uts --ipc --net --pid
root@##############:/#