redis-cluster部署及數據遷移

工作原理

節選自redis官方文檔：http://www.redis.cn/topics/cluster-tutorial.html

Redis集群介紹

Redis 集群是一個提供在多個Redis間節點間共享數據的程序集。
Redis集群并不支持處理多個keys的命令,因為這需要在不同的節點間移動數據,從而達不到像Redis那樣的性能,在高負載的情況下可能會導致不可預料的錯誤.
Redis 集群通過分區來提供一定程度的可用性,在實際環境中當某個節點宕機或者不可達的情況下繼續處理命令. Redis 集群的優勢:

• 自動分割數據到不同的節點上。
• 整個集群的部分節點失敗或者不可達的情況下能夠繼續處理命令。

  Redis 集群的數據分片

  Redis 集群沒有使用一致性hash, 而是引入了 哈希槽的概念.
  Redis 集群有16384個哈希槽,每個key通過CRC16校驗后對16384取模來決定放置哪個槽.集群的每個節點負責一部分hash槽,舉個例子,比如當前集群有3個節點,那么:

• 節點 A 包含 0 到 5500號哈希槽.
• 節點 B 包含5501 到 11000 號哈希槽.
• 節點 C 包含11001 到 16384號哈希槽.

這種結構很容易添加或者刪除節點. 比如如果我想新添加個節點D, 我需要從節點 A, B, C中得部分槽到D上. 如果我像移除節點A,需要將A中得槽移到B和C節點上,然后將沒有任何槽的A節點從集群中移除即可. 由于從一個節點將哈希槽移動到另一個節點并不會停止服務,所以無論添加刪除或者改變某個節點的哈希槽的數量都不會造成集群不可用的狀態.

Redis 集群的主從復制模型

為了使在部分節點失敗或者大部分節點無法通信的情況下集群仍然可用，所以集群使用了主從復制模型,每個節點都會有N-1個復制品.
在我們例子中具有A，B，C三個節點的集群,在沒有復制模型的情況下,如果節點B失敗了，那么整個集群就會以為缺少5501-11000這個范圍的槽而不可用.
然而如果在集群創建的時候（或者過一段時間）我們為每個節點添加一個從節點A1，B1，C1,那么整個集群便有三個master節點和三個slave節點組成，這樣在節點B失敗后，集群便會選舉B1為新的主節點繼續服務，整個集群便不會因為槽找不到而不可用了
不過當B和B1 都失敗后，集群是不可用的.

redis安裝過程

1、下載redis安裝包

2、解壓

cd /app
tar zxf redis-3.2.11.tar.gz 
mv redis-3.2.11.tar.gz redis

3、編譯安裝

cd redis

make && make install

4、創建相關目錄,將相關命令從src目錄復制到bin目錄

mkdir -pv /app/redis/{bin,conf,data,logs}

5、使用utils中的install_server.sh安裝redis server

部署過程

Redis-cluster搭建

1、從測試打包已經編譯redis文件夾，上傳至需要安裝的服務器

2、解壓至app目錄，目錄結構如下，其中data,conf,logs為編譯完成后自行創建的目錄，分別用來存放數據文件，配置文件，日志文件，data目錄下自行創建以端口號命名的文件夾，分別用來存放各個端口號實例的數據文件

-rw-rw-r--  1 root root 92766 Sep 21 22:20 00-RELEASENOTES
drwxr-xr-x  2 root root  4096 Dec 20 16:40 bin
-rw-rw-r--  1 root root    53 Sep 21 22:20 BUGS
drwxr-xr-x  2 root root  4096 Jan  4 15:37 conf
-rw-rw-r--  1 root root  1805 Sep 21 22:20 CONTRIBUTING
-rw-rw-r--  1 root root  1487 Sep 21 22:20 COPYING
drwxr-xr-x 10 root root  4096 Jan  3 11:24 data
drwxrwxr-x  7 root root  4096 Dec 20 14:50 deps
-rw-rw-r--  1 root root    11 Sep 21 22:20 INSTALL
drwxr-xr-x  2 root root  4096 Jan  4 15:37 logs
-rw-rw-r--  1 root root   151 Sep 21 22:20 Makefile
-rw-rw-r--  1 root root  4223 Sep 21 22:20 MANIFESTO
-rw-rw-r--  1 root root  6834 Sep 21 22:20 README.md
-rw-rw-r--  1 root root 46695 Sep 21 22:20 redis.conf
-rwxrwxr-x  1 root root   271 Sep 21 22:20 runtest
-rwxrwxr-x  1 root root   280 Sep 21 22:20 runtest-cluster
-rwxrwxr-x  1 root root   281 Sep 21 22:20 runtest-sentinel
-rw-rw-r--  1 root root  7606 Sep 21 22:20 sentinel.conf
drwxrwxr-x  2 root root  4096 Dec 20 14:51 src
drwxrwxr-x 10 root root  4096 Sep 21 22:20 tests
drwxrwxr-x  7 root root  4096 Jan  4 14:36 utils

3、修改各個端口的配置文件，以node1:7000的配置文件為例，根據情況修改

bind node1
protected-mode yes
port 7000
tcp-backlog 511
timeout 0
tcp-keepalive 300
daemonize yes
supervised no
pidfile /var/run/redis_7000.pid
loglevel notice
logfile /app/redis/logs/redis_7000.log
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir /app/redis/data/7000
slave-serve-stale-data yes
slave-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
slave-priority 100
appendonly no
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
lua-time-limit 5000
cluster-enabled yes
cluster-config-file nodes-7000.conf
cluster-node-timeout 15000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes

4、啟動對應端口的服務（所有節點的主從都啟動）,以7000為例

/app/redis/bin/redis-server  /app/redis/conf/7000.conf

5、創建集群

 /app/redis/bin/redis-trib.rb create node1:7000 node2:7000 node2:7000

[root@node1 bin]# ./redis-trib.rb check node1:7000
>>> Performing Cluster Check (using node node1:7000)
M: 3b485951a72e16133464585dc2920c10ce75967b node1:7000
   slots:5461-10922 (5461 slots) master
   0 additional replica(s)
M: 54d5eb9415289228b05b28fe585bd5ab6e9bc0bc node3:7000
   slots:0-5460 (5461 slots) master
   0 additional replica(s)
M: b6dd7c3d5af4ffc37e70d7eb538a3f9b0ae5a84f node2:7000
   slots:10923-16383 (5461 slots) master
   0 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

6、從節點的添加需要在數據遷移之后

數據遷移

根據實際情況，參考：https://www.18188.org/articles/2016/04/23/1461374145366.html 進行數據遷移。

1、將所有的slot都轉移到一臺主實例

[root@node1 bin]# ./redis-trib.rb reshard --from b6dd7c3d5af4ffc37e70d7eb538a3f9b0ae5a84f   --to 3b485951a72e16133464585dc2920c10ce75967b  --slots 5462 --yes node1:7000

[root@node1 bin]# ./redis-trib.rb reshard --from 54d5eb9415289228b05b28fe585bd5ab6e9bc0bc   --to 3b485951a72e16133464585dc2920c10ce75967b  --slots 5462 --yes node1:7000

2、將原有的阿里云redis備份rdb文件下載放到對應目錄

cp /app/hins2533955_data_20180105195042.rdb /app/redis/data/7000/dump.rdb

3、重啟將備份文件讀入,重啟完成連接進入使用dbsize查看數據是否導入

/app/redis/bin/redis-cli -h node1 -p 7000 shutdown
/app/redis/bin/redis-server /app/redis/conf/7000.conf

4、重新分配slot

[root@node1 bin]# ./redis-trib.rb reshard --from 3b485951a72e16133464585dc2920c10ce75967b   --to b6dd7c3d5af4ffc37e70d7eb538a3f9b0ae5a84f  --slots 5462 --yes node2:7000

[root@node1 bin]# ./redis-trib.rb reshard --from 3b485951a72e16133464585dc2920c10ce75967b   --to 54d5eb9415289228b05b28fe585bd5ab6e9bc0bc  --slots 5462 --yes node3:7000

5、增加從節點,各節點對應關系：

1.node1:7000(主)---》node4:7000(從)
2.node2:7000(主)---》node5:7000(從)
3.node3:7000(主)---》 node6:7000(從)

[root@node1 bin]# ./redis-trib.rb check node1.7000
Invalid IP or Port (given as node1.7000) - use IP:Port format
[root@node1 bin]# ./redis-trib.rb check node1:7000
>>> Performing Cluster Check (using node node1:7000)
M: 3b485951a72e16133464585dc2920c10ce75967b node1:7000
   slots:10924-16383 (5460 slots) master
   0 additional replica(s)
M: b6dd7c3d5af4ffc37e70d7eb538a3f9b0ae5a84f node2:7000
   slots:0-5461 (5462 slots) master
   0 additional replica(s)
M: 54d5eb9415289228b05b28fe585bd5ab6e9bc0bc node3:7000
   slots:5462-10923 (5462 slots) master
   0 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@node1 bin]# ./redis-trib.rb add-node --slave --master-id 3b485951a72e16133464585dc2920c10ce75967b  node4:7000 node1:7000
>>> Adding node node4:7000 to cluster node1:7000
>>> Performing Cluster Check (using node node1:7000)
M: 3b485951a72e16133464585dc2920c10ce75967b node1:7000
   slots:10924-16383 (5460 slots) master
   0 additional replica(s)
M: b6dd7c3d5af4ffc37e70d7eb538a3f9b0ae5a84f node2:7000
   slots:0-5461 (5462 slots) master
   0 additional replica(s)
M: 54d5eb9415289228b05b28fe585bd5ab6e9bc0bc node3:7000
   slots:5462-10923 (5462 slots) master
   0 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node node4:7000 to make it join the cluster.
Waiting for the cluster to join...
>>> Configure node as replica of node1:7000.
[OK] New node added correctly.
[root@node1 bin]# ./redis-trib.rb add-node --slave --master-id b6dd7c3d5af4ffc37e70d7eb538a3f9b0ae5a84f  node5:7000 node2:7000
>>> Adding node node5:7000 to cluster node2:7000
>>> Performing Cluster Check (using node node2:7000)
M: b6dd7c3d5af4ffc37e70d7eb538a3f9b0ae5a84f node2:7000
   slots:0-5461 (5462 slots) master
   0 additional replica(s)
S: 8ff59145b1d3d7fd57923f3cb9c3444f7236d7b1 node4:7000
   slots: (0 slots) slave
   replicates 3b485951a72e16133464585dc2920c10ce75967b
M: 54d5eb9415289228b05b28fe585bd5ab6e9bc0bc node3:7000
   slots:5462-10923 (5462 slots) master
   0 additional replica(s)
M: 3b485951a72e16133464585dc2920c10ce75967b node1:7000
   slots:10924-16383 (5460 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node node5:7000 to make it join the cluster.
Waiting for the cluster to join.
>>> Configure node as replica of node2:7000.
[OK] New node added correctly.
[root@node1 bin]# ./redis-trib.rb add-node --slave --master-id 54d5eb9415289228b05b28fe585bd5ab6e9bc0bc  node6:7000 node3:7000
>>> Adding node node6:7000 to cluster node3:7000
>>> Performing Cluster Check (using node node3:7000)
M: 54d5eb9415289228b05b28fe585bd5ab6e9bc0bc node3:7000
   slots:5462-10923 (5462 slots) master
   0 additional replica(s)
S: c09449b9dce5af4ac485ccedb664188229d75430 node5:7000
   slots: (0 slots) slave
   replicates b6dd7c3d5af4ffc37e70d7eb538a3f9b0ae5a84f
S: 8ff59145b1d3d7fd57923f3cb9c3444f7236d7b1 node4:7000
   slots: (0 slots) slave
   replicates 3b485951a72e16133464585dc2920c10ce75967b
M: b6dd7c3d5af4ffc37e70d7eb538a3f9b0ae5a84f node2:7000
   slots:0-5461 (5462 slots) master
   1 additional replica(s)
M: 3b485951a72e16133464585dc2920c10ce75967b node1:7000
   slots:10924-16383 (5460 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node node6:7000 to make it join the cluster.
Waiting for the cluster to join.
>>> Configure node as replica of node3:7000.
[OK] New node added correctly.

[root@node1 bin]# ./redis-trib.rb check node1:7000
>>> Performing Cluster Check (using node node1:7000)
M: 3b485951a72e16133464585dc2920c10ce75967b node1:7000
   slots:10924-16383 (5460 slots) master
   1 additional replica(s)
S: 8ff59145b1d3d7fd57923f3cb9c3444f7236d7b1 node4:7000
   slots: (0 slots) slave
   replicates 3b485951a72e16133464585dc2920c10ce75967b
S: 7501954b635724d66285afbb95841f4b49df5645 node6:7000
   slots: (0 slots) slave
   replicates 54d5eb9415289228b05b28fe585bd5ab6e9bc0bc
S: c09449b9dce5af4ac485ccedb664188229d75430 node5:7000
   slots: (0 slots) slave
   replicates b6dd7c3d5af4ffc37e70d7eb538a3f9b0ae5a84f
M: 54d5eb9415289228b05b28fe585bd5ab6e9bc0bc node3:7000
   slots:5462-10923 (5462 slots) master
   1 additional replica(s)
M: b6dd7c3d5af4ffc37e70d7eb538a3f9b0ae5a84f node2:7000
   slots:0-5461 (5462 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.