Redis異構集群數據實時遷移

背景

由于歷史原因，公司的緩存方案使用的是Codis，并且一個大部門公用一個集群，我們計劃廢棄Codis，用Redis原生的集群架構，為什么要廢棄Codis呢，主要有兩個原因：1、Codis官方已經很久沒有更新維護了，Redis官方版本已經迭代到5.x.x了，codis-server還是3.x.x，Redis的一些新特性無法支持；2、基于風險均攤、雞蛋不放一個籃子的原則，目前我們這樣的用法違背了這一原則，如果一個集群出問題，那么整個部門的全部服務都受影響。在前期和業務部門調研的過程中發現，大家用Codis不僅僅是做緩存，有些業務場景還當儲存用，比如計數器等；所以我們需要一個數據實時遷移方案，這樣業務才能無感知的從Codis遷移到Redis。

方案選型

需求

1、支持從Codis到Redis Cluster做數據遷移
2、支持從Codis到哨兵集群做數據遷移
3、支持只遷移部分key
4、支持查看遷移進度

調研

1、redis-migrate-tool
redis-migrate-tool是唯品會開源的一款Redis異構集群之間的數據實時遷移工具，不過已經有兩年沒有更新了，我個人覺得這是一款比較完善的工具，特別是數據校驗，詳細功能介紹見GitHub：
https://github.com/vipshop/redis-migrate-tool
2、RedisShake
RedisShake是阿里云基于豌豆莢開源的redis-port進行二次開發的一個支持Redis異構集群實時同步的工具，它和redis-migrate-tool相比較，我覺得它的優點在于支持前綴key的同步，支持多DB同步，而redis-migrate-tool 只能全量同步，并且如果源做了分庫，同步到目標Redis的時候都同步到了db0一個庫里面了，這對于做了分庫場景的業務是不可行的，關于RedisShake的詳細功能介紹見GitHub：
https://github.com/alibaba/RedisShake

3、redis-port
redis-port是豌豆莢當年為了讓大家方便從redis遷移到Codis開源的一個Redis數據遷移工具，現在也已經很久沒更新了，關于它的功能也用法見GitHub：
https://github.com/CodisLabs/redis-port

實踐

環境

codis---》哨兵

codis---》Redis Cluster

使用redis-migrate-tool進行數據遷移

遷移工具安裝

按官方文檔進行編譯安裝即可

編寫配置文件

遷移哨兵的配置文件

vim /chj/app/redis-migrate-tool/rmt_sentinel.conf
[source]
type: single
servers :
- 192.168.46.150:10379
- 192.168.47.150:10379
redis_auth: xxx

[target]
type: single
servers:
- 192.168.9.87:6384
redis_auth: 123456

[common]
listen: 0.0.0.0:8888

遷移redis cluster的配置文件

vim /chj/app/redis-migrate-tool/rmt_cluster.conf

[source]
type: single
servers :
- 192.168.46.150:10379
- 192.168.47.150:10379
redis_auth: xxx

[target]
type: redis cluster
servers:
- 192.168.9.87:6383
- 192.168.9.89:6382
- 192.168.9.88:6381
redis_auth: 123456

[common]
listen: 0.0.0.0:8889

啟動同步程序

cd /chj/app/redis-migrate-tool
#condis遷移數據到哨兵集群
./src/redis-migrate-tool -c rmt_sentinel.conf -o rmt.log -d 
#condis遷移數據到redis cluster
./src/redis-migrate-tool -c rmt_cluster.conf -o rmt_cluster.log -d

數據校驗

cd /chj/app/redis-migrate-tool
[root@devops-template-test redis-migrate-tool]# ./src/redis-migrate-tool -c rmt_sentinel.conf -C "redis_check 60000"
Check job is running...
[2019-06-25 11:12:09.414] rmt_check.c:848 ERROR: key checked failed: check key's value error, value is inconsistent. key(len:17, type:hash): BigData-IpParse:4

Checked keys: 60000
Inconsistent value keys: 1
Inconsistent expire keys : 0
Other check error keys: 0
Checked OK keys: 59999

Check job finished, used 16.622s
PS
1、"-C "redis_check 60000" 指定要執行數據校驗，60000指的是校驗數據的樣本數，默認是1000
2、如果有異常，需要確認執行異常key的情況

同步狀態確認

total_msgs_outqueue可以判斷是否有oplog在隊列中等待處理，如果total_msgs_outqueue>0，請繼續等待，直到total_msgs_outqueue=0才能切換

[root@devops-template-test redis-migrate-tool]# redis-cli -h 127.0.0.1 -p 8889 info
 Server
version:0.1.0
os:Linux 3.10.0-693.5.2.el7.x86_64 x86_64
multiplexing_api:epoll
gcc_version:4.8.5
process_id:10137
tcp_port:8889
uptime_in_seconds:1201
uptime_in_days:0
config_file:/chj/app/redis-migrate-tool/rmt_cluster.conf

 Clients
connected_clients:1
max_clients_limit:100
total_connections_received:1

 Memory
mem_allocator:jemalloc-0.0.0

 Group
source_nodes_count:2
target_nodes_count:4

Stats
all_rdb_received:1
all_rdb_parsed:1
all_aof_loaded:0
rdb_received_count:2
rdb_parsed_count:2
aof_loaded_count:0
total_msgs_recv:357666
total_msgs_sent:357666
total_net_input_bytes:78804395
total_net_output_bytes:1688068278
total_net_input_bytes_human:75.15M
total_net_output_bytes_human:1.57G
total_mbufs_inqueue:0
total_msgs_outqueue:0

使用RedisShake進行數據遷移

工具安裝

mkdir /chj/app/redis-shake
cd  /chj/app/redis-shake
wget https://github.com/alibaba/RedisShake/releases/download/release-v1.6.9-20190624/redis-shake.tar.gz
tar -zxvf redis-shake.tar.gz

編寫配置文件

在原來的配置文件上修改，只修改下面有注釋的項，其他保持不變

id = redis-shake
log.file = ./redis-shake.log
log.level = info
pid_path =
system_profile = 9310
http_profile = 9320
ncpu = 0
parallel = 32
source.type = cluster  #源類型選擇cluster
source.address = 192.168.46.150:10379;192.168.47.150:10379  #codis 分片master的地址
source.password_raw = xxx #codis的密碼
source.auth_type = auth
source.tls_enable = false
target.type = sentinel  #目標的類型是哨兵
#target.type = cluster #目標是redis cluster
target.address = sentinel-zhj2-redis-sentinel-dev-6384@192.168.9.87:6385;192.168.9.88:6385;192.168.9.89:6385 #目標哨兵集群的地址
#target.address = 192.168.9.87:6383;192.168.9.89:6382;192.168.9.88:6381 #目標redis cluster的地址
target.password_raw = 123456 #目標redis的密碼
target.auth_type = auth
target.db = -1
target.tls_enable = false
rdb.input = local
rdb.output = local_dump
rdb.parallel = 0
rdb.special_cloud =
fake_time =
rewrite = true
filter.db = 0 #只同步db0
filter.key =mms;vcc #只同步mms和vcc開頭的key
filter.slot =
filter.lua = false
big_key_threshold = 524288000
psync = false
metric = true
metric.print_log = false
heartbeat.url =
heartbeat.interval = 3
heartbeat.external = test external
heartbeat.network_interface =
sender.size = 104857600
sender.count = 5000
sender.delay_channel_size = 65535
keep_alive = 0
scan.key_number = 50
scan.special_cloud =
scan.key_file =
qps = 200000
replace_hash_tag = false
extra = false

啟動同步程序

/chj/app/redis-shake/start.sh /chj/app/redis-shake/redis-shake.conf sync

查看同步狀態

通過比較PullCommandTotal - BypassCommandTotal == PushCommandTotal 確定同步是否完成

curl  http://192.168.47.253:9320/metric| python -m json.tool
[
    {
        "AvgDelay": "0.43 ms",
        "BypassCmdCount": 0,
        "BypassCmdCountTotal": 0,
        "Delay": "null ms",
        "Details": null,
        "FailCmdCount": 0,
        "FailCmdCountTotal": 0,
        "FullSyncProgress": 100,
        "NetworkFlowTotal": 42006,
        "NetworkSpeed": 0,
        "ProcessingCmdCount": 0,
        "PullCmdCount": 0,
        "PullCmdCountTotal": 897,
        "PushCmdCount": 0,
        "PushCmdCountTotal": 839,
        "SenderBufCount": 0,
        "SourceAddress": "192.168.46.150:10379",
        "SourceDBOffset": 0,
        "StartTime": "2019-06-25T17:45:23Z",
        "Status": "incr",
        "SuccessCmdCount": 0,
        "SuccessCmdCountTotal": 839,
        "TargetAddress": [
            "192.168.9.87:6384"
        ],
        "TargetDBOffset": 0
    },
    {
        "AvgDelay": "0.60 ms",
        "BypassCmdCount": 1,
        "BypassCmdCountTotal": 4067,
        "Delay": "null ms",
        "Details": null,
        "FailCmdCount": 0,
        "FailCmdCountTotal": 0,
        "FullSyncProgress": 100,
        "NetworkFlowTotal": 37629,
        "NetworkSpeed": 0,
        "ProcessingCmdCount": 0,
        "PullCmdCount": 1,
        "PullCmdCountTotal": 5106,
        "PushCmdCount": 0,
        "PushCmdCountTotal": 333,
        "SenderBufCount": 0,
        "SourceAddress": "192.168.47.150:10379",
        "SourceDBOffset": 0,
        "StartTime": "2019-06-25T17:45:23Z",
        "Status": "incr",
        "SuccessCmdCount": 0,
        "SuccessCmdCountTotal": 333,
        "TargetAddress": [
            "192.168.9.87:6384"
        ],
        "TargetDBOffset": 0
    }
]