Kafka 版本: 0.10.2.1
准备对 __consumer_offsets topic 副本迁移。
生成迁移计划
`cat<<EOF > topics-to-move.json`
`{`
` "version": 1,`
` "topics": [`
` {"topic": "__consumer_offsets"}`
` ]`
`}`
`EOF`
`ZK="172.16.255.20/kafka"`
`bin/kafka-reassign-partitions.sh \`
`--zookeeper ${ZK} \`
`--topics-to-move-json-file topics-to-move.json \`
`--broker-list "0,1,2,3,4,5,6,7" \`
`--generate`
输出如下
Current partition replica assignment
`{"version":1,"partitions":[`
`{"topic":"__consumer_offsets","partition":0,"replicas":[3,6,8]},`
`{"topic":"__consumer_offsets","partition":1,"replicas":[4,8,9]},`
`{"topic":"__consumer_offsets","partition":2,"replicas":[5,9,0]},`
`{"topic":"__consumer_offsets","partition":3,"replicas":[6,0,1]},`
`{"topic":"__consumer_offsets","partition":4,"replicas":[8,1,2]},`
`{"topic":"__consumer_offsets","partition":5,"replicas":[9,2,3]},`
`{"topic":"__consumer_offsets","partition":6,"replicas":[0,3,4]},`
`{"topic":"__consumer_offsets","partition":7,"replicas":[1,4,5]},`
`{"topic":"__consumer_offsets","partition":8,"replicas":[2,5,6]},`
`{"topic":"__consumer_offsets","partition":9,"replicas":[3,8,9]},`
`{"topic":"__consumer_offsets","partition":10,"replicas":[4,9,0]},`
`{"topic":"__consumer_offsets","partition":11,"replicas":[5,0,1]},`
`{"topic":"__consumer_offsets","partition":12,"replicas":[6,1,2]},`
`{"topic":"__consumer_offsets","partition":13,"replicas":[8,2,3]},`
`{"topic":"__consumer_offsets","partition":14,"replicas":[9,3,4]},`
`{"topic":"__consumer_offsets","partition":15,"replicas":[0,4,5]},`
`{"topic":"__consumer_offsets","partition":16,"replicas":[1,5,6]},`
`{"topic":"__consumer_offsets","partition":17,"replicas":[2,6,8]},`
`{"topic":"__consumer_offsets","partition":18,"replicas":[3,9,0]},`
`{"topic":"__consumer_offsets","partition":19,"replicas":[4,0,1]},`
`{"topic":"__consumer_offsets","partition":20,"replicas":[5,1,2]},`
`{"topic":"__consumer_offsets","partition":21,"replicas":[6,2,3]},`
`{"topic":"__consumer_offsets","partition":22,"replicas":[8,3,4]},`
`{"topic":"__consumer_offsets","partition":23,"replicas":[9,4,5]},`
`{"topic":"__consumer_offsets","partition":24,"replicas":[0,5,6]},`
`{"topic":"__consumer_offsets","partition":25,"replicas":[1,6,8]},`
`{"topic":"__consumer_offsets","partition":26,"replicas":[2,8,9]},`
`{"topic":"__consumer_offsets","partition":27,"replicas":[3,0,1]},`
`{"topic":"__consumer_offsets","partition":28,"replicas":[4,1,2]},`
`{"topic":"__consumer_offsets","partition":29,"replicas":[5,2,3]},`
`{"topic":"__consumer_offsets","partition":30,"replicas":[6,3,4]},`
`{"topic":"__consumer_offsets","partition":31,"replicas":[8,4,5]},`
`{"topic":"__consumer_offsets","partition":32,"replicas":[9,5,6]},`
`{"topic":"__consumer_offsets","partition":33,"replicas":[0,6,8]},`
`{"topic":"__consumer_offsets","partition":34,"replicas":[1,8,9]},`
`{"topic":"__consumer_offsets","partition":35,"replicas":[2,9,0]},`
`{"topic":"__consumer_offsets","partition":36,"replicas":[3,1,2]},`
`{"topic":"__consumer_offsets","partition":37,"replicas":[4,2,3]},`
`{"topic":"__consumer_offsets","partition":38,"replicas":[5,3,4]},`
`{"topic":"__consumer_offsets","partition":39,"replicas":[6,4,5]},`
`{"topic":"__consumer_offsets","partition":40,"replicas":[8,5,6]},`
`{"topic":"__consumer_offsets","partition":41,"replicas":[9,6,8]},`
`{"topic":"__consumer_offsets","partition":42,"replicas":[0,8,9]},`
`{"topic":"__consumer_offsets","partition":43,"replicas":[1,9,0]},`
`{"topic":"__consumer_offsets","partition":44,"replicas":[2,0,1]},`
`{"topic":"__consumer_offsets","partition":45,"replicas":[3,2,4]},`
`{"topic":"__consumer_offsets","partition":46,"replicas":[4,3,5]},`
`{"topic":"__consumer_offsets","partition":47,"replicas":[5,4,6]},`
`{"topic":"__consumer_offsets","partition":48,"replicas":[6,5,8]},`
`{"topic":"__consumer_offsets","partition":49,"replicas":[8,6,9]}`
`]}`
Proposed partition reassignment configuration
`{"version":1,"partitions":[`
`{"topic":"__consumer_offsets","partition":0,"replicas":[3,2,4]},`
`{"topic":"__consumer_offsets","partition":1,"replicas":[4,3,5]},`
`{"topic":"__consumer_offsets","partition":2,"replicas":[5,4,6]},`
`{"topic":"__consumer_offsets","partition":3,"replicas":[6,5,7]},`
`{"topic":"__consumer_offsets","partition":4,"replicas":[7,6,0]},`
`{"topic":"__consumer_offsets","partition":5,"replicas":[0,7,1]},`
`{"topic":"__consumer_offsets","partition":6,"replicas":[1,0,2]},`
`{"topic":"__consumer_offsets","partition":7,"replicas":[2,1,3]},`
`{"topic":"__consumer_offsets","partition":8,"replicas":[3,4,5]},`
`{"topic":"__consumer_offsets","partition":9,"replicas":[4,5,6]},`
`{"topic":"__consumer_offsets","partition":10,"replicas":[5,6,7]},`
`{"topic":"__consumer_offsets","partition":11,"replicas":[6,7,0]},`
`{"topic":"__consumer_offsets","partition":12,"replicas":[7,0,1]},`
`{"topic":"__consumer_offsets","partition":13,"replicas":[0,1,2]},`
`{"topic":"__consumer_offsets","partition":14,"replicas":[1,2,3]},`
`{"topic":"__consumer_offsets","partition":15,"replicas":[2,3,4]},`
`{"topic":"__consumer_offsets","partition":16,"replicas":[3,5,6]},`
`{"topic":"__consumer_offsets","partition":17,"replicas":[4,6,7]},`
`{"topic":"__consumer_offsets","partition":18,"replicas":[5,7,0]},`
`{"topic":"__consumer_offsets","partition":19,"replicas":[6,0,1]},`
`{"topic":"__consumer_offsets","partition":20,"replicas":[7,1,2]},`
`{"topic":"__consumer_offsets","partition":21,"replicas":[0,2,3]},`
`{"topic":"__consumer_offsets","partition":22,"replicas":[1,3,4]},`
`{"topic":"__consumer_offsets","partition":23,"replicas":[2,4,5]},`
`{"topic":"__consumer_offsets","partition":24,"replicas":[3,6,7]},`
`{"topic":"__consumer_offsets","partition":25,"replicas":[4,7,0]},`
`{"topic":"__consumer_offsets","partition":26,"replicas":[5,0,1]},`
`{"topic":"__consumer_offsets","partition":27,"replicas":[6,1,2]},`
`{"topic":"__consumer_offsets","partition":28,"replicas":[7,2,3]},`
`{"topic":"__consumer_offsets","partition":29,"replicas":[0,3,4]},`
`{"topic":"__consumer_offsets","partition":30,"replicas":[1,4,5]},`
`{"topic":"__consumer_offsets","partition":31,"replicas":[2,5,6]},`
`{"topic":"__consumer_offsets","partition":32,"replicas":[3,7,0]},`
`{"topic":"__consumer_offsets","partition":33,"replicas":[4,0,1]},`
`{"topic":"__consumer_offsets","partition":34,"replicas":[5,1,2]},`
`{"topic":"__consumer_offsets","partition":35,"replicas":[6,2,3]},`
`{"topic":"__consumer_offsets","partition":36,"replicas":[7,3,4]},`
`{"topic":"__consumer_offsets","partition":37,"replicas":[0,4,5]},`
`{"topic":"__consumer_offsets","partition":38,"replicas":[1,5,6]},`
`{"topic":"__consumer_offsets","partition":39,"replicas":[2,6,7]},`
`{"topic":"__consumer_offsets","partition":40,"replicas":[3,0,1]},`
`{"topic":"__consumer_offsets","partition":41,"replicas":[4,1,2]},`
`{"topic":"__consumer_offsets","partition":42,"replicas":[5,2,3]},`
`{"topic":"__consumer_offsets","partition":43,"replicas":[6,3,4]},`
`{"topic":"__consumer_offsets","partition":44,"replicas":[7,4,5]},`
`{"topic":"__consumer_offsets","partition":45,"replicas":[0,5,6]},`
`{"topic":"__consumer_offsets","partition":46,"replicas":[1,6,7]},`
`{"topic":"__consumer_offsets","partition":47,"replicas":[2,7,0]},`
`{"topic":"__consumer_offsets","partition":48,"replicas":[3,1,2]},`
`{"topic":"__consumer_offsets","partition":49,"replicas":[4,2,3]}`
`]}`
将新的分区配置保存到文件
保存到 reassign-partition.json
`cat<<EOF > reassign-partition.json`
`{"version":1,"partitions":[`
`{"topic":"__consumer_offsets","partition":0,"replicas":[3,2,4]},`
`{"topic":"__consumer_offsets","partition":1,"replicas":[4,3,5]},`
`{"topic":"__consumer_offsets","partition":2,"replicas":[5,4,6]},`
`{"topic":"__consumer_offsets","partition":3,"replicas":[6,5,7]},`
`{"topic":"__consumer_offsets","partition":4,"replicas":[7,6,0]},`
`{"topic":"__consumer_offsets","partition":5,"replicas":[0,7,1]},`
`{"topic":"__consumer_offsets","partition":6,"replicas":[1,0,2]},`
`{"topic":"__consumer_offsets","partition":7,"replicas":[2,1,3]},`
`{"topic":"__consumer_offsets","partition":8,"replicas":[3,4,5]},`
`{"topic":"__consumer_offsets","partition":9,"replicas":[4,5,6]},`
`{"topic":"__consumer_offsets","partition":10,"replicas":[5,6,7]},`
`{"topic":"__consumer_offsets","partition":11,"replicas":[6,7,0]},`
`{"topic":"__consumer_offsets","partition":12,"replicas":[7,0,1]},`
`{"topic":"__consumer_offsets","partition":13,"replicas":[0,1,2]},`
`{"topic":"__consumer_offsets","partition":14,"replicas":[1,2,3]},`
`{"topic":"__consumer_offsets","partition":15,"replicas":[2,3,4]},`
`{"topic":"__consumer_offsets","partition":16,"replicas":[3,5,6]},`
`{"topic":"__consumer_offsets","partition":17,"replicas":[4,6,7]},`
`{"topic":"__consumer_offsets","partition":18,"replicas":[5,7,0]},`
`{"topic":"__consumer_offsets","partition":19,"replicas":[6,0,1]},`
`{"topic":"__consumer_offsets","partition":20,"replicas":[7,1,2]},`
`{"topic":"__consumer_offsets","partition":21,"replicas":[0,2,3]},`
`{"topic":"__consumer_offsets","partition":22,"replicas":[1,3,4]},`
`{"topic":"__consumer_offsets","partition":23,"replicas":[2,4,5]},`
`{"topic":"__consumer_offsets","partition":24,"replicas":[3,6,7]},`
`{"topic":"__consumer_offsets","partition":25,"replicas":[4,7,0]},`
`{"topic":"__consumer_offsets","partition":26,"replicas":[5,0,1]},`
`{"topic":"__consumer_offsets","partition":27,"replicas":[6,1,2]},`
`{"topic":"__consumer_offsets","partition":28,"replicas":[7,2,3]},`
`{"topic":"__consumer_offsets","partition":29,"replicas":[0,3,4]},`
`{"topic":"__consumer_offsets","partition":30,"replicas":[1,4,5]},`
`{"topic":"__consumer_offsets","partition":31,"replicas":[2,5,6]},`
`{"topic":"__consumer_offsets","partition":32,"replicas":[3,7,0]},`
`{"topic":"__consumer_offsets","partition":33,"replicas":[4,0,1]},`
`{"topic":"__consumer_offsets","partition":34,"replicas":[5,1,2]},`
`{"topic":"__consumer_offsets","partition":35,"replicas":[6,2,3]},`
`{"topic":"__consumer_offsets","partition":36,"replicas":[7,3,4]},`
`{"topic":"__consumer_offsets","partition":37,"replicas":[0,4,5]},`
`{"topic":"__consumer_offsets","partition":38,"replicas":[1,5,6]},`
`{"topic":"__consumer_offsets","partition":39,"replicas":[2,6,7]},`
`{"topic":"__consumer_offsets","partition":40,"replicas":[3,0,1]},`
`{"topic":"__consumer_offsets","partition":41,"replicas":[4,1,2]},`
`{"topic":"__consumer_offsets","partition":42,"replicas":[5,2,3]},`
`{"topic":"__consumer_offsets","partition":43,"replicas":[6,3,4]},`
`{"topic":"__consumer_offsets","partition":44,"replicas":[7,4,5]},`
`{"topic":"__consumer_offsets","partition":45,"replicas":[0,5,6]},`
`{"topic":"__consumer_offsets","partition":46,"replicas":[1,6,7]},`
`{"topic":"__consumer_offsets","partition":47,"replicas":[2,7,0]},`
`{"topic":"__consumer_offsets","partition":48,"replicas":[3,1,2]},`
`{"topic":"__consumer_offsets","partition":49,"replicas":[4,2,3]}`
`]}`
`EOF`
执行迁移计划
记得一定要限速 --throttle
此处限速 50MB/s
`ZK="172.16.255.20:2181/kafka"`
`bin/kafka-reassign-partitions.sh \`
`--zookeeper ${ZK} \`
`--reassignment-json-file reassign-partition.json \`
`--throttle 500000000 \`
`--execute`
查看迁移进度
`ZK="172.16.255.20/kafka"`
`bin/kafka-reassign-partitions.sh \`
`--zookeeper ${ZK} \`
`--reassignment-json-file reassign-partition.json \`
`--verify`