大量机器掉电后修复osd

2023-01-17

字数统计: 1.9k字 | 阅读时长≈ 9分

机器断电导致集群无法正常，手工处理

写在前面

松鼠哥的ceph专业课程上线啦！
面向新手同学，从0实战，全面入门ceph安装部署与运维，有需要的同学赶紧扫码订购吧：

咳咳，开始之前，照例还是要打一波招聘广告，我们这边是天翼云科技的媒体存储组，目前我们仍在招聘存储研发、存储运维及相关组件开发的人才，有意向换工作的老铁可以试试，简历发一发，我们这边还是很不错的，千万不要错过

开始

早上接到消息，说我们有个生产集群所在的机房发生断电，十几台机器断电后重新开起来了，然后发现有3台机器的osd没起来，对于我们12+4的EC配比来说，3台机器挂掉应该不至于丢数据，不过也不能掉以轻心

先看下集群的情况

# ceph -s
data:
    pools:   8 pools, 22536 pgs
    objects: 275.2 M objects, 865 TiB
    usage:   1.4 PiB used, 14 PiB / 15 PiB avail
    pgs:     7.588% pgs not active
             104965191/3302772936 objects degraded (3.178%)
             15591 active+clean
             5198  active+undersized+degraded
             1710  down
             31    active+clean+scrubbing+deep+repair
             6     active+clean+scrubbing+deep
 
  io:
    client:   10 MiB/s rd, 15 KiB/s wr, 1.87 kop/s rd, 13 op/s wr
# ceph health detail|more
HEALTH_WARN noout flag(s) set; 52 osds down; 1 host (30 osds) down; Reduced data availability: 1710 pgs inactive, 1710 pgs down; Degraded da
ta redundancy: 104965191/3302772936 objects degraded (3.178%), 5198 pgs degraded, 5198 pgs undersized; application not enabled on 1 pool(s);
 237 slow ops, oldest one blocked for 139 sec, daemons [osd.666,osd.668,osd.669,osd.670,osd.671,osd.674,osd.676,osd.679,osd.685,osd.686]... 
have slow ops.
OSDMAP_FLAGS noout flag(s) set
OSD_DOWN 52 osds down
    osd.570 (root=hdd-root-2,host=node-20) is down
    ....
    osd.666 (root=hdd-root-2,host=node-23) is down
    ....
    osd.988 (root=hdd-root-2,host=node-33) is down

分布到3个host的52个osd是down状态，有1710个pg是down状态的，其中node-20是整台起不来，另外node-23和node-33是分别挂掉十几个osd，down状态的pg估计是卡住了，因为ec不超过m个故障域出现osd挂掉，应该不会down才对

因为故障域是host，只挂掉3台还行，数据很安全，最简单粗暴的方式就是重建这些起不来的osd，但是，重建这么多osd必然会产生非常多的数据回填，能不能尽量减小重建的osd的数量呢？

抢救osd

先手工重启一下起不来的osd，发现不行，报错

  -394> 2022-08-11 11:48:56.343 7fceda304b80 -1 bluestore(/var/lib/ceph/osd/ceph-988) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x52000, got 0x8246d7a2, expected 0xd3488b37, device location [0x50b01847000~1000], logical extent 0x52000~1000, object #-1:1ada1050:::osdmap.3757:0#
  -394> 2022-08-11 11:48:56.343 7fceda304b80 -1 osd.988 0 failed to load OSD map for epoch 3757, got 0 bytes
  -394> 2022-08-11 11:48:56.343 7fceda304b80 -1 osd.988 3760 load_pgs: have pgid 12.facs9 at epoch 3757, but missing map.  Crashing.
  -394> 2022-08-11 11:48:56.347 7fceda304b80 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/13.2.10/rpm/el7/BUILD/ceph-13.2.10/src/osd/OSD.cc: In function 'void OSD::load_pgs()' thread 7fceda304b80 time 2022-08-11 11:48:56.344513
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/13.2.10/rpm/el7/BUILD/ceph-13.2.10/src/osd/OSD.cc: 4112: FAILED assert(0 == "Missing map in load_pgs")

看了下，是osdmap出问题了，有问题的版本是3757，查了一圈，说可以通过注入osdmap的方式解决

1	ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-988/ --type bluestore --op set-osdmap --file osdmap.3757

注入后再重启，看看怎么样

  -834> 2022-08-11 15:12:44.559 7f6e4a69bb80 -1 bluestore(/var/lib/ceph/osd/ceph-988) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0xe95feac5, expected 0xbf010d3f, device location [0x50b01849000~1000], logical extent 0x0~1000, object #-1:171a1050:::osdmap.3758:0#
  -834> 2022-08-11 15:12:44.559 7f6e4a69bb80 -1 osd.988 0 failed to load OSD map for epoch 3758, got 0 bytes
  -834> 2022-08-11 15:12:44.559 7f6e4a69bb80 -1 osd.988 3760 load_pgs: have pgid 12.10b5s6 at epoch 3758, but missing map.  Crashing.
  -834> 2022-08-11 15:12:44.563 7f6e4a69bb80 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/13.2.10/rpm/el7/BUILD/ceph-13.2.10/src/osd/OSD.cc: In function 'void OSD::load_pgs()' thread 7f6e4a69bb80 time 2022-08-11 15:12:44.559875
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/13.2.10/rpm/el7/BUILD/ceph-13.2.10/src/osd/OSD.cc: 4112: FAILED assert(0 == "Missing map in load_pgs")

这回变成3758有问题了，说明注入是有效的，那么怎么搞？看了下当前最新的osdmap版本

[root@node-33 temp]# ceph osd dump|more
epoch 6496
fsid xxxxxx-8ceb-4da7-822d-xxxxx
...

高很多，嗯，用脚本搞起来

#!/bin/bash

for x in $( seq 3757 3761 )
do
    ceph osd getmap $x -o osdmap.$x
    ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-$1/ --type bluestore --op set-osdmap --file osdmap.$x
done
[root@node-33 temp]# sh -x ./test_osdmap.sh 988
++ seq 3756 3761
+ for x in '$( seq 3756 3760 )'
+ ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-988/ --type bluestore --op set-osdmap --file osdmap.3756
+ for x in '$( seq 3756 3760 )'
+ ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-988/ --type bluestore --op set-osdmap --file osdmap.3757
+ for x in '$( seq 3756 3760 )'
+ ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-988/ --type bluestore --op set-osdmap --file osdmap.3758
+ for x in '$( seq 3756 3760 )'
+ ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-988/ --type bluestore --op set-osdmap --file osdmap.3759
+ for x in '$( seq 3756 3760 )'
+ ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-988/ --type bluestore --op set-osdmap --file osdmap.3760
+ for x in '$( seq 3756 3760 )'
+ ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-988/ --type bluestore --op set-osdmap --file osdmap.3761
osdmap (#-1:15fa1050:::osdmap.3761:0#) does not exist.

来到3761这个版本时，报错了，说明在这个版本的osdmap中不存在这个obj的信息，我们就此打住，可以不往上继续增加osdmap的版本，尝试启动osd

重启后，osd正常起来了！

如法炮制！仍然有部份osd起不来，报错是superblock坏掉了

  -347> 2022-08-11 10:53:38.784 7f452d13cb80 -1 bluestore(/var/lib/ceph/osd/ceph-689) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0xd4c4ec7d, expected 0xbf010d3f, device location [0x4e0d551b000~1000], logical extent 0x0~1000, object #-1:171a1050:::osdmap.3758:0#
  -347> 2022-08-11 10:53:38.784 7f452d13cb80 -1 osd.689 0 failed to load OSD map for epoch 3758, got 0 bytes
  -347> 2022-08-11 10:53:38.788 7f452d13cb80 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/13.2.10/rpm/el7/BUILD/ceph-13.2.10/src/osd/OSD.h: In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7f452d13cb80 time 2022-08-11 10:53:38.785190
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/13.2.10/rpm/el7/BUILD/ceph-13.2.10/src/osd/OSD.h: 823: FAILED assert(ret)

看来单纯让它的osdmap版本提高不能解决问题，所幸的是我们的故障域还算安全，重建这个osd算了！

对所有注入osdmap后起不来的osd进行重建后，集群所有pg都正常了，火力全开做恢复

data:
  pools:   8 pools, 22536 pgs
  objects: 269.5 M objects, 848 TiB
  usage:   1.3 PiB used, 14 PiB / 15 PiB avail
  pgs:     123847796/3233427210 objects degraded (3.830%)
           2621/269462262 objects unfound (0.001%)
           16328 active+clean
           4026  active+undersized+degraded
           1038  active+undersized+degraded+remapped+backfill_wait
           852   active+undersized+degraded+remapped+backfilling
           257   active+recovery_unfound+undersized+degraded+remapped
           31    active+clean+scrubbing+deep+repair
           4     active+clean+scrubbing+deep
 
io:
  client:   13 MiB/s rd, 193 KiB/s wr, 2.69 kop/s rd, 322 op/s wr
  recovery: 24 GiB/s, 7.34 kobjects/s

总结

断电导致十几台机器挂掉，这个跟机器放置的方式有很大关系。我们设置故障域为host，不就是为了避免这种情况吗？当一个机柜掉电，也不至于太多host同时挂掉，所以，是不是机房人员上机器的时候，贪方便没按要求放置机器=.=

本文作者： 奋斗的松鼠
本文链接： http://www.strugglesquirrel.com/2023/01/17/大量机器掉电后修复osd/
版权声明： 本博客所有文章除特别声明外，创作版权均为作者个人所有，未经允许禁止转载！