烹茶细论

MFS 文件missing处理

发表于 | 分类于 | | 阅读次数

中午突然发现收到一条zabix的报警邮件，提示cpu iowait很高，于是进去发现nginx的进程都是D的状态，跟踪进程发现进程卡在读mfs的文件，于是发现mfs提示两百多个文件missing

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
currently unavailable chunk 0000000000464B18 (inode: 55 ; index: 0)
* currently unavailable file 55: reader/reader/articlelist/index/581_3.json
currently unavailable chunk 0000000000464B19 (inode: 56 ; index: 0)
* currently unavailable file 56: reader/reader/articlelist/index/581_5.json
currently unavailable chunk 0000000000464B1A (inode: 57 ; index: 0)
* currently unavailable file 57: reader/reader/articlelist/index/581.json
currently unavailable chunk 0000000000464A77 (inode: 4482049 ; index: 0)
* currently unavailable file 4482049: reader/reader/articlelist/20150816/700_3.json
currently unavailable chunk 0000000000464A78 (inode: 4482065 ; index: 0)
* currently unavailable file 4482065: reader/reader/articlelist/20150816/700_5.json
currently unavailable chunk 0000000000464A7A (inode: 4482082 ; index: 0)
* currently unavailable file 4482082: reader/reader/articlelist/20150816/700.json
currently unavailable chunk 0000000000464A7F (inode: 4482086 ; index: 0)
* currently unavailable file 4482086: reader/reader/articlelist/20150816/700_3_new.json
``` 
<!-- more -->
排查原因大概是昨天停电导致的，于是开始修复这些文件
使用客户端的mfsfilerepair对文件进行修复
1、将日志文件存放于a文件
2、提取出所有missing的文件路径到b
```bash
cat a | awk '{for(i=0;i&lt;NF;i++)if(NR%2==0){print}}' |  awk -F ':' '{print $2}'

3、然后写一个shell脚本执行，脚本如下，相当简单

#!/bin/bash
for line in $(cat b)
do
    mfsfilerepair /mnt$line   #mfs我挂载在mnt下
done

这样就批量修复好了