烹茶细论

发表于 | 分类于 | | 阅读次数

监控系统用的是zabbix，机器一多，告警邮件就多，每次看邮件都很烦躁，于是有了想要收敛下告警邮件的想法。

根据峰云大神的博客收敛的思路，自己动手实现了这个收敛平台。

1、重复性的邮件统一发

2、根据报警内容再做相关报警检查，减少误报可能

3、web展示，未处理未恢复每天定时通知

阅读全文 »
烹茶细论

发表于 | 分类于 | | 阅读次数

内部的运维平台，安全测试之后需要添加验证码功能防止爆破。然后开始网上找教程，照着抄，备忘

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
from PIL import Image,  ImageDraw, ImageFont
import random, hashlib, datetime, cStringIO
def get_check_code_image(request,image='static/resources/images/code.png'):
    try:
        im = Image.open(image)
        draw = ImageDraw.Draw(im)
        mp = hashlib.md5()
        # mp = md5.new()
        mp_src = mp.update(str(datetime.datetime.now()))
        mp_src = mp.hexdigest()
        rand_str = mp_src[0:4]
        draw.text((10,10), rand_str[0], font=ImageFont.truetype("static/resources/font/wqy.ttc", random.randrange(25,50)))
        draw.text((48,10), rand_str[1], font=ImageFont.truetype("static/resources/font/wqy.ttc", random.randrange(25,50)))
        draw.text((85,10), rand_str[2], font=ImageFont.truetype("static/resources/font/wqy.ttc", random.randrange(25,50)))
        draw.text((120,10), rand_str[3], font=ImageFont.truetype("static/resources/font/wqy.ttc", random.randrange(25,50)))
        del draw
        request.session['checkcode'] = rand_str
        buf = cStringIO.StringIO()
        im.save(buf, 'png')
    except Exception,e:
        logging.error('get_check_code_image:'+str(e))
    return HttpResponse(buf.getvalue(),'/login/png')
阅读全文 »
烹茶细论

发表于 | 分类于 | | 阅读次数

#一、bond0作用及要求

bond0:负载均衡，提高网络的高可用性，将两块网卡绑定成一块虚拟网卡对外服务。这样就是有一个线路出现问题，也不会导致服务器连接中断。

系统环境：Centos6.4 64位
网卡1：em1和em2
网卡2：p1p1和p1p2
目的：将em1和p1p1绑定在一起成为bond0

阅读全文 »
烹茶细论

发表于 | 分类于 | | 阅读次数

最近几天，MFS服务器到每小时的01-05分钟都会有失去响应，资源访问失败的情况。查看log

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
Oct 10 13:01:49 VM_200_107 mfsmaster[31003]: connection with client(ip:172.16.200.81) has been closed by peer
Oct 10 13:01:49 VM_200_107 mfsmaster[31003]: connection with client(ip:172.16.115.33) has been closed by peer
##表示客户端和master的连接中断
Oct  4 08:01:28 VM_200_107 mfsmaster[2843]: connection with CS(172.16.200.102) has been closed by peer
Oct  4 08:01:28 VM_200_107 mfsmaster[2843]: connection with CS(172.16.200.103) has been closed by peer
Oct  4 08:01:28 VM_200_107 mfsmaster[2843]: connection with CS(172.16.200.104) has been closed by peer
##表示ChunkServer和Master的连接中断
Oct  4 08:01:28 VM_200_107 mfsmaster[2843]: chunkserver disconnected - ip: 172.16.200.102, port: 9422, usedspace: 463146741760 (431.34 GiB), totalspace: 2054567444480 (1913.47 GiB)
Oct  4 08:01:30 VM_200_107 mfsmaster[2843]: chunkserver disconnected - ip: 172.16.200.103, port: 9422, usedspace: 459528495104 (427.97 GiB), totalspace: 2054567444480 (1913.47 GiB)
Oct  4 08:01:31 VM_200_107 mfsmaster[2843]: chunkserver disconnected - ip: 172.16.200.104, port: 9422, usedspace: 461537153024 (429.84 GiB), totalspace: 2054567444480 (1913.47 GiB)
##ChunkServer 中断连接
Oct 10 13:01:52 VM_200_107 mfsmaster[31003]: connection with ML(127.0.0.1) has been closed by peer
##表示Metalogger和Master的连接中断
Oct 10 13:01:52 VM_200_107 mfsmaster[31003]: chunkserver register begin (packet version: 5) - ip: 172.16.200.102, port: 9422
Oct 10 13:01:52 VM_200_107 mfsmetalogger[31700]: connection was reset by Master
Oct 10 13:01:53 VM_200_107 mfsmaster[31003]: chunkserver register begin (packet version: 5) - ip: 172.16.200.103, port: 9422
Oct 10 13:01:54 VM_200_107 mfsmaster[31003]: chunkserver register end (packet version: 5) - ip: 172.16.200.102, port: 9422, usedspace: 490665570304 (456.97 GiB), totalspace: 2054567444480 
Oct 10 13:01:54 VM_200_107 mfsmaster[31003]: chunkserver register end (packet version: 5) - ip: 172.16.200.103, port: 9422, usedspace: 486941249536 (453.50 GiB), totalspace: 2054567444480 
##chunkserver重新连接
阅读全文 »
烹茶细论

发表于 | 分类于 | | 阅读次数

1、查看虚拟机的ID
在宿主机里面执行 virsh list 可以查看所有虚拟机信息

2、进入/data/kvm_img/虚拟机UUID 目录下

qemu-img create -f raw disk_add.img 100G （这条命令将在当前目录创建一个100G的文件）

3、使用 virsh 命令使虚拟机挂载刚才创建的文件，相当于添加硬盘

virsh attach-disk a3997428-bf53-7c2b-86ad-d93fc0f48f64 /data/kvm_img/a3997428-bf53-7c2b-86ad-d93fc0f48f64/disk_add.img vdc –cache none

阅读全文 »
烹茶细论

发表于 | 分类于 | | 阅读次数

中午突然发现收到一条zabix的报警邮件，提示cpu iowait很高，于是进去发现nginx的进程都是D的状态，跟踪进程发现进程卡在读mfs的文件，于是发现mfs提示两百多个文件missing

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
currently unavailable chunk 0000000000464B18 (inode: 55 ; index: 0)
* currently unavailable file 55: reader/reader/articlelist/index/581_3.json
currently unavailable chunk 0000000000464B19 (inode: 56 ; index: 0)
* currently unavailable file 56: reader/reader/articlelist/index/581_5.json
currently unavailable chunk 0000000000464B1A (inode: 57 ; index: 0)
* currently unavailable file 57: reader/reader/articlelist/index/581.json
currently unavailable chunk 0000000000464A77 (inode: 4482049 ; index: 0)
* currently unavailable file 4482049: reader/reader/articlelist/20150816/700_3.json
currently unavailable chunk 0000000000464A78 (inode: 4482065 ; index: 0)
* currently unavailable file 4482065: reader/reader/articlelist/20150816/700_5.json
currently unavailable chunk 0000000000464A7A (inode: 4482082 ; index: 0)
* currently unavailable file 4482082: reader/reader/articlelist/20150816/700.json
currently unavailable chunk 0000000000464A7F (inode: 4482086 ; index: 0)
* currently unavailable file 4482086: reader/reader/articlelist/20150816/700_3_new.json
``` 
<!-- more -->
排查原因大概是昨天停电导致的，于是开始修复这些文件
使用客户端的mfsfilerepair对文件进行修复
1、将日志文件存放于a文件
2、提取出所有missing的文件路径到b
```bash
cat a | awk '{for(i=0;i&lt;NF;i++)if(NR%2==0){print}}' |  awk -F ':' '{print $2}'

3、然后写一个shell脚本执行，脚本如下，相当简单

#!/bin/bash
for line in $(cat b)
do
    mfsfilerepair /mnt$line   #mfs我挂载在mnt下
done

这样就批量修复好了

烹茶细论

发表于 | 分类于 | | 阅读次数

之前用的是cacti监控，最近尝试搭建了zabbix，主机添加、图像展示、自动发现主机的功能都已经设置好。于是开始设置你邮件报警功能

阅读全文 »
烹茶细论

发表于 | 分类于 | | 阅读次数

salt-ssh是依赖ssh进行通讯的，最大的优点就是可以不通过salt-minion客户端去执行远程的批量操作。所以在没有安装salt-minion之前使用salt-ssh是一个很不错的选择。
salt-ssh还能使用salt的标准模块和命令

阅读全文 »