etcd容量不足报错解决
报错日志信息:“etcdserver:mvcc:database space exceded”
解决步骤:
需要用到etcdctl
工具,一般二进制部署的k8s,都会带有这个工具,如果没有,可以去github下载指定版本的etcd二进制文件即可
1.查看节点状态获取db size
ETCDCTL_API=3 etcdctl \
--cacert=/etc/kubernetes/cert/ca.pem \
--cert=/etc/kubernetes/cert/etcd.pem \
--key=/etc/kubernetes/cert/etcd-key.pem \
--endpoints=https://172.31.243.179:2379 \
endpoint status --write-out="table"
在etcd启动的时候,如果没有配置--quota-backend-bytes
的大小,默认只有2G,超过此大小将导致了apiserver无法写入etcd
pem认证文件路径根据实际环境做修改
2.获取旧版本号
ETCDCTL_API=3 etcdctl \
--cacert=/etc/kubernetes/cert/ca.pem \
--cert=/etc/kubernetes/cert/etcd.pem \
--key=/etc/kubernetes/cert/etcd-key.pem \
--endpoints=https://172.31.243.179:2379 \
endpoint status --write-out="json" | egrep -o '"revision":[0-9]*' | egrep -o '[0-9]*'
3.压缩旧版本
ETCDCTL_API=3 etcdctl \
--cacert=/etc/kubernetes/cert/ca.pem \
--cert=/etc/kubernetes/cert/etcd.pem \
--key=/etc/kubernetes/cert/etcd-key.pem \
--endpoints=https://172.31.243.179:2379 \
compact 15151255 #(由步骤2获得)
4.清理碎片
ETCDCTL_API=3 etcdctl \
--cacert=/etc/kubernetes/cert/ca.pem \
--cert=/etc/kubernetes/cert/etcd.pem \
--key=/etc/kubernetes/cert/etcd-key.pem \
--endpoints=https://172.31.243.179:2379 \
defrag
5.再次查看节点db size
ETCDCTL_API=3 etcdctl \
--cacert=/etc/kubernetes/cert/ca.pem \
--cert=/etc/kubernetes/cert/etcd.pem \
--key=/etc/kubernetes/cert/etcd-key.pem \
--endpoints=https://172.31.243.179:2379 \
endpoint status --write-out="table"
此时db size应变为1Mb
6.清除告警
ETCDCTL_API=3 etcdctl \ --cacert=/etc/kubernetes/cert/ca.pem \ --cert=/etc/kubernetes/cert/etcd.pem \ --key=/etc/kubernetes/cert/etcd-key.pem \ --endpoints=https://172.31.243.179:2379 \ alarm disarm
结束