Prometheus 监控docker

查看docker运行状态

docker stats

启动docker cadvisor

• 直接运行docker

docker run -d \
? --volume=/:/rootfs:ro \
? --volume=/var/run:/var/run:rw \
? --volume=/sys:/sys:ro \
? --volume=/var/lib/docker/:/var/lib/docker:ro \
? --publish=8080:8080 \
? --name=cadvisor \
? gcr.io/cadvisor/cadvisor:v0.44.0

• docker-compose.yml方式

mkdir /data/cadvisor
cat >docker-compose.yml << 'EOF'
version: '3.3'
networks:
? monitoring:
? ? driver: bridge

services:
? cadvisor:
? ? image: gcr.io/cadvisor/cadvisor:v0.44.0
# ? image: google/cadvisor:latest
? ? container_name: cadvisor
? ? restart: always
? ? volumes:
? ? ? - /etc/localtime:/etc/localtime:ro
? ? ? - /:/rootfs:ro
? ? ? - /var/run:/var/run:rw
? ? ? - /sys:/sys:ro
? ? ? - /var/lib/docker/:/var/lib/docker:ro
? ? networks:
? ? ? - monitoring
? ? ports:
? ? ? - 8080:8080
EOF

运行

docker-compose up -d

check

通过访问 http://10.19.1.220:8080可以查看，当前主机上容器的运行状态，
采集样本 http://10.19.1.220:8080/metrics)

?更新 prometheus.yml配置

? ?- job_name: 'cadvisor'
? ? scrape_interval: 15s
? ? static_configs:
? ? - targets: ['cadvisor:8080']
? ? ? labels:
? ? ? ? instance: Prometheus服务器
#增加新的cadvisor配置
? ? - targets: ['10.19.1.220:8080']
? ? ? labels:
? ? ? ? instance: cadvisor-220服务器?
?

docker触发器

cat >prometheus/rules/docker.yml << 'EOF'
groups:
- name: DockerContainers
? rules:
? - alert: ContaonerKilled
? ? expr: time() -Container_last_seen > 60
? ? for: 0m
? ? labels:
? ? ? severiry: warning
? ? annotations:
? ? ? summary: "Docker容器被杀死,容器: {{ $labels.instance }}"
? ? ? description: "{{ $value }} 容器消失了"
EOF

热加载

docker exec -it prometheus promtool check config /etc/prometheus/prometheus.yml
curl -X POST http://localhost:9090/-/reload

添加dashboard

11600