prometheus federate ingesting out-of-order samples error

参考

Prometheus error - Error on ingesting samples
Federation Job “Error on ingesting out-of-order samples”

环境

Prometheus 通过联邦获取 A 和 B 的 metrics， 3 套环境都是一样的搭建方式，简单来说无差异部署

问题

在主 prometheus 上查看日志，有很多 msg="Error on ingesting out-of-order samples" num_dropped 的日志

ts=2024-01-17T03:00:26.288Z caller=scrape.go:1723 level=warn component="scrape manager" scrape_pool=federateA target="https://federateA:443/federate?match%5B%5D=%7Bjob%3D~%22apisix-gateway%22%7D&match%5B%5D=%7Bjob%3D~%22kube-state-metrics%22%7D&match%5B%5D=%7Bjob%3D~%22node-exporter%22%7D&match%5B%5D=%7Bjob%3D~%22rook-ceph-mgr%22%7D&match%5B%5D=%7Bjob%3D~%22kubelet%22%7D&match%5B%5D=%7Bjob%3D~%22apiserver%22%7D&match%5B%5D=%7Bjob%3D~%22business-metrics%22%7D&match%5B%5D=%7Bjob%3D~%22redis%22%7D&match%5B%5D=%7Bjob%3D~%22probe-alive%22%7D&match%5B%5D=%7Bjob%3D~%22elastic-exporter%22%7D&match%5B%5D=%7Bjob%3D~%22etcd-metrics%22%7D&match%5B%5D=%7Bjob%3D~%22pg-exporter%22%7D&match%5B%5D=%7Bjob%3D~%22node-problem-detector%22%7D&match%5B%5D=%7Bjob%3D~%22pv-metrics%22%7D&match%5B%5D=%7Bjob%3D~%22kube-summary-exporter%22%7D" msg="Error on ingesting out-of-order samples" num_dropped=2292

处理

Enable debugging mode in Prometheus by passing --log.level=debug as an argument to prometheus
Check the log. In my case the metric that was dropped, was returned:
msg=“Out of order sample” series="some-metric{a=“b”,c=“d”}
Check the prometheus configuration and rule files that could return a duplicated some-metric. In my case a duplicated rule was causing the issue as the same metric was scraped from the federated prometheus and calculated again by my prometheus. Removing the duplicated rule solved the issue.

按照上述操作，日志中确实发现很多类似 series="some-metric{a="b",c="d"} 的信息

### 删掉另外 2 套环境中的 rules，问题解决
for x in $(kubectl -n monitoring get prometheusrules.monitoring.coreos.com  | awk '{print $1}' | grep -v NAME);do kubectl -n monitoring get prometheusrules.monitoring.coreos.com $x -oyaml > ${x}.yaml; kubectl -n monitoring delete prometheusrules.monitoring.coreos.com $x;done

初步检查，在主 prometheus 环境中无影响，可以获取 federate 相关的信息(但是在 federate 环境中就没有了相关的聚合函数了，比如：instance:node_network_receive_bytes_excluding_lo:rate5m)

总结

遇事不要忘记了 --log.level=debug