Hello! Our prometheus contains metrics of our kubernetes cluster.
Today, I wanted to look at data of a node (EC2 instance on AWS) that has been removed yesterday. But all the data regarding the VM is gone. In contrast, data of a Kubernetes Pod that has been terminated is still visible later. The nodes are discovered using ec2_sd_configs, the Kuberneted Pod data is scraped via federation from another prometheus instance running inside the K8s cluster. Apparently there is a difference in behaviour regarding removed targets. But I cannot find a configuration parameter. How can I keep historic data of removed VMs (or other targets) when the target itself is no longer there? I bet that is a FAQ, but I did not find information on how data removal works (beyond the storage configuration, which I expect works the same for all series) Thanks, Martin -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/0fc5db05-21bb-43ca-9503-adb354355763n%40googlegroups.com.
