Hello Stuart! My retention period is 90 days. I try to see the data via a grafana dashboard, as well as via the prometheus graphs feature, same effect.
All the kubernetes-related data is pulled via federate from another prometheus instance running inside the K8s cluster. Does the federate protocol include a mechanism to tell the pulling prometheus to delete series? Do the scrape jobs delete series implicitly? Thanks Martin Stuart Clark schrieb am Donnerstag, 10. September 2020 um 11:53:24 UTC+2: > On 10/09/2020 10:48, Martin Emrich wrote: > > Hello! > > > > Our prometheus contains metrics of our kubernetes cluster. > > > > Today, I wanted to look at data of a node (EC2 instance on AWS) that > > has been removed yesterday. But all the data regarding the VM is gone. > > In contrast, data of a Kubernetes Pod that has been terminated is > > still visible later. > > > > The nodes are discovered using ec2_sd_configs, the Kuberneted Pod data > > is scraped via federation from another prometheus instance running > > inside the K8s cluster. > > > > Apparently there is a difference in behaviour regarding removed > > targets. But I cannot find a configuration parameter. > > > > How can I keep historic data of removed VMs (or other targets) when > > the target itself is no longer there? I bet that is a FAQ, but I did > > not find information on how data removal works (beyond the storage > > configuration, which I expect works the same for all series) > > > > How are you trying to see the older data & what is your retention period? > > If a pod/EC2 instance goes away it just means no new data will be > scraped and any timeseries that were being scraped will go stale, but > the existing data should remain until the retention period is reached. > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/8e1cf1c1-77d6-42b5-83f2-4a3d5c31599an%40googlegroups.com.
