When time series label values are substituted for new ones regularly and the old series becomes active with no new data points. Keeping ingestion rate constant, what is the performance impact of high churn rate on ingestion, querying, compaction, and other operations.
For example, container_memory_usage_bytes gives memory usage of a pod's containers in kubernetes. It contains a *pod* and *container *labels contains pod name and container name, respectively, these may change often due to change in deployment, auto scaling, restart of crashed pods or for load balancing for a large scale real time system. I believe the inactive time series should be regularly flushed out of active memory. High cardinality caused by the high churn rate should not cause high RAM usage. Compaction operation runs in background packing time series blocks into bigger blocks. If ingestion rate is constant, I don't foresee an impact of compaction runtime or resource usage. Querying for a specific pod/container (most commonly container) would touch blocks within the time horizon of query. Unless query spans a large time range, I don't think there should be a significant impact on query run time and use too much CPU/RAM. Is my understanding correct? Are there other performance considerations for a high churn rate? -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/69beccfa-907d-459a-bbfa-4bd80b258ee3n%40googlegroups.com.
