Also consider this: at 30s run intervals, if you were to look at a longer time (say a week), would you still be interested in each individual run's timing? How would you aggregate them?
Prometheus' answer is to construct time series of the number of runs, and cumulative run time, starting at some arbitrary point in time (together these are a summary). By looking at the change in these numbers over time, we can calculate the duty cycle (what fraction of time is spent running vs. idle) or average run time (cumulative run time divided by the number of runs in the same timeframe). Note that this is all phrased in terms of numbers that exist continuously (time spent since …) rather than individual events (time spent in the fifteenth run). Unfortunately there is no trivial way to keep these accumulated counters over multiple process invocations z since the client libraries only hold them in memory. Ideally, you could get them from the long-running process that starts these individual runs. If that is not possible, the third party aggregating pushgateway may be useful to you. I hope this helps clarify how Prometheus sees the world! /MR On Wed, Mar 17, 2021, 00:06 Matthias Rampke <matth...@prometheus.io> wrote: > There is a mismatch of models here. You are asking about plotting a set of > (x,y) points; Prometheus fundamentally thinks in terms of continuous time > series that happen to be sampled at the scrape interval. > > One way to resolve this is to consider the continuous time series of "how > long did the last run take". This can be sampled at any time, whether 2 or > 22 seconds after the job has finished. You need to sample (scrape) at least > twice per job run to reliably get all counts, and you will not be directly > able to determine whether a run happened to have exactly the same duration > as the one before. > > However, you can use the knowledge of how often the job runs to make the > graph look nice: play with the interval and overall time window alignment > in Grafana to do that. > > /MR > > > On Tue, Mar 16, 2021, 18:48 Dennis Kanygin <dennis.kany...@gmail.com> > wrote: > >> Struggling with Prometheus use case. >> >> Periodic job runs every 30 seconds or so. Duration varies. Need a bar >> graph with x-axis as time of run and y-axis as duration. Pushing duration >> metrics to pushgateway. With this setup time stamp that is being attached >> to duration metrics is the time of scrape of pushgateway. Timestamp of >> duration is therefore "off" when graphing. As I understand, this is a >> feature, not a bug. Any recommendation on how to accomplish what I need? >> Seems like this would be a fairly common scenario so curious how others are >> solving it. >> >> cheers, >> >> Dennis Kanygin >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to prometheus-users+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/e69c1cb3-683a-4652-9bd4-4cd93d3b996fn%40googlegroups.com >> <https://groups.google.com/d/msgid/prometheus-users/e69c1cb3-683a-4652-9bd4-4cd93d3b996fn%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAMV%3D_gY_uj7mfCVUn099te_xmCUd4NfG9YT-apzF-f9UdBeL6A%40mail.gmail.com.
