For a (conservative) guide see this article
<https://www.robustperception.io/cardinality-is-key>. You have the right
intuition – ~10 buckets is a good number.
You *can* go higher if you use high-powered machines (mostly: lots of RAM)
for Prometheus, but you will run into increasing problems as either the
cardinality of a single label, or the cardinality of a metric, gets large.
Some of these you can work around by aggressively pre-aggregating in
recording rules; otherwise it will be impossible to graph any reasonable
timeframe.
/MR
On Wed, Sep 2, 2020 at 7:38 AM rs vas <rsvas.t...@gmail.com> wrote:
> We are seeing an issue when we have used and generated a histogram with
> 69le's it created a unique time series per le. this metric has few labels
> for example: method, operation, identifier, ip
>
> This single metric itself is generating a 17k unique time series from a
> single host and from all hosts its total time series is totalling up to
> 300k for this xxxxx_seconds_bucket series.
>
> I was reading some best practices on defining buckets, we will have to
> consider SLO and all, but my questions are:
>
> - any good number we can cross when defining buckets for example not
> to define more than 10 le's.
> - any good number of max labels you can have on a single metric
> - any good number on the cardinality of a single time series at a
> given point? If I query the count(xxxxx_seconds_bucket) what is that number
> we should not cross?
>
> I have a feeling like it is going to kill the Prometheus at some point, if
> we can't fix the number of le's. Any input is appreciated.
>
>
> *Example defined buckets: (looks to be a real concern here having 69
> le's).*
> xxxxx_seconds_bucket{...="none",le="0.001",} 71849.0
> xxxxx_seconds_bucket{...="none",le="0.001048576",} 72078.0
> xxxxx_seconds_bucket{...="none",le="0.001398101",} 73083.0
> xxxxx_seconds_bucket{...="none",le="0.001747626",} 73600.0
> xxxxx_seconds_bucket{...="none",le="0.002097151",} 73943.0
> xxxxx_seconds_bucket{...="none",le="0.002446676",} 74160.0
> xxxxx_seconds_bucket{...="none",le="0.002796201",} 74399.0
> xxxxx_seconds_bucket{...="none",le="0.003145726",} 74936.0
> xxxxx_seconds_bucket{...="none",le="0.003495251",} 75109.0
> xxxxx_seconds_bucket{...="none",le="0.003844776",} 75227.0
> xxxxx_seconds_bucket{...="none",le="0.004194304",} 75336.0
> xxxxx_seconds_bucket{...="none",le="0.005592405",} 75674.0
> xxxxx_seconds_bucket{...="none",le="0.006990506",} 75885.0
> xxxxx_seconds_bucket{...="none",le="0.008388607",} 75958.0
> xxxxx_seconds_bucket{...="none",le="0.009786708",} 75981.0
> xxxxx_seconds_bucket{...="none",le="0.011184809",} 75995.0
> xxxxx_seconds_bucket{...="none",le="0.01258291",} 76004.0
> xxxxx_seconds_bucket{...="none",le="0.013981011",} 76005.0
> xxxxx_seconds_bucket{...="none",le="0.015379112",} 76008.0
> xxxxx_seconds_bucket{...="none",le="0.016777216",} 76013.0
> xxxxx_seconds_bucket{...="none",le="0.022369621",} 76033.0
> xxxxx_seconds_bucket{...="none",le="0.027962026",} 76039.0
> xxxxx_seconds_bucket{...="none",le="0.033554431",} 76039.0
> xxxxx_seconds_bucket{...="none",le="0.039146836",} 76039.0
> xxxxx_seconds_bucket{...="none",le="0.044739241",} 76039.0
> xxxxx_seconds_bucket{...="none",le="0.050331646",} 76039.0
> xxxxx_seconds_bucket{...="none",le="0.055924051",} 76039.0
> xxxxx_seconds_bucket{...="none",le="0.061516456",} 76042.0
> xxxxx_seconds_bucket{...="none",le="0.067108864",} 76043.0
> xxxxx_seconds_bucket{...="none",le="0.089478485",} 76044.0
> xxxxx_seconds_bucket{...="none",le="0.111848106",} 76044.0
> xxxxx_seconds_bucket{...="none",le="0.134217727",} 76044.0
> xxxxx_seconds_bucket{...="none",le="0.156587348",} 76044.0
> xxxxx_seconds_bucket{...="none",le="0.178956969",} 76044.0
> xxxxx_seconds_bucket{...="none",le="0.20132659",} 76045.0
> xxxxx_seconds_bucket{...="none",le="0.223696211",} 76045.0
> xxxxx_seconds_bucket{...="none",le="0.246065832",} 76046.0
> xxxxx_seconds_bucket{...="none",le="0.268435456",} 76046.0
> xxxxx_seconds_bucket{...="none",le="0.357913941",} 76046.0
> xxxxx_seconds_bucket{...,le="0.447392426",} 76057.0
> xxxxx_seconds_bucket{...,le="0.536870911",} 76061.0
> xxxxx_seconds_bucket{...,le="0.626349396",} 76061.0
> xxxxx_seconds_bucket{...,le="0.715827881",} 76064.0
> xxxxx_seconds_bucket{...,le="0.805306366",} 76085.0
> xxxxx_seconds_bucket{...,le="0.894784851",} 76085.0
> xxxxx_seconds_bucket{...,le="0.984263336",} 76086.0
> xxxxx_seconds_bucket{...,le="1.073741824",} 76086.0
> xxxxx_seconds_bucket{...,le="1.431655765",} 76086.0
> xxxxx_seconds_bucket{...,le="1.789569706",} 76088.0
> xxxxx_seconds_bucket{...,le="2.147483647",} 76116.0
> xxxxx_seconds_bucket{...,le="2.505397588",} 76116.0
> xxxxx_seconds_bucket{...,le="2.863311529",} 76116.0
> xxxxx_seconds_bucket{...,le="3.22122547",} 76116.0
> xxxxx_seconds_bucket{...,le="3.579139411",} 76116.0
> xxxxx_seconds_bucket{...,le="3.937053352",} 76116.0
> xxxxx_seconds_bucket{...,le="4.294967296",} 76116.0
> xxxxx_seconds_bucket{...,le="5.726623061",} 76116.0
> xxxxx_seconds_bucket{...,le="7.158278826",} 76116.0
> xxxxx_seconds_bucket{...,le="8.589934591",} 76116.0
> xxxxx_seconds_bucket{...,le="10.021590356",} 76116.0
> xxxxx_seconds_bucket{...,le="11.453246121",} 76116.0
> xxxxx_seconds_bucket{...,le="12.884901886",} 76116.0
> xxxxx_seconds_bucket{...,le="14.316557651",} 76116.0
> xxxxx_seconds_bucket{...,le="15.748213416",} 76116.0
> xxxxx_seconds_bucket{...,le="17.179869184",} 76116.0
> xxxxx_seconds_bucket{...,le="22.906492245",} 76116.0
> xxxxx_seconds_bucket{...,le="28.633115306",} 76116.0
> xxxxx_seconds_bucket{...,le="30.0",} 76116.0
> xxxxx_seconds_bucket{...,le="+Inf",} 76116.0
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/CAPs_AfjxPf5mOLb1s%2BtgFXfjAqcZ7uOyMT1WrX3n3KymYNp_4Q%40mail.gmail.com
> <https://groups.google.com/d/msgid/prometheus-users/CAPs_AfjxPf5mOLb1s%2BtgFXfjAqcZ7uOyMT1WrX3n3KymYNp_4Q%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/CAMV%3D_gbiZ%3DruT0e%3D%3DaHyJ5dLEYpLQd4XraubS7DWEFBu%3Drj%3DoQ%40mail.gmail.com.
