[prometheus-users] Re: Problems with Regex Relabel and Optional Characters
It's not clear what your requirements are. Is it possible that values after the double-dash might themselves contain a single dash, like mykub-test--another-test--test-123 and if so, what should happen? Could there be additional double-dash separated sections like mykub-test--anothertest--test123--test456 and if so, what should happen? If you want mykub-test--a-a--b-b--c-c to become mykub-test--a-a, then you could try this (untested): - source_labels: [__meta_kubernetes_namespace] regex: (mykub-test--.*?)(--.*)? target_label: kubernetes_customer_env_name replacement: $1 action: replace .*? applies the "non-greedy" modifier to make the first .* match the smallest possible number of times, whereas (xxx)? means match the subexpression zero or one times. For more info see https://github.com/google/re2/wiki/Syntax If you need to do more complex multi-step processing then you can use temporary labels. Names starting __tmp_ are reserved for this purpose (i.e. are guaranteed not to clash with other prometheus-internal labels). The following will split the label into two, on the *first* instance of "--" which appears in the string, requiring at least one instance to occur: - source_labels: [__meta_kubernetes_namespace] regex: (.*?)--(.*) target_label: __tmp_nsprefix replacement: $1 action: replace - source_labels: [__meta_kubernetes_namespace] regex: (.*?)--(.*) target_label: __tmp_nssuffix replacement: $2 action: replace On Tuesday, 14 June 2022 at 18:38:55 UTC+2 eagl...@gmail.com wrote: > Wonder if I can get some help. We are seeing an oddity here. Within my > configuration, we are using kubernetes_sd_configs and I have the following > in the relabel section of my scrape config: > > - source_labels: [__meta_kubernetes_namespace] > separator: ; > regex: (^mykub-test--[^-]*) > target_label: kubernetes_customer_env_name > replacement: $1 > action: replace > > This works fine if the namespace is just mykub-test--herenow but not for > namespaces like mykub-test--anothertest2--test123 . Basically, I want to > strip off the "--test123" (or ignore it) in the later example. Any > suggestions? > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ebec1e70-d9d3-415d-970d-44bed596e9a9n%40googlegroups.com.
[prometheus-users] Problems with Regex Relabel and Optional Characters
Wonder if I can get some help. We are seeing an oddity here. Within my configuration, we are using kubernetes_sd_configs and I have the following in the relabel section of my scrape config: - source_labels: [__meta_kubernetes_namespace] separator: ; regex: (^mykub-test--[^-]*) target_label: kubernetes_customer_env_name replacement: $1 action: replace This works fine if the namespace is just mykub-test--herenow but not for namespaces like mykub-test--anothertest2--test123 . Basically, I want to strip off the "--test123" (or ignore it) in the later example. Any suggestions? -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/aaa440a4-2711-4de8-8c6e-e0533fcc308fn%40googlegroups.com.
[prometheus-users] Re: Does Prometheus recommend exposing 2M timeseries per scrape endpoint?
Total number of time series scraped would be more important I think, so you also need to know how many targets you'll have. I had Prometheus servers scraping 20-30M time series total and that was eating pretty much all memory on server with 256GB ram. In general when doing capacity planning we expect 4KB of memory per time series for base Go memory, and then we need to double that for garbage collector (you can try to tweak GOGC env variable to trade some cpu for less gc memory overhead). With 25M time series 4KB per series means 100GB of Go allocations, and 200GB to account for garbage collector, which usually fits 256GB. But we do run a huge number of services, so Prometheus will scrape lots of targets and get a small number of metrics from each. You want to scrape 2M from a single target and that means Prometheus will have to request, read and parse a huge response body, this might require more peak memory and it might be slow, so your scrape interval would have to allow for that. Another thing to remember is churn - if your time series have labels that keep changing all the time then you might run out of memory, since everything that prometheus scrapes (even only once) ends up in memory until it persists data to disk, which is by default every 2h AFAIR. If the list of values of your APN is not a fixed set and you keep seeing random values over time, then that will accumulate in memory, so your capacity planning would have to take into account how many unique values of APN (and other values) are there and if this is going to grow over time. That's assuming you want to stick with a single prometheus instance, if you can shard your scrapes then you can scale horizontally. It's always hard to give a concrete answer to question like this since it all depends, but it's usually a matter of having enough memory, cpu is typically (in my environment at least) less important. On Tuesday, 14 June 2022 at 12:13:24 UTC+1 vteja...@gmail.com wrote: > I have a use case where a particular service (that can be horizontally > scaled to desired replica count) exposes a 2 Million time series. > Prometheus might expect huge resources to scrape such service (this is > normal). But I'm not sure if there is a recommendation from the community > on instrumentation best practices and maximum count to expose. > > Thanks, > Teja > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/9c494e40-fe19-4252-a9f0-5d024c04d8b4n%40googlegroups.com.
Re: [prometheus-users] Does Prometheus recommend exposing 2M timeseries per scrape endpoint?
Telco system, that creates series per Access Point Name (APN) name as label value (similar to a user-id or a pod-IP but very much important to record) /Teja On Tuesday, June 14, 2022 at 1:52:19 PM UTC+2 Stuart Clark wrote: > On 14/06/2022 12:32, tejaswini vadlamudi wrote: > > Thanks Stuart, this expectation is coming as a legacy requirement for > > a Telco cloud-native application that has huge cardinality (3000 > > values for a label-1) and little dimensionality (2 labels) for 300 > > metrics. > > > > Is there any recommendation like not more than 1k or 10k series per > > endpoint? > > The general expectation is that each Prometheus server would have no > more than a few million time series in total. The normal use case is to > have 10s/100s of jobs & targets, each exposing 100s/1000s of time > series, rather than a single target exposing significant number of time > series itself. > > If this is a single application what is the need for that level of > cardinality? For example something like a high volume e-commerce system > with millions of users might only need labels with say a cardinality of > a few 10s/100s each (for example HTTP status code, section of website, > etc.). What is it about this system that you think it needs very high > cardinality labels? > > -- > Stuart Clark > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/4686a9de-20e6-466a-a26d-1eef8b04300an%40googlegroups.com.
Re: [prometheus-users] Does Prometheus recommend exposing 2M timeseries per scrape endpoint?
On 14/06/2022 12:32, tejaswini vadlamudi wrote: Thanks Stuart, this expectation is coming as a legacy requirement for a Telco cloud-native application that has huge cardinality (3000 values for a label-1) and little dimensionality (2 labels) for 300 metrics. Is there any recommendation like not more than 1k or 10k series per endpoint? The general expectation is that each Prometheus server would have no more than a few million time series in total. The normal use case is to have 10s/100s of jobs & targets, each exposing 100s/1000s of time series, rather than a single target exposing significant number of time series itself. If this is a single application what is the need for that level of cardinality? For example something like a high volume e-commerce system with millions of users might only need labels with say a cardinality of a few 10s/100s each (for example HTTP status code, section of website, etc.). What is it about this system that you think it needs very high cardinality labels? -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/fba029fa-6fb0-4b26-daae-8fe4377ed3e6%40Jahingo.com.
Re: [prometheus-users] Does Prometheus recommend exposing 2M timeseries per scrape endpoint?
Thanks Stuart, this expectation is coming as a legacy requirement for a Telco cloud-native application that has huge cardinality (3000 values for a label-1) and little dimensionality (2 labels) for 300 metrics. Is there any recommendation like not more than 1k or 10k series per endpoint? Br, Teja On Tuesday, June 14, 2022 at 1:22:32 PM UTC+2 Stuart Clark wrote: > On 14/06/2022 12:13, tejaswini vadlamudi wrote: > > I have a use case where a particular service (that can be horizontally > > scaled to desired replica count) exposes a 2 Million time series. > > Prometheus might expect huge resources to scrape such service (this is > > normal). But I'm not sure if there is a recommendation from the > > community on instrumentation best practices and maximum count to expose. > > > Two million time series returned from a single scrape request? > > That's way out of the expected ballpark for Prometheus, and also sounds > totally outside what I'd expect from any metrics system. > > Would you be able to explain a bit more about what you are wanting to be > able to achieve and we can suggest an alternative to Prometheus/metrics > that would help? > > -- > Stuart Clark > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/c9cef2c4-d583-4ab5-9c45-5ea4e8af5ffcn%40googlegroups.com.
Re: [prometheus-users] Does Prometheus recommend exposing 2M timeseries per scrape endpoint?
On 14/06/2022 12:13, tejaswini vadlamudi wrote: I have a use case where a particular service (that can be horizontally scaled to desired replica count) exposes a 2 Million time series. Prometheus might expect huge resources to scrape such service (this is normal). But I'm not sure if there is a recommendation from the community on instrumentation best practices and maximum count to expose. Two million time series returned from a single scrape request? That's way out of the expected ballpark for Prometheus, and also sounds totally outside what I'd expect from any metrics system. Would you be able to explain a bit more about what you are wanting to be able to achieve and we can suggest an alternative to Prometheus/metrics that would help? -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/001c5333-a851-c781-83a4-05391824323e%40Jahingo.com.
[prometheus-users] Does Prometheus recommend exposing 2M timeseries per scrape endpoint?
I have a use case where a particular service (that can be horizontally scaled to desired replica count) exposes a 2 Million time series. Prometheus might expect huge resources to scrape such service (this is normal). But I'm not sure if there is a recommendation from the community on instrumentation best practices and maximum count to expose. Thanks, Teja -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ac52e2b1-292c-49fe-b906-8463d98897f3n%40googlegroups.com.
[prometheus-users] Version
How to add an alert to the grafana dashboard when the version of API is about to expire? -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/2c469438-adc0-4631-88af-f04036a7aa1an%40googlegroups.com.
