Re: [prometheus-users] OTel/Prometheus Interoperability Survey

cause it doesn't follow > normal conventions: > > Options are: > node_time > node_time_nanoseconds > > There is no option for the real Prometheus metric: node_time_seconds > :( > > On Thu, May 2, 2024 at 5:02 PM Bartłomiej Płotka > wrote: > >> Hi

[prometheus-users] Re: OTel/Prometheus Interoperability Survey

PS: The deadline for this survey is 31st May 2024 Kind Regards, Bartek Płotka (@bwplotka) -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+u

[prometheus-users] OTel/Prometheus Interoperability Survey

Hi! The OpenTelemetry end-user group , in collaboration with the Prometheus team, shared a survey to learn more about some naming and label conventions between both projects (e.g. the dot in the label names and unit suffix). Your opinion matters!

Re: [prometheus-users] A solution for counters automatic initialization to 0

While searching for some patterns I noticed this thread. Looked on this counter warm-up solution - quite clever, thanks for sharing! We are working on different solution, utilizing created timestamps feature. It's being designed here

[prometheus-users] Re: Questions on Prometheus Agent Mode

Hi > Can we configure multiple remote-write receivers in agent mode? Yes, you can. All what's written in Remote Write tuning post (https://prometheus.io/docs/practices/remote_write/) applies to both agent and normal mode. Remote Write code in Prometheus does not care what type of TSDB DB

[prometheus-users] Re: Questions on Prometheus Agent Mode

Hi > Can we configure multiple remote-write receivers in agent mode? Yes, you can. All what's written in Remote Write tuning post applies to both agent and normal mode . Remote Write code in Prometheus does not care what type of TSDB DB P

[prometheus-users] Re: Questions on Prometheus Agent Mode

Hi > Can we configure multiple remote-write receivers in agent mode? Yes, you can. All what's written in Remote Write tuning post applies to both agent and normal mode . Remote Write code in Prometheus does not care what type of TSDB DB P

Re: [prometheus-users] Thanos - partial response behavior

Hi, this question might better be suited for either https://github.com/thanos-io/thanos Github Discussion/Issue of #thanos Slack channel. Thanos querier declares a response is partial when atleast one instance > exposing Store APIs is down. Systems preferring accuracy will “abort” rule > evaluatio

Re: [prometheus-users] Does Prometheus support pagination of the results?

Yet, there is an open question if any streaming API would be useful (e.g similar to what we have with streaming remote read now). Kind Regards, Bartek Płotka (@bwplotka) On Mon, May 10, 2021 at 8:57 AM Hello World wrote: > Thanks for the clarification, Julien. > > On Wednesday, 28 April 2021 a

[prometheus-users] Reminder: Prometheus Ecosystem Community Meeting

Hi All! We have Prometheus Ecosystem Community Meeting today at 5pm UTC. Please suggest agenda items for related projects and/or questions! It's your chance to learn more and ask live questions to the Community and Team. Agenda: https://bit.ly/prometheus-community-agenda 🤗 Thanks! Kind Reg

Re: [prometheus-users] Collector already registered that provides name

Hey, Your application has probably already registered such metrics. The most popular cause of this is using globals (please avoid those) or and registration of metric on request path or something (it should be on binary/application start only). Kind Regards, Bartek Płotka (@bwplotka) On Thu, 1

Re: [prometheus-users] Re: Tool for performance/load test

Hi, We wrote also some tooling for generating blocks with arbitrary data: http://github.com/thanos-io/thanosbench Somo yolo part of podcast we did showing how to do it: https://youtu.be/j4TAGO019HU?t=1251 Kind Regards, Bartek Płotka (@bwplotka) On Wed, 11 Nov 2020 at 16:34, Anoop Mohan wrote:

Re: [prometheus-users] Sample Code in Go/C++ to Publish Metrics to a Remote Write Endpoint

I think this question should go to the TimescaleDB support list rather because it's up to Timescale what exactly semantics they allow on their APIs. In terms of official Prometheus Remote Write, it's hard to get exactly the same semantics that is accurate as Prometheus client. Think about stalenes

Re: [prometheus-users] Output Modifiers for Query API

Please share your ideas for example post-processors you would love to see! (: Also shared on Twitter: https://twitter.com/bwplotka/status/1324760431177682947 Kind Regards, Bartek Płotka (@bwplotka) On Fri, 6 Nov 2020 at 15:17, Gayathri Venkatesh wrote: > Hi everyone, > > My name is Gayathri,

Re: [prometheus-users] Prometheus eats up all memory with constant load

Hi, That's expected with the applications built with the newer Go versions. On Linux, they will "reserve" whatever memory is available in the host, but it's actually might not be allocations. This might be a memory that the kernel can take whenever it needs, so those cases will not trigger OOMs

[prometheus-users] ODIC/Oauth2 client for Remote Write

Hi As per https://github.com/prometheus/prometheus/issues/8132 we would like to leverage OIDC for remote write natively. While we can propose and slowly discuss this upstream there is already efficient way to have sidecar that does the OIDC/Oauth2 dance and replaces bearer token on Prometheus

Re: [prometheus-users] Disable remote write retry

Hey, Unless there is some bug on the receiving side (maybe your front proxy masking the actual status code) or Cortex - both Cortex and Thanos Receive in cases of not accepting write for reasons like this (something that there is no point retrying for) returns the status code that tells Prometheus

Re: [prometheus-users] Re: Memory usage discrepancy

Hey, I wrote about this here https://www.bwplotka.dev/2019/golang-memory-monitoring/ at some point as well. Unless something changes by the exporter that actually gives your this metric (it's not Prometheus question, Prometheus is merely collecting those from other exporters) container_memory_work

Re: [prometheus-users] Annotations to scrape multiple containers in same pod?

Hey, Years ago I created this doc https://docs.google.com/document/d/1S6O1czHtjR2DGfK2zeZDLr88wxyLRXB2JpH5g7YM1J8/edit with an overview of options, and nothing changed from that time. Maybe except maturity and more adoption of one of the options: Prometheus Operator: See: https://github.com/promet

Re: [prometheus-users] Issues with prometheus rules once thanos is enabled

n Pivotto wrote: > > Can we know which prometheus version you are running? > > > On 25 Aug 13:42, Julien Pivotto wrote: > > On 25 Aug 12:12, Bartłomiej Płotka wrote: > > > Hey, > > > > > > Interesting. Can you double-check your rules, what are they asking f

Re: [prometheus-users] Issues with prometheus rules once thanos is enabled

Hey, Interesting. Can you double-check your rules, what are they asking for? What queries are they making? Sounds like some typo in rule configuration you have with double asked matchers in the PromQL? Not sure how this is related to Thanos if you see this on Prometheus PromQL log. Kind Regards,

[prometheus-users] Re: empty external labels are not allowed for Thanos block

Hej, For Thanos questions it's best to visit #thanos channel on CNCF slack. You can find a link to the slack here https://slack.cncf.io/ TL;DR It would be nice to know if sidecar connects to correct Promethus server, otherwise it takes labels driectly from config API, so we did not see any issu

Re: [prometheus-users] Thanos/Prometheus Query

I would say it's best to go to #thanos Slack (CNCF Slack workspace) with those questions. We have more than 1k users there willing to answer some of those! TL;DR 1. Why you need that? Why not putting Alertmanager to the central location instead? Alertmanager is to

[prometheus-users] Recommendations for Open Source Analytic (OLAP) system / API to mine Prometheus/Thanos data.

Hi Prometheus Community! 👋 I recently noticed that many Prometheus and Thanos users often desire to use their metric data collected by Prometheus for more advanced Analytics cases. Something more suitable for Business Intelligence / OLAP use cases. As the Prometheus maintainers, we designed Prome

Re: [prometheus-users] Thanos Querier HA

Hi, Thanos Querier is stateless so you can run as many as you want and load balance between them (e.g put behind L4 k8s service). There is also a new component coming which will make it even easier to distribute requests among those (plus retries). This is also for scale-out, not only for high av

Re: [prometheus-users] Re: Prometheus remote_read for multiple Prometheus instances or Thanos Querier?

Also this: https://prometheus.io/blog/2019/10/10/remote-read-meets-streaming/ Actually Brian you have a very nice point here. It's not well documented - it's definitely easy to miss this! There is even a recent discussion about it thanks to Shay: https://github.com/prometheus/prometheus/issues/719

Re: [prometheus-users] Re: Prometheus remote_read for multiple Prometheus instances or Thanos Querier?

Hi, Thanks, Brian for tips, I will quickly jump in, as there is a little bit of misinformation here, maybe because things are moving quickly in this area. (: > I believe that Thanos querier component has small footprint (compare to prometheus remote_read). Why? There is literally no overhead vs

Re: [prometheus-users] Remote Write Server Side Traffic Mirroring + Obfuscator for Prometheus Ecosystem

ent -> filtering -> remote target2 (staging) > -> obfuscation -> remote target3 (dev) > > > On Mon, Apr 20, 2020 at 9:24 PM Bartłomiej Płotka > wrote: > >> Hi! >> >> This question is not strictly related to Prometheus, b

[prometheus-users] Remote Write Server Side Traffic Mirroring + Obfuscator for Prometheus Ecosystem

Hi! This question is not strictly related to Prometheus, but rather to server-side Remote Write APIs. We are looking at how to have more realistic staging environments for servers like that. In order to achieve so, we want to "mirror" / "fork" portion of production remote write traffic to other cl