Re: [prometheus-users] Re: Alert Duplication in HA Prometheus & Alertmanager setup.

I don't know about the original poster, but in my case, I have an 2 instances of Prometheus using the same config file, with an external label set to "prom" on each. I have both pointed to 2 alertmanagers which are clustered, and if I create a silence in one I can see it appear in the other.

Re: [prometheus-users] Setting up a Prometheus environment with a push mechanism

Regarding the other points: however you judge the differences, they are still significant departures from the rest of the Prometheus ecosystem (including compatible third-party providers). It's ok to build something incompatible, it's just IMO not ok to superficially make it sound everywhere like

Re: [prometheus-users] Setting up a Prometheus environment with a push mechanism

On Thu, Mar 25, 2021 at 3:15 PM Aliaksandr Valialkin wrote: > > > On Thu, Mar 25, 2021 at 3:54 PM Julius Volz > wrote: > >> What should be noted when bringing up VictoriaMetrics in the Prometheus >> context is that it is deliberately incompatible with Prometheus in multiple >> ways: >> >> -

Re: [prometheus-users] Consul SD and adding labels via static_config

The `labels` section is unavailable in `consul_sd_configs`. If you want adding labels to all the metrics collected from discovered Consul targets, then add the following relabeling rules under consul_sd_configs section: - target_label: env replacement: foo - target_label: region replacement:

Re: [prometheus-users] Setting up a Prometheus environment with a push mechanism

On Thu, Mar 25, 2021 at 3:54 PM Julius Volz wrote: > What should be noted when bringing up VictoriaMetrics in the Prometheus > context is that it is deliberately incompatible with Prometheus in multiple > ways: > > - VM's MetricsQL behaves differently from PromQL in a multitude of ways > and is

Re: [prometheus-users] fix yamllint errors specific for Prometheus.yml

On 24.03.21 14:37, David Leadbeater wrote: I’ve not used it but https://github.com/jumanjihouse/pre-commit-hook-yamlfmt looks like it could be useful for you. Thank you. This tool did very well the job and the result was compatible

Re: [prometheus-users] Setting up a Prometheus environment with a push mechanism

Another option if your security will allow opening one port to Prometheus data, there is the "exporter_exporter" https://github.com/QubitProducts/exporter_exporter On the subject of external labels, you should have those anyway on your Prometheus instances otherwise you can't identify them

Re: [prometheus-users] Setting up a Prometheus environment with a push mechanism

Alright, did not know all of that. We are now setting up the Prometheus Proxy and are hoping that it does what we want it to do. VictoriaMetrics would be a last case scenario anyway Op donderdag 25 maart 2021 om 14:54:22 UTC+1 schreef juliu...@promlabs.com: > What should be noted when bringing

Re: [prometheus-users] Setting up a Prometheus environment with a push mechanism

What should be noted when bringing up VictoriaMetrics in the Prometheus context is that it is deliberately incompatible with Prometheus in multiple ways: - VM's MetricsQL behaves differently from PromQL in a multitude of ways and is *not* backwards-compatible with PromQL (see also

Re: [prometheus-users] invalid UTF-8 label value on Swedish text

On 25 Mar 12:48, John Dexter wrote: > Thanks David, > > That indeed flags the error and shows the special character is all messed > up. I don't know iconv but if this is the issue and there's no way around > it than fixing the raw text input, I can find a code solution. Shame > Prometheus doesn't

Re: [prometheus-users] Setting up a Prometheus environment with a push mechanism

It is correct that Thanos' deduplication works when using just 2 or more Prometheus servers. The thing is, Thanos cannot use existing labels to do deduplication. I want it to deduplicate using the label called exported_instance, but i cannot predefine this label in the external labels for

Re: [prometheus-users] `count by` looks like lost data in Prometheus

Hi, The count() aggregator counts how many time series there are at a single point in time (at each evaluation step). So it makes sense that the answer is 2 no matter what resolution you choose (as long as there are two time series at each time step). Maybe you were looking for the

Re: [prometheus-users] Setting up a Prometheus environment with a push mechanism

Probably VictoriaMetrics could help with this setup. It supports data push via various protocols , it supports data deduplication and it is compatible with Prometheus datasource in

Re: [prometheus-users] Setting up a Prometheus environment with a push mechanism

If you're monitoring within a single network, but can't make any connections to your targets, PushProx might be a good idea https://github.com/prometheus-community/PushProx On Thu, Mar 25, 2021 at 2:01 PM robbe vaes wrote: > Okay so, we want to have an environment using Prometheus, where we

Re: [prometheus-users] Setting up a Prometheus environment with a push mechanism

Deduplication works fine with Thanos, you need to make sure you setup replica labels in your Prometheus external labels. On Thu, Mar 25, 2021 at 2:20 PM robbe vaes wrote: > The thing is we can implement multiple prometheus instances as well, thats > no issue and will probably happen anyway. The

Re: [prometheus-users] Setting up a Prometheus environment with a push mechanism

The thing is we can implement multiple prometheus instances as well, thats no issue and will probably happen anyway. The thing is, we tried using Thanos to manage multiple prometheus servers and do deduplication, but the deduplication does not work for the collectd exporters. The problem its

Re: [prometheus-users] Setting up a Prometheus environment with a push mechanism

The way this is usually solved is by duplicating prometheus - it seems that now you have moved the SPOF from the exporter to prometheus. Regards, On 25 Mar 06:01, robbe vaes wrote: > Okay so, we want to have an environment using Prometheus, where we can > monitor our servers etc with a push

Re: [prometheus-users] Setting up a Prometheus environment with a push mechanism

Okay so, we want to have an environment using Prometheus, where we can monitor our servers etc with a push method rather then pull due to network security aspects. As of now, we managed to set up collectd together with collectd exporter for prometheus. This way we can have the clients or

Re: [prometheus-users] invalid UTF-8 label value on Swedish text

Thanks David, That indeed flags the error and shows the special character is all messed up. I don't know iconv but if this is the issue and there's no way around it than fixing the raw text input, I can find a code solution. Shame Prometheus doesn't support any other encodings but heigh ho.

Re: [prometheus-users] invalid UTF-8 label value on Swedish text

Can you try: curl host/metrics | promtool check metrics Which will avoid any issues due to copying and means Prometheus sees the raw bytes. C++ std::string types (which is what prometheus-cpp uses: https://jupp0r.github.io/prometheus-cpp/structprometheus_1_1ClientMetric_1_1Label.html) don't

Re: [prometheus-users] Setting up a Prometheus environment with a push mechanism

Can you describe more about what your network topology is exactly? There are a number of solutions for dealing with distributed monitoring. On Thu, Mar 25, 2021 at 12:45 PM robbe vaes wrote: > Hi, > > I am trying to setup a monitoring environment with Prometheus, but it has > to be using a push

Re: [prometheus-users] invalid UTF-8 label value on Swedish text

A little more digging (I am not a web-dev so excuse any dumb things): - when I look at the endpoint URL in Chrome, Developer tools->Network->Header I get the following response header: [image: image.png] - when I check what encoding Chrome has actually *used *(

[prometheus-users] Setting up a Prometheus environment with a push mechanism

Hi, I am trying to setup a monitoring environment with Prometheus, but it has to be using a push mechanism instead of the standard pull mechanism Prometheus uses. I was wondering what options there are to create an environment like this. It would also have to perfom data deduplication. The

Re: [prometheus-users] invalid UTF-8 label value on Swedish text

It's the latest version as of a few weeks ago. Other modules which are identical except their label values contain no special characters work fine so I am fairly confident this is the reason - should I be able to find more verbose logs telling me exactly what it didn't like because I could not

Re: [prometheus-users] invalid UTF-8 label value on Swedish text

Do you have an up to date version of prometheus? I tryied this and it workd for me. Are you sure that those are the offending metrics? Thanks. On 25 Mar 10:47, John Dexter wrote: > I wouldn't be surprised though it does validate online (unless copy-paste > in chrome automatically changes the

Re: [prometheus-users] invalid UTF-8 label value on Swedish text

I wouldn't be surprised though it does validate online (unless copy-paste in chrome automatically changes the encoding!) I'm using the prometheus-cpp library and I cannot find any mention of character support. Does Prometheus not have any way to convert incoming data? The strings I am using as

Re: [prometheus-users] invalid UTF-8 label value on Swedish text

Sounds like you may have some data that's encoded in a pre-UTF-8 format like https://en.wikipedia.org/wiki/Windows-1257 or https://en.wikipedia.org/wiki/ISO/IEC_8859. You'll have a character encoding converter in your exporter in order to ingest the data. On Thu, Mar 25, 2021 at 11:32 AM John

[prometheus-users] Consul SD and adding labels via static_config

Hello! I wanted to ask if it would be possible to add labels via static_config to a scrape job that uses the Consul Service Discovery mechanism? Right now I have tested two solutions and with the static target, the labels are being applied while the Consul SD does not apply. Consul SD: -

[prometheus-users] invalid UTF-8 label value on Swedish text

I understand that metric and label *names* are standard ASCII but *values* support Unicode, at least according to docs. I am working on a Swedish-language system and just realised several of my targets are reporting this error, however their endpoints work just fine. I have a metric with:

Re: [prometheus-users] Alerts sent to sms

On 25/03/2021 09:22, Mohan Nagandlla wrote: Hi team Can any one guide me to send the alerts from alertmanager to sms That very much depends on which SMS provider you are wanting to use. Alertmanager has direct support for Pagerduty and OpsGenie (amongst others) which can send things to SMS.

[prometheus-users] Alerts sent to sms

Hi team Can any one guide me to send the alerts from alertmanager to sms -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to