Re: [prometheus-users] Re: Proxy authentication credentials using proxy_connect_header not working

2023-03-07 Thread Julien Pivotto
On 02 Mar 02:28, Brian Candler wrote: > On Thursday, 2 March 2023 at 06:32:07 UTC Kesavanand Chavali wrote: > > we did a wireshark capture and found that below works. This we figured out > from other already working clients with other software; e.g. some clients > working with same proxy. > Coul

Re: [prometheus-users] Why does the federate endpoint return all metrics as untyped?

2023-03-07 Thread Bjoern Rabenstein
[Apologies for late answer.] On 15.12.22 01:41, 心若寒江雪 wrote: > > How does upper Prometheus in the Prometheus federated cluster know what > kind of Metrics they are getting from the federate interface of lower > Prometheus? Sadly, Prometheus doesn't really persist the metric type. It does some

Re: [prometheus-users] Re: Joining two metrics in promtheus and having all labels from both

2023-03-07 Thread Nick Carlton
Thanks, I think I understand what you mean now. In theory both of the metrics that I’m using to would work just without any manipulation, for example ltmPoolMbrStatusAvailState will return everything I care about and all the labels I need to use to match it’s just once I manipulate the values tha

Re: [prometheus-users] Re: Joining two metrics in promtheus and having all labels from both

2023-03-07 Thread Brian Candler
On Tuesday, 7 March 2023 at 16:47:34 UTC Nick Carlton wrote: I suppose that if there are for example no nodes that are enabled but not available, would Prometheus actually know the pools that don’t match this because they theory they would not be returned at all in the query, nevermind the cou

Re: [prometheus-users] Re: Joining two metrics in promtheus and having all labels from both

2023-03-07 Thread Nick Carlton
Thanks I’ll have a look into that. Are you able to advise on substituting my query into the format ‘foo or on (x,y,z) bar * 0’ like you suggested. I tried one but got unexpected expression so I’m obviously getting the syntax wrong….! I suppose that if there are for example no nodes that are enab

Re: [prometheus-users] Re: Joining two metrics in promtheus and having all labels from both

2023-03-07 Thread Brian Candler
On Tuesday, 7 March 2023 at 16:20:14 UTC Nick Carlton wrote: This is the first I’ve heard of recording rules. So I would be able to convert the group_right query into a new metric using this method? Yes. The documentation seems to suggest it’s a colon string with an expression. Not necessari

Re: [prometheus-users] Re: Joining two metrics in promtheus and having all labels from both

2023-03-07 Thread Nick Carlton
This is the first I’ve heard of recording rules. So I would be able to convert the group_right query into a new metric using this method? The documentation seems to suggest it’s a colon string with an expression. Does that mean the metric becomes the string containing the colons? And I guess these

Re: [prometheus-users] Re: Joining two metrics in promtheus and having all labels from both

2023-03-07 Thread Brian Candler
> My issue with the above is that I don't have the `ltmPoolMbrStatusAvailState` metric with both the `ltmPoolMbrStatusAvailState` and `ltmPoolMbrStatusEnabledState` label values. Im not sure how I would manage to get the merged metric in this way. You can nest queries as far as you need, includ

Re: [prometheus-users] Re: Joining two metrics in promtheus and having all labels from both

2023-03-07 Thread Nick Carlton
Thanks, The reason the metrics have the same name as the label is because in order to bring the value in from the SNMP exporter as a lablel, you have to set the type as DisplayString. As from what I have read, doing a 'join' would not allow you yo get the value of the metric in as a label value

Re: [prometheus-users] Re: Joining two metrics in promtheus and having all labels from both

2023-03-07 Thread Brian Candler
Something like: ltmPoolMbrStatusAvailState * on(instance,ltmPoolMbrStatusNodeName,ltmPoolMbrStatusPoolName,ltmPoolMbrStatusPort) group_left(ltmPoolMbrStatusEnabledState) ltmPoolMbrStatusEnabledState (Slightly confusing that "ltmPoolMbrStatusEnabledState" is both a label name, and a metric name)

Re: [prometheus-users] Re: Joining two metrics in promtheus and having all labels from both

2023-03-07 Thread Brian Candler
Well, "many" means "one or more", and therefore "one" is a valid case of "many" :-) But yes, you'll have to treat it as a many-to-one in order to pick up the extra labels. If you know for sure that metrics A and B are always matched 1:1, then it doesn't matter whether you use group_left or gro

[prometheus-users] Re: {\"message\":\"Not Found\"}\n" - mail Alertmanager (and prometheus-msteams)

2023-03-07 Thread Brian Candler
You showed that you are using the following config: receivers: ... - name: 'teams' webhook_configs: - url: 'https://company.webhook.office.com/webhookb2/XXX' send_resolved: true This means you are attempting to send the alertmanager webhook message directly to Microsoft (*.office.com

Re: [prometheus-users] Re: Joining two metrics in promtheus and having all labels from both

2023-03-07 Thread Nick Carlton
Ah ok, I would have thought this was one-to-one because for each metric that exists in ltmPoolMbrStatusAvailState there is one exact match within ltmPoolMbrStatusEnabledState. Not multiple. Unless I’m reading the definitions wrong. Thanks On Tue, 7 Mar 2023 at 12:06, Brian Candler wrote: > To

[prometheus-users] Re: {\"message\":\"Not Found\"}\n" - mail Alertmanager (and prometheus-msteams)

2023-03-07 Thread Brian Candler
This one? https://github.com/prometheus-msteams/prometheus-msteams As your problem is with configuring that piece of software, then you're probably best off asking your question there. ISTM you'd be best off starting with their supplied default-message-card.tmpl, and get it working with that, b

[prometheus-users] Re: Joining two metrics in promtheus and having all labels from both

2023-03-07 Thread Brian Candler
To pick up labels from the other side, you need to list them as part of your group_right. e.g. (untested) ltmPoolMbrStatusAvailState * on(instance,ltmPoolMbrStatusNodeName,ltmPoolMbrStatusPoolName,ltmPoolMbrStatusPort) group_right(*foo,bar,baz*) ltmPoolMbrStatusEnabledState will pick up label

[prometheus-users] Re: {\"message\":\"Not Found\"}\n" - mail Alertmanager (and prometheus-msteams)

2023-03-07 Thread Kolja Krückmann
I am trying to use prometheus-msteams as middleware. there I am using my .tmpl earlyer attached. The error sais that it's missing a summary or text. But my tmpl should have that. any suggestions? Brian Candler schrieb am Dienstag, 7. März 2023 um 11:29:31 UTC+1: > On Tuesday, 7 March 2023 at 08

[prometheus-users] Re: {\"message\":\"Not Found\"}\n" - mail Alertmanager (and prometheus-msteams)

2023-03-07 Thread Brian Candler
On Tuesday, 7 March 2023 at 08:48:37 UTC Kolja Krückmann wrote: I just checked and prom is using the correct yml. I just missed that I actually changed the alertmanager to localhost:9093 - thats why my error sais localhost:9093 not 127.0.0.1:9093 Furthermore, I actually don't know why but I

[prometheus-users] Re: Joining two metrics in promtheus and having all labels from both

2023-03-07 Thread Nick Carlton
Thanks for your response. If I do `ltmPoolMbrStatusAvailState * on(instance,ltmPoolMbrStatusNodeName,ltmPoolMbrStatusPoolName,ltmPoolMbrStatusPort) group_right ltmPoolMbrStatusEnabledState` I get: `{instance="ltm01", job="f5_ltm_test", ltmPoolMbrStatusEnabledState="1", ltmPoolMbrStatusNodeNam

[prometheus-users] Re: {\"message\":\"Not Found\"}\n" - mail Alertmanager (and prometheus-msteams)

2023-03-07 Thread Kolja Krückmann
Thanks for the quick response. I just checked and prom is using the correct yml. I just missed that I actually changed the alertmanager to localhost:9093 - thats why my error sais localhost:9093 not 127.0.0.1:9093 Furthermore, I actually don't know why but I just restartet the prom.exe an

[prometheus-users] Re: Joining two metrics in promtheus and having all labels from both

2023-03-07 Thread Brian Candler
It has to be done in a query - the relabelling phase of a scrape job cannot see other metrics. What you are looking for is one-to-many queries, which can pick up labels from the "one" side and apply them to the "many": https://www.robustperception.io/how-to-have-labels-for-machine-roles https:/