[prometheus-users] Re: snmp_exporter. server returned HTTP status 500 Internal Server Error

[Brian Candler]

Aside: curl is useful for debugging:
curl -g `http://127.0.0.1:9116/snmp?module=if_mib=x.x.x.x=true`

However in your case, you've found the problem: you need to set a community 
string.

Unfortunately, snmp_exporter makes it hard to set this, because the MIB and 
the device authentication are bundled together in the "module".  Here's the 
easiest way I've found to do this:

1. Edit snmp.yml (that was supplied with snmp_exporter, or that you 
generated)

2. Find the MIB that you're interested in using: in this case if_mib.  
Modify the first line like this:

*if_mib:*

becomes

*if_mib: _mib*

(this is called an "anchor" in YAML)

3. Add to the very *end* of the file:





*if_mib_secret:  <<: *if_mib  version: 2  auth:community: BlahBlah123*

What you're doing is taking a copy of the "if_mib" section, and then 
overriding some of the parameters (the 'auth' and 'version' keys) without 
changing anything else.

[Aside: you should be able to do steps 1-3 using the snmp generator 
instead, but you'll have to duplicate the if_mib section in generator.yml 
instead, and then run the generator]

4. Restart snmp_exporter, and check this works using curl:


*curl -g 
`http://127.0.0.1:9116/snmp?module=if_mib_secret=x.x.x.x=true`*

5. Change your scrape job for these devices to use module if_mib_secret 
instead of if_mib.

Oddly, your scrape job shows a static target of only 127.0.0.1, i.e. it 
looks like you're sending queries to the snmpd running on prometheus 
itself, not to any remote devices.  I would suggest that you use the 
file_sd mechanism, and also that you pick up the module from this, so that 
a single scrape job can handle any combination of targets and SNMP modules.

  - job_name: snmp
metrics_path: /snmp
file_sd_configs:
  - files:
  - /etc/prometheus/targets.d/snmp_targets.yml
relabel_configs:
  - source_labels: [__address__]
target_label: instance
  - source_labels: [__address__]
target_label: __param_target
  - source_labels: [module]
target_label: __param_module
  - target_label: __address__
replacement: 127.0.0.1:9116  # SNMP exporter

then in snmp_targets.yml you can put:

- labels:
module: if_mib
  targets:
- foo
- bar
- labels:
module: if_mib_secret
  targets:
- baz
- qux


-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/aa9f124e-e927-4ad6-91c5-e261d3c52312n%40googlegroups.com.

[prometheus-users] Re: failed to sync configmap cache: timed out waiting for the condition

[Brian Candler]

> "MountVolume.SetUp failed for volume "prometheus-config-volume" : failed 
to sync configmap cache: timed out waiting for the condition"

That appears to be either a Kubernetes or a Longhorn error.  Either way, 
it's not anything that Prometheus is responsible for.

On Monday, 28 February 2022 at 16:45:04 UTC rnanda...@gmail.com wrote:

> Hi Team,
>
> I have a prometheus container running in k8s and is using longhorn as 
> storage to store config files and DB. The Prometheus container starts 
> successfully and could be accessed by Prometheus UI as well for around 2-3 
> mins and crashes immediately. When I describe the k8s pod, it shows below 
> error message during crash.
>
> "MountVolume.SetUp failed for volume "prometheus-config-volume" : failed 
> to sync configmap cache: timed out waiting for the condition"
>
> The configmap being referred by prometheus-config-volume does exist. So, I 
> am not sure what is going wrong here as prometheus logs too didn't have any 
> helpful message for. Could someone help me find the issue here?
>
> Regards,
> Nanda
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/18b3410d-6d84-4c29-9b38-03f929aa2395n%40googlegroups.com.

[prometheus-users] failed to sync configmap cache: timed out waiting for the condition

[Nanda kumar R]

Hi Team,

I have a prometheus container running in k8s and is using longhorn as 
storage to store config files and DB. The Prometheus container starts 
successfully and could be accessed by Prometheus UI as well for around 2-3 
mins and crashes immediately. When I describe the k8s pod, it shows below 
error message during crash.

"MountVolume.SetUp failed for volume "prometheus-config-volume" : failed to 
sync configmap cache: timed out waiting for the condition"

The configmap being referred by prometheus-config-volume does exist. So, I 
am not sure what is going wrong here as prometheus logs too didn't have any 
helpful message for. Could someone help me find the issue here?

Regards,
Nanda

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/45eebf7b-746d-470c-aab7-6d8ff1839fben%40googlegroups.com.

Re: [prometheus-users] Scrape_interval

[Stuart Clark]

On 28/02/2022 11:41, BHARATH KUMAR wrote:

Hello all,

Q1) What will happen if we set scrape interval as 5 minutes or 10 minutes?
Q2) I am observing some misbehavior when we set different scrape 
interval. what is the reason behind that?


The standard maximum scape interval is about 2 minutes, due to staleness 
- less frequent scrapes would likely cause series to be marked as stale 
and therefore disappear from alerts/dashboards.


--
Stuart Clark

--
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/0c4fa543-07ab-a119-5344-c10d514055a2%40Jahingo.com.

[prometheus-users] Inhibit Rules

[Raslen Ben Hadj Salah]

Hi, 

Using Prometheus v2.27.1 with Alertmanager v0.22.2 deployed by the 
kube-prometheus-stack Helm chart (adjusted by Rancher) on a K8s-cluster we 
want to achieve, that all alerts are automatically muted when a certain 
alert is firing. 

In our case, all alerts should be muted when the availability check (using 
the up-metric) triggers an alert, so that in case of a server failure we 
only get an alerting that the server is down. Although this is a common 
concern in our eyes, unfortunately we could not find any helpful advice on 
how to implement it. Apparently, the usual procedure here is to put a 
fake-alerting along e.g. 
https://stackoverflow.com/questions/57305247/how-to-inhibit-alerts-outside-business-hours-with-prometheus-alertmanager
 

 

What is the basic recommended procedure here? 

Regards, 

Raslen 

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/8789af47-680f-4371-879e-7cb2c281443cn%40googlegroups.com.

[prometheus-users] Errors building sample remote storage application

[hartfordfive]

I'm attempting to build the following sample application for remote storage:

https://github.com/prometheus/prometheus/blob/release-2.33/documentation/examples/remote_storage/example_write_adapter/server.go

Although when initializing my mod file and attempting to run "go mod tidy", 
I always get the following error:

github.com/hartfordfive/prom-remote-storage-example imports
github.com/prometheus/prometheus/storage/remote imports
github.com/prometheus/prometheus/config imports
github.com/prometheus/prometheus/discovery/config imports
github.com/prometheus/prometheus/discovery/azure imports
github.com/Azure/azure-sdk-for-go/arm/compute: module 
github.com/Azure/azure-sdk-for-go@latest found (v62.0.0+incompatible), but 
does not contain package github.com/Azure/azure-sdk-for-go/arm/compute
github.com/hartfordfive/prom-remote-storage-example imports
github.com/prometheus/prometheus/storage/remote imports
github.com/prometheus/prometheus/config imports
github.com/prometheus/prometheus/discovery/config imports
github.com/prometheus/prometheus/discovery/azure imports
github.com/Azure/azure-sdk-for-go/arm/network: module 
github.com/Azure/azure-sdk-for-go@latest found (v62.0.0+incompatible), but 
does not contain package github.com/Azure/azure-sdk-for-go/arm/network


Seems like this dependency is missing although I can't seem to solve this.  
 Any help would be appreciated.

Thanks.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/b1ab28e1-98e9-4cef-815c-da0c939a27acn%40googlegroups.com.

Re: [prometheus-users] Drop label for single metric only, not for entire scrape job

[Julien Pivotto]

On 25 Feb 03:47, 'Peter Burrows' via Prometheus Users wrote:
> Hi folks, I have a scenario where I wish to drop label *L* from metric *A*. 
> I easily found documentation on how to drop label *L* from all metrics 
> within a job, however in my case I have another metric we'll call *B* 
> that's in the same scrape job and for metric *B* label *L* is useful and 
> shouldn't be dropped. I've been unable to find anything online to help with 
> such a scenario. I basically want to drop a label for a specific metric but 
> leave it in place for all other metrics in the same job. Is anyone aware of 
> a way to achieve this, perhaps with metric_relabel_configs?

You can do:

metric_relabel_configs:
- source_labels: [__name__,L]
  target_label: L
  replacement: ""
  regex: 'A,.*'

Regards

> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/prometheus-users/2b6620a2-36e8-48c5-ab87-2a1faf30fe83n%40googlegroups.com.


-- 
Julien Pivotto
@roidelapluie

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/20220228125702.GA349220%40hydrogen.

[prometheus-users] Drop label for single metric only, not for entire scrape job

['Peter Burrows' via Prometheus Users]

Hi folks, I have a scenario where I wish to drop label *L* from metric *A*. 
I easily found documentation on how to drop label *L* from all metrics 
within a job, however in my case I have another metric we'll call *B* 
that's in the same scrape job and for metric *B* label *L* is useful and 
shouldn't be dropped. I've been unable to find anything online to help with 
such a scenario. I basically want to drop a label for a specific metric but 
leave it in place for all other metrics in the same job. Is anyone aware of 
a way to achieve this, perhaps with metric_relabel_configs?

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/2b6620a2-36e8-48c5-ab87-2a1faf30fe83n%40googlegroups.com.

[prometheus-users] snmp_exporter. server returned HTTP status 500 Internal Server Error

[Hemedi Salim]

Hi all,

I am babie to this world(prometheus, snmp_exporter).  I recently configure 
Prometheus/snmp_exporter to monitors swithes. I want to monitor 5 switches  
but Prometheus scrap metrics form 2 switches and the other three shows an 
error ( *server returned HTTP status 500 Internal Server Error).*

am using the default smnp.yml which I generate form snmp generator. and my 
Prometheus.yml setting look like this:

 job_name: snmp
metrics_path: /snmp params: module: [if_mib] static_configs: - targets: - 
127.0.0.1 relabel_configs: - source_labels: [__address__] target_label: 
__param_target - source_labels: [__param_target] target_label: instance - 
target_label: __address__ replacement: 127.0.0.1:9116 # URL as shown on the 
UI


I tried to do snmp walk to troubleshout with command (snmpwalk -v2c -c 
public x.x.x.x) the x.x.x.x represent switch ip. but it gives me time out 
error. 

then I figure out the switches which gives error has snmp community string 
set up. so I tried to run snmpwalk again replacing default public communit 
string with the community string set on this switches and I was able to 
snmpwalk on these swathes.

my question is, how can I make the Prometheus scrape from these remaining 
switches, is there any setting am missing to facilitate that?

please help.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/f9e4f237-11f3-4873-86b2-4a2bb8968a2an%40googlegroups.com.

[prometheus-users] Scrape_interval

[BHARATH KUMAR]

Hello all,

Q1) What will happen if we set scrape interval as 5 minutes or 10 minutes?
Q2) I am observing some misbehavior when we set different scrape interval. 
what is the reason behind that?


Could you please answer those questions?

Thanks in advance.

Regards,
Bharath kumar

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/c52853a4-502a-4b80-bd18-a0b8452c6915n%40googlegroups.com.