[prometheus-users] Re: Saving alerts generated in a database for further procesing like offline querying and ML

[Adarsh Kumar Pandey]

Thanks for the hint I will try that but I'm not much of a coder so I will 
have to learn to do that ..meanwhile if you could make my life easier by 
pointing me to the right direction I will be humbled by that. 

Rgards,
Adarsh

On Thursday, 2 April 2020 08:48:02 UTC+5:30, Adarsh Kumar Pandey wrote:
>
> I wanted to store the alerts generated by prometheus into a separate 
> database and I know that alertsnitch provides this feature but what if I 
> want to use another database then how to do that?
> Does prometheus logs the alerts firing internally and how to access that 
> any idea?
>
> Thanks and much Respect for your support guyz !!!
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/b5b89581-8260-46f4-bc9d-c48f232190bc%40googlegroups.com.

[prometheus-users] Re: Saving alerts generated in a database for further procesing like offline querying and ML

[Matt Palmer]

On Wed, Apr 01, 2020 at 08:18:02PM -0700, Adarsh Kumar Pandey wrote:
> I wanted to store the alerts generated by prometheus into a separate 
> database and I know that alertsnitch provides this feature but what if I 
> want to use another database then how to do that?
> Does prometheus logs the alerts firing internally and how to access that 
> any idea?

Write a small service that accepts the HTTP requests that Prometheus sends
for alerts and writes them to a database of your choice, configure
Prometheus to send alerts to that service, job done.

- Matt

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/20200402033817.656vqi4pe53we4xh%40hezmatt.org.

[prometheus-users] Saving alerts generated in a database for further procesing like offline querying and ML

[Adarsh Kumar Pandey]

I wanted to store the alerts generated by prometheus into a separate 
database and I know that alertsnitch provides this feature but what if I 
want to use another database then how to do that?
Does prometheus logs the alerts firing internally and how to access that 
any idea?

Thanks and much Respect for your support guyz !!!

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/6dc1b378-e7e7-4a2f-bb56-98ed7095ca68%40googlegroups.com.

[prometheus-users] Re: Intergrating Fortigate SNMP with promethes

[Andrew Meyer]

You need a new scrape job for each device type.  However I am also 
struggling with the same issue.  I am trying to figure out if I should add 
fortinet or fortigate to the generate.yml

On Thursday, January 30, 2020 at 1:54:22 PM UTC-6, Elvin cbl wrote:
>
> Hi Team ,
>
> I tried to intergrate Fortigate SNMP with prometheus but i am unable to 
> get a proper connection
> Steps Followed
>
> In Fortigate enabled SNMP 
> In Network --> interface added the SNMP to the local network
>
> on my prometheus linux machine i did wire shark and i was getting the logs
>
> Step on SNMP_exporter 
>  i used the github link to get the latest Release 
> https://github.com/prometheus/snmp_exporter 
>  downloaded it and ran the file using ./snmpexporter
>
> in my prometheus.yml file added the below
>
> scrape_configs:
>   - job_name: 'snmp'
> static_configs:
>   - targets:
> - 192.168.1.2  # SNMP device.
> metrics_path: /snmp
> params:
>   module: [if_mib]
> relabel_configs:
>   - source_labels: [__address__]
> target_label: __param_target
>   - source_labels: [__param_target]
> target_label: instance
>   - target_label: __address__
> replacement: 127.0.0.1:9116  # The SNMP exporter's real hostname:port
>
>
> and also tried to run  SNMP Exporter Config Generator
>
> sudo apt-get install unzip build-essential libsnmp-dev # Debian-based distros
>
>
> go get github.com/prometheus/snmp_exporter/generator
> cd ${GOPATH-$HOME/go}/src/github.com/prometheus/snmp_exporter/generator
> go build
> make mibs
>
>
> The FORTIGATE mibs that i downloaded from the fortigte UI were i enabled 
> SNMP i added the same in /usr/share/snmp/mibs/FORTINET-CORE-MIB.mib
>
> Im not able to run the ./generator generate to crearte the snmp.yml file
>
> i also edited the generate.yml file
>
> fortigate_snmp:
>   walk:
> - ifXTable
> - fgVpn
> - fgSystem
> - fgIntf
> - fgInetProto
>   version: 3
>   max_repetitions: 25
>   timeout: 10s
>   auth:
> username: your_username  # Required, no default. -u option to NetSNMP.
>   
>
> security_level: authNoPriv  # Defaults to noAuthNoPriv. -l option to 
> NetSNMP.  
> 
> # Can be noAuthNoPriv, authNoPriv or 
> authPriv. 
>   
> password: your_password  # Has no default. Also known as authKey, -A 
> option to NetSNMP.
>  
> # Required if security_level is authNoPriv or authPriv.   
>   
>
> auth_protocol: SHA  # MD5 or SHA, defaults to SHA. -a option to NetSNMP.  
>   
>
>   # Used if security_level is authNoPriv or authPriv. 
>
>
> Please help 
> Regards 
> Elvin Fernandez
>
>
>
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/25d22ab5-2ada-49f8-98f2-14c4dcb99eae%40googlegroups.com.

Re: [prometheus-users] Federation for cloudwatch_exporter - best practice?

[Albert Serrallé]

That's how is usually done in my company (and I suspect that's very 
prevalent but *wrong* in the end, if I understand that link correctly). In 
my company, at team level we have a lot of metrics scraped into a single 
prometheus. Then, at company level we just "replicate" everything with the 
corresponding team label.

The *official* recommendation would be to create aggregation rules in the 
"team" prometheus, and federate that from the "company" prometheus, right?

On Wednesday, 1 April 2020 21:42:20 UTC+2, Brian Brazil wrote:
>
> On Wed, 1 Apr 2020 at 20:11, Albert Serrallé  > wrote:
>
>> Hello,
>>
>> I'm trying to figure out how to federate my cloudwatch_exporter metrics. 
>>
>> The default metrics are 10m old, I understand that this is needed because 
>> cloudwatch consolidate metrics over time. My prometheus is scraping the 
>> exporter and saving the values with the original timestamp. Federation 
>> queries (as they are instant queries) cannot retrieve those metrics.
>>
>> So, I think that my only options are:
>>
>>- set_timestamp -> false (fake illusion of real time metric, which 
>>are really 10m old)
>>- delay_seconds -> 1s-5m (metric might not be consolidated in 
>>cloudwatch)
>>
>> There's anything else I can try? I don't like any of those two methods 
>> for the exposed reasons. I've been looking at the changes in stale logic 
>> for 2.0, but I don't know if this applies to me or how to do it: 
>>
>> If you're federating instance-level metrics, switch to only aggregated 
>>> metrics
>>>
>>
>> So, what's the recommended way to overcome this?
>>
>
> You can't with federation, it's merely a more obvious case where 
> instance-level metrics don't work. See 
> https://www.robustperception.io/federation-what-is-it-good-for
>
> In this case if you need the metrics in a different Prometheus, get that 
> Prometheus to scrape the cloudwatch exporter.
>
> Brian
>  
>
>>
>> Many thanks.
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to promethe...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/prometheus-users/030245a2-9e9e-4493-9a39-eebf53753e47%40googlegroups.com
>>  
>> 
>> .
>>
>
>
> -- 
> Brian Brazil
> www.robustperception.io
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/f03663b6-1dba-4930-a016-077a2328745c%40googlegroups.com.

[prometheus-users] Re: Federation for cloudwatch_exporter - best practice?

[Albert Serrallé]

For the records, it looks like there's a third option:

   - honor_timestamps -> false (fake ilusion of real time metric)


On Wednesday, 1 April 2020 21:11:29 UTC+2, Albert Serrallé wrote:
>
> Hello,
>
> I'm trying to figure out how to federate my cloudwatch_exporter metrics. 
>
> The default metrics are 10m old, I understand that this is needed because 
> cloudwatch consolidate metrics over time. My prometheus is scraping the 
> exporter and saving the values with the original timestamp. Federation 
> queries (as they are instant queries) cannot retrieve those metrics.
>
> So, I think that my only options are:
>
>- set_timestamp -> false (fake illusion of real time metric, which are 
>really 10m old)
>- delay_seconds -> 1s-5m (metric might not be consolidated in 
>cloudwatch)
>
> There's anything else I can try? I don't like any of those two methods for 
> the exposed reasons. I've been looking at the changes in stale logic for 
> 2.0, but I don't know if this applies to me or how to do it: 
>
> If you're federating instance-level metrics, switch to only aggregated 
>> metrics
>>
>
> So, what's the recommended way to overcome this?
>
> Many thanks.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/3ca678a8-9ffb-4a47-9372-76297f56e136%40googlegroups.com.

[prometheus-users] Federation for cloudwatch_exporter - best practice?

[Albert Serrallé]

Hello,

I'm trying to figure out how to federate my cloudwatch_exporter metrics. 

The default metrics are 10m old, I understand that this is needed because 
cloudwatch consolidate metrics over time. My prometheus is scraping the 
exporter and saving the values with the original timestamp. Federation 
queries (as they are instant queries) cannot retrieve those metrics.

So, I think that my only options are:

   - set_timestamp -> false (fake illusion of real time metric, which are 
   really 10m old)
   - delay_seconds -> 1s-5m (metric might not be consolidated in cloudwatch)

There's anything else I can try? I don't like any of those two methods for 
the exposed reasons. I've been looking at the changes in stale logic for 
2.0, but I don't know if this applies to me or how to do it: 

If you're federating instance-level metrics, switch to only aggregated 
> metrics
>

So, what's the recommended way to overcome this?

Many thanks.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/030245a2-9e9e-4493-9a39-eebf53753e47%40googlegroups.com.

[prometheus-users] Blackbox exporter does not available on network

[Gergely Fehérvári]

Hi,

I started my blackbox exporter on a MacOs computer, and it is not available 
from other devices through the network. When I start the blackbox exporter 
process, it does not ask for port listen permission tho. What do I need to 
do, to access the exporter from different devices?

Here is more detail about the issue: 
https://github.com/prometheus/blackbox_exporter/issues/590

Thanks
Gergo

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/f0e4337c-04ba-4d12-8f4b-038424954322%40googlegroups.com.

Re: [prometheus-users] Getting status of services in CentOS 6.

[Yagyansh S. Kumar]

Alas, only if migration from CentOS 6 to 7 was in my hand :P. 
But thanks for the suggestion, will give process exporter a try.

On Wednesday, April 1, 2020 at 2:50:39 AM UTC+5:30, Christian Hoffmann 
wrote:
>
> Hi, 
>
> On 3/31/20 12:22 PM, Yagyansh S. Kumar wrote: 
> > Hi. We have systemd collector in Prometheus that enables us to get the 
> > metrics - status of services from systemd. This works well in CentOS 7. 
> > But CentOS 6 does not have systemd, hence, the collector keeps throwing 
> > error. Is there any alternative for this for CentOS 6? 
>
> I don't think there is any alternative which is as integrated/powerful 
> as systemd is on CentOS 7 and onwards. 
> There are lots of service supervisors. Technically, CentOS 6 uses 
> Upstart. However, almost no service seemed to make use of it, not even 
> system ones. Instead, most of the services were plain old SysV init 
> scripts. For those, there is no such supervision or integrated API for 
> monitoring. 
>
> Basically, I would go for process-based monitoring in this case (using 
> process_exporter). 
>
> On the other hand, support for CentOS 6 will end in only a few months so 
> I guess any efforts should be invested regarding migrations towards 
> CentOS 7 or 8... 
>
> Kind regards, 
> Christian 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/9b7330d0-ee3c-4eb6-a6b8-9a2f32107095%40googlegroups.com.

[prometheus-users] Re: Getting status of services in CentOS 6.

[Yagyansh S. Kumar]

Hi Joey,

As far as I know, if you have created a service file(in 
/etc/systemd/system/) for your custom service, you don't need to whitelist 
it separately, node_exporter's systemd collector itself will scrape the 
metrics for your custom service. 

On Wednesday, April 1, 2020 at 3:37:51 AM UTC+5:30, Joey Jojo wrote:
>
> Hey Yagyansh,
>
> Sorry to hijack your post but I am wondering if you made this work in Cent 
> OS, if you could help me with my issue with monitoring Systemd Services. 
> Basically I need to add custom services to the 
> --collector.systemd.unit-whitelist="(apache2|ssh|rsyslog|nginx).service" 
> but not sure how to do it? 
>
> Again sorry to hijack your post and sorry can't add anything to help
>
> On Tuesday, March 31, 2020 at 6:22:17 AM UTC-4, Yagyansh S. Kumar wrote:
>>
>> Hi. We have systemd collector in Prometheus that enables us to get the 
>> metrics - status of services from systemd. This works well in CentOS 7.
>> But CentOS 6 does not have systemd, hence, the collector keeps throwing 
>> error. Is there any alternative for this for CentOS 6?
>> Also, in my 6 VMs the URL http://localhost:9001 was giving 404 status 
>> and supervisord collector was enabled, and the load of the machines sky 
>> rocketed and node_exporter started consuming over 98%+ CPU. I haven't seen 
>> any abnormality in the server, so can this be cause for the high load, 
>> because I think these kind of bugs were fixed in late 2017.
>>
>> Thanks!
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/87479859-c9d5-4513-999e-28bcabc3aba8%40googlegroups.com.

Re: [prometheus-users] Re: Data retention policy

[REMI DRUILHE]

No problem :)

At least it is pretty clear for me now. I will discuss with my DPO again to 
have its point of view on this. Maybe I am over-reacting about those data.

Anyway, thanks for the help and the talk. It has been very interesting.

Le mercredi 1 avril 2020 11:12:01 UTC+2, Stuart Clark a écrit :
>
> On 2020-04-01 09:58, REMI DRUILHE wrote: 
> > You are right, most of the time this kind de-anonymisation is extreme. 
> > And right again when you say that there is no personal data stored in 
> > Prometheus. 
> > 
> > I am also not a lawyer but I know from my DPO that the national data 
> > protection authority (NDPA) might be very very very meticulous, 
> > especially in my domain of video processing... We had several meetings 
> > about it and we had to review our data processing multiple time. I was 
> > just looking for a way to delete data with a hard deadline if the NDPA 
> > say that the current solution is not good enough (the one with 
> > storage.tsdb.retention.time option). I think it is better to come with 
> > an answer than saying that we did not thought about it. 
>
> Unfortunately there are no guarantees around deletion. 
>
> In addition to the fuzziness around exactly when a block might be 
> removed you can also end up with data files hanging around in certain 
> error scenarios (e.g. tmp files if there are issues loading the WAL on 
> startup or during block rotation) 
>
> > 
> > Le mardi 31 mars 2020 17:51:31 UTC+2, Stuart Clark a écrit : 
> > 
> >> No that sounds fairly normal. One thing to note is that those 
> >> timestamps are not the times the methods were called. They are when 
> >> Prometheus scraped your application. So if you scrape once a minute 
> >> the actual call could have been at any point during that minute. 
> >> Equally if there are multiple calls during that minute you'd have no 
> >> idea when they happened either. 
> >> 
> >> I'm not a lawyer or GDPR expert, but I think the type of extreme 
> >> de-anonymisation you are suggesting is not generally something you'd 
> >> be expected to be worrying about. Equally even if you do have an 
> >> idea of who might have called an API there still isn't any personal 
> >> data in Prometheus. 
> >> 
> >> On 31 March 2020 15:27:36 BST, REMI DRUILHE  
> >> wrote: 
> >> In our code, we are using a counter to count the accesses to the 
> >> various methods of the API. We have one counter per method. We do 
> >> not store the timestamp. But when we ask Prometheus with a 
> >> "query_range" (see request below), it returns the list of all the 
> >> methods that have been accessed. 
> >> 
> >> curl 
> >> 
> > '
> http://172.22.0.15:9090/api/v1/query_range?query=bea_nb_request=2020-03-31T00:01:00.000Z=2020-03-31T17:00:00.000Z=60s
>  
> >> [1]' 
> >> 
> >> For each of our API method, it also returns a list of key-value 
> >> where the key is the timestamp and the value is the value of the 
> >> counter at that time (see example below). Thus, in some way, you are 
> >> able to track when the method has been called. And if our system is 
> >> used by a single user, then it is easy to follow which methods he 
> >> called. It is a bit twisted, but the national data protection 
> >> authority might also be twisted sometimes... But according to your 
> >> previous answers, maybe we did not used the counter in a proper way 
> >> and we should change the way it is designed. 
> >> 
> >> { 
> >> "status":"success", 
> >> "data":{ 
> >> "resultType":"matrix", 
> >> "result":[ 
> >> { 
> >> "metric":{ 
> >> "__name__":"bea_nb_request", 
> >> "action":"my_api_method", 
> >> "instance":"bea:8081", 
> >> "job":"bea" 
> >> }, 
> >> "values":[ 
> >> [ 
> >> 1585663440, 
> >> "1" 
> >> ], 
> >> [ 
> >> 1585663500, 
> >> "2" 
> >> ], 
> >> [ 
> >> 1585663560, 
> >> "3" 
> >> ], 
> >> [ 
> >> 1585663620, 
> >> "3" 
> >> ], 
> >> [ 
> >> 1585663680, 
> >> "3" 
> >> ], 
> >> [ 
> >> 1585663740, 
> >> "3" 
> >> ], 
> >> [ 
> >> 1585663800, 
> >> "3" 
> >> ], 
> >> [ 
> >> 1585663860, 
> >> "3" 
> >> ] 
> >> ] 
> >> }, 
> >> others_api_methods... 
> >> } 
> >> ] 
> >> } 
> >> } 
> >> 
> >> Le mardi 31 mars 2020 13:40:03 UTC+2, Stuart Clark a écrit : 
> >> How are you storing the timestamp? Is that in a label or a metric 
> >> value as the last call to the API? 
> >> 
> >> In general these are sounding like you are trying to store events 
> >> within Prometheus rather than metrics. Normally you'd not have a 
> >> timestamp but a counter of the number of calls to the API. 
> >> 
> >> On 31 March 2020 12:27:38 BST, REMI DRUILHE  
> >> wrote: 
> >> 
> >> Le lundi 30 mars 2020 16:37:11 UTC+2, Brian Candler a écrit : 
> >> On Monday, 30 March 2020 09:34:01 UTC+1, REMI DRUILHE wrote: 
> >> In our context, Prometheus is storing system metrics and business 
> >> metrics, especially the number of accesses to the methods of our 
> >> API. 
> > 
> > That presumably is an aggegate of all calls to a particular method. 
> > 
> > If you recorded counts as separate 

[prometheus-users] Re: Can Prometheus be configured for multiple Kubernetes clusters?

[Surbhi]

hello Nayab Zehra,
  Can i know if you found any alternative solution to your problem. I m 
facing the same issues. It would be very helpful if i get some guidance 
from you regarding this issue.
On Wednesday, December 20, 2017 at 7:23:54 PM UTC+5:30, nayab zehra wrote:
>
> I am trying to configure Prometheus outside of the Kubernetes cluster and 
> I want to monitor more than one Kubernetes cluster, however in the 
> Kubernetes sd Config (for outside cluster monitoring) *api_server *option 
> is there. I have configured one cluster successfully, but *api_server* 
> only takes one *host.*
> I know one solution to achieve this is to write multiple jobs for multiple 
> clusters, but what i need is that to have one Kubernetes sd config job and 
> same job can scrape multiple Kubernetes clusters.
> Is there any way to achieve that?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/a19552b0-0e93-4f2f-889b-ad5dc9100763%40googlegroups.com.

[prometheus-users] Problems with starting JMX exporter

[Asulan Kenzhebekov]

Hello guys! i hava a ZooKeeper and Kafka cluster.
So i have script to run the ZooKeeper instance, and i add the following 
line as EXTRA_ARGS
EXTRA_ARGS=${EXTRA_ARGS-'-name kafkaServer -loggc 
"-javaagent:/opt/jmx-exporter/jmx-exporter.jar=7071:/etc/jmx-exporter/zookeeper.yml"'}

and when i start my ZooKeeper instance i get the following ERROR message in 
my log file
Error: Could not find or load main class 
"-javaagent:.opt.jmx-exporter.jmx_exporter_javaagent.jar=7071:.etc.jmx-exporter.zookeeper.yml"

here is my zookeeper.yml

rules:
  - pattern: 
"org.apache.ZooKeeperService<>(\\w+)"
name: "zookeeper_$2"
  - pattern: "org.apache.ZooKeeperService<>(\\w+)"
name: "zookeeper_$3"
labels:
  replicaId: "$2"
  - pattern: "org.apache.ZooKeeperService<>(\\w+)"
name: "zookeeper_$4"
labels:
  replicaId: "$2"
  memberType: "$3"
  - pattern: "org.apache.ZooKeeperService<>(\\w+)"
name: "zookeeper_$4_$5"
labels:
  replicaId: "$2"
  memberType: "$3"



P.S i use the https://alex.dzyoba.com/blog/jmx-exporter/ as reference to 
install and use the JMX_EXPORTER for Monitor Kafka and ZooKeeper status

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/a406d7ee-e9f0-4cf4-b1bd-109029474b04%40googlegroups.com.

[prometheus-users] Re: Too many scrape configs on prometheus config file

[Ishvar B]

Hi Brain,

Thanks for you reply. I was able to figure out yesterday. Is there a way to 
give basic auth details in the yaml file. I have been trying to add basic 
auth and prometheus is not picking them up. sample config.

- targets: ['**.**.**']

  scheme: "https" 

  basic_auth: 

username: ""

password: ""

  labels:

job: '**'

Thanks
Eswar

On Tuesday, 31 March 2020 16:59:52 UTC+2, Ishvar B wrote:
>
> Hi,
>
> I have a prometheus set up which has too many (roughly 50 static configs) 
> and this number would increase as we have lot of vms still be scraped for 
> prometheus. The file is becoming huge , especially when some applications 
> require basic auth. is there any best practice to maintain the config file 
> as with such huge file some times changes would cause accidental errors.
>
> Thanks
> Eswar
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/b0a038f1-29a5-4c77-8f58-5630969c2abb%40googlegroups.com.

[prometheus-users] Re: Monitor number of seconds since metric change as prometheus time series

[Weston Greene]

In the stackoverflow post about this same topic, I was encouraged to reduce 
my evaluation frequency since `last-update` was likely going stale by the 
default TTL (Time To Live) of 5 minutes.

Now I can't get passed the `vector contains metrics with the same labelset 
after applying rule labels`.

I do add labels in the recording rule:
```
  stat: true
  monitor: false
```

I believe this is because `last-update` already has all the labels that 
`metric-name` has plus the labels that the recording rule adds, so when the 
`or` is triggered `last-update` conflicts since it already has the labels.

How do I get around this? Thank you again for your creativity!


On Monday, March 30, 2020 at 10:23:20 AM UTC-4, Weston Greene wrote:
>
> This was already partially answered in 
> https://stackoverflow.com/questions/54148451
>
> But not sufficiently, so I'm asking here and in the Stack Overflow: 
> https://stackoverflow.com/questions/60928468
>
> Here is the image of the graph: 
>
> [image: Screen Shot 2020-03-30 at 06.18.07.png]
>
>
>
> On Monday, March 30, 2020 at 10:21:01 AM UTC-4, Weston Greene wrote:
>>
>>
>> I have the Recording rule pattern:
>> ```yaml
>>   - record: last-update
>> expr: |
>>   timestamp(changes(metric-name[450s]) > 0)
>> or
>>   last-update
>> ```
>>
>> However, that doesn't work. The `or last-update` part doesn't return a 
>> value.
>>
>> I have tried using an offset,
>> ` or (last-update offset 450s)`, 
>> to no avail.
>>
>>
>> My evaluation frequency is 5 minutes (the frequency that prometheus runs 
>> my Recording rules). I tried the 7.5 minutes offset because I theorized 
>> that the OR was attempting to write last-update as last-update but 
>> last-update was null in that second; if the OR were to attempt writing 
>> last-update as the value it was during it's previous evaluation, then it 
>> should find a value in last-update, but that returned no value as well.
>>
>>
>> This is what the metric looks like graphed: 
>>
>> [choppy rather than a complete staircase][1] (I don't have enough 
>> reputation to post pictures...)
>>
>>
>>
>> Thank you in advance for your help.
>>
>> Why I care:
>> If a time series plateaus for an extended period of time then I want to 
>> know as that may mean it has begun to fail to return accurate data.
>>
>>
>>   [1]: I think the image link is preventing me from posting
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/f50bb7cf-ba77-49a7-89b3-733b84703e41%40googlegroups.com.

Re: [prometheus-users] Re: Data retention policy

[Stuart Clark]

On 2020-04-01 09:58, REMI DRUILHE wrote:

You are right, most of the time this kind de-anonymisation is extreme.
And right again when you say that there is no personal data stored in
Prometheus.

I am also not a lawyer but I know from my DPO that the national data
protection authority (NDPA) might be very very very meticulous,
especially in my domain of video processing... We had several meetings
about it and we had to review our data processing multiple time. I was
just looking for a way to delete data with a hard deadline if the NDPA
say that the current solution is not good enough (the one with
storage.tsdb.retention.time option). I think it is better to come with
an answer than saying that we did not thought about it.


Unfortunately there are no guarantees around deletion.

In addition to the fuzziness around exactly when a block might be 
removed you can also end up with data files hanging around in certain 
error scenarios (e.g. tmp files if there are issues loading the WAL on 
startup or during block rotation)




Le mardi 31 mars 2020 17:51:31 UTC+2, Stuart Clark a écrit :


No that sounds fairly normal. One thing to note is that those
timestamps are not the times the methods were called. They are when
Prometheus scraped your application. So if you scrape once a minute
the actual call could have been at any point during that minute.
Equally if there are multiple calls during that minute you'd have no
idea when they happened either.

I'm not a lawyer or GDPR expert, but I think the type of extreme
de-anonymisation you are suggesting is not generally something you'd
be expected to be worrying about. Equally even if you do have an
idea of who might have called an API there still isn't any personal
data in Prometheus.

On 31 March 2020 15:27:36 BST, REMI DRUILHE 
wrote:
In our code, we are using a counter to count the accesses to the
various methods of the API. We have one counter per method. We do
not store the timestamp. But when we ask Prometheus with a
"query_range" (see request below), it returns the list of all the
methods that have been accessed.

curl


'http://172.22.0.15:9090/api/v1/query_range?query=bea_nb_request=2020-03-31T00:01:00.000Z=2020-03-31T17:00:00.000Z=60s

[1]'

For each of our API method, it also returns a list of key-value
where the key is the timestamp and the value is the value of the
counter at that time (see example below). Thus, in some way, you are
able to track when the method has been called. And if our system is
used by a single user, then it is easy to follow which methods he
called. It is a bit twisted, but the national data protection
authority might also be twisted sometimes... But according to your
previous answers, maybe we did not used the counter in a proper way
and we should change the way it is designed.

{
"status":"success",
"data":{
"resultType":"matrix",
"result":[
{
"metric":{
"__name__":"bea_nb_request",
"action":"my_api_method",
"instance":"bea:8081",
"job":"bea"
},
"values":[
[
1585663440,
"1"
],
[
1585663500,
"2"
],
[
1585663560,
"3"
],
[
1585663620,
"3"
],
[
1585663680,
"3"
],
[
1585663740,
"3"
],
[
1585663800,
"3"
],
[
1585663860,
"3"
]
]
},
others_api_methods...
}
]
}
}

Le mardi 31 mars 2020 13:40:03 UTC+2, Stuart Clark a écrit :
How are you storing the timestamp? Is that in a label or a metric
value as the last call to the API?

In general these are sounding like you are trying to store events
within Prometheus rather than metrics. Normally you'd not have a
timestamp but a counter of the number of calls to the API.

On 31 March 2020 12:27:38 BST, REMI DRUILHE 
wrote:

Le lundi 30 mars 2020 16:37:11 UTC+2, Brian Candler a écrit :
On Monday, 30 March 2020 09:34:01 UTC+1, REMI DRUILHE wrote:
In our context, Prometheus is storing system metrics and business
metrics, especially the number of accesses to the methods of our
API.


That presumably is an aggegate of all calls to a particular method.

If you recorded counts as separate metrics labelled by source IP
address or username, then that would be identifiable.  But prometheus
does not work well with such high cardinality metrics anyway.

Yeah, it is just the timestamp of the call that is stored, not the IP
or the user name. Thus, it is not identifiable with Prometheus only.
But, the system aims at being used by 1 or 2 persons at the same time
in a closed network. In this context, I think it could be easy for
someone to associate the timestamp with the person that was using the
application at a specific time.

Anyway, I will figure out another way to achieve what we would like to
do.

Thanks for the help.

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

 --
You received this message because you are subscribed to the Google
Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to prometheus-users+unsubscr...@googlegroups.com.
To view this 

Re: [prometheus-users] Re: Data retention policy

[REMI DRUILHE]

You are right, most of the time this kind de-anonymisation is extreme. And 
right again when you say that there is no personal data stored in 
Prometheus.

I am also not a lawyer but I know from my DPO that the national data 
protection authority (NDPA) might be very very very meticulous, especially 
in my domain of video processing... We had several meetings about it and we 
had to review our data processing multiple time. I was just looking for a 
way to delete data with a hard deadline if the NDPA say that the current 
solution is not good enough (the one with storage.tsdb.retention.time 
option). I think it is better to come with an answer than saying that we 
did not thought about it.

Le mardi 31 mars 2020 17:51:31 UTC+2, Stuart Clark a écrit :
>
> No that sounds fairly normal. One thing to note is that those timestamps 
> are not the times the methods were called. They are when Prometheus scraped 
> your application. So if you scrape once a minute the actual call could have 
> been at any point during that minute. Equally if there are multiple calls 
> during that minute you'd have no idea when they happened either.
>
> I'm not a lawyer or GDPR expert, but I think the type of extreme 
> de-anonymisation you are suggesting is not generally something you'd be 
> expected to be worrying about. Equally even if you do have an idea of who 
> might have called an API there still isn't any personal data in Prometheus. 
>
> On 31 March 2020 15:27:36 BST, REMI DRUILHE  > wrote:
>>
>> In our code, we are using a counter to count the accesses to the various 
>> methods of the API. We have one counter per method. We do not store the 
>> timestamp. But when we ask Prometheus with a "query_range" (see request 
>> below), it returns the list of all the methods that have been accessed. 
>>
>> curl '
>> http://172.22.0.15:9090/api/v1/query_range?query=bea_nb_request=2020-03-31T00:01:00.000Z=2020-03-31T17:00:00.000Z=60s
>> '
>>
>> For each of our API method, it also returns a list of key-value where the 
>> key is the timestamp and the value is the value of the counter at that time 
>> (see example below). Thus, in some way, you are able to track when the 
>> method has been called. And if our system is used by a single user, then it 
>> is easy to follow which methods he called. It is a bit twisted, but the 
>> national data protection authority might also be twisted sometimes... But 
>> according to your previous answers, maybe we did not used the counter in a 
>> proper way and we should change the way it is designed.
>>
>> {
>>"status":"success",
>>"data":{
>>   "resultType":"matrix",
>>   "result":[
>>  {
>> "metric":{
>>"__name__":"bea_nb_request",
>>"action":"my_api_method",
>>"instance":"bea:8081",
>>"job":"bea"
>> },
>> "values":[
>>[
>>   1585663440,
>>   "1"
>>],
>>[
>>   1585663500,
>>   "2"
>>],
>>[
>>   1585663560,
>>   "3"
>>],
>>[
>>   1585663620,
>>   "3"
>>],
>>[
>>   1585663680,
>>   "3"
>>],
>>[
>>   1585663740,
>>   "3"
>>],
>>[
>>   1585663800,
>>   "3"
>>],
>>[
>>   1585663860,
>>   "3"
>>]
>> ]
>>  },
>>  others_api_methods...
>>  }
>>   ]
>>}
>> }
>>
>>
>>
>> Le mardi 31 mars 2020 13:40:03 UTC+2, Stuart Clark a écrit :
>>>
>>> How are you storing the timestamp? Is that in a label or a metric value 
>>> as the last call to the API?
>>>
>>> In general these are sounding like you are trying to store events within 
>>> Prometheus rather than metrics. Normally you'd not have a timestamp but a 
>>> counter of the number of calls to the API. 
>>>
>>> On 31 March 2020 12:27:38 BST, REMI DRUILHE  wrote:



 Le lundi 30 mars 2020 16:37:11 UTC+2, Brian Candler a écrit :
>
> On Monday, 30 March 2020 09:34:01 UTC+1, REMI DRUILHE wrote:
>>
>> In our context, Prometheus is storing system metrics and business 
>> metrics, especially the number of accesses to the methods of our API.
>>
>>>

> That presumably is an aggegate of all calls to a particular method.
>
> If you recorded counts as separate metrics labelled by source IP 
> address or username, then that would be identifiable.  But prometheus 
> does 
> not work well with such high cardinality metrics anyway.
>

 Yeah, it is just the timestamp of the call that is stored, not the IP 
 or the user 

Re: [prometheus-users] how can I monitor custom systemd services?

[Christian Hoffmann]

Hi,

you need to change your pattern: _* matches zero or more underscores. What you 
want instead is _.+ which matches an underscore followed by one or more 
arbitrary characters.

The pattern is a regular expression, not a glob.

Kind regards,
Christian

Am March 31, 2020 10:04:22 PM UTC schrieb Joey Jojo :
>Hey Christian,
>
>Thanks for getting back to me. Yes I just ran this and looks active and 
>running:
>
>root@aws-mtl-dev3:~# systemctl list-units |grep nuxt
>nuxt_site-a.service  loaded active 
>running   Nuxt Vue JS Service
>nuxt_site-b.service  loaded active 
>running   Nuxt Vue JS Service
>nuxt_site-c.service  loaded active 
>running   Nuxt Vue JS Service
>nuxt_usa-a.serviceloaded 
>active running   Nuxt Vue JS Service
>nuxt_usa-b.serviceloaded 
>active running   Nuxt Vue JS Service
>nuxt_usa-c.serviceloaded 
>active running   Nuxt Vue JS Service
>
>And this is my Node Exporter file:
>
>And this is what I have in the collector
>
> --collector.systemd \
> --collector.systemd.unit-whitelist=
>"(apache2|ssh|rsyslog|nginx|nuxt_*).service"
>
>I am trying to use nuxt_* (wildcard) to see if it an pickup all of those 
>custom services.
>Any idea why it doesn't show up in Prometheus when I query it?
>
>
>On Tuesday, March 31, 2020 at 5:15:46 PM UTC-4, Christian Hoffmann wrote:
>>
>> Hi, 
>>
>>
>> On 3/31/20 10:56 PM, Joey Jojo wrote: 
>> > I have a node exporter setup and installed in a Linux Ubuntu server and 
>> > everything works fine. I've had to setup a few different custom SystemD 
>> > services located in /etc/systemd/system/ and I'd like to know how I can 
>> > whitelist them into the node_exporter.service which is also located in 
>> > /etc/systemd/system 
>> > 
>> > This is the configuration in there right now: 
>> > 
>> [...] 
>> > Question is, how can I add a custom SystemD service called let's say, 
>> > "nuxt_sitename-a.service" as well as "nuxt_sitename-b.service" etc... 
>> > When I try to add it in the /*--collector.systemd.unit-whitelist= */I 
>> > don't see anything with that name in Prometheus query 
>> > /*node_systemd_unit_state */related to the nuxt_sitename-a/b but I do 
>> > see all the other whitelist services like SSH, Apache etc.. 
>> > **/**/ 
>> > 
>> > Does anybody know how to get node exporter to see 
>>
>> Your approach sounds correct -- this is what we are using as well. 
>> Can you confirm that the unit is loaded? systemctl list-units must 
>> return it; it being returned from systemctl list-unit-files is not 
>> enough. Maybe you still need to start and/or reference/enable it? 
>>
>> Can you confirm that node_exporter is running using the updated 
>> whitelist, e.g. did you perform a full node_exporter restart? 
>>
>> Kind regards, 
>> Christian 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/7864BB83-6287-42F4-A779-131660F1F137%40hoffmann-christian.info.

[prometheus-users] Re: Prometheus Operator and auto discovery of AlertManagers

[Richard Moe]

Answering my own mail:)

Autodiscovery of AlertManagers is not supported in the Prom custom 
resource. It
requires a custom configuration.
https://github.com/coreos/prometheus-operator/blob/master/Documentation/custom-configuration.md

On Tuesday, March 31, 2020 at 3:54:10 PM UTC+2, Richard Moe wrote:
>
> Hi!
>
> I am struggling a bit to figure out how to correctly configure 
> autodiscovery of AlertManagers in a Kubernetes cluster.
>
> Anyone have a fully working example Prometheus custom resource of this?
>
> Just putting example config from Prometheus doc doesn't seem to work as 
> the resource
> expects a list of alertmanager endpoints (
> https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#alertmanagerendpoints
> )
>
> Example config not working:
>
> alerting:
>   alertmanagers:
>   - path_prefix: /admin/alertmanager
> kubernetes_sd_configs:
>   - role: pod
> tls_config:
>   ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
> bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
> relabel_configs:
> - source_labels: [__meta_kubernetes_pod_label_name]
>   regex: alertmanager
>   action: keep
> - source_labels: [__meta_kubernetes_namespace]
>   regex: default
>   action: keep
> - source_labels: [__meta_kubernetes_pod_container_port_number]
>   regex:
>   action: drop
>
>
>
> Thanks :)
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/da8108fb-f2fb-4ce4-ba34-e4b089c21106%40googlegroups.com.

Re: [prometheus-users] SNMP_GENERATOR

[sayf eddine Hammemi]

Oh ! did you build the project? make sure to cd to the folder containing
the binary or add it to you $PATH

On Tue, Mar 31, 2020 at 7:45 PM yesmine kessentini <
yesminekessentin...@gmail.com> wrote:

> but when writing this command : which generator
> the response is :
> /usr/bin/which: no generator in
> (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/go/bin)
>
>
> Le mar. 31 mars 2020 à 18:42, yesmine kessentini <
> yesminekessentin...@gmail.com> a écrit :
>
>> thanks ,
>>
>> but i have already this package ,
>> /usr/lib/libnetsnmp.so.35
>> /usr/lib/libnetsnmp.so.30
>> /usr/lib/libnetsnmp.so.30.0.3
>> /usr/lib64/libnetsnmp.so.31
>> /usr/lib64/libnetsnmp.so.31.0.2
>> /usr/lib64/libnetsnmp.so
>> /usr/local/lib/libnetsnmp.so.35.0.0
>> /usr/local/lib/libnetsnmp.so.35
>> /usr/local/lib/libnetsnmp.so
>>
>>
>>
>>
>> Le mar. 31 mars 2020 à 17:06, sayf eddine Hammemi <
>> sayf.eddine.hamm...@gmail.com> a écrit :
>>
>>> This is a stackoverflow question
>>>
>>> ╭─sayf@degla ~
>>> ╰─$ sudo apt-file search libnetsnmp.so
>>> libsnmp-dev: /usr/lib/x86_64-linux-gnu/libnetsnmp.so
>>> libsnmp30: /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30
>>> libsnmp30: /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.3
>>>
>>>
>>> You need this package
>>>
>>> On Tue, Mar 31, 2020 at 4:17 PM yesmine kessentini <
>>> yesminekessentin...@gmail.com> wrote:
>>>
 HI ,

 i have this error while writing ./generator generate

 ./generator: error while loading shared libraries: libnetsnmp.so.35:
 cannot open shared object file: No such file or directory

  Can you help me in finding where is the issue?

 --
 You received this message because you are subscribed to the Google
 Groups "Prometheus Users" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to prometheus-users+unsubscr...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/prometheus-users/13049112-2117-4c04-840d-a6dd5f3edf49%40googlegroups.com
 
 .

>>> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/CAM6migbX%2BSmadX1aWLyJ%2BmoAKS7E6XuW1ZOhpSiAj%2B1T%3Dz5kpw%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8WiussVSVKiZkupbxVUwTXZmyR8D-6Q0zTK9tZdcgVBwFA%40mail.gmail.com.