Re: [prometheus-users] Re: node_exporter stopping when run as systemd service on Linux

Hi Brian, Are you getting any error when you start it manually as the user? /opt/node_exporter/node_exporter --collector.textfile.directory=/var/lib/node_exporter --collector.systemd --collector.ntp Thanks n Regards, Chalapathi On Wed, Nov 10, 2021 at 12:39 AM Brian Candler wrote: > Here is t

[prometheus-users] Query regarding Alertmanager

Hi All, Can you please confirm if the understanding below is correct? 1. when an alert is fired in prometheus, it fills startsAt to the time when it got fired. Even when the alert is resolved, startsAt will point to the time at which this alert was fired? 2. Alert manager sends the same startsAt

[prometheus-users] Some timeseries getting dropped

Hi All, I am seeing some time series from the target are getting dropped and not written to tsdb. Initially i thought it might be related to wrong metric relabel configs, but i have removed all the relabel configs, even then the time series is not being written. Target is giving the time series.

Re: [prometheus-users] Re: regarding pprof

gt; > On Sat, Jul 24, 2021, 08:22 Venkata Bhagavatula > wrote: > >> Queries are happening only on Node1. Node0 is only scrapping targets. >> >> On Fri, Jul 23, 2021 at 4:54 PM Stuart Clark >> wrote: >> >>> On 23/07/2021 12:20, Venkata Bhagavatula wrote

Re: [prometheus-users] Re: regarding pprof

Queries are happening only on Node1. Node0 is only scrapping targets. On Fri, Jul 23, 2021 at 4:54 PM Stuart Clark wrote: > On 23/07/2021 12:20, Venkata Bhagavatula wrote: > > Forgot to mention, we are using prometheus version 2.16.0 and cannot > update to the latest version. >

[prometheus-users] regarding pprof

Hi All, In one of our production setups, we have configured prometheus HA on Virtual machines(node0, node1). I see that node0 prometheus takes around 5gb of ram and node1 takes just 1gb of ram. user has changed min.block-duration to 30m and max block duration to 2h. I told them to not modify them

[prometheus-users] % cpu utilization

Hi All, I am trying to add an alert rule, which raises an alert when %utilization of container cpu usage is above a certain threshold. Our scrape interval is 1min Following is expression, i started with: sum(rate (container_cpu_usage_seconds_total{container=~'.+'}[4m])*100) by (namespace,containe

[prometheus-users] Regarding retention.size

Hi All, Can you please clarify the below queries? 1. Does retention.size consider chunks_head size while limiting the tsdb size to configured limit? 2. I am upgrading from prometheus version 2.16.0 to 2.20.1, In 2.20.1 wal compression is enabled by default. So the old wal segments are not compress

[prometheus-users] Wal inclusion in retention.size

Hi, In the https://prometheus.io/docs/prometheus/2.20/storage/ link, Following is mentioned regarding retention.size --storage.tsdb.retention.size: [EXPERIMENTAL] This determines the maximum number of bytes that storage blocks can use (note that this does not include the WAL size, which can be su

Re: [prometheus-users] Re: Alert manager GUI does not open using IPV6 address

Hi Brian, Thanks for the response. Can i raise a issue in alertmanager linking to the above Elm/Url issue? Once the Elm/url is fixed, then version can be upgraded in alertmanager. Thanks n Regards, Chalapathi On Tuesday, July 28, 2020 at 11:07:40 PM UTC+5:30, Brian Candler wrote: > > I note tha

Re: [prometheus-users] Re: Alert manager GUI does not open using IPV6 address

Hi Brian, I have installed alertmanager on a VM. I am still seeing the same issue. Following is the command line argument we are using for starting alertmanager: */bin/alertmanager --config.file=/etc/alertmanager/alertmanager.yml --storage.path=/alertmanager --log.level=debug --cluster.listen-ad

[prometheus-users] Re: prometheus not scrapping targets when timestamp field is present

ards, Chalapathi On Thu, Jul 2, 2020 at 3:03 PM Venkata Bhagavatula wrote: > Hi, > > We are using prometheus version 2.11.1, In our application, the scrape > target has timestamp field. when timestamp field is present, then > prometheus is not scrapping any metrics. > Following is

[prometheus-users] prometheus not scrapping targets when timestamp field is present

Hi, We are using prometheus version 2.11.1, In our application, the scrape target has timestamp field. when timestamp field is present, then prometheus is not scrapping any metrics. Following is the output of the curl request for scrape target: - *cmd: curl http://:24231/metrics* meas_gauge

[prometheus-users] Re: Restricting Prometheus to a particular Namespace

Able to solve the issue. There is a configuration error in one config file where namespaces were not added. Also if we add node role, then clusterrole, clusterolebinding is needed, as node resource is cluster scoped. Thanks n Regards, Chalapathi On Tue, May 26, 2020 at 10:31 PM Venkata

[prometheus-users] Restricting Prometheus to a particular Namespace

Hi All, Currently Prometheus needs ClusterRole and ClusterRoleBinding for scrapping the metrics on Kubernetes. We want to restrict the prometheus to a particular namespace. So we changed RBAC to using Role and RoleBinding and in the Prometheus configuration we added namespaces to kubernetes_sd_con

Re: [prometheus-users] Alert manager in HA forwards multiple alarms for the same alert

Hi Julius, I am from same team as that of Rajalakshmi, Both alertmanager instances are clustered together via --cluster.peer. Following is way the alertmanager process are started: *Alertmanager1:* */bin/alertmanager --config.file=/etc/alertmanager/alertmanager.yml --storage.path=/appdata/cpro/

[prometheus-users] Re: Deleting timeseries

Venkata Bhagavatula wrote: > Hi, > > I was using the admin rest api for deleting the timeseries. Following is > what i executed: > > *curl -XPOST -g > 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={__name__=~ > <http://localhost:9090/api/v

Re: [prometheus-users] usage of rate function on recording metric

why we see a drop of the counter value in the above charts ? Thanks n Regards, Chalapathi. On Mon, Mar 9, 2020 at 5:09 PM Julien Pivotto wrote: > On 09 Mar 11:35, Stuart Clark wrote: > > On 2020-03-09 11:21, Venkata Bhagavatula wrote: > > > Hi Stuart, > > &g

Re: [prometheus-users] usage of rate function on recording metric

On Thu, Feb 27, 2020 at 12:59 PM Stuart Clark wrote: > The graph showed a reduction at various points. Is there a bug in the > recording rule calculation that can cause reduction which needs fixing? > > On 27 February 2020 06:24:11 GMT, Venkata Bhagavatula < > venkat.cha..

[prometheus-users] Deleting timeseries

Hi, I was using the admin rest api for deleting the timeseries. Following is what i executed: *curl -XPOST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={__name__=~ ".+"}'* followed by *curl -XPOST

Re: [prometheus-users] usage of rate function on recording metric

is seen as a counter reset. > > If this isn't a counter use the derivative function rather than rate > > On 26 February 2020 14:05:31 GMT, Venkata Bhagavatula < > venkat.cha...@gmail.com> wrote: >> >> Hi, >> >> In our application, there is one met