[prometheus-users] scrape_interval is not working properly.

[Byungkwon Choi]

Hello,

I want to collect the HTTP requests per second every second.
To do so, I'm using Prometheus and Prometheus Adapter.

I set the value of 'scrape_interval', 'scrape_timeout', and 
'evaluation_interval' to 1s in the Prometheus ConfigMaps resource of K8S as 
follows:

[image: 스크린샷 2020-08-05 오전 12.20.46.png]


I use the following query to collect HTTP requests per second:
sum(irate(request_total{namespace!="", direction="inbound"}[30s])) by 
(deployment)

However, the value of the query is updated almost every 20 seconds.
What I expect to this value is to be updated every second because I use 
'irate' here and set 'scrape_interval' to 1 second.


My current environment is as follows:

   - Service mesh: Linkerd v2.8.1
   - K8S version: v1.17.9
   - CNI: Calico
   - Other components deployed together: Jaeger, Prometheus-adapter


Did I miss something?
Thank you in advance!

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/aa2fcdf9-dddc-4c36-b1be-82cd9bae8542o%40googlegroups.com.

Re: [prometheus-users] Re: Select metric only if another does not exist

[Aliaksandr Valialkin]

Try the following query:

metric_name unless on (hostname) another_metrics{type="value"}

On Tue, Aug 4, 2020 at 2:44 PM Sam Lee  wrote:

> try metric{type != "two"} ?
>
> 在 2020年8月4日星期二 UTC+8下午3:30:05，Seitan写道：
>>
>> Hello,
>> I'm trying to do a promql query to select only hosts, that don't have
>> specific label on them.
>> For example, selecting metrics from hosts who DO have label (type=value)
>> works:
>>
>> metrics_name * on (hostname) another_metrics{type="value"}
>>
>> Problem is when i try to do same select against hosts without that label:
>>
>>
>> metrics_name * on (hostname) (absent(another_metrics{type="value"}))
>>
>> this query gives no results.
>> Is there any way to do such query?
>> Thank you
>>
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/bce85d3d-80c0-4ffb-ae40-9b0476a5b553o%40googlegroups.com
> 
> .
>


-- 
Best Regards,

Aliaksandr Valialkin, CTO VictoriaMetrics

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAPbKnmB31V3jrgbcMkCQUrtKUY8KSs1JoiQTZ4-2S9OTMwYYOA%40mail.gmail.com.

[prometheus-users] Re: Able to specify bind port, but not address

[jumble]

"latest prometheus" -> prometheus-2.20.0.linux-amd64
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/8d4948f7-efda-420a-858b-d91df1f17d86o%40googlegroups.com.

[prometheus-users] Able to specify bind port, but not address

[jumble]

Latest prometheus, on RHEL8.


Tried command: ./prometheus
Expected behavior: bind to 0.0.0.0:9090
Observed behavior: bound to 127.0.0.1:9090


Tried command: ./prometheus --web.listen-address=0.0.0.0:9090
Expected behavior: bind to 0.0.0.0:9090
Observed behavior: bound to 127.0.0.1:9090


Tried command: ./prometheus --web.listen-address=0.0.0.0:9000
Expected behavior: bind to 0.0.0.0:9000
Observed behavior: bound to 127.0.0.1:9000

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/ab43a493-7f37-42df-a391-4ad3f0ce154eo%40googlegroups.com.

[prometheus-users] How to prevent sending resolve notification after resolve_timeout?

[shiqi chai]

Hey guys,I have a problem with configuration of resolve_timeout. As it means, a 
notication of resolved will be send after the timeout. But actually the issue 
still be firing, it disturb the correct resolved notification
How can I prevent it?

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/2cdbef0e-d132-4c4f-b5b7-740b6d43e55bo%40googlegroups.com.

[prometheus-users] Re: Select metric only if another does not exist

[Sam Lee]

try metric{type != "two"} ?

在 2020年8月4日星期二 UTC+8下午3:30:05，Seitan写道：
>
> Hello,
> I'm trying to do a promql query to select only hosts, that don't have 
> specific label on them.
> For example, selecting metrics from hosts who DO have label (type=value) 
> works:
>
> metrics_name * on (hostname) another_metrics{type="value"}
>
> Problem is when i try to do same select against hosts without that label:
>
>
> metrics_name * on (hostname) (absent(another_metrics{type="value"}))
>
> this query gives no results.
> Is there any way to do such query?
> Thank you
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/bce85d3d-80c0-4ffb-ae40-9b0476a5b553o%40googlegroups.com.

Re: [prometheus-users] external_labels from prometheus not able to use as filter in Grafana Dashboard variables

[Ben Kochie]

This sounds like a `label="$var"` vs `label=~"$var"` issue. Grafana
sometimes formats the variable as a regexp, so you need to use `=~` for it
to work.

On Tue, Aug 4, 2020 at 6:52 AM Pavan Vasisht 
wrote:

> Hi,
>
> Using the helm(*Version:"v3.2.4",
> GitCommit:"0ad800ef43d3b826f31a5ad8dfbb4fe05d143688"*) - stable/prometheus
>  was
> configuring *external_label* so that this can be used as
> parameter(drop-down value) to filter in grafana dashbaord. The
> external_label is getting propagated across all the metrics. Manually
> setting the label and fetching metrics works, but the label when tried to
> set as variable, no luck.
> Please refer the github issue which I had raised for Grafana
> , as per the suggestion
> asking here.
>
> Some backgroud -
> Using thanos side car within prometheus server to push the metrics to s3
> bucket and fetching the stored metrics into grafana via thanos querier.
>
> Thanks
> Pavan
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/be329a82-8681-4450-989d-402126f2bf97o%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CABbyFmpVfkLBrz7DMgJZsGj9z%2Bb5LYqfTPvoaEZrigUQF3Vm2Q%40mail.gmail.com.

[prometheus-users] Re: Select metric only if another does not exist

[Sam Lee]

 metric{type != "two"} ?

在 2020年8月4日星期二 UTC+8下午3:30:05，Seitan写道：
>
> Hello,
> I'm trying to do a promql query to select only hosts, that don't have 
> specific label on them.
> For example, selecting metrics from hosts who DO have label (type=value) 
> works:
>
> metrics_name * on (hostname) another_metrics{type="value"}
>
> Problem is when i try to do same select against hosts without that label:
>
>
> metrics_name * on (hostname) (absent(another_metrics{type="value"}))
>
> this query gives no results.
> Is there any way to do such query?
> Thank you
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/6e29bc99-53b3-44b3-b720-f36465285ad1o%40googlegroups.com.

[prometheus-users] can I add for remote_read

[Sam Lee]

I have to remote_read from influxdb and another prometheus, can I add 
labels to distinguish the two remote_read just as below:

# read data from kafka
remote_read:
- url: http://192.168.11.10:30070/api/v1/prom/read?db=prometheus
  read_recent: true
  labels:
 from: influxdb

# read data from prometheus
remote_read:
- read_recent: true
  url: http://192.168.11.15:30086/api/v1/read
  read_recent: true
  labels:
 from: prometheus
   
# remote write to kafka adapter
remote_write:
- url: http://127.0.0.1:18080/receive



-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/5e6dc8da-748e-4634-8b98-ed0fded00ba9o%40googlegroups.com.

[prometheus-users] Prometheus memory issue

[Vinod M V]

Hi All ,

  I am facing Memory usage with Prometheus service and Maintaining 
30 days of data from Node exporter, Process exporter and JMX exporter for 
95 servers in Prometheus Database. 

 Grafana and Prometheus are running on the same node. When loading 
node_exporter or JVM Grafana dashboards, memory usage will shoot up to more 
than 10 GB and sometime usage will hit max and Prometheus will be 
restarted. Moreover memory usage will not reduce even after closing 
JVM/Node exporter Grafana Dashboards. 

 Always need to restart Prometheus to release the memory. I am 
expecting 300 more servers to this configuration in future. 

Can someone please suggest a solution for this memory issue ( or 
suggest a best database for save the metrics) and suggest a open source 
solution for High availability 

Monitoring Server config: ( VM server):
Server Arch: x86_64
CPU core count: 4
Memory: 16 GB
Prometheus Version: 2.20.0
Prom Config Flags: --log.level=error --query.max-concurrency=10 
--query.max-samples=1000 --storage.tsdb.retention 
30d --web.route-prefix=/

Node_exporter Version on node: 0.16.0
Process_exporter Version on node: 0.5.0
Grafana Version: 6.4.4  

*Thanks and Regards,*
*Vinod M V*

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/4aea5f48-60d8-4b5d-bee7-899e3544ed9bo%40googlegroups.com.

[prometheus-users] blackbox dns probe failed

[e huang]

Host operating system: output of uname -a

Linux office-monitor 3.10.0-957.el7.x86_64 #1 
 SMP Thu Nov 8 
23:39:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
blackbox_exporter version: output of blackbox_exporter -version

blackbox_exporter, version 0.17.0 (branch: HEAD, revision: 1bc7680 

)
build user: root@626fb3899f41
build date: 20200619-11:54:41
go version: go1.14.4
What is the blackbox.yml module config.

modules:
http_2xx:
prober: http
http_post_2xx:
prober: http
http:
method: POST
tcp_connect:
prober: tcp
pop3s_banner:
prober: tcp
tcp:
query_response:
- expect: "^+OK"
tls: true
tls_config:
insecure_skip_verify: false
ssh_banner:
prober: tcp
tcp:
query_response:
- expect: "^SSH-2.0-"
irc_banner:
prober: tcp
tcp:
query_response:
- send: "NICK prober"
- send: "USER prober prober prober :prober"
- expect: "PING :([^ ]+)"
send: "PONG ${1}"
- expect: "^:[^ ]+ 001"
icmp:
prober: icmp

##
dns_awg.enmonster.com:
prober: dns
timeout: 5s
dns:
query_type: "A"
preferred_ip_protocol: "ip4"
query_name: "awg.enmonster.com"
valid_rcodes:
- NOERROR
dns_azg.enmonster.com:
prober: dns
timeout: 5s
dns:
query_type: "A"
preferred_ip_protocol: "ip4"
query_name: "azg.enmonster.com"
valid_rcodes:
- NOERROR
dns_www.baidu.com:
prober: dns
timeout: 5s
dns:
query_type: "A"
preferred_ip_protocol: "ip4"
query_name: "www.baidu.com"
dns_eboss.enmonster.com:
prober: dns
timeout: 5s
dns:
query_type: "A"
preferred_ip_protocol: "ip4"
query_name: "eboss.enmonster.com"
valid_rcodes:
- NOERROR
dns_hydra.enmonster.com:
prober: dns
timeout: 5s
dns:
query_type: "A"
preferred_ip_protocol: "ip4"
query_name: "hydra.enmonster.com"
dns_www.dingtalk.com:
prober: dns
timeout: 5s
dns:
query_type: "A"
preferred_ip_protocol: "ip4"
query_name: "www.dingtalk.com"
What is the prometheus.yml scrape config.my global config

global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default 
is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is 
every 1 minute.
scrape_timeout is set to the global default (10s).Alertmanager configuration

alerting:
alertmanagers:

   - static_configs:
  - targets:- alertmanager:9093
   
Load rules once and periodically evaluate them according to the global 
'evaluation_interval'.

rule_files:
- "first_rules.yml"- "second_rules.yml"A scrape configuration containing 
exactly one endpoint to scrape:Here it's Prometheus itself.

scrape_configs:
The job name is added as a label job= to any timeseries scraped 
from this config.
   
   - 
   
   job_name: 'prometheus'
   metrics_path defaults to '/metrics'scheme defaults to 'http'.
   
   static_configs:
   - targets: ['localhost:9090']
   
#dns_check

   - job_name: 'blackbox-dns_azg.enmonster.com'
   metrics_path: /probe
   params:
   module:
  - dns_azg.enmonster.com
  static_configs:
  - targets:
 - 10.208.100.9
 - 10.208.100.8
 labels:
 type: dns_resolve
 zone: office_sy
 relabel_configs:
  - source_labels: [address]
  target_label: "__param_target"
  - source_labels: [address]
  target_label: "instance"
  - source_labels: [_
*param_module] regex: 'dns*(.*)'
  target_label: 'domain_name'
  - target_label: address
  replacement: "127.0.0.1:9115"
   - job_name: 'blackbox-dns_awg.enmonster.com'
   metrics_path: /probe
   params:
   module:
  - dns_awg.enmonster.com
  static_configs:
  - targets:
 - 10.208.100.9
 - 10.208.100.8
 labels:
 type: dns_resolve
 zone: office_sy
 relabel_configs:
  - source_labels: [address]
  target_label: "__param_target"
  - source_labels: [address]
  target_label: "instance"
  - source_labels: [_
*param_module] regex: 'dns*(.*)'
  target_label: 'domain_name'
  - target_label: address
  replacement: "127.0.0.1:9115"
   - job_name: 'blackbox-dns_sso.enmonster.com'
   metrics_path: /probe
   params:
   module:
  - dns_sso.enmonster.com
  static_configs:
  - targets:
 - 10.208.100.9
 - 10.208.100.8
 labels:
 type: dns_resolve
 zone: office_sy
 relabel_configs:
  - source_labels: [address]
  target_label: "__param_target"
  - source_labels: [address]
  target_label: "instance"
  - source_labels: [_
*param_module] regex: 'dns*(.*)'
  target_label: 'domain_name'
  - target_label: address
  replacement: "127.0.0.1:9115"
   - job_name: 'blackbox-dns_hydra.enmonster.com'
   metrics_path: /probe
   params:
   module:
  - dns_hydra.enmonster.com
  static_configs:
  - targets:
 - 10.208.100.9
 - 10.208.100.8
 labels:
 type: dns_resolve
 zone: 

[prometheus-users] Re: Select metric only if another does not exist

[Seitan]

Hello,
the problem is that the node has label with many values:

metric{type="one"},
metric{type="two"},
metric{type="three"}

I just need do query against those nodes, who has no
metric{type="two"}



On Tuesday, August 4, 2020 at 10:30:05 AM UTC+3, Seitan wrote:
>
> Hello,
> I'm trying to do a promql query to select only hosts, that don't have 
> specific label on them.
> For example, selecting metrics from hosts who DO have label (type=value) 
> works:
>
> metrics_name * on (hostname) another_metrics{type="value"}
>
> Problem is when i try to do same select against hosts without that label:
>
>
> metrics_name * on (hostname) (absent(another_metrics{type="value"}))
>
> this query gives no results.
> Is there any way to do such query?
> Thank you
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/ebf6bff8-ecd8-47de-8b1e-90d47cefd6aao%40googlegroups.com.

[prometheus-users] Re: Select metric only if another does not exist

[Brian Candler]

On Tuesday, 4 August 2020 08:30:05 UTC+1, Seitan wrote:
>
> I'm trying to do a promql query to select only hosts, that don't have 
> specific label on them.
>

The empty value is equivalent to "no label":

metric{type=""}

will select timeseries 'metric' which have no label 'type'.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/01e4001e-f11f-4ca4-aebd-4e56ef499a3do%40googlegroups.com.

[prometheus-users] Select metric only if another does not exist

[Seitan]

Hello,
I'm trying to do a promql query to select only hosts, that don't have 
specific label on them.
For example, selecting metrics from hosts who DO have label (type=value) 
works:

metrics_name * on (hostname) another_metrics{type="value"}

Problem is when i try to do same select against hosts without that label:


metrics_name * on (hostname) (absent(another_metrics{type="value"}))

this query gives no results.
Is there any way to do such query?
Thank you


-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/71b92451-bb5d-40bd-8cff-9d4078683427o%40googlegroups.com.