Re: [prometheus-users] Incorrect Node Labels in kube_pod_container_info

[Murali Krishna Kanagala]

Thanks for the clarification Julius! I will read through the links you have
shared.

On Fri, Jan 20, 2023, 5:16 AM Julius Volz  wrote:

> Hi Murali,
>
> So I guess both of those metrics are coming from the same
> kube-state-metrics target? While I'm not sure why those two metrics would
> be behaving differently, I'm not sure you'd want put the
> "__meta_kubernetes_pod_node_name" service discovery meta label into the
> "node" target label, because this discovery meta label contains only the
> node name of the discovered kube-state-metrics target and not of the
> individual specific pods that the metric from kube-state-metrics is about.
> So I would actually expect both metrics to only show one single "node"
> value ever after your relabeling rule (or as many values as you have k-s-m
> instances).
>
> One thing I did notice on
> https://github.com/kubernetes/kube-state-metrics/blob/main/docs/pod-metrics.md
> is that "kube_pod_info" already has a "node" label to begin with (coming
> from kube-state-metrics), while "kube_pod_container_info" does not. But
> your relabeling rule sets a "node" label on the target itself, which will
> conflict during the scrape with the "node" label on the "kube_pod_info"
> metric. That should cause the original "node" label on "kube_pod_info"
> to be moved to an "exported_node" label to resolve the label collision.
> That is, unless you are setting the "honor_labels: true" option in the
> scrape config (see
> https://github.com/prometheus/prometheus/blob/64842f137e1ae6e041e12a2707d99d6da4ba885b/docs/configuration/configuration.md?plain=1#L135-L155),
> which would 100% explain why you'd see the behavior you're describing,
> since that would keep the original label from kube-state-metrics and throw
> away the conflicting target label.
>
> Anyway, for diagnosing discovery and relabeling issues in general, it's
> always a good idea to check out the /service-discovery page on your
> Prometheus server. There you can inspect the labels of any target both
> before and after relabeling ("Discovered Labels" column vs. "Target Labels"
> column). You can also use the relabeling simulator / explainer tool at
> https://relabeler.promlabs.com/ (shameless plug) to enter your relabeling
> rules and the target labels before relabeling (both in YAML format) to see
> what effect the rules have on the labels.
>
> Regards,
> Julius
>
> On Thu, Jan 19, 2023 at 2:36 PM Murali Krishna Kanagala <
> kanagalamur...@gmail.com> wrote:
>
>> Hi All,
>>
>> While working on some alert rules I have noticed that the metric*
>> kube_pod_container_Info* has wrong values under the node label. This
>> label uses the below scrape/relabel config which looks right. Whereas the
>> other metric from the same job *kube_pod_info *has the correct node
>> label value. The node value in the *kube_pod_container_info *is giving
>> the name of the node where Prometheus is running (only returns 1 node value
>> for all the pods in the cluster).
>>
>>   - source_labels: [__meta_kubernetes_pod_node_name]
>> separator: ;
>> regex: (.*)
>> target_label: node
>> replacement: $1
>> action: replace
>>
>> Another observation is that a bunch of metrics under this job
>> *kubernetes-service-endpoints* (like 
>> *kube_pod_container_status_restarts_total)
>> *are also reporting only one node value for all the pods on the cluster.
>>
>> This is deployed using helm chart version 19.3.1 with the default scrape
>> config.
>>
>> Any suggestions/ recommendations?
>> --
>> Regards,
>> Murali Krishna Kanagala
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to prometheus-users+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-users/CAKimyZvTqw9jqW%3DoWFa8StAUV2UnBFOZCp6xE-YwaUz0ZmjF4g%40mail.gmail.com
>> <https://groups.google.com/d/msgid/prometheus-users/CAKimyZvTqw9jqW%3DoWFa8StAUV2UnBFOZCp6xE-YwaUz0ZmjF4g%40mail.gmail.com?utm_medium=email_source=footer>
>> .
>>
>
>
> --
> Julius Volz
> PromLabs - promlabs.com
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZv4MODLT47YSn0dOrN1GO7DbG6Ty%3Dicaa%3DUuTv0uuXXng%40mail.gmail.com.

[prometheus-users] Incorrect Node Labels in kube_pod_container_info

[Murali Krishna Kanagala]

Hi All,

While working on some alert rules I have noticed that the metric*
kube_pod_container_Info* has wrong values under the node label. This label
uses the below scrape/relabel config which looks right. Whereas the other
metric from the same job *kube_pod_info *has the correct node label value.
The node value in the *kube_pod_container_info *is giving the name of the
node where Prometheus is running (only returns 1 node value for all the
pods in the cluster).

  - source_labels: [__meta_kubernetes_pod_node_name]
separator: ;
regex: (.*)
target_label: node
replacement: $1
action: replace

Another observation is that a bunch of metrics under this job
*kubernetes-service-endpoints* (like *kube_pod_container_status_restarts_total)
*are also reporting only one node value for all the pods on the cluster.

This is deployed using helm chart version 19.3.1 with the default scrape
config.

Any suggestions/ recommendations?
-- 
Regards,
Murali Krishna Kanagala

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZvTqw9jqW%3DoWFa8StAUV2UnBFOZCp6xE-YwaUz0ZmjF4g%40mail.gmail.com.

[prometheus-users] GaugeMetricFamily Timestamp with Pushgateway

[Murali Krishna Kanagala]

Hello everyone,

I am working on a tool to push historical metrics (1-5 min old) to push
gateway.
I have tried using the Gauge form prometheus-client but it does not support
setting an older timestamp.
GaugeMetricFamily seems to support it but i am not able to find an example
of how to use it with push gateway.

Can someone help me with this?
-- 
Regards,
Murali Krishna Kanagala

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZucGgpaG9oreSc3FvmJuFx17fkSqXp9bGdv8jZ9ii9OjA%40mail.gmail.com.

[prometheus-users] Re: Alertmanager webhook timeout

[Murali Krishna Kanagala]

Guys,

Can I get some help with this?

On Mon, Jul 6, 2020, 10:25 AM Murali Krishna Kanagala <
kanagalamur...@gmail.com> wrote:

> Hello everyone,
>
> Can someone help  me on how can
> I change the webhook timeout settings for a webhook?
> Also in an alert route if AM is waiting for a webhook response does it
> holds of the rest of the pipeline until that
>
> Thanks!
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZtOrdmZA00kyuqZbGUg6msAXXj4njL_7y9b%2BRFPPMW8Mw%40mail.gmail.com.

[prometheus-users] Alertmanager webhook timeout

[Murali Krishna Kanagala]

Hello everyone,

Can someone help  me on how can
I change the webhook timeout settings for a webhook?
Also in an alert route if AM is waiting for a webhook response does it
holds of the rest of the pipeline until that

Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZuZzGuD2zc8Ygnz3D-_TktBWvrcQ2LWbQq4EO3DyADvxg%40mail.gmail.com.

[prometheus-users] Relabeling Question

[Murali Krishna Kanagala]

Hello everyone,

We are moving from wmi_exporter to windows_exporter and trying to do it in
a smooth manner.
Our Alerts and Graphs are heavily dependent on WMI metrics and moving to
Windows exporter may cause some issues because of a different metric prefix
(wmi to windows).
We are looking for a temporary solution where we can keep both wmi and
windows exporter metrics. This can be done by running both exporters on the
server and by adding that to scrape config but I am looking for an option
if I can use the relabel features to create a new metric with wmi_ prefix
for every windows_ prefix metric.

For ex. If i scrape windows_cpu_seconds_total, i want to keep that metric
and also create a new one like wmi_cpu_seconds_total.

Is there a way in metric relabeling to keep both old and relabeled metrics?

Thanks,
Murali Kanagala

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZvGKaiaQVtBmTt%3Dcd2pSr%3DMPH_2M%3Ds%3D4VViv6Bp8taQWA%40mail.gmail.com.

Re: [prometheus-users] Re: Prometheus not able to start and gives error "opening storage failed:: found unsequential head chunk error"

[Murali Krishna Kanagala]

Try deleting the contents WAL folder and start the service. Ideally you can
delete the last WAL where the service is failing to start. It is in the log
just before Prometheus stopping.

On Thu, Jun 11, 2020, 2:26 AM neel patel  wrote:

> Hi Julien,
>
> Unfortunately i don't have previous logs. And i don't think, it is related
> to disk as i have ~28GB storage available. Below is the output of my root
> file system.
>
> /dev/sda396G   69G   28G  72% /
> /dev/sda1   297M  207M   91M  70% /boot
> tmpfs   378M 0  378M   0% /run/user/26
> tmpfs   378M   24K  378M   1% /run/user/1000
>
>
> Let me know in case of more help.
>
> On Thursday, June 11, 2020 at 12:20:46 PM UTC+5:30, neel patel wrote:
>>
>> Prometheus was working fine before 2 days and today when i stop and start
>> it again, it shows below error and not able to start the prometheus.
>>
>> I build prometheus from master branch and using that binary since 2 days.
>> Below are the logs when i start prometheus.
>>
>>
>> [neel@localhost prometheus]$ ./prometheus --config.file=prometheus.yml
>> --storage.tsdb.path=data/
>> level=info ts=2020-06-11T06:25:10.967Z caller=main.go:302 msg="No time or
>> size retention was set so using the default time retention" duration=15d
>> level=info ts=2020-06-11T06:25:10.967Z caller=main.go:337 msg="Starting
>> Prometheus" version="(version=2.18.1, branch=master,
>> revision=18d9ebf0ffc26b8bd0e136f552c8e9886d29ade4)"
>> level=info ts=2020-06-11T06:25:10.967Z caller=main.go:338
>> build_context="(go=go1.14.3, user=neel@localhost.localdomain,
>> date=20200604-05:51:34)"
>> level=info ts=2020-06-11T06:25:10.967Z caller=main.go:339
>> host_details="(Linux 3.10.0-1062.9.1.el7.x86_64 #1 SMP Fri Dec 6 15:49:49
>> UTC 2019 x86_64 localhost.localdomain (none))"
>> level=info ts=2020-06-11T06:25:10.967Z caller=main.go:340
>> fd_limits="(soft=1024, hard=4096)"
>> level=info ts=2020-06-11T06:25:10.967Z caller=main.go:341
>> vm_limits="(soft=unlimited, hard=unlimited)"
>> level=info ts=2020-06-11T06:25:10.973Z caller=main.go:678 msg="Starting
>> TSDB ..."
>> level=info ts=2020-06-11T06:25:10.973Z caller=web.go:524 component=web
>> msg="Start listening for connections" address=0.0.0.0:9090
>> level=info ts=2020-06-11T06:25:10.974Z caller=repair.go:59 component=tsdb
>> msg="Found healthy block" mint=1591250134254 maxt=159125760
>> ulid=01EACBSE6H4EP5CS2K2G6KWJ0W
>> level=info ts=2020-06-11T06:25:10.974Z caller=repair.go:59 component=tsdb
>> msg="Found healthy block" mint=159168240 maxt=159174720
>> ulid=01EAE4FPDZ932CDRBAGH019TK7
>> level=info ts=2020-06-11T06:25:10.974Z caller=repair.go:59 component=tsdb
>> msg="Found healthy block" mint=159174720 maxt=159176880
>> ulid=01EAEQAB8X70893C6ZC2T2ZK38
>> level=info ts=2020-06-11T06:25:10.975Z caller=repair.go:59 component=tsdb
>> msg="Found healthy block" mint=159179040 maxt=159179760
>> ulid=01EAGR8C1PMZZACQMN185NC254
>> level=info ts=2020-06-11T06:25:10.975Z caller=repair.go:59 component=tsdb
>> msg="Found healthy block" mint=159179760 maxt=159180480
>> ulid=01EAGR8D3YVDV517XCMBMMKCS7
>> level=info ts=2020-06-11T06:25:10.975Z caller=repair.go:59 component=tsdb
>> msg="Found healthy block" mint=159176880 maxt=159179040
>> ulid=01EAGR8ETQQC8MZ09SSC62N1D3
>> level=info ts=2020-06-11T06:25:10.982Z caller=main.go:547 msg="Stopping
>> scrape discovery manager..."
>> level=info ts=2020-06-11T06:25:10.983Z caller=main.go:561 msg="Stopping
>> notify discovery manager..."
>> level=info ts=2020-06-11T06:25:10.983Z caller=main.go:583 msg="Stopping
>> scrape manager..."
>> level=info ts=2020-06-11T06:25:10.983Z caller=main.go:557 msg="Notify
>> discovery manager stopped"
>> level=info ts=2020-06-11T06:25:10.983Z caller=main.go:543 msg="Scrape
>> discovery manager stopped"
>> level=info ts=2020-06-11T06:25:10.983Z caller=manager.go:882
>> component="rule manager" msg="Stopping rule manager..."
>> level=info ts=2020-06-11T06:25:10.983Z caller=manager.go:892
>> component="rule manager" msg="Rule manager stopped"
>> level=info ts=2020-06-11T06:25:10.983Z caller=notifier.go:601
>> component=notifier msg="Stopping notification manager..."
>> level=info ts=2020-06-11T06:25:10.983Z caller=main.go:749 msg="Notifier
>> manager stopped"
>> level=info ts=2020-06-11T06:25:10.983Z caller=main.go:577 msg="Scrape
>> manager stopped"
>> level=error ts=2020-06-11T06:25:10.983Z caller=main.go:758 err="opening
>> storage failed: found unsequential head chunk files 144 and 148"
>>
>> #
>>
>> Data directory content as below
>>
>> 01EACBSE6H4EP5CS2K2G6KWJ0W  01EAEQAB8X70893C6ZC2T2ZK38
>> 01EAGR8D3YVDV517XCMBMMKCS7  chunks_head  queries.active
>> 01EAE4FPDZ932CDRBAGH019TK7  01EAGR8C1PMZZACQMN185NC254
>> 01EAGR8ETQQC8MZ09SSC62N1D3  lock wal
>>
>> ##
>>
>>
>>
>> Prometheus config file as below.
>>
>> # my global config
>> global:
>>   scrape_interval: 15s # Set the scrape interval to every 

Re: [prometheus-users] correlating counters across services

[Murali Krishna Kanagala]

Metrics use the labels from kubernetes. If your kubernetes tenant use
specific labels like namespace etc. those will be used in the metrics
automatically.

On Mon, Jun 8, 2020, 4:16 PM Manoj Guglani  wrote:

> Hi,
>
> I am working on integrating prometheus for our microservices platform
> deployed on kubernetes. A question came up during review about being able
> to correlate counters across services so thought of asking it here.
> Consider a case where adding a new tenant results in multiple services
> changing some state, e.g. one service could be making update to a database,
> another one updating configuration, etc. Can we use labels or some other
> mechanism to correlate the counters or is this not recommended?
>
> Thanks,
>
> Manoj
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/f49de2d8-7b7d-4b39-9225-839a2522b93co%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZuTsU02Y%2BxKXXdJJi_krWD_JTVn4W%3DmXcxzyupFL4T74Q%40mail.gmail.com.

Re: [prometheus-users] Ways to Mitigate on huge label value pair during high cardinalities

[Murali Krishna Kanagala]

This should be taken care by the exporter you are collecting metrics from.
If you are writing your own exporter then make validate what labels stay
unique all the time. For ex. If you are collecting ngnix request metrics
then passing request uuid as a label makes the metrics highly cardinal.

On Thu, Jun 4, 2020, 2:01 PM Dinesh N 
wrote:

> Hi Team,
>
> Figuring out ways to optimise high cardinality labels, any suggestions
> are welcomed here.
>
> Regards
> Dinesh
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/CAA6KEskjY1TpjVddTSAAx8AL3aH7Zc7yyNcg0McpOBOfrrow0g%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZszRG4msyHP7pT6C6%2B1QLLu62aB-6pK1d-9V5LFpqh3Tw%40mail.gmail.com.

Re: [prometheus-users] Re: (Alertmanager) Ignore instance label to prevent same alert multiple times

[Murali Krishna Kanagala]

I guess your scrape config has some static labels like environment: prod,
region: central, etc. so add another static label with cluster name. Then
do a label replacement on your alert query to replace the instance value to
your cluster name. Then you have 1 instance name for the entire cluster. If
you want to add your vhost to it then try label_join which can add your
vhost name to the instance label.


On Thu, Jun 4, 2020, 12:48 PM 'ping...@hioscar.com' via Prometheus Users <
prometheus-users@googlegroups.com> wrote:

> +1
>
> We get the same alert multiple times in the same email, because the
> monitor label (prometheus instance) being different for our simple
> replicated setup. Would be nice to be able to ignore certain labels so that
> alert bodies are higher signal.
>
> On Thursday, May 9, 2019 at 11:29:58 AM UTC-4, tommy@gmail.com wrote:
>>
>> Did you ever find a solution for this? I'm looking at the exact same
>> issue.
>>
>> // Tommy
>>
>> Den måndag 26 februari 2018 kl. 09:45:29 UTC+1 skrev Robin Ong:
>>>
>>> Hi all,
>>>
>>> I have a question regarding filtering messages to prevent getting
>>> multiple alerts on same error.
>>>
>>> *Case:*
>>> I have a rabbitMQ cluster that has 3 nodes. Each running on a different
>>> server. This rabbitMQ cluster duplicates the messages on a queue to all
>>> instances.
>>> So when an Error hits an Error queue I receive 3 alerts for each
>>> instance. The only value that makes each error unique is the instance value.
>>>
>>> I would like Alert Manager to just look at VHOST and queue name. This
>>> way I should only get 1 alert?
>>>
>>>
>>> Do you guys know a solution?
>>>
>>> Regards in advance.
>>>
>>>
>>> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/ffdeacd5-4bf8-48dc-83ab-a0e06869fc2f%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZuKvHeWqthHd1Rfd-FBme9EW8PLt29bPaQXi_3OMthpkg%40mail.gmail.com.

Re: [prometheus-users] Windows Exporter SetUp

[Murali Krishna Kanagala]

You need Prometheus installed and add the wmi exporter target to Prometheus
scrape config. Then point grafana to Prometheus.

On Thu, Jun 4, 2020, 1:20 PM Freddy Mack  wrote:

> Hello Krishna,
> This is the link
>
> https://github.com/prometheus-community/windows_exporter/issues/519
>
> I have just downloaded the wmi_exporter on clients , what else need to be
> done on clients windows servers? to get the Metrics displayed on Grafana,
> Also I have Linux and Windows as client servers on Grafana.
>
>
>
> On Thursday, June 4, 2020 at 12:43:44 PM UTC-5, Murali Krishna Kanagala
> wrote:
>>
>> Wmi exporter is the windows equivalent of node exporter. You need to run
>> it as a service and add that machine to scrape config. Never heard of
>> windows exporter.
>>
>> On Thu, Jun 4, 2020, 11:24 AM Freddy Mack  wrote:
>>
>>> I have Download the wmi_exporter on localhost and clients- but I see the
>>> blogs that they changed into wmi into windows_exporter is that necessary ?
>>> If I use wmi_exporter - I have just downloaded on client windows
>>> machines- what all Steps required it to work and get the metrics on Grafana.
>>> I have added all the client windows servers in prometheus.yml as well
>>>
>>>
>>> On Thursday, June 4, 2020 at 10:58:11 AM UTC-5, Stuart Clark wrote:
>>>>
>>>> On 2020-06-04 16:42, Freddy Mack wrote:
>>>> > Hello,
>>>> >
>>>> > I have installed wmi_exporter on the client Windows servers and the
>>>> > localhost (CentOS7) on which Grafana 7.0.3 and prometheus installed
>>>> > on.
>>>> > I need the procedure to get windows_exporter to download and install
>>>> > and configure on all the environment and to pull the Metrics on
>>>> > Grafana Dashboard.
>>>> > I have already set the Linux Metrics in place on Grafana and want to
>>>> > Add the Windows as well.
>>>> >
>>>> > Metrics are:
>>>> >
>>>> > §CPU
>>>> > utilization   Warning at 89%, Alarm
>>>> at
>>>> > 90%
>>>> >
>>>> > §  Memory utilization  Warning at 90%, Alarm
>>>> > at 95%
>>>> >
>>>> > §  File System utilization Warning at 90%, Alarm
>>>> > at 95%
>>>> >
>>>> > §  System messages (ex. Errors in /var/log/messages)  Any
>>>> > critical errors reported in logs
>>>> >
>>>> > §  Disk INODE utilization Warning at 90%, Alarm
>>>> > at 95%
>>>> >
>>>> > §  System process monitoring (ex. Set of services on a server if they
>>>> > are running or not)  Alarm if services are not running.
>>>> >
>>>> > §  Server Network Interface Up or Down (Alarm if server doesn’t
>>>> > respond)
>>>> >  Regards
>>>> >
>>>>
>>>> You would add an additional job in the prometheus.yaml file to scrape
>>>> metrics from each Windows machine containing the WMI Exporter. Then
>>>> you'd add additional alerting rules & dashboards as needed.
>>>>
>>>>
>>>> --
>>>> Stuart Clark
>>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Prometheus Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to promethe...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/prometheus-users/d0ba4aa0-9a3a-4759-b581-fdf9b4dbf657%40googlegroups.com
>>> <https://groups.google.com/d/msgid/prometheus-users/d0ba4aa0-9a3a-4759-b581-fdf9b4dbf657%40googlegroups.com?utm_medium=email_source=footer>
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/77d7bdef-e172-48c4-a905-e454bca92526o%40googlegroups.com
> <https://groups.google.com/d/msgid/prometheus-users/77d7bdef-e172-48c4-a905-e454bca92526o%40googlegroups.com?utm_medium=email_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZutuOLR09TFe-jxnuKAr0%3DOKDxuhq%3DWYxZW1WSXC-gm1w%40mail.gmail.com.

Re: [prometheus-users] Prometheus Querying

[Murali Krishna Kanagala]

Take a look these queries
https://www.robustperception.io/which-are-my-biggest-metrics

Based on these you can write queries to get the data you are looking for.

On Thu, Jun 4, 2020, 5:44 AM Nishant Ketu  wrote:

> Hi All ,
>
> I need few values about my Prometheus server for calculation the RAM
> required for cardinality and ingestion. The required values are:
>
>-
> *Average Labels Per Time Series* *
>-
> *Number of Unique Label Pairs* *
>- *Average Bytes per Label Pair**
>
> How could I query for the above values? Please help.
>
> Thanks
>
> The information in this message may be proprietary and/or confidential,
> and protected from disclosure. If the reader of this message is not the
> intended recipient, you are hereby notified that any dissemination,
> distribution or copying of this communication is strictly prohibited. If
> you have received this communication in error, please notify ATMECS and
> delete it from your computer.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/a1d772e4-0d56-48a6-b381-f148db5cafcf%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZuiZ_yA4XSi-ANa49SVLq0bO-cysd1UdzKZLSzONvPnjA%40mail.gmail.com.

Re: [prometheus-users] Windows Exporter SetUp

[Murali Krishna Kanagala]

Wmi exporter is the windows equivalent of node exporter. You need to run it
as a service and add that machine to scrape config. Never heard of windows
exporter.

On Thu, Jun 4, 2020, 11:24 AM Freddy Mack  wrote:

> I have Download the wmi_exporter on localhost and clients- but I see the
> blogs that they changed into wmi into windows_exporter is that necessary ?
> If I use wmi_exporter - I have just downloaded on client windows machines-
> what all Steps required it to work and get the metrics on Grafana.
> I have added all the client windows servers in prometheus.yml as well
>
>
> On Thursday, June 4, 2020 at 10:58:11 AM UTC-5, Stuart Clark wrote:
>>
>> On 2020-06-04 16:42, Freddy Mack wrote:
>> > Hello,
>> >
>> > I have installed wmi_exporter on the client Windows servers and the
>> > localhost (CentOS7) on which Grafana 7.0.3 and prometheus installed
>> > on.
>> > I need the procedure to get windows_exporter to download and install
>> > and configure on all the environment and to pull the Metrics on
>> > Grafana Dashboard.
>> > I have already set the Linux Metrics in place on Grafana and want to
>> > Add the Windows as well.
>> >
>> > Metrics are:
>> >
>> > §CPU
>> > utilization   Warning at 89%, Alarm at
>> > 90%
>> >
>> > §  Memory utilization  Warning at 90%, Alarm
>> > at 95%
>> >
>> > §  File System utilization Warning at 90%, Alarm
>> > at 95%
>> >
>> > §  System messages (ex. Errors in /var/log/messages)  Any
>> > critical errors reported in logs
>> >
>> > §  Disk INODE utilization Warning at 90%, Alarm
>> > at 95%
>> >
>> > §  System process monitoring (ex. Set of services on a server if they
>> > are running or not)  Alarm if services are not running.
>> >
>> > §  Server Network Interface Up or Down (Alarm if server doesn’t
>> > respond)
>> >  Regards
>> >
>>
>> You would add an additional job in the prometheus.yaml file to scrape
>> metrics from each Windows machine containing the WMI Exporter. Then
>> you'd add additional alerting rules & dashboards as needed.
>>
>>
>> --
>> Stuart Clark
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/d0ba4aa0-9a3a-4759-b581-fdf9b4dbf657%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZsdkHpSYZzBeSV%2BD9m4Q3LFsVnxApL3wsu5E2vP%2BX0oyg%40mail.gmail.com.

Re: [prometheus-users] Converting value from process exporter to percent in grafana for better representation

[Murali Krishna Kanagala]

The metrics you have shared above are actual memory usage in bytes. To
calculate the percentage we need to the memory capacity of the server you
are monitoring. There should be another metric giving that value.
Then you can calculate the percent usage by 100*  (memory_used_bytes/Total
memory bytes).

On Thu, May 28, 2020, 11:15 AM Pooja Chauhan  wrote:

> Hi ,
> I am using process-exporter to monitor processes in my linux machine. The
> output of process exporter gives me value of memory as below :
>  namedprocess_namegroup_memory_bytes{groupname="",memtype="resident"}
> 1.56147712e+08
> namedprocess_namegroup_memory_bytes{groupname="***",memtype="swapped"}
> 0
> namedprocess_namegroup_memory_bytes{groupname="***",memtype="virtual"}
> 1.24284928e+09
>
> Can you pls suggest how to convert these resident and virtaul memory
> values to percenatege for better representaion in grafana panel.I have also
> enabled percent nob in
> Visualization tab as in the screen shot attached but that give weird value
> too.Need help.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/246d487c-a78e-4c66-b5b0-6bf0541f73f8%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZtb3Tq7oVZ8otJQNfmKKzMgFX-WszHUBgDxqxv2fs78nQ%40mail.gmail.com.

Re: [prometheus-users] Re: I want to monitor ssl cert expiry of my urls using ssl exporter.Need help

[Murali Krishna Kanagala]

Okay. Try changing the query as i suggested above.

On Wed, May 13, 2020 at 8:45 AM Pooja Chauhan  wrote:

>  I have  imported the grafana dashboard for blackbox exporter
> here(dashboard no-7587) https://grafana.com/grafana/dashboards/7587
>
> On Wednesday, May 13, 2020 at 6:58:21 PM UTC+5:30, Murali Krishna Kanagala
> wrote:
>>
>> What Dashboard are you using? Do you have a link from Grafana website?
>> If you use the metric *probe_ssl_earliest_cert_expiry *you should
>> multiply it with 1000 and use -MM-DD HH:mm:ss (on any date specific
>> unit) unit. This will give absolute date when the cert expires.
>> If you want to get how much time left then use time *from **now *unit in
>> Grafana. it will give you how much time left for the expiry.
>>
>> So your query in Grafana would be  *1000 *
>> probe_ssl_earliest_cert_expiry{your_labels}.  *Thats all, no need of
>> subtracting current time.
>>
>>
>> On Wed, May 13, 2020 at 4:36 AM Pooja Chauhan  wrote:
>>
>>> The other one where it is not working is here
>>>
>>> On Wednesday, May 13, 2020 at 3:04:09 PM UTC+5:30, Pooja Chauhan wrote:
>>>>
>>>> I update grafana to 6.7.3 after discussing here then also it is the
>>>> same issue. Both my panel is configured same way only then I don't know
>>>> what is the issue . I have  imported the grafana dashboard for blackbox
>>>> exporter here(dashboard no-7587) the query is same for both panels.
>>>>
>>>> On Wednesday, May 13, 2020 at 2:35:20 PM UTC+5:30, Julius Volz wrote:
>>>>>
>>>>> It seems your issue is not with the Blackbox Exporter, but with
>>>>> getting a Grafana panel correctly configured then. Or as your comment "BTW
>>>>> i am using grafana 6.7.2 and 6.7.3 , for 6.7.3 it works and for 6.7.2 it
>>>>> does not" suggests, it might be a problem with the specific Grafana
>>>>> version, who knows.
>>>>>
>>>>> On Wed, May 13, 2020 at 10:54 AM Pooja Chauhan 
>>>>> wrote:
>>>>>
>>>>>> Hi I already tried through blackbox exporter but grafana panel does
>>>>>> not show the correct information of ssl expiry related data.So checking
>>>>>> other alternatives here .I have already asked about that in this group :
>>>>>> https://groups.google.com/forum/#!topic/prometheus-users/LA0X83H24Fg
>>>>>>   check my problem here.
>>>>>>
>>>>>> On Wednesday, May 13, 2020 at 2:19:29 PM UTC+5:30, Pooja Chauhan
>>>>>> wrote:
>>>>>>>
>>>>>>> Hi ,
>>>>>>> Need help in monitoring ssl cert of urls.I was exploring ssl
>>>>>>> exporter but could not get much info on how to download and use 
>>>>>>> it.Please
>>>>>>> suggest.
>>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "Prometheus Users" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to promethe...@googlegroups.com.
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/prometheus-users/12db4602-2ccc-46cc-8bf3-dce73491847f%40googlegroups.com
>>>>>> <https://groups.google.com/d/msgid/prometheus-users/12db4602-2ccc-46cc-8bf3-dce73491847f%40googlegroups.com?utm_medium=email_source=footer>
>>>>>> .
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Julius Volz
>>>>> PromLabs - promlabs.com
>>>>>
>>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Prometheus Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to promethe...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/prometheus-users/17013b45-dd69-44e8-8ef6-820ee1c95242%40googlegroups.com
>>> <https://groups.google.com/d/msgid/prometheus-users/17013b45-dd69-44e8-8ef6-820ee1c95242%40googlegroups.com?utm_medium=email_source=footer>
>>> .
>>>
>>
>>
>> --
>> Regards,
>> Murali Krishna Kanagala
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/a324ad0a-4628-40fb-a746-ad63a2e3da31%40googlegroups.com
> <https://groups.google.com/d/msgid/prometheus-users/a324ad0a-4628-40fb-a746-ad63a2e3da31%40googlegroups.com?utm_medium=email_source=footer>
> .
>


-- 
Regards,
Murali Krishna Kanagala

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZve3dn8_%3DpCuU%3DOk9g4YxOD2XKyqBDxt8Zd%2Bi3FtSHPwQ%40mail.gmail.com.

Re: [prometheus-users] Re: I want to monitor ssl cert expiry of my urls using ssl exporter.Need help

[Murali Krishna Kanagala]

What Dashboard are you using? Do you have a link from Grafana website?
If you use the metric *probe_ssl_earliest_cert_expiry *you should multiply
it with 1000 and use -MM-DD HH:mm:ss (on any date specific unit) unit.
This will give absolute date when the cert expires.
If you want to get how much time left then use time *from **now *unit in
Grafana. it will give you how much time left for the expiry.

So your query in Grafana would be  *1000 *
probe_ssl_earliest_cert_expiry{your_labels}.  *Thats all, no need of
subtracting current time.


On Wed, May 13, 2020 at 4:36 AM Pooja Chauhan  wrote:

> The other one where it is not working is here
>
> On Wednesday, May 13, 2020 at 3:04:09 PM UTC+5:30, Pooja Chauhan wrote:
>>
>> I update grafana to 6.7.3 after discussing here then also it is the same
>> issue. Both my panel is configured same way only then I don't know what is
>> the issue . I have  imported the grafana dashboard for blackbox exporter
>> here(dashboard no-7587) the query is same for both panels.
>>
>> On Wednesday, May 13, 2020 at 2:35:20 PM UTC+5:30, Julius Volz wrote:
>>>
>>> It seems your issue is not with the Blackbox Exporter, but with getting
>>> a Grafana panel correctly configured then. Or as your comment "BTW i am
>>> using grafana 6.7.2 and 6.7.3 , for 6.7.3 it works and for 6.7.2 it does
>>> not" suggests, it might be a problem with the specific Grafana version, who
>>> knows.
>>>
>>> On Wed, May 13, 2020 at 10:54 AM Pooja Chauhan 
>>> wrote:
>>>
>>>> Hi I already tried through blackbox exporter but grafana panel does not
>>>> show the correct information of ssl expiry related data.So checking other
>>>> alternatives here .I have already asked about that in this group :
>>>> https://groups.google.com/forum/#!topic/prometheus-users/LA0X83H24Fg
>>>>   check my problem here.
>>>>
>>>> On Wednesday, May 13, 2020 at 2:19:29 PM UTC+5:30, Pooja Chauhan wrote:
>>>>>
>>>>> Hi ,
>>>>> Need help in monitoring ssl cert of urls.I was exploring ssl exporter
>>>>> but could not get much info on how to download and use it.Please suggest.
>>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Prometheus Users" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to promethe...@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/prometheus-users/12db4602-2ccc-46cc-8bf3-dce73491847f%40googlegroups.com
>>>> <https://groups.google.com/d/msgid/prometheus-users/12db4602-2ccc-46cc-8bf3-dce73491847f%40googlegroups.com?utm_medium=email_source=footer>
>>>> .
>>>>
>>>
>>>
>>> --
>>> Julius Volz
>>> PromLabs - promlabs.com
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/17013b45-dd69-44e8-8ef6-820ee1c95242%40googlegroups.com
> <https://groups.google.com/d/msgid/prometheus-users/17013b45-dd69-44e8-8ef6-820ee1c95242%40googlegroups.com?utm_medium=email_source=footer>
> .
>


-- 
Regards,
Murali Krishna Kanagala

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZuzKMXm9BC9EKHraCo8CDB0xt4BBUQ0pj8_qaYb%3DAfqwQ%40mail.gmail.com.

Re: [prometheus-users] Prometheus Adapter

[Murali Krishna Kanagala]

 In order to expose metrics beyond CPU and memory to Kubernetes for
autoscaling, you’ll need an “adapter” that serves the custom metrics API.
Since you’ve got Prometheus metrics, it makes sense to use the Prometheus
adapter to serve metrics out of Prometheus.

On Wed, Apr 29, 2020 at 1:43 PM Radha R4 
wrote:

> https://github.com/DirectXMan12/k8s-prometheus-adapter
>
> On Wednesday, April 29, 2020 at 11:39:16 AM UTC-7, Julien Pivotto wrote:
>>
>> On 29 Apr 11:36, Radha R4 wrote:
>> > What is a Prometheus adapter in simple terms?
>>
>>
>> Can you please specify the context?
>>
>> We have a glossary here:
>> https://prometheus.io/docs/introduction/glossary/#prometheus
>>
>> But I don't see 'adapter'.
>>
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> Groups "Prometheus Users" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> an email to promethe...@googlegroups.com.
>> > To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-users/7dd93818-bfce-49ab-a2eb-9dd014220670%40googlegroups.com.
>>
>>
>>
>> --
>>  (o-Julien Pivotto
>>  //\Open-Source Consultant
>>  V_/_   Inuits - https://www.inuits.eu
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/efde4ac0-1376-4504-bc15-504d16a11956%40googlegroups.com
> <https://groups.google.com/d/msgid/prometheus-users/efde4ac0-1376-4504-bc15-504d16a11956%40googlegroups.com?utm_medium=email_source=footer>
> .
>


-- 
Regards,
Murali Krishna Kanagala

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZsvj7UHrkgzjgLhck9TsOQsa46YXbMxWfqW0zm16cn4vA%40mail.gmail.com.

Re: [prometheus-users] Blackbox exporter probe_ssl_earliest_cert_expiry giving wrong value in grafana

[Murali Krishna Kanagala]

Grafana query:
probe_ssl_earliest_cert_expiry{instance=~"$target"} * 1000

Grafana unit: any date unit

On Thu, Apr 23, 2020, 5:33 AM James Eduard Andaya <
jameseduard.and...@gmail.com> wrote:

> Hi sir thanks for the input how, im using those on grafana
>
> probe_ssl_earliest_cert_expiry{instance=~"$target"} - time()
>
>
> [image: probe.PNG]
>
>
> Regards,
> James
>
> On Thursday, April 23, 2020 at 6:05:24 PM UTC+8, Murali Krishna Kanagala
> wrote:
>>
>> Hi James,
>>
>> Multiply the expiry value you receive from
>> probe_ssl_earliest_cert_expiry  with thousand when you use it on Grafana.
>> We can directly apply the units on Grafana without subtracting the time()
>> from it.
>>
>> Thanks,
>> Murali Kanagala
>>
>> On Thu, Apr 23, 2020, 4:26 AM James Eduard Andaya 
>> wrote:
>>
>>> I have configured and installed blackbox exporter to monitor application
>>> url and SSL expiry in grafana dashboard , I see the SSL expiry is showing
>>> wrong
>>>
>>> probe_ssl_earliest_cert_expiry{instance=~"$target"} -time()
>>>
>>> but on priometheus panel i was able to get the correct data of
>>> expiration.
>>>
>>>
>>> https://github.com/prometheus/blackbox_exporter/issues/607
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Prometheus Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to promethe...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/prometheus-users/77995789-75a7-4d14-93df-9c7d20d97221%40googlegroups.com
>>> <https://groups.google.com/d/msgid/prometheus-users/77995789-75a7-4d14-93df-9c7d20d97221%40googlegroups.com?utm_medium=email_source=footer>
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/35888b1a-4c08-4f9c-8db3-2ff8e8416104%40googlegroups.com
> <https://groups.google.com/d/msgid/prometheus-users/35888b1a-4c08-4f9c-8db3-2ff8e8416104%40googlegroups.com?utm_medium=email_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZtVB_LyegyAwstAM78nZMzAHr217xGPgGNFgUtLfxjLnQ%40mail.gmail.com.

Re: [prometheus-users] Blackbox exporter probe_ssl_earliest_cert_expiry giving wrong value in grafana

[Murali Krishna Kanagala]

Hi James,

Multiply the expiry value you receive from probe_ssl_earliest_cert_expiry
with thousand when you use it on Grafana. We can directly apply the units
on Grafana without subtracting the time() from it.

Thanks,
Murali Kanagala

On Thu, Apr 23, 2020, 4:26 AM James Eduard Andaya <
jameseduard.and...@gmail.com> wrote:

> I have configured and installed blackbox exporter to monitor application
> url and SSL expiry in grafana dashboard , I see the SSL expiry is showing
> wrong
>
> probe_ssl_earliest_cert_expiry{instance=~"$target"} -time()
>
> but on priometheus panel i was able to get the correct data of expiration.
>
>
> https://github.com/prometheus/blackbox_exporter/issues/607
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/77995789-75a7-4d14-93df-9c7d20d97221%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZvqaOmMYyx0QZwcLL1LEPn1Vda5OuAEy0hTOZT0h2n5Fw%40mail.gmail.com.

Re: [prometheus-users] Prometheus dashboard exposed over alb-ingress-controller is not loading fully

[Murali Krishna Kanagala]

Nice! Here are some more details about the web external url flag
https://www.robustperception.io/using-external-urls-and-proxies-with-prometheus

On Tue, Apr 21, 2020, 10:29 PM vikram yerneni  wrote:

> Actually, I fixed the issue Murali.
> I added the /* after the url defined under hosts section and it worked
> like charm...
>
>
> Thanks
>
> On Tuesday, April 21, 2020 at 10:10:13 PM UTC-5, vikram yerneni wrote:
>>
>> I tried to add it over here Murali:
>> https://github.com/helm/charts/blob/master/stable/prometheus/values.yaml#L555
>> It did nothing... Still same...
>>
>> Thanks again for helping me dude...
>>
>> On Tuesday, April 21, 2020 at 9:10:30 PM UTC-5, Murali Krishna Kanagala
>> wrote:
>>>
>>> You can add the arguments here.
>>>
>>>
>>> https://github.com/helm/charts/blob/master/stable/prometheus/values.yaml#L615
>>>
>>> On Tue, Apr 21, 2020, 7:47 PM vikram yerneni  wrote:
>>>
>>>> Hi Murali, the container gets created by Helm charts. Let me see if I
>>>> can modify the values here.
>>>>
>>>> However, the page is loading (I just want to reiterate here). Its just
>>>> the additional *.css & *.js files are not loading...
>>>>
>>>> Thanks
>>>>
>>>> On Tuesday, April 21, 2020 at 6:29:33 PM UTC-5, Murali Krishna Kanagala
>>>> wrote:
>>>>>
>>>>> Try setting the --web.external-url in the container arguments.
>>>>>
>>>>>
>>>>> On Tue, Apr 21, 2020, 5:47 PM vikram yerneni 
>>>>> wrote:
>>>>>
>>>>>> Fellas, I am using Helm/Stable/Prometheus Server for my Metrics
>>>>>> datasource and the Prometheus Server Dashboard is exposed using 
>>>>>> alb-ingress
>>>>>> controller in AWS. Somehow the prometheus webpage is not loading fully 
>>>>>> (few
>>>>>> parts of the webpage are not getting loaded and throwing 404 errors). 
>>>>>> Here
>>>>>> is the Ingress configuration
>>>>>>
>>>>>> ingress:
>>>>>> ## If true, Prometheus server Ingress will be created
>>>>>>
>>>>>> ##
>>>>>>
>>>>>> enabled: true
>>>>>>
>>>>>>
>>>>>> ## Prometheus server Ingress annotations
>>>>>>
>>>>>> ##
>>>>>>
>>>>>> annotations:
>>>>>>
>>>>>>kubernetes.io/ingress.class: 'alb'
>>>>>>
>>>>>>#kubernetes.io/tls-acme: 'true'
>>>>>>
>>>>>>alb.ingress.kubernetes.io/scheme: internet-facing
>>>>>>
>>>>>>alb.ingress.kubernetes.io/load-balancer-attributes:
>>>>>> 'routing.http2.enabled=true,idle_timeout.timeout_seconds=60'
>>>>>>
>>>>>>alb.ingress.kubernetes.io/certificate-arn: certname
>>>>>>
>>>>>>alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80},
>>>>>> {"HTTPS":443}]'
>>>>>>
>>>>>>alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type":
>>>>>> "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443",
>>>>>> "StatusCode": "HTTP_301"}}'
>>>>>>
>>>>>> service:
>>>>>>
>>>>>>   annotations:
>>>>>>
>>>>>> alb.ingress.kubernetes.io/target-type: ip
>>>>>>
>>>>>> labels: {}
>>>>>>
>>>>>> path: /*
>>>>>>
>>>>>> hosts:
>>>>>>
>>>>>>- prometheus.company.com
>>>>>>
>>>>>>
>>>>>> ## Extra paths to prepend to every host configuration. This is
>>>>>> useful when working with annotation based services.
>>>>>>
>>>>>> extraPaths:
>>>>>>
>>>>>>  - path: /*
>>>>>>
>>>>>>backend:
>>>>>>
>>>>>>  serviceName: ssl-redirect
>>>>>>
>>>>

Re: [prometheus-users] Prometheus dashboard exposed over alb-ingress-controller is not loading fully

[Murali Krishna Kanagala]

You can add the arguments here.

https://github.com/helm/charts/blob/master/stable/prometheus/values.yaml#L615

On Tue, Apr 21, 2020, 7:47 PM vikram yerneni  wrote:

> Hi Murali, the container gets created by Helm charts. Let me see if I can
> modify the values here.
>
> However, the page is loading (I just want to reiterate here). Its just the
> additional *.css & *.js files are not loading...
>
> Thanks
>
> On Tuesday, April 21, 2020 at 6:29:33 PM UTC-5, Murali Krishna Kanagala
> wrote:
>>
>> Try setting the --web.external-url in the container arguments.
>>
>>
>> On Tue, Apr 21, 2020, 5:47 PM vikram yerneni  wrote:
>>
>>> Fellas, I am using Helm/Stable/Prometheus Server for my Metrics
>>> datasource and the Prometheus Server Dashboard is exposed using alb-ingress
>>> controller in AWS. Somehow the prometheus webpage is not loading fully (few
>>> parts of the webpage are not getting loaded and throwing 404 errors). Here
>>> is the Ingress configuration
>>>
>>> ingress:
>>> ## If true, Prometheus server Ingress will be created
>>>
>>> ##
>>>
>>> enabled: true
>>>
>>>
>>> ## Prometheus server Ingress annotations
>>>
>>> ##
>>>
>>> annotations:
>>>
>>>kubernetes.io/ingress.class: 'alb'
>>>
>>>#kubernetes.io/tls-acme: 'true'
>>>
>>>alb.ingress.kubernetes.io/scheme: internet-facing
>>>
>>>alb.ingress.kubernetes.io/load-balancer-attributes:
>>> 'routing.http2.enabled=true,idle_timeout.timeout_seconds=60'
>>>
>>>alb.ingress.kubernetes.io/certificate-arn: certname
>>>
>>>alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80},
>>> {"HTTPS":443}]'
>>>
>>>alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type":
>>> "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443",
>>> "StatusCode": "HTTP_301"}}'
>>>
>>> service:
>>>
>>>   annotations:
>>>
>>> alb.ingress.kubernetes.io/target-type: ip
>>>
>>> labels: {}
>>>
>>> path: /*
>>>
>>> hosts:
>>>
>>>- prometheus.company.com
>>>
>>>
>>> ## Extra paths to prepend to every host configuration. This is
>>> useful when working with annotation based services.
>>>
>>> extraPaths:
>>>
>>>  - path: /*
>>>
>>>backend:
>>>
>>>  serviceName: ssl-redirect
>>>
>>>  servicePort: use-annotation
>>>
>>>
>>> When I access prometheus.company.com, its getting properly redirected
>>> to prometheus.company.com/graph (assuming the redirect is working
>>> fine). However, some parts (*.js & *.css files) of the webpage is throwing
>>> 404 errors Anyone been through this issue before and if so, how y'all got
>>> it resolved?
>>>
>>> This configuration is set in the values.yaml file:
>>> https://github.com/helm/charts/blob/master/stable/prometheus/values.yaml
>>>
>>> Cheers!
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Prometheus Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to promethe...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/prometheus-users/da00c725-157c-49cb-b544-24aa2c9aa7d1%40googlegroups.com
>>> <https://groups.google.com/d/msgid/prometheus-users/da00c725-157c-49cb-b544-24aa2c9aa7d1%40googlegroups.com?utm_medium=email_source=footer>
>>> .
>>>
>>> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/0538e51b-9093-4347-adc0-c86f333bddcf%40googlegroups.com
> <https://groups.google.com/d/msgid/prometheus-users/0538e51b-9093-4347-adc0-c86f333bddcf%40googlegroups.com?utm_medium=email_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZt8iPPVdXysoWVMMbnOOFr59qiSbkaF6wSoU_mzpYF%2BGg%40mail.gmail.com.

Re: [prometheus-users] Prometheus dashboard exposed over alb-ingress-controller is not loading fully

[Murali Krishna Kanagala]

Try setting the --web.external-url in the container arguments.


On Tue, Apr 21, 2020, 5:47 PM vikram yerneni  wrote:

> Fellas, I am using Helm/Stable/Prometheus Server for my Metrics datasource
> and the Prometheus Server Dashboard is exposed using alb-ingress controller
> in AWS. Somehow the prometheus webpage is not loading fully (few parts of
> the webpage are not getting loaded and throwing 404 errors). Here is the
> Ingress configuration
>
> ingress:
> ## If true, Prometheus server Ingress will be created
>
> ##
>
> enabled: true
>
>
> ## Prometheus server Ingress annotations
>
> ##
>
> annotations:
>
>kubernetes.io/ingress.class: 'alb'
>
>#kubernetes.io/tls-acme: 'true'
>
>alb.ingress.kubernetes.io/scheme: internet-facing
>
>alb.ingress.kubernetes.io/load-balancer-attributes:
> 'routing.http2.enabled=true,idle_timeout.timeout_seconds=60'
>
>alb.ingress.kubernetes.io/certificate-arn: certname
>
>alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80},
> {"HTTPS":443}]'
>
>alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type":
> "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443",
> "StatusCode": "HTTP_301"}}'
>
> service:
>
>   annotations:
>
> alb.ingress.kubernetes.io/target-type: ip
>
> labels: {}
>
> path: /*
>
> hosts:
>
>- prometheus.company.com
>
>
> ## Extra paths to prepend to every host configuration. This is useful
> when working with annotation based services.
>
> extraPaths:
>
>  - path: /*
>
>backend:
>
>  serviceName: ssl-redirect
>
>  servicePort: use-annotation
>
>
> When I access prometheus.company.com, its getting properly redirected to
> prometheus.company.com/graph (assuming the redirect is working fine).
> However, some parts (*.js & *.css files) of the webpage is throwing 404
> errors Anyone been through this issue before and if so, how y'all got it
> resolved?
>
> This configuration is set in the values.yaml file:
> https://github.com/helm/charts/blob/master/stable/prometheus/values.yaml
>
> Cheers!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/da00c725-157c-49cb-b544-24aa2c9aa7d1%40googlegroups.com
> 
> .
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZsbfVbEoK28Z-fopoqkxts1tpBsTVmt3%2BDUvkTpxge%2Bug%40mail.gmail.com.

Re: [prometheus-users] inhibit all severity but how??

[Murali Krishna Kanagala]

Hi Danny,

Did you try using a reflex for the target_match?
And also adding another inhibitor rule  with target_match as critical?

On Wed, Apr 15, 2020, 7:17 AM Danny de Waard  wrote:

> I have a alrtmanger setup and prometheus setup that are working good.
> I cab inhibit severity 'warning' when maintenance is on.
>
> But how do i also inhibit critical (and/or) info messages when maintenance
> is on.
>
> So if my maintenance_mode == 1 all warning alerts are inhibited. But i
> also want critical to be inhibited.
>
> How do i do that?
>
> prometheus rules
> groups:
> - name: targets
>   rules:
>   - alert: MaintenanceMode
> expr: maintenance_mode == 1
> for: 1m
> labels:
>   severity: warning
> annotations:
>   summary: "This is a maintenance alert"
>   description: "Fires during maintenance mode and is routed to a
> blackhole by Alertmanager"
>   - alert: monitor_service_down
> expr: up == 0
> for: 40s
> labels:
>   severity: critical
> annotations:
>   summary: "A exporter service is non-operational"
>   description: "One of the exporters on {{ $labels.instance }} is or
> was down. Check the up/down dashboard in grafana.
> http://lsrv2289.linux.rabobank.nl:3000/d/RSNFpMXZz/up-down-monitor?refresh=1m
> "
>   - alert: server_down
> expr: probe_success == 0
> for: 30s
> labels:
>   severity: critical
> annotations:
>   summary: "Server is down (no probes are up)"
>   description: "Server {{ $labels.instance }} is down."
>   - alert: loadbalancer_down
> expr: loadbalancer_stats < 1
> for: 30s
> labels:
>   severity: critical
> annotations:
>   summary: "A loadbalancer is down"
>   description: "Loadbalancer for {{ $labels.instance }} is down."
>   - alert: high_cpu_load15
> expr: node_load15 > 4.5
> for: 900s
> labels:
>   severity: critical
> annotations:
>   summary: "Server under high load (load 15m) for 15 minutes."
>   description: "Host is under high load, the avg load 15m is at {{
> $value}}. Reported by instance {{ $labels.instance }} of job {{ $labels.job
> }}."
>
>
>
>
>
>
> alertmanager.yml
>
> global:
> route:
>   group_by: [instance,severity,job]
>   receiver: 'default'
>   routes:
>- match:
>   alertname: 'MaintenanceMode'
>  receiver: 'blackhole'
>- match:
>   severity: warning
>   job: PAT
>  receiver: 'pat'
>- match:
>   severity: warning
>   job: PROD
>  receiver: 'prod'
>- match:
>   severity: critical
>   job: PAT
>  receiver: 'pat-crit'
>- match:
>   severity: critical
>   job: PROD
>  receiver: 'prod-crit'
>  continue: true
>- match:
>   severity: critical
>   job: PROD
>  receiver: 'sms-waard'
>- match:
>   severity: info
>  receiver: 'info'
>- match:
>   severity: atombomb
>  receiver: 'webhook'
> receivers:
>   - name: 'default'
> email_configs:
>  - to: 'mailaddress' ##fill in your email
>from: 'alertmanager_defa...@superheroes.com'
>smarthost: 'localhost:25'
>require_tls: false
>   - name: 'pat'
> email_configs:
>  - to: 'mailaddress' ##fill in your email
>from: 'alertmanager_...@superheroes.com'
>smarthost: 'localhost:25'
>require_tls: false
>   - name: 'prod'
> email_configs:
>  - to: 'mailaddress, mailaddress' ##fill in your email
>from: 'alertmanager_p...@superheroes.com'
>smarthost: 'localhost:25'
>require_tls: false
>   - name: 'pat-crit'
> email_configs:
>  - to: 'mailaddress' ##fill in your email
>from: 'critical-alertmanager_...@superheroes.com'
>smarthost: 'localhost:25'
>require_tls: false
>   - name: 'prod-crit'
> email_configs:
>  - to: 'mailaddress, mailaddress' ##fill in your email
>from: 'critical-alertmanager_p...@superheroes.com'
>smarthost: 'localhost:25'
>require_tls: false
>   - name: 'info'
> email_configs:
>  - to: 'mailaddress' ##fill in your email
>from: 'alertmanager_i...@superheroes.com'
>smarthost: 'localhost:25'
>require_tls: false
>   - name: 'sms-waard'
> email_configs:
>  - to: 'mailaddress' ##fill in your email
>from: 'alertmanager_i...@superheroes.com'
>smarthost: 'localhost:25'
>require_tls: false
>   - name: 'webhook'
> webhook_configs:
>   - url: 'http://127.0.0.1:9000'
>   - name: 'blackhole'
>
> inhibit_rules:
> - source_match:
> alertname: MaintenanceMode
>   target_match:
> severity: warning
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> 

Re: [prometheus-users] Prometheus failing after few hours of restart.

[Murali Krishna Kanagala]

Looks like a memory issue. How much RAM you have on the server?

https://github.com/prometheus/prometheus/issues/4945

On Wed, Apr 15, 2020, 11:09 AM Ben Kochie  wrote:

> That looks like an incomplete stack trace. Can you get the full stack
> trace from your syslog?
>
> There's not enough information to go on here. Please mention what version
> of Prometheus this is, and include your prometheus.yml.
>
> On Wed, Apr 15, 2020 at 5:49 PM Pooja Chauhan 
> wrote:
>
>> Hi ,
>> For the past 2 days after restart of prometheus ,it stops in few hours.
>>
>> output of journalctl command is :
>>
>> -- Logs begin at Wed 2020-03-11 05:15:30 UTC, end at Wed 2020-04-15
>> 15:40:24 UTC. --
>> Apr 15 14:23:38 ip-172-28-48-25 systemd[1]: prometheus.service: Failed
>> with result 'exit-code'.
>> Apr 15 14:23:38 ip-172-28-48-25 systemd[1]: prometheus.service: Main
>> process exited, code=exited, status=2/INVALIDARGUMENT
>> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: created by
>> github.com/prometheus/prometheus/scrape.(*scrapePool).sync
>> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]:
>>  /app/scrape/scrape.go:854 +0x822
>> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]:
>> github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0xc051b85e80,
>> 0x14f46b0400, 0x6fc23ac00, 0x0)
>> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: goroutine 573 [select]:
>> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]:
>>  /app/scrape/scrape.go:336 +0x45d
>> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: created by
>> github.com/prometheus/prometheus/scrape.(*scrapePool).sync
>> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]:
>>  /app/scrape/scrape.go:774 +0x113
>> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]:
>> github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0xc051b85d80,
>> 0x14f46b0400, 0x6fc23ac00, 0x0)
>> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: goroutine 572 [select,
>> 1 minutes]:
>> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]:
>>  /app/scrape/scrape.go:336 +0x45d
>> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: created by
>> github.com/prometheus/prometheus/scrape.(*scrapePool).sync
>> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]:
>>  /app/scrape/scrape.go:774 +0x113
>> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]:
>> github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0xc051b85c80,
>> 0x14f46b0400, 0x6fc23ac00, 0x0)
>> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: goroutine 571 [select,
>> 1 minutes]:
>> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]:
>>  /app/scrape/scrape.go:336 +0x45d
>> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: created by
>> github.com/prometheus/prometheus/scrape.(*scrapePool).sync
>> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]:
>>  /app/scrape/scrape.go:813 +0x487
>> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]:
>> github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0xc051b85b80,
>> 0x14f46b0400, 0x6fc23ac00, 0x0)
>> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]:
>>  /app/scrape/scrape.go:471 +0x111
>>
>>
>> and ubuntu@ip-172-28-48-25:/etc/systemd/system$ sudo service prometheus
>> status
>> ● prometheus.service - Prometheus Server
>>Loaded: loaded (/etc/systemd/system/prometheus.service; enabled;
>> vendor preset: enabled)
>>Active: failed (Result: exit-code) since Wed 2020-04-15 14:23:38 UTC;
>> 1h 25min ago
>>  Docs: https://prometheus.io/docs/introduction/overview/
>>   Process: 6138 ExecStart=/etc/prometheus/prometheus
>> --config.file=/etc/prometheus/prometheus.yml
>> --storage.tsdb.path=/var/lib/prometheus --storage.tsdb.retention=7d (c
>>  Main PID: 6138 (code=exited, status=2)
>>
>> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]:
>> github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0xc051b85d80,
>> 0x14f46b0400, 0x6fc23ac00, 0x0)
>> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]:
>>  /app/scrape/scrape.go:774 +0x113
>> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: created by
>> github.com/prometheus/prometheus/scrape.(*scrapePool).sync
>> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]:
>>  /app/scrape/scrape.go:336 +0x45d
>> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: goroutine 573 [select]:
>> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]:
>> github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0xc051b85e80,
>> 0x14f46b0400, 0x6fc23ac00, 0x0)
>> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]:
>>  /app/scrape/scrape.go:854 +0x822
>> Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: created by
>> github.com/prometheus/prometheus/scrape.(*scrapePool).sync
>> Apr 15 14:23:38 ip-172-28-48-25 systemd[1]: prometheus.service: Main
>> process exited, code=exited, status=2/INVALIDA
>>
>>
>>
>>
>> Please suggest how to resolve this
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to 

Re: [prometheus-users] Re: Pattern(s) for having a counter in short-lived applications (batch jobs)

[Murali Krishna Kanagala]

Thanks Brian!

On Wed, Apr 15, 2020, 5:49 AM Brian Candler  wrote:

> pushgateway doesn't aggregate counters, but statsd_exporter does.  This
> sounds more like what you need.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/f710266e-0cb5-439a-a19c-cbd338a24f27%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZuYy3TPD5q47YkcSTp8%2By9JGwTryPVbfmKrqkHoTkpHGw%40mail.gmail.com.

Re: [prometheus-users] Grafana support for Prometheus Webhook Alerts?

[Murali Krishna Kanagala]

It can take alertmanager as a data source.

On Sun, Apr 5, 2020, 11:34 AM Steve  wrote:

> Hi
> Does Grafana support HTTP/HTTPS webhook alerts originating from the
> Prometheus alert manager(s)?
>
> Thanks
>
> -Steve
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/3463faa3-fec6-4b77-be09-a66c80698fa2%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZuPjFO11aZg%2BMOQVfN5VAsG5zEtNcDgjkHvWHwCxHcSmQ%40mail.gmail.com.

Re: [prometheus-users] Re: Start recording rule until N mins of data is collected

[Murali Krishna Kanagala]

Every time the data is scrapped Prometheus does that calculation and record
the custom metric. 5min here is a sliding window with latest metric's time
stamp as the curren time and the previous one (if it exists within last
5min) as the previous metric. If there is no metric in last 5 min you'll
not see any result.

On Tue, Apr 7, 2020, 1:23 PM Brian Candler  wrote:

> > What happens when I do rate(metricsX[5m]) but only 1 min of data is
> present?
>
> rate() takes the first and last point within the given time window, and
> calculates the rate between them.  So: as long as there are at least two
> data points in the window(*), you'll get a result.  If not, you'll get no
> result.
>
> (*) And the second is higher than the first, since rate works with
> counters.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/686f17dc-988c-4c14-a6bd-a648a9f80a56%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZsFsAC8PqQ1_eGJnRYyyyxx_EaW24fm0rJa4ib65Oz%3DfA%40mail.gmail.com.

Re: [prometheus-users] Removed target sends resolved alert

[Murali Krishna Kanagala]

If the target is removed from the scrape config the alert will resolve
anyway. If you want to persist the alert set send_resolved to false in
alertmanager.

On Thu, Mar 26, 2020, 3:16 PM Stuart Clark  wrote:

> On 26/03/2020 19:09, Amanda H. L. de Andrade wrote:
> > Hi,
> >
> > If a remove a target that has a alert in firing state, alertmanager
> > sends a "resolved" alert to ours receivers.
> >
> > How can I prevent that?
>
>
> That's expected as the alert is no longer firing.
>
>
> --
> Stuart Clark
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/01b812c3-a941-5dd7-449a-f6f86c30b9ed%40Jahingo.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZt_-xYD85GyYWtORg3Pg_VeRmh4P32_Dm6nPi1gEOy2uQ%40mail.gmail.com.

Re: [prometheus-users] Postman Exporter

[Murali Krishna Kanagala]

Thank you Sebastian!

On Fri, Mar 27, 2020, 5:51 PM Sebastian Ebling 
wrote:

> Hi,
>
> I created an exporter that runs Postman collections and returns metrics on
> those runs.
>
> You can find the project here:
> https://github.com/hudeldudel/postman-exporter
>
> I would appreciate feedback.
>
> The included example is very simple and could also have been done with the
> blackbox exporter. But if you know Postman a little better, you also know
> that a lot is possible with it that is difficult or impossible to achieve
> with the blackbox exporter.
>
> I use it to monitor APIs that are secured with OAuth2 and mTLS.
> In another case, a complete OpenID Connect login with code flow (including
> login to the IDP, cookie handling, CSRF token handling) is carried out and
> monitored.
>
> regards
>
> Sebastian
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/0a4ff8e8-e7a0-44e6-b357-d3288de33c81%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZv1r9-q6QD0J2oBy9HEDYDBOG6T_XAszzt-CZ7sZSu9Nw%40mail.gmail.com.

Re: [prometheus-users] Filter alertmanager notification like slack using templates

[Murali Krishna Kanagala]

I dont think the webhook receiver config supports templates. If your api
receiver does not accept the json payload sent by alertmanager then you may
need to write a custom translator something like a small flask app that can
receive the data from alertmanager, process it and push it to your api.

On Tue, Mar 24, 2020, 11:13 PM Rahul Hada  wrote:

> I am able to get json data, but was looking for more appropriate template
> so that could send only required parameters to the api. Any suggestions for
> that would be appreciated.
>
> On Wed, 25 Mar, 2020, 9:30 AM Murali Krishna Kanagala, <
> kanagalamur...@gmail.com> wrote:
>
>> To send an alert to a web hook there is no need for a template. The
>> receiver api receivs the alerts in as json payload.
>>
>> https://prometheus.io/docs/alerting/configuration/#webhook_config
>>
>>
>> On Tue, Mar 24, 2020, 12:46 AM Rahul Hada  wrote:
>>
>>> Hi All,
>>> We are using templates to refine the parameters going to the slack
>>> channel from alertmanager, we do have one api to which we are sending alert
>>> notifications but while using same templates for api, reloading the
>>> alertmanager.yml gives error " field title not found in type config.plain".
>>> I have rechecked if there is any indentation error in yaml file. Please
>>> suggest whether it is possible for api or not. If yes , please guide me
>>> towards it.
>>>
>>> Template file :-
>>> {{ define "webhook.default.title" }}{{ template "__subject" . }}{{ end }}
>>> {{ define "webhook.default.username" }}{{ template "__alertmanager" .
>>> }}{{ end }}
>>> {{ define "webhook.default.fallback" }}{{ template
>>> "webhook.default.title" . }} | {{ template "webhook.default.titlelink" .
>>> }}{{ end }}
>>> {{ define "webhook.default.pretext" }}{{ end }}
>>> {{ define "webhook.default.titlelink" }}{{ template "__alertmanagerURL"
>>> . }}{{ end }}
>>> {{ define "webhook.default.iconemoji" }}{{ end }}
>>> {{ define "webhook.default.iconurl" }}{{ end }}
>>> {{ define "webhook.default.text" }}{{ end }}
>>> {{ define "webhook.default.footer" }}{{ end }}
>>>
>>>
>>> Alertmanager.yml file :-
>>> - name: "en_api"
>>>   webhook_configs:
>>>   - url: http://dticket.xxx.com/prometheus_api.php
>>> title: '{{ template "webhook.default.title" . }}'
>>> text: |-
>>>   {{ range .Alerts }}
>>>  *ALERT:* {{ .Annotations.summary }} - `{{ .Labels.severity }}`
>>> *DESCRIPTION:* {{ .Annotations.description }}
>>> {{ .Annotations.identifier }}
>>>   {{ end }}
>>>
>>>
>>> Thanks in Advance
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Prometheus Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to prometheus-users+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/prometheus-users/c01b22eb-828d-4df2-bcea-f6e448ab2ca5%40googlegroups.com
>>> <https://groups.google.com/d/msgid/prometheus-users/c01b22eb-828d-4df2-bcea-f6e448ab2ca5%40googlegroups.com?utm_medium=email_source=footer>
>>> .
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to prometheus-users+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-users/CAKimyZvybwitKXoPBy5MbxS6NDab-bybKecsr2DiqK8Z55s20w%40mail.gmail.com
>> <https://groups.google.com/d/msgid/prometheus-users/CAKimyZvybwitKXoPBy5MbxS6NDab-bybKecsr2DiqK8Z55s20w%40mail.gmail.com?utm_medium=email_source=footer>
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZu2zpHV2zLYR7Rnh9FpXe0zoQon3J59tAA47_kdX_tfWg%40mail.gmail.com.

Re: [prometheus-users] Filter alertmanager notification like slack using templates

[Murali Krishna Kanagala]

To send an alert to a web hook there is no need for a template. The
receiver api receivs the alerts in as json payload.

https://prometheus.io/docs/alerting/configuration/#webhook_config


On Tue, Mar 24, 2020, 12:46 AM Rahul Hada  wrote:

> Hi All,
> We are using templates to refine the parameters going to the slack channel
> from alertmanager, we do have one api to which we are sending alert
> notifications but while using same templates for api, reloading the
> alertmanager.yml gives error " field title not found in type config.plain".
> I have rechecked if there is any indentation error in yaml file. Please
> suggest whether it is possible for api or not. If yes , please guide me
> towards it.
>
> Template file :-
> {{ define "webhook.default.title" }}{{ template "__subject" . }}{{ end }}
> {{ define "webhook.default.username" }}{{ template "__alertmanager" . }}{{
> end }}
> {{ define "webhook.default.fallback" }}{{ template "webhook.default.title"
> . }} | {{ template "webhook.default.titlelink" . }}{{ end }}
> {{ define "webhook.default.pretext" }}{{ end }}
> {{ define "webhook.default.titlelink" }}{{ template "__alertmanagerURL" .
> }}{{ end }}
> {{ define "webhook.default.iconemoji" }}{{ end }}
> {{ define "webhook.default.iconurl" }}{{ end }}
> {{ define "webhook.default.text" }}{{ end }}
> {{ define "webhook.default.footer" }}{{ end }}
>
>
> Alertmanager.yml file :-
> - name: "en_api"
>   webhook_configs:
>   - url: http://dticket.xxx.com/prometheus_api.php
> title: '{{ template "webhook.default.title" . }}'
> text: |-
>   {{ range .Alerts }}
>  *ALERT:* {{ .Annotations.summary }} - `{{ .Labels.severity }}`
> *DESCRIPTION:* {{ .Annotations.description }}
> {{ .Annotations.identifier }}
>   {{ end }}
>
>
> Thanks in Advance
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/c01b22eb-828d-4df2-bcea-f6e448ab2ca5%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZvybwitKXoPBy5MbxS6NDab-bybKecsr2DiqK8Z55s20w%40mail.gmail.com.

Re: [prometheus-users] Working of Textfile Collector.

[Murali Krishna Kanagala]

Good to know that vmstat is for a different purpose.

On Sun, Mar 8, 2020, 1:52 AM Brian Candler  wrote:

> The vmstat collector isn't anything to do with VMWare: it returns stats
> about the Linux virtual memory subsystem from /proc/vmstat.
>
> $ curl /metrics | grep -i vmstat
> ...
> node_scrape_collector_duration_seconds{collector="vmstat"} 0.000644674
> node_scrape_collector_success{collector="vmstat"} 1
> # HELP node_vmstat_oom_kill /proc/vmstat information field oom_kill.
> # TYPE node_vmstat_oom_kill untyped
> node_vmstat_oom_kill 0
> # HELP node_vmstat_pgfault /proc/vmstat information field pgfault.
> # TYPE node_vmstat_pgfault untyped
> node_vmstat_pgfault 2.48255587e+09
> # HELP node_vmstat_pgmajfault /proc/vmstat information field pgmajfault.
> # TYPE node_vmstat_pgmajfault untyped
> node_vmstat_pgmajfault 1.4914231e+07
> # HELP node_vmstat_pgpgin /proc/vmstat information field pgpgin.
> # TYPE node_vmstat_pgpgin untyped
> node_vmstat_pgpgin 1.4020742e+08
> # HELP node_vmstat_pgpgout /proc/vmstat information field pgpgout.
> # TYPE node_vmstat_pgpgout untyped
> node_vmstat_pgpgout 1.040636696e+09
> # HELP node_vmstat_pswpin /proc/vmstat information field pswpin.
> # TYPE node_vmstat_pswpin untyped
> node_vmstat_pswpin 19131
> # HELP node_vmstat_pswpout /proc/vmstat information field pswpout.
> # TYPE node_vmstat_pswpout untyped
> node_vmstat_pswpout 311500
>
> For examples of exposing the software version as a prometheus label, see:
> https://www.robustperception.io/exposing-the-software-version-to-prometheus
> https://www.robustperception.io/how-to-have-labels-for-machine-roles
>
> 
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/6f8cccd6-b8b2-4455-9cb0-6d5dd0188368%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZtB%2BZji9phf9oQS5EcysKYao20DSYdDxohxmL7JAKs9rg%40mail.gmail.com.

Re: [prometheus-users] Working of Textfile Collector.

[Murali Krishna Kanagala]

Also, did you try using the vmstat command line flag on node exporter?

On Sat, Mar 7, 2020, 12:31 PM Murali Krishna Kanagala <
kanagalamur...@gmail.com> wrote:

> Just drop a text file with data in Prometheus format and pass it as a
> command line arguments on your node exporter.
>
> https://prometheus.io/docs/concepts/data_model/
>
> https://github.com/prometheus/node_exporter/blob/master/README.md
>
> On Sat, Mar 7, 2020, 11:01 AM Yagyansh S. Kumar 
> wrote:
>
>> Hi. I am just now able to get my head around the working of textfile
>> collector. Can someone explain or provide a good link to understand its
>> working?
>>
>> Thanks!
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to prometheus-users+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-users/19715622-31e6-402e-8497-49448f6f0b95%40googlegroups.com
>> <https://groups.google.com/d/msgid/prometheus-users/19715622-31e6-402e-8497-49448f6f0b95%40googlegroups.com?utm_medium=email_source=footer>
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZsFA6aCv-FxOHmDbe629miqK1a8FcXOTtHO8d2YAAmaBg%40mail.gmail.com.

Re: [prometheus-users] Working of Textfile Collector.

[Murali Krishna Kanagala]

Just drop a text file with data in Prometheus format and pass it as a
command line arguments on your node exporter.

https://prometheus.io/docs/concepts/data_model/

https://github.com/prometheus/node_exporter/blob/master/README.md

On Sat, Mar 7, 2020, 11:01 AM Yagyansh S. Kumar 
wrote:

> Hi. I am just now able to get my head around the working of textfile
> collector. Can someone explain or provide a good link to understand its
> working?
>
> Thanks!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/19715622-31e6-402e-8497-49448f6f0b95%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZv_CY647%3D%2BbkJUHEz%2BA1CtRuCxMz5i14d4NN1_GMcD9nQ%40mail.gmail.com.

Re: [prometheus-users] How to expose a custom metrics from a pod/service in Kubernetes ?

[Murali Krishna Kanagala]

Sorry for the wrong link
https://medium.com/@zhimin.wen/custom-prometheus-metrics-for-apps-running-in-kubernetes-498d69ada7aa

On Tue, Mar 3, 2020, 10:04 PM Murali Krishna Kanagala <
kanagalamur...@gmail.com> wrote:

> I think this page can help you to write the sd config for kubernetes pods.
> https://github.com/nlighten/tomcat_exporter
>
> For scraping only master pod of a given service it should have a special
> label so that Prometheus can discover it. An alternative approach would be
> a stateful set where every pod in a service can be given different labels.
>
> Did you try reading about kube dd config to scrape service endpoints? If
> that is possible then you will be scraping the master pod if you scrape the
> service endpoint.
>
>
>
> On Fri, Feb 28, 2020, 12:01 AM SS. OO.  wrote:
>
>> Hi!
>>
>> I have a Kubernetes/Prometeus newbie question.
>>
>> I have a Kubernetes application that I want to scale up via Horizontal
>> Pod Autoscaler based on custom metrics reflecting the length of the app's
>> work queue backlog in the app's internal database.
>>
>> I can expose the metrics on a pod port in Prometheus format, so there is
>> more than one scrape endpoint for fail-over purposes. However I do not want
>> Prometheus to call the scrape endpoint on every pod's instance, only one
>> instance at a time and failover to another instance only if necessary. All
>> instance endpoints report the same data, and calling them in parallel would
>> serve no useful purpose and only increase a workload on storage queries
>> behind the app runtime.
>>
>> Right now the application is packaged as a Kubernetes Deployment. Do I
>> need to convert it for the above as a Service, or can Prometeus understand
>> intrinsically or be instructed in the configuration that only one instance
>> is to be called at a time?
>>
>> Also, can someone please provide an example of the snippet I need to
>> insert into
>>
>> https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml
>> so Prometheus picks up the endpoint? Assuming, for example, it is exposed
>> on pod port , path /metrics, service/pod named ABC or with label
>> appname=LABC?
>> Or do I simply tag the pod/service/deployment with some tag and Prometeus
>> starts to collect on it? If yes, how do I pack the endpoint
>> port+protocol+path into the tag?
>>
>> Thanks for the advice.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to prometheus-users+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-users/cabf6ba3-79c8-4f5e-865d-1fc51ba52c72%40googlegroups.com
>> <https://groups.google.com/d/msgid/prometheus-users/cabf6ba3-79c8-4f5e-865d-1fc51ba52c72%40googlegroups.com?utm_medium=email_source=footer>
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZuSwA9V2EQcc%2BcGLak%3D2mQcLT06A6s99yWVGQta2AHoUA%40mail.gmail.com.

Re: [prometheus-users] How to expose a custom metrics from a pod/service in Kubernetes ?

[Murali Krishna Kanagala]

I think this page can help you to write the sd config for kubernetes pods.
https://github.com/nlighten/tomcat_exporter

For scraping only master pod of a given service it should have a special
label so that Prometheus can discover it. An alternative approach would be
a stateful set where every pod in a service can be given different labels.

Did you try reading about kube dd config to scrape service endpoints? If
that is possible then you will be scraping the master pod if you scrape the
service endpoint.



On Fri, Feb 28, 2020, 12:01 AM SS. OO.  wrote:

> Hi!
>
> I have a Kubernetes/Prometeus newbie question.
>
> I have a Kubernetes application that I want to scale up via Horizontal Pod
> Autoscaler based on custom metrics reflecting the length of the app's work
> queue backlog in the app's internal database.
>
> I can expose the metrics on a pod port in Prometheus format, so there is
> more than one scrape endpoint for fail-over purposes. However I do not want
> Prometheus to call the scrape endpoint on every pod's instance, only one
> instance at a time and failover to another instance only if necessary. All
> instance endpoints report the same data, and calling them in parallel would
> serve no useful purpose and only increase a workload on storage queries
> behind the app runtime.
>
> Right now the application is packaged as a Kubernetes Deployment. Do I
> need to convert it for the above as a Service, or can Prometeus understand
> intrinsically or be instructed in the configuration that only one instance
> is to be called at a time?
>
> Also, can someone please provide an example of the snippet I need to
> insert into
>
> https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml
> so Prometheus picks up the endpoint? Assuming, for example, it is exposed
> on pod port , path /metrics, service/pod named ABC or with label
> appname=LABC?
> Or do I simply tag the pod/service/deployment with some tag and Prometeus
> starts to collect on it? If yes, how do I pack the endpoint
> port+protocol+path into the tag?
>
> Thanks for the advice.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/cabf6ba3-79c8-4f5e-865d-1fc51ba52c72%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZsGX9GLMQLhB7evuXOo2FGS_c%2BdCcVo2S6HiZMNRbCf%2Bg%40mail.gmail.com.

Re: [prometheus-users] Checking if NFS is hanged or not using node_exporter.

[Murali Krishna Kanagala]

The other option would be to run a custom exporter from one box that can
ssh to the rest and run the needful commands.

On Tue, Mar 3, 2020, 9:25 AM Yagyansh S. Kumar 
wrote:

> I also thought about that, but I am keeping that as a last resort. But
> that would require me to push a script to all my 2500+ servers.
>
> On Tuesday, March 3, 2020 at 8:46:27 PM UTC+5:30, Murali Krishna Kanagala
> wrote:
>>
>> I would write a small shell script that tries to write to the nfs mount
>> path and writes the status to a file which can be read by the text file
>> collector. And schedule that shell script cron. I think this is the easiest
>> solution.
>>
>> On Tue, Mar 3, 2020, 9:12 AM Yagyansh S. Kumar 
>> wrote:
>>
>>> Already enabled the nfs and nfsd collectors. Till now I haven't found
>>> anything that can accurately give me the information about NFS hang.
>>> Correct me if I am wrong, but I don't think it is a good indicator of
>>> NFS hang as there may be times where no activity is happening on the NFS,
>>> but that does not mean that NFS is hanged. (eg. I have 25 NFS mounts on one
>>> of my servers, some of them are used rarely, so we won't find any
>>> substantial IO on those mounts, but I need to know whether they are
>>> accessible or not). Still, thanks for the suggestion, will try it out once.
>>>
>>>
>>> On Tuesday, March 3, 2020 at 8:35:03 PM UTC+5:30, Murali Krishna
>>> Kanagala wrote:
>>>>
>>>> Try enabling the nfs options in the node exporter config. It will spit
>>>> out some metrics about the nfs status.
>>>>
>>>> Also look at the disk IO metrics from node exporter and if you see no
>>>> activity which indicates the nfs is not doing anything.
>>>>
>>>> On Tue, Mar 3, 2020, 7:10 AM Yagyansh S. Kumar 
>>>> wrote:
>>>>
>>>>> I want to check if the NFS is hanged(i.e whether it is accessible from
>>>>> the server or not, and if yes then what is the response time it is
>>>>> getting). I know using the mountstats and nfs collector we have a lot of
>>>>> metrics for NFS, but haven't found any that can tell me every time the NFS
>>>>> hangs correctly.
>>>>> Thanks in advance.
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Prometheus Users" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to promethe...@googlegroups.com.
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/prometheus-users/06929518-d3b5-4c2f-9490-b08cc664d26b%40googlegroups.com
>>>>> <https://groups.google.com/d/msgid/prometheus-users/06929518-d3b5-4c2f-9490-b08cc664d26b%40googlegroups.com?utm_medium=email_source=footer>
>>>>> .
>>>>>
>>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Prometheus Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to promethe...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/prometheus-users/1dda60cc-0b20-47da-87ff-4f1c76ce076f%40googlegroups.com
>>> <https://groups.google.com/d/msgid/prometheus-users/1dda60cc-0b20-47da-87ff-4f1c76ce076f%40googlegroups.com?utm_medium=email_source=footer>
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/3822b8a0-697f-4efe-87ab-5a3f90de0786%40googlegroups.com
> <https://groups.google.com/d/msgid/prometheus-users/3822b8a0-697f-4efe-87ab-5a3f90de0786%40googlegroups.com?utm_medium=email_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZvQvbqQkoOe%2BR04b6EeQMvsE_-Z2UeOabdqtm7VO8dRmg%40mail.gmail.com.

Re: [prometheus-users] Checking if NFS is hanged or not using node_exporter.

[Murali Krishna Kanagala]

I would write a small shell script that tries to write to the nfs mount
path and writes the status to a file which can be read by the text file
collector. And schedule that shell script cron. I think this is the easiest
solution.

On Tue, Mar 3, 2020, 9:12 AM Yagyansh S. Kumar 
wrote:

> Already enabled the nfs and nfsd collectors. Till now I haven't found
> anything that can accurately give me the information about NFS hang.
> Correct me if I am wrong, but I don't think it is a good indicator of NFS
> hang as there may be times where no activity is happening on the NFS, but
> that does not mean that NFS is hanged. (eg. I have 25 NFS mounts on one of
> my servers, some of them are used rarely, so we won't find any substantial
> IO on those mounts, but I need to know whether they are accessible or not).
> Still, thanks for the suggestion, will try it out once.
>
>
> On Tuesday, March 3, 2020 at 8:35:03 PM UTC+5:30, Murali Krishna Kanagala
> wrote:
>>
>> Try enabling the nfs options in the node exporter config. It will spit
>> out some metrics about the nfs status.
>>
>> Also look at the disk IO metrics from node exporter and if you see no
>> activity which indicates the nfs is not doing anything.
>>
>> On Tue, Mar 3, 2020, 7:10 AM Yagyansh S. Kumar 
>> wrote:
>>
>>> I want to check if the NFS is hanged(i.e whether it is accessible from
>>> the server or not, and if yes then what is the response time it is
>>> getting). I know using the mountstats and nfs collector we have a lot of
>>> metrics for NFS, but haven't found any that can tell me every time the NFS
>>> hangs correctly.
>>> Thanks in advance.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Prometheus Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to promethe...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/prometheus-users/06929518-d3b5-4c2f-9490-b08cc664d26b%40googlegroups.com
>>> <https://groups.google.com/d/msgid/prometheus-users/06929518-d3b5-4c2f-9490-b08cc664d26b%40googlegroups.com?utm_medium=email_source=footer>
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/1dda60cc-0b20-47da-87ff-4f1c76ce076f%40googlegroups.com
> <https://groups.google.com/d/msgid/prometheus-users/1dda60cc-0b20-47da-87ff-4f1c76ce076f%40googlegroups.com?utm_medium=email_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZubrdH7_%2BcfiXuQ4ESGMfm%2By7HNAVekSEv9K3iXE8RWEQ%40mail.gmail.com.

Re: [prometheus-users] Checking if NFS is hanged or not using node_exporter.

[Murali Krishna Kanagala]

Try enabling the nfs options in the node exporter config. It will spit out
some metrics about the nfs status.

Also look at the disk IO metrics from node exporter and if you see no
activity which indicates the nfs is not doing anything.

On Tue, Mar 3, 2020, 7:10 AM Yagyansh S. Kumar 
wrote:

> I want to check if the NFS is hanged(i.e whether it is accessible from the
> server or not, and if yes then what is the response time it is getting). I
> know using the mountstats and nfs collector we have a lot of metrics for
> NFS, but haven't found any that can tell me every time the NFS hangs
> correctly.
> Thanks in advance.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/06929518-d3b5-4c2f-9490-b08cc664d26b%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZso4WJE%2BFy7zoTQOsCMpO30hbH%2Bh5vfQ6AADbto7nqhYQ%40mail.gmail.com.

Re: [prometheus-users] Re: Alert for high-frequency changes of a metric?

[Murali Krishna Kanagala]

Check if the  absent function helps you.

https://prometheus.io/docs/prometheus/latest/querying/functions/#absent


On Mon, Mar 2, 2020, 9:49 AM Moses Moore 
wrote:

>  >Did you try something like "changes(foo[30m]) > 10" ?
>  > That would alert if the value changed 10 times in the last 30 minutes.
>
> Good guess.  If it were a boolean ([0, 1]) then your idea would detect
> flip-flopping (flapping?) over time.
>
> A metric can be healthy while changing constantly, but gently -- a
> smoothly increasing slope or a long sine-wave.  I'm hoping to detect
> high frequency of deviant amplitudes of change.  Think of a temperature
> sensor that rises and falls over the course of a day, but when something
> overheats it will spike, then the device downgrades until the
> temperature is under a threshold, then spikes again as it tries
> operating at full capacity... every time it drops into the "healthy"
> range, so an alert looking merely for exceeding a threshold will never
> leave the alertstate="pending".
>
> Really wish I could post a picture to illustrate the pattern I'm looking
> for.  It's obvious to a human eye.
>
> Maybe something like a combination of changes() and rate() ? with some
> *_over_time aggregation?  Seems too complex, I hope I'm overthinking it.
>
> On 2020-02-28 11:26 a.m., Łukasz Mierzwa wrote:
> >
> > On Friday, 28 February 2020 16:21:53 UTC, Moses Moore wrote:
> >
> > (Looks like my previous ask of this question got spamblocked
> > because I included a screenshot.  c'est la vie.)
> >
> > I have alerts for when a metric's value passes above or below a
> > threshold.  I can ask for the minimum or maximum over a time
> > range, I can as for a prediction based on the slope of a graph.
> >
> > I have some resources that I know will fail soon after their
> > metrics fluctuate wildly over a short period of time.  They may
> > never exceed the absolute value of 85% during their fluctuations,
> > or they may exceed this briefly but not long enough to cause
> > concern if it was a smooth line.  I.E.  If the samples over time
> > were [30, 30, 31, 70, 5, 69, 6, 71, 5, 69, null, null, null]  I
> > want to detect it before the metric goes absent (because the
> > resource crashed).
> >
> > Setting the threshold at ">69" doesn't work because the value
> > drops below the threshold on the next scrape, closing the alert;
> > besides, if it were at a steady 69 that would be healthy.
> > Setting the threshold at "avg(metric[interval)" doesn't work
> > because the average of an oscillating metric will be well within
> > the healthy range.
> > I thought of setting an alert for "max_over_time - min_over_time >
> > 50" but that would trigger on a smooth ascension -- a false positive.
> >
> > What's the question should I ask Prometheus to detect a metric
> > that vibrates too much?
> >
> > --
> > You received this message because you are subscribed to a topic in the
> > Google Groups "Prometheus Users" group.
> > To unsubscribe from this topic, visit
> >
> https://groups.google.com/d/topic/prometheus-users/6BCaoU4WCS8/unsubscribe
> .
> > To unsubscribe from this group and all its topics, send an email to
> > prometheus-users+unsubscr...@googlegroups.com
> > .
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/prometheus-users/5f7ef3b8-a311-4d98-8bcb-2594e6aaef80%40googlegroups.com
> > <
> https://groups.google.com/d/msgid/prometheus-users/5f7ef3b8-a311-4d98-8bcb-2594e6aaef80%40googlegroups.com?utm_medium=email_source=footer
> >.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/9ab78b23-8eee-a201-213b-8ed979f237e9%40gmail.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZtKAStKwnB5gY2%3DuAGb9%2BA7qZ82WdcwsLymsQZuU5ba6A%40mail.gmail.com.

Re: [prometheus-users] 'insert metric at cursor' is showing all metrics

[Murali Krishna Kanagala]

Did you try a restart of Prometheus service?

On Mon, Mar 2, 2020, 1:04 AM Ankita Khot  wrote:

> In Prometheus UI within the dropdown that says 'insert metric at cursor'
> it shows the previous scraped metrics as well. How to remove those? I would
> like it to show only the metrics which i am keeping through
> 'metric-relabel-config'.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/5f10618e-4d2b-4958-89cd-e9ba1f4feec9%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZtuRL4dxPdK8QVT5XW%2BF2zUKB6K7w_Nvw31BxCLeOLZ1w%40mail.gmail.com.

Re: [prometheus-users] Read access log as metrics

[Murali Krishna Kanagala]

Prometheus is not designed to handle logs. Try Loki, or rsyslog along with
grok exporter to translate logs into metrics.

On Mon, Mar 2, 2020, 5:03 AM Stuart Clark  wrote:

> On 2020-03-02 11:00, Julian Ade Putra wrote:
> > Hello Experts,
> >
> > I have access log that contains API list and status code that keep
> > updating base on the transaction. May i know what is the best method
> > to read it as metric in Prometheus?
> > Below the format of access log :
>
>
> You can use a tool suck as mtail or the grok-exporter to create metrics
> from a log file
>
> >
> > 2020-03-02  14:38:44POST
> > 89218a53-181b-4ef3-bd64-cc159832accc-00021a11   0:13
> > /oem/proxy_services/inbound_sync_process_http_prepaid   200 514
> > 2020-03-02  14:38:49POST
> > 89218a53-181b-4ef3-bd64-cc159832accc-00021a13   0:17
> > /oem/proxy_services/inbound_sync_process_http_prepaid   200 7831
> > 2020-03-02  14:38:51POST
> > 89218a53-181b-4ef3-bd64-cc159832accc-00021a15   0:13
> > /oem/proxy_services/inbound_sync_process_http_prepaid   200 306
> > 2020-03-02  14:39:02POST
> > 89218a53-181b-4ef3-bd64-cc159832accc-00021a18   0:5
> > /Catalyst/ProxyServices/NotificationActivityProxyService200
> >  770
> > 2020-03-02  14:39:03POST
> > 89218a53-181b-4ef3-bd64-cc159832accc-00021a19   0:11
> > /oem/proxy_services/inbound_sync_process_http_postpaid  200 777
> > 2020-03-02  14:39:03POST
> > 89218a53-181b-4ef3-bd64-cc159832accc-00021a1a   0:16
> > /oem/proxy_services/inbound_sync_process_http_postpaid  200 2793
> > 2020-03-02  14:39:03POST
> > 89218a53-181b-4ef3-bd64-cc159832accc-00021a1b   0:13
> > /oem/proxy_services/inbound_sync_process_eai_http_prepaid   200
> >  713
> > 2020-03-02  14:39:03POST
> > 89218a53-181b-4ef3-bd64-cc159832accc-00021a1d   0:13
> > /oem/proxy_services/inbound_sync_process_eai_http_prepaid   200
> >  713
> > 2020-03-02  14:39:04POST
> > 89218a53-181b-4ef3-bd64-cc159832accc-00021a1e   0:13
> > /oem/proxy_services/inbound_sync_process_eai_http_prepaid   200
> >  713
> > 2020-03-02  14:39:04POST
> > 89218a53-181b-4ef3-bd64-cc159832accc-00021a1f   0:13
> > /oem/proxy_services/inbound_sync_process_eai_http_prepaid   200
> >  714
> > 2020-03-02  14:39:06POST
> > 89218a53-181b-4ef3-bd64-cc159832accc-00021a21   0:13
> > /oem/proxy_services/inbound_sync_process_http_prepaid   200 1923
> > 2020-03-02  14:39:10POST
> > 89218a53-181b-4ef3-bd64-cc159832accc-00021a24   0:17
> > /oem/proxy_services/inbound_sync_process_http_prepaid   200 5013
> > 2020-03-02  14:39:15POST
> > 89218a53-181b-4ef3-bd64-cc159832accc-00021a27   0:14
> > /oem/proxy_services/inbound_sync_process_http_postpaid  200 49994
> > 2020-03-02  14:39:16POST
> > 89218a53-181b-4ef3-bd64-cc159832accc-00021a28   0:12
> > /oem/proxy_services/inbound_sync_process_eai_http_prepaid   200
> >  2577
> > 2020-03-02  14:39:16POST
> > 89218a53-181b-4ef3-bd64-cc159832accc-00021a2a   0:15
> > /oem/proxy_services/inbound_sync_process_http_prepaid   200 4556
> > 2020-03-02  14:39:17POST
> > 89218a53-181b-4ef3-bd64-cc159832accc-00021a2b   0:15
> > /oem/proxy_services/inbound_sync_process_http_prepaid   200 307
> > 2020-03-02  14:39:21POST
> > 89218a53-181b-4ef3-bd64-cc159832accc-00021a2c   0:15
> > /oem/proxy_services/inbound_sync_process_eai_http_prepaid   200
> >  643
> > 2020-03-02  14:39:22POST
> > 89218a53-181b-4ef3-bd64-cc159832accc-00021a20   0:13
> > /oem/proxy_services/inbound_sync_process_eai_http_prepaid   200
> >  713
> > 2020-03-02  14:39:22POST
> > 89218a53-181b-4ef3-bd64-cc159832accc-00021a1c   0:13
> > /oem/proxy_services/inbound_sync_process_eai_http_prepaid   200
> >  716
> > 2020-03-02  14:39:23POST
> > 89218a53-181b-4ef3-bd64-cc159832accc-00021a2e   0:15
> > /oem/proxy_services/inbound_sync_process_eai_http_prepaid   200
> >  643
> > 2020-03-02  14:39:28POST
> > 89218a53-181b-4ef3-bd64-cc159832accc-00021a30   0:15
> > /oem/proxy_services/inbound_sync_process_eai_http_prepaid   200
> >  899
> >
> > Thanks,
> > Julian
> >
> >  --
> > You received this message because you are subscribed to the Google
> > Groups "Prometheus Users" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> > an email to prometheus-users+unsubscr...@googlegroups.com.
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/prometheus-users/a5750a63-3da7-4496-bd40-4a3b6ea70e28%40googlegroups.com
> > [1].
> >
> >
> > Links:
> > --
> > [1]
> >
> https://groups.google.com/d/msgid/prometheus-users/a5750a63-3da7-4496-bd40-4a3b6ea70e28%40googlegroups.com?utm_medium=email_source=footer
>
> --
> Stuart Clark
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus 

Re: [prometheus-users] Prometheus metrics to a url

[Murali Krishna Kanagala]

Use remote write option. That remote service should be able to parse the
metrics and forward them wherever you want them to be.

On Mon, Mar 2, 2020, 4:42 AM adi garg  wrote:

> Hello experts,
>
> Is there a way to send Prometheus metrics to a URL, where there can be an
> exporter to send metrics to another system?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/980590a4-45b0-41ec-a7c3-b26371091e7d%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZsqO20AdQk%2BFmUopCcbQv7324dNwpbY1EkKj53f5H1ZPw%40mail.gmail.com.

Re: [prometheus-users] Invalid token error while scraping a target

[Murali Krishna Kanagala]

Metric format is not as as exp6by Prometheus. Promtool can validate the
metric format.
Follow the steps in this link
https://www.robustperception.io/invalid-is-not-a-valid-start-token-and-other-scrape-errors

On Sun, Mar 1, 2020, 8:57 PM Tasleem Halai  wrote:

> I am using Prometheus : 2.8.1
>
> prometheus config:
>
> scrape_configs:
>   # The job name is added as a label `job=` to any timeseries
> scraped from this config.
>   - job_name: 'prometheus'
>
> # metrics_path defaults to '/metrics'
> # scheme defaults to 'http'.
>
> static_configs:
> - targets: ['localhost:9090']
>
>   - job_name: 'grafana'
> static_configs:
> - targets: ['localhost:3000']
>
>   - job_name: 'TestApp'
> metrics_path: '/actuator/prometheus'
> static_configs:
> - targets: ['localhost:8080']
>
> http://localhost:8080/actuator/prometheus
>  gives following metrics
> like:
>
> # HELP jvm_buffer_memory_used_bytes An estimate of the memory that the Java 
> virtual machine is using for this buffer pool
> # TYPE jvm_buffer_memory_used_bytes gauge
> jvm_buffer_memory_used_bytes{id="mapped",} 0.0
> jvm_buffer_memory_used_bytes{id="direct",} 81920.0
> # HELP hikaricp_connections Total connections
> # TYPE hikaricp_connections gauge
> hikaricp_connections{pool="HikariPool-1",} 10.0
> # HELP cache_size The number of entries in this cache. This may be an 
> approximation, depending on the type of cache.
> # TYPE cache_size gauge
> cache_size{cache="addressTypesCache",cacheManager="cachceManager",name="addressTypesCache",}
>  0.0
> cache_size{cache="feeTypesCache",cacheManager="cachceManager",name="feeTypesCache",}
>  0.0
>
>
> But when prometheus tries to scrape it, it gives below error:
>
> "INVALID" is not a valid start token
> Its not clear from the logs or from this error as to whats wrong and how
> to fix it.
> The spring boot app has security enabled & is protected by username and
> password.
>
> I tried prometheus with few other versions and specifying basic auth in
> config but that dint work either.
>
> Can someone please help me understand what could be missing here?
>
>
> --
> Thanks,
> Tasleem
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/f5fccada-59fd-49eb-9ad8-430043f1963e%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZtbJNK-4EjiWeKEJSJeK%3D20h%2BLwjSgotyUThQzYdwm3Gw%40mail.gmail.com.

Re: [prometheus-users] Re: Process Monitoring

[Murali Krishna Kanagala]

Node exporter can monitor all the processes running on a machine.
Just pass the commandline arguments as explained in the below link.

https://github.com/prometheus/node_exporter



On Sun, Mar 1, 2020, 3:21 AM Arun Raj  wrote:

> Can someone help me with details that which metrics provide me to monitor
> the process in the NodeExporter ?
>
> On Friday, October 11, 2019 at 3:05:12 AM UTC+9, Satish P wrote:
>>
>> Shivali, nodeexporter from prometheus is a good option to monitor the
>> processes.
>>
>> On Wednesday, October 9, 2019 at 4:51:23 PM UTC-4, Satish P wrote:
>>>
>>> Kafka is a open source project written in Scala and Java. We need to
>>> scrape the monitoring metrics from JVM using JMX exporter.
>>>
>>> Read more about it: https://github.com/prometheus/jmx_exporter
>>>
>>>
>>> On Wednesday, October 9, 2019 at 4:49:35 AM UTC-4, Shivali Chauhan wrote:

 Hi Team,

 I am trying to set up process monitoring for my infra(Process like:
 Kafka process , opentsdb process java process...etc) using prometheus. I am
 totally new to prometheus. Can you please help me out on this and please
 suggest some good blogs which i can follow for process monitoring.

 Thank you
 Kind regards
 Shivali Chauhan

 --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/0f1ad5b4-ebd1-4db8-94de-c89d9ff4308c%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZtpEYq91z5arUM7ZGFV4eSTyo1wqLpREvzjaBwMEGubsw%40mail.gmail.com.

Re: [prometheus-users]

[Murali Krishna Kanagala]

It's not a good idea to keep prometheus.yml and the rest of the scrape
config files in the same path.

Create a sub directory under /etc/prometheus to keep the scrape config
files.

On Sun, Mar 1, 2020, 5:22 PM rs vas  wrote:

> Hi, I would like to include all the rule files in prometheus.yml
> configuration but with no luck:
>
> rule_files:
>   - "/etc/prometheus/*.yml"
>
> I am trying to follow the syntax from
> https://prometheus.demo.do.prometheus.io/config can someone please help!
> Getting error on prometheus start:
>
> Thanks!
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/CAPs_Afg%3DHv%3Dz-45-f70aKw7JgLOcTYepbt4ipaG8dnmwZS3mtA%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZvn79RBD6zJv7JuZiHp8bCRFrFAYtjYevfEnP%2BzP-qfqA%40mail.gmail.com.

Re: [prometheus-users] Re: monitoring Openshift Cluster with External Promethus

[Murali Krishna Kanagala]

You can scrape the Prometheus on the openshift using federation
configuration on the external Prometheus. With this approach you don't have
to persist the metrics on the openshift cluster(no need of persist storage)

Just expose the Prometheus on the open shift cluster using nodeport or
something like that and add it to the federation configuration on your
external Prometheus.

https://prometheus.io/docs/prometheus/latest/federation/

On Fri, Feb 14, 2020, 12:28 PM Tarun Gupta  wrote:

> Were you able to figure this out. I have exactly the same problem to solve.
>
> Thanks
>
> On Friday, June 7, 2019 at 11:31:09 AM UTC-7, IndGirl6 wrote:
>>
>> Hi,
>>
>> would appreciate it if some one can detail step by step or point me to a
>> good document for this.
>> I have an Openshift Cluster setup. It already has its own prometheus and
>> grafana  setup internally within the cluster.
>>
>> External to this cluster, i also have another standalone Prometheus
>> setup, that monitors my other devices (physical servers / databases /
>> application etc).
>>
>> I would like to be able to monitor my openshift cluster using the
>> External standalone prometheus server.
>>
>> Is this possible, can some one guide me on the setup.
>>
>> Thanks
>> IG6
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/ae87e731-f57b-4a6b-8b4e-0e26160c8f37%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZsm4GzkahGEX%3DgObF46UVssYN%3DaAfFAyG-pDLLuL1d1AQ%40mail.gmail.com.