[ceph-users] Re: prometheus - figure out which mgr (metrics endpoint) that is active

[Karsten Nielsen]

Hi Ernesto, 


Thank you. That did the trick :)

Kind Regards,
- Karsten 


On 29-09-2021 19:59, Ernesto Puerta wrote:

Hi Karsten, 

Endpoints returning no data shouldn't be an issue. If all endpoints are scraped under the same job, they'll only differ on the "instance" label. 

The "instance" label is being progressively removed from the ceph_* metric queries (as it only makes sense for node exporter ones). In the meantime, you could use the " [1]relabel_configs" setting [1] to override the different values: 


relabel_configs:
- source_labels: [__address__]
regex: '.*'
target_label: instance
replacement: 'ceph-mgr'

Kind Regards, Ernesto 

On Wed, Sep 29, 2021 at 4:08 PM Karsten Nielsen  wrote: 

OK thanks for that explanation. Would be awesome if you got time to do 
the patches upstream. It does seem like a lot of work.

I will get cracking at it.

On 28-09-2021 22:38, David Orman wrote:

We scrape all mgr endpoints since we use external Prometheus clusters,
as well. The query results will have {instance=activemgrhost}. The
dashboards in upstream don't have multiple cluster support, so we have
to modify them to work with our deployments since we have multiple
ceph clusters being polled by Prometheus clusters. We effectively add
instance regular expressions to all the queries on the dashboards, and
a variable for the dashboard itself, to support getting the list of
clusters via a label_values call on one of the ceph_exporter metrics +
regular expression to parse out the part after the hostname portion of
the fqdn.

I don't think the current dashboards are intended for use outside the
internal Prometheus deployments, but we definitely intended (at some
point when time permitted) to try and submit patches that would work
for both use-cases, since it's painful to continually update the
dashboards on every release.

On Tue, Sep 28, 2021 at 12:45 PM Karsten Nielsen  
wrote:


Hi,
I am running ceph 16.2.6 installed with cephadm.
I have enabled prometheus to be able scrape metrics from an external
promethus server.
I have 3 nodes with mgr daeamon all reply to the query against
node:9283/metrics 2 is returning a empty reply - the none active 
mgr's.
Is there a node:9283/health or other path to query for the once that 
is

not active ?
I am asking as I am getting empty dashboards 2 of 3 times as there are
no metrics when the wrong endpoint is getting scraped.

Thanks,
- Karsten
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



Links:
--
[1]
https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: prometheus - figure out which mgr (metrics endpoint) that is active

[Ernesto Puerta]

Hi Karsten,

Endpoints returning no data shouldn't be an issue. If all endpoints are
scraped under the same job, they'll only differ on the "instance" label.

The "instance" label is being progressively removed from the ceph_* metric
queries (as it only makes sense for node exporter ones). In the meantime,
you could use the "
relabel_configs"
setting

to
override the different values:

  relabel_configs:
  - source_labels: [__address__]
regex: '.*'
target_label: instance
replacement: 'ceph-mgr'

Kind Regards,
Ernesto


On Wed, Sep 29, 2021 at 4:08 PM Karsten Nielsen  wrote:

> OK thanks for that explanation. Would be awesome if you got time to do
> the patches upstream. It does seem like a lot of work.
> I will get cracking at it.
>
> On 28-09-2021 22:38, David Orman wrote:
> > We scrape all mgr endpoints since we use external Prometheus clusters,
> > as well. The query results will have {instance=activemgrhost}. The
> > dashboards in upstream don't have multiple cluster support, so we have
> > to modify them to work with our deployments since we have multiple
> > ceph clusters being polled by Prometheus clusters. We effectively add
> > instance regular expressions to all the queries on the dashboards, and
> > a variable for the dashboard itself, to support getting the list of
> > clusters via a label_values call on one of the ceph_exporter metrics +
> > regular expression to parse out the part after the hostname portion of
> > the fqdn.
> >
> > I don't think the current dashboards are intended for use outside the
> > internal Prometheus deployments, but we definitely intended (at some
> > point when time permitted) to try and submit patches that would work
> > for both use-cases, since it's painful to continually update the
> > dashboards on every release.
> >
> > On Tue, Sep 28, 2021 at 12:45 PM Karsten Nielsen 
> > wrote:
> >>
> >> Hi,
> >> I am running ceph 16.2.6 installed with cephadm.
> >> I have enabled prometheus to be able scrape metrics from an external
> >> promethus server.
> >> I have 3 nodes with mgr daeamon all reply to the query against
> >> node:9283/metrics 2 is returning a empty reply - the none active
> >> mgr's.
> >> Is there a node:9283/health or other path to query for the once that
> >> is
> >> not active ?
> >> I am asking as I am getting empty dashboards 2 of 3 times as there are
> >> no metrics when the wrong endpoint is getting scraped.
> >>
> >> Thanks,
> >> - Karsten
> >> ___
> >> ceph-users mailing list -- ceph-users@ceph.io
> >> To unsubscribe send an email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: prometheus - figure out which mgr (metrics endpoint) that is active

[Karsten Nielsen]

OK thanks for that explanation. Would be awesome if you got time to do 
the patches upstream. It does seem like a lot of work.

I will get cracking at it.

On 28-09-2021 22:38, David Orman wrote:

We scrape all mgr endpoints since we use external Prometheus clusters,
as well. The query results will have {instance=activemgrhost}. The
dashboards in upstream don't have multiple cluster support, so we have
to modify them to work with our deployments since we have multiple
ceph clusters being polled by Prometheus clusters. We effectively add
instance regular expressions to all the queries on the dashboards, and
a variable for the dashboard itself, to support getting the list of
clusters via a label_values call on one of the ceph_exporter metrics +
regular expression to parse out the part after the hostname portion of
the fqdn.

I don't think the current dashboards are intended for use outside the
internal Prometheus deployments, but we definitely intended (at some
point when time permitted) to try and submit patches that would work
for both use-cases, since it's painful to continually update the
dashboards on every release.

On Tue, Sep 28, 2021 at 12:45 PM Karsten Nielsen  
wrote:


Hi,
I am running ceph 16.2.6 installed with cephadm.
I have enabled prometheus to be able scrape metrics from an external
promethus server.
I have 3 nodes with mgr daeamon all reply to the query against
node:9283/metrics 2 is returning a empty reply - the none active 
mgr's.
Is there a node:9283/health or other path to query for the once that 
is

not active ?
I am asking as I am getting empty dashboards 2 of 3 times as there are
no metrics when the wrong endpoint is getting scraped.

Thanks,
- Karsten
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: prometheus - figure out which mgr (metrics endpoint) that is active

[David Orman]

We scrape all mgr endpoints since we use external Prometheus clusters,
as well. The query results will have {instance=activemgrhost}. The
dashboards in upstream don't have multiple cluster support, so we have
to modify them to work with our deployments since we have multiple
ceph clusters being polled by Prometheus clusters. We effectively add
instance regular expressions to all the queries on the dashboards, and
a variable for the dashboard itself, to support getting the list of
clusters via a label_values call on one of the ceph_exporter metrics +
regular expression to parse out the part after the hostname portion of
the fqdn.

I don't think the current dashboards are intended for use outside the
internal Prometheus deployments, but we definitely intended (at some
point when time permitted) to try and submit patches that would work
for both use-cases, since it's painful to continually update the
dashboards on every release.

On Tue, Sep 28, 2021 at 12:45 PM Karsten Nielsen  wrote:
>
> Hi,
> I am running ceph 16.2.6 installed with cephadm.
> I have enabled prometheus to be able scrape metrics from an external
> promethus server.
> I have 3 nodes with mgr daeamon all reply to the query against
> node:9283/metrics 2 is returning a empty reply - the none active mgr's.
> Is there a node:9283/health or other path to query for the once that is
> not active ?
> I am asking as I am getting empty dashboards 2 of 3 times as there are
> no metrics when the wrong endpoint is getting scraped.
>
> Thanks,
> - Karsten
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io