Re: [prometheus-developers] Re: auto discover of targets in Prometheus

2021-02-21 Thread Gajendra D Ambi 
the ipam sd sounds intriguing but i wonder if this is true with prometheus
on k8s since the configmap is presented as a secret to prometheus.

On Sun, Feb 21, 2021, 6:33 PM Stuart Clark  wrote:

> On 20/02/2021 15:08, Gajendra D Ambi wrote:
> > Hi Team,
> > Yes we do have a IPAM available. When we run prometheus node exporter
> > agent it starts showing up metrics at ip/metrics. I was hoping for a
> > way to give a range of ip addresses to prometheus, rather than adding
> > them each as a static target. What would be the best way for us then!
> > with 100s of baremetal servers, Also these baremetals are managed by
> > openstack, I do see openstack auto discovery but will it also discover
> > bare metal servers managed by openstack or just the VM instances?
> > because prometheus wasn't explicit on that part in the documentation.
> > If you could clarify in here or documentation then it would be awesome.
>
> The OpenStack SD can do both -
>
> https://prometheus.io/docs/prometheus/latest/configuration/configuration/#openstack_sd_config
> - It has support for both hypervisor and instance for all Nova servers.
>
> You can use the File SD
> (
> https://prometheus.io/docs/prometheus/latest/configuration/configuration/#file_sd_config)
>
> in conjunction with your IPAM to manage scrape configs too. A common
> method is to run a regular scheduled job to interrogate the IPAM for
> new, changed or removed devices and then produce JSON or YAML files for
> your jobs - depending on the level of information you could produce
> lists for the SNMP exporter (e.g. network equipment), node exporter
> (Unix servers), MySQL exporter (and other databases), IPMI exporter, etc.
>
> --
> Stuart Clark
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CAHy9ZTh_Dm%2BNGDbB16Xj7ehHrAmyJSdtNybj8RE346k8B9moDQ%40mail.gmail.com.

Re: [prometheus-developers] Re: auto discover of targets in Prometheus

2021-02-21 Thread Stuart Clark 

On 20/02/2021 15:08, Gajendra D Ambi wrote:

Hi Team,
Yes we do have a IPAM available. When we run prometheus node exporter 
agent it starts showing up metrics at ip/metrics. I was hoping for a 
way to give a range of ip addresses to prometheus, rather than adding 
them each as a static target. What would be the best way for us then! 
with 100s of baremetal servers, Also these baremetals are managed by 
openstack, I do see openstack auto discovery but will it also discover 
bare metal servers managed by openstack or just the VM instances? 
because prometheus wasn't explicit on that part in the documentation. 
If you could clarify in here or documentation then it would be awesome.


The OpenStack SD can do both - 
https://prometheus.io/docs/prometheus/latest/configuration/configuration/#openstack_sd_config 
- It has support for both hypervisor and instance for all Nova servers.


You can use the File SD 
(https://prometheus.io/docs/prometheus/latest/configuration/configuration/#file_sd_config) 
in conjunction with your IPAM to manage scrape configs too. A common 
method is to run a regular scheduled job to interrogate the IPAM for 
new, changed or removed devices and then produce JSON or YAML files for 
your jobs - depending on the level of information you could produce 
lists for the SNMP exporter (e.g. network equipment), node exporter 
(Unix servers), MySQL exporter (and other databases), IPMI exporter, etc.


--
Stuart Clark

--
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/91d867e1-00db-1fdc-bf08-984ef9970d2c%40Jahingo.com.

Re: [prometheus-developers] Re: auto discover of targets in Prometheus

2021-02-20 Thread Gajendra D Ambi 
Hi Team,
Yes we do have a IPAM available. When we run prometheus node exporter agent
it starts showing up metrics at ip/metrics. I was hoping for a way to give
a range of ip addresses to prometheus, rather than adding them each as a
static target. What would be the best way for us then! with 100s of
baremetal servers, Also these baremetals are managed by openstack, I do see
openstack auto discovery but will it also discover bare metal servers
managed by openstack or just the VM instances? because prometheus wasn't
explicit on that part in the documentation. If you could clarify in here or
documentation then it would be awesome.

Thanks and Regards,
https://ambig.one/2/
*Gajendra D Ambi *|AWS-CSAA, VCAP6-DCA, 2xVCA, 7xVCP, VCE-CIA, EMC-ISA,
Comptia Server+/A+, MCSE, 2xMCTS vExpert (2017/2018) |



On Sat, Feb 20, 2021 at 12:56 PM Ben Kochie  wrote:

> Prometheus absolutely does do dynamic discovery.
>
> See the scrape_config section of the docs:
> https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config
>
> There are 16 available XXX_sd_configs options there. If your source of
> truth isn't listed, you can either write your own adapter using
> file_sd_configs as an API. Or if you think your system is popular enough,
> contribute it to the codebase.
>
> There is a growing list of adapters here:
> https://prometheus.io/docs/operating/integrations/#file-service-discovery
>
> On Fri, Feb 19, 2021 at 11:19 PM Gajendra Ambi  wrote:
>
>> Hi,
>> I too just discovered that it does not do dynamic discovery. I have to
>> tell prometheus to see it by using file based discovery in prometheus. This
>> is very unfortunate where we have 100s of nodes and growing and each group
>> of devs who own them get to add them when they want. Currently it seems We
>> have to do it manually. Thanks for the quick info.
>>
>> On Saturday, February 20, 2021 at 3:33:47 AM UTC+5:30 Julien Pivotto
>> wrote:
>>
>>> On 19 Feb 14:00, Gajendra Ambi wrote:
>>> > Everyone gives the same link
>>> >
>>> https://prometheus.io/docs/prometheus/latest/configuration/configuration
>>> when
>>> > asked about auto discovery of physical nodes but the entire page has
>>> only 2
>>> > occurences of the word auto-discovery.
>>> > We have 700+ physical servers and Prometheus is running on k8s
>>> (targets are
>>> > not part of k8s). Can someone please point to a blog or the actual
>>> part of
>>> > the documentation which has auto discovery of physical nodes, not the
>>> same
>>> > link please. I could not find anything there. It says, you can do it
>>> using
>>> > 1 of the auto discovery methods but the link does not have a single
>>> method
>>> > of auto discovery listed there.
>>>
>>> Hello,
>>>
>>> Prometheus does not do *auto* discovery. We do *service* discovery. You
>>> will need a source of truth where your register your nodes.
>>>
>>>
>>> >
>>> > On Monday, September 3, 2018 at 7:44:49 PM UTC+5:30 HARSH AGARWAL
>>> wrote:
>>> >
>>> > > You can use Prometheus-service-discovery(SD) to automatically
>>> discover the
>>> > > targets.
>>> > >
>>> https://prometheus.io/docs/prometheus/latest/configuration/configuration
>>> > >
>>> > > Using relabel-configs
>>> > > <
>>> https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Crelabel_config%3E>,
>>>
>>> > > you can specify which targets you want to
>>> keep/discard/change-labels, etc.
>>> > >
>>> > > Prometheus supports SD for many providers. You can also use your
>>> > > own custom-service-discovery mechanisms using file_sd
>>> > > <
>>> https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cfile_sd_config%3E>
>>>
>>> > > .
>>> > >
>>> > >
>>> > > On Monday, September 3, 2018 at 7:36:28 PM UTC+5:30, HJS wrote:
>>> > >>
>>> > >> For a particular job in Prometheus, it seems like the typical
>>> config is
>>> > >> something like this:
>>> > >>
>>> > >> static_configs:
>>> > >> - targets: ['localhost:9090']
>>> > >>
>>> > >> But in the case where I want a dynamic list of hosts((auto discover
>>> of
>>> > >> targets)), this option is available in prometheus and if available
>>> how can
>>> > >> config??
>>> > >>
>>> > >>
>>> > >> Thank you in advance!
>>> > >>
>>> > >>
>>> >
>>> > --
>>> > You received this message because you are subscribed to the Google
>>> Groups "Prometheus Developers" group.
>>> > To unsubscribe from this group and stop receiving emails from it, send
>>> an email to prometheus-devel...@googlegroups.com.
>>> > To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/prometheus-developers/cfc35736-9321-438e-8692-c3591ef2e550n%40googlegroups.com.
>>>
>>>
>>>
>>> --
>>> Julien Pivotto
>>> @roidelapluie
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to prometheus-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web

Re: [prometheus-developers] Re: auto discover of targets in Prometheus

2021-02-19 Thread Ben Kochie 
Prometheus absolutely does do dynamic discovery.

See the scrape_config section of the docs:
https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config

There are 16 available XXX_sd_configs options there. If your source of
truth isn't listed, you can either write your own adapter using
file_sd_configs as an API. Or if you think your system is popular enough,
contribute it to the codebase.

There is a growing list of adapters here:
https://prometheus.io/docs/operating/integrations/#file-service-discovery

On Fri, Feb 19, 2021 at 11:19 PM Gajendra Ambi  wrote:

> Hi,
> I too just discovered that it does not do dynamic discovery. I have to
> tell prometheus to see it by using file based discovery in prometheus. This
> is very unfortunate where we have 100s of nodes and growing and each group
> of devs who own them get to add them when they want. Currently it seems We
> have to do it manually. Thanks for the quick info.
>
> On Saturday, February 20, 2021 at 3:33:47 AM UTC+5:30 Julien Pivotto wrote:
>
>> On 19 Feb 14:00, Gajendra Ambi wrote:
>> > Everyone gives the same link
>> >
>> https://prometheus.io/docs/prometheus/latest/configuration/configuration
>> when
>> > asked about auto discovery of physical nodes but the entire page has
>> only 2
>> > occurences of the word auto-discovery.
>> > We have 700+ physical servers and Prometheus is running on k8s (targets
>> are
>> > not part of k8s). Can someone please point to a blog or the actual part
>> of
>> > the documentation which has auto discovery of physical nodes, not the
>> same
>> > link please. I could not find anything there. It says, you can do it
>> using
>> > 1 of the auto discovery methods but the link does not have a single
>> method
>> > of auto discovery listed there.
>>
>> Hello,
>>
>> Prometheus does not do *auto* discovery. We do *service* discovery. You
>> will need a source of truth where your register your nodes.
>>
>>
>> >
>> > On Monday, September 3, 2018 at 7:44:49 PM UTC+5:30 HARSH AGARWAL
>> wrote:
>> >
>> > > You can use Prometheus-service-discovery(SD) to automatically
>> discover the
>> > > targets.
>> > >
>> https://prometheus.io/docs/prometheus/latest/configuration/configuration
>> > >
>> > > Using relabel-configs
>> > > <
>> https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Crelabel_config%3E>,
>>
>> > > you can specify which targets you want to keep/discard/change-labels,
>> etc.
>> > >
>> > > Prometheus supports SD for many providers. You can also use your
>> > > own custom-service-discovery mechanisms using file_sd
>> > > <
>> https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cfile_sd_config%3E>
>>
>> > > .
>> > >
>> > >
>> > > On Monday, September 3, 2018 at 7:36:28 PM UTC+5:30, HJS wrote:
>> > >>
>> > >> For a particular job in Prometheus, it seems like the typical config
>> is
>> > >> something like this:
>> > >>
>> > >> static_configs:
>> > >> - targets: ['localhost:9090']
>> > >>
>> > >> But in the case where I want a dynamic list of hosts((auto discover
>> of
>> > >> targets)), this option is available in prometheus and if available
>> how can
>> > >> config??
>> > >>
>> > >>
>> > >> Thank you in advance!
>> > >>
>> > >>
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> Groups "Prometheus Developers" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> an email to prometheus-devel...@googlegroups.com.
>> > To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-developers/cfc35736-9321-438e-8692-c3591ef2e550n%40googlegroups.com.
>>
>>
>>
>> --
>> Julien Pivotto
>> @roidelapluie
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/be0fbf93-357f-403a-8f8f-b70140ead2den%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmqHj_SHfATx2DcHrxCB0mu56d0gwrx5cQsG23mcm%3DdSmg%40mail.gmail.com.

Re: [prometheus-developers] Re: auto discover of targets in Prometheus

2021-02-19 Thread Stuart Clark 

On 19/02/2021 22:19, Gajendra Ambi wrote:

Hi,
I too just discovered that it does not do dynamic discovery. I have to 
tell prometheus to see it by using file based discovery in prometheus. 
This is very unfortunate where we have 100s of nodes and growing and 
each group of devs who own them get to add them when they want. 
Currently it seems We have to do it manually. Thanks for the quick info.


Prometheus needs some way of obtaining a list of the targets to scrape 
for each job.


For cloud based instances that is fairly easy - Prometheus can ask them 
to list all instances and using features like tags you can decide what 
is scraped.


For virtual machines you could do something similar. For physical 
machines it really depends on how you have things setup. Do you have a 
IPAM/asset management system? Do you run a tool like Consul?


If you ignore Prometheus for a second: If you were asked to produce a 
list of all the servers with details about what is running on them (so 
you knew what metrics to fetch) how would you do that?


Whatever the answer, that's what you should aim to have Prometheus use. 
If the answer is to produce a manual list, then unfortunately that's 
what you need to do - Prometheus has to look somewhere for the list.


--
Stuart Clark

--
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/9bc46dfc-25a0-39b5-4d50-55b4c9a252bd%40Jahingo.com.

Re: [prometheus-developers] Re: auto discover of targets in Prometheus

2021-02-19 Thread Gajendra Ambi 
Hi,
I too just discovered that it does not do dynamic discovery. I have to tell 
prometheus to see it by using file based discovery in prometheus. This is 
very unfortunate where we have 100s of nodes and growing and each group of 
devs who own them get to add them when they want. Currently it seems We 
have to do it manually. Thanks for the quick info.

On Saturday, February 20, 2021 at 3:33:47 AM UTC+5:30 Julien Pivotto wrote:

> On 19 Feb 14:00, Gajendra Ambi wrote:
> > Everyone gives the same link 
> > https://prometheus.io/docs/prometheus/latest/configuration/configuration 
> when 
> > asked about auto discovery of physical nodes but the entire page has 
> only 2 
> > occurences of the word auto-discovery. 
> > We have 700+ physical servers and Prometheus is running on k8s (targets 
> are 
> > not part of k8s). Can someone please point to a blog or the actual part 
> of 
> > the documentation which has auto discovery of physical nodes, not the 
> same 
> > link please. I could not find anything there. It says, you can do it 
> using 
> > 1 of the auto discovery methods but the link does not have a single 
> method 
> > of auto discovery listed there.
>
> Hello,
>
> Prometheus does not do *auto* discovery. We do *service* discovery. You
> will need a source of truth where your register your nodes.
>
>
> > 
> > On Monday, September 3, 2018 at 7:44:49 PM UTC+5:30 HARSH AGARWAL wrote:
> > 
> > > You can use Prometheus-service-discovery(SD) to automatically discover 
> the 
> > > targets.
> > > 
> https://prometheus.io/docs/prometheus/latest/configuration/configuration
> > >
> > > Using relabel-configs 
> > > <
> https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Crelabel_config%3E>,
>  
>
> > > you can specify which targets you want to keep/discard/change-labels, 
> etc.
> > >
> > > Prometheus supports SD for many providers. You can also use your 
> > > own custom-service-discovery mechanisms using file_sd 
> > > <
> https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cfile_sd_config%3E
> >
> > > .
> > >
> > >
> > > On Monday, September 3, 2018 at 7:36:28 PM UTC+5:30, HJS wrote:
> > >>
> > >> For a particular job in Prometheus, it seems like the typical config 
> is 
> > >> something like this:
> > >>
> > >> static_configs:
> > >> - targets: ['localhost:9090']
> > >>
> > >> But in the case where I want a dynamic list of hosts((auto discover 
> of 
> > >> targets)), this option is available in prometheus and if available 
> how can 
> > >> config??
> > >>
> > >>
> > >> Thank you in advance!
> > >>
> > >>
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups "Prometheus Developers" group.
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to prometheus-devel...@googlegroups.com.
> > To view this discussion on the web visit 
> https://groups.google.com/d/msgid/prometheus-developers/cfc35736-9321-438e-8692-c3591ef2e550n%40googlegroups.com
> .
>
>
> -- 
> Julien Pivotto
> @roidelapluie
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/be0fbf93-357f-403a-8f8f-b70140ead2den%40googlegroups.com.

Re: [prometheus-developers] Re: auto discover of targets in Prometheus

2021-02-19 Thread Julien Pivotto 
On 19 Feb 14:00, Gajendra Ambi wrote:
> Everyone gives the same link 
> https://prometheus.io/docs/prometheus/latest/configuration/configuration when 
> asked about auto discovery of physical nodes but the entire page has only 2 
> occurences of the word auto-discovery. 
> We have 700+ physical servers and Prometheus is running on k8s (targets are 
> not part of k8s). Can someone please point to a blog or the actual part of 
> the documentation which has auto discovery of physical nodes, not the same 
> link please. I could not find anything there. It says, you can do it using 
> 1 of the auto discovery methods but the link does not have a single method 
> of auto discovery listed there.

Hello,

Prometheus does not do *auto* discovery. We do *service* discovery. You
will need a source of truth where your register your nodes.


> 
> On Monday, September 3, 2018 at 7:44:49 PM UTC+5:30 HARSH AGARWAL wrote:
> 
> > You can use Prometheus-service-discovery(SD) to automatically discover the 
> > targets.
> > https://prometheus.io/docs/prometheus/latest/configuration/configuration
> >
> > Using relabel-configs 
> > ,
> >  
> > you can specify which targets you want to keep/discard/change-labels, etc.
> >
> > Prometheus supports SD for many providers. You can also use your 
> > own custom-service-discovery mechanisms using file_sd 
> > 
> > .
> >
> >
> > On Monday, September 3, 2018 at 7:36:28 PM UTC+5:30, HJS wrote:
> >>
> >> For a particular job in Prometheus, it seems like the typical config is 
> >> something like this:
> >>
> >> static_configs:
> >> - targets: ['localhost:9090']
> >>
> >> But in the case where I want a dynamic list of hosts((auto discover of 
> >> targets)), this option is available in prometheus and if available how can 
> >> config??
> >>
> >>
> >> Thank you in advance!
> >>
> >>
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/prometheus-developers/cfc35736-9321-438e-8692-c3591ef2e550n%40googlegroups.com.


-- 
Julien Pivotto
@roidelapluie

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/20210219220315.GA364881%40oxygen.