Re: [Nagios-users] Dynamically generated service checks

[Marc Powell]

On Jul 16, 2009, at 10:43 AM, David Rosenstrauch wrote:

> Sounds like it might be what we'd want.  I'm a bit unclear about how
> this would work in practice though.
>
> What I'd want is that when I go check the Nagios "Service Problems"  
> page
> in the GUI, I'd want to see multiple lines for the service - one line
> for each of the tags/files that we're having problems with.

This isn't possible without actually creating those host{} and  
service{} definitions in the config files and reloading. I'm not aware  
of any other way to do it. As far as I know it's not possible for an  
Event Broker module to modify the config data so you couldn't even  
write a custom module to do it if you saw something new.

> But if this is configured as just a single (passive, volatile)  
> service,
> then wouldn't the GUI show only one line for the service, containing
> details for only the most recent tag/file that failed the check?

Yes, that would be the compromise. You'd have a greater dependence on  
the actual e-mail alerts.

--
Marc

--
Enter the BlackBerry Developer Challenge  
This is your chance to win up to $100,000 in prizes! For a limited time, 
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize  
details at: http://p.sf.net/sfu/Challenge
___
Nagios-users mailing list
Nagios-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when reporting 
any issue. 
::: Messages without supporting info will risk being sent to /dev/null

Re: [Nagios-users] Dynamically generated service checks

[David Rosenstrauch]

Marc Powell wrote:
> On Jul 15, 2009, at 11:13 AM, David Rosenstrauch wrote:
> 
>> IMO, the ideal solution here would be if I could just submit passive
>> check results for services that aren't explicitly configured in  
>> Nagios.
>>  But alas, that's not allowed and it fails with messages like  
>> "Warning:
>>  Passive check result was received for service 'foo' on host
>> 'mysql-dev', but the service could not be found!"
> 
> That's correct but this would be a lot like submitting SNMP traps to  
> nagios. You might glance over that documentation to get ideas. You  
> could create a generic service, set it as passive and volatile and  
> then just start submitting passive results to it with descriptive/ 
> useful output. Each non-ok result would generate an alert so you  
> wouldn't miss anything but if you don't build in some protections, you  
> could repeat notify about the same problem.
> 
> --
> Marc

Hmmm ...

Sounds like it might be what we'd want.  I'm a bit unclear about how 
this would work in practice though.

What I'd want is that when I go check the Nagios "Service Problems" page 
in the GUI, I'd want to see multiple lines for the service - one line 
for each of the tags/files that we're having problems with.

But if this is configured as just a single (passive, volatile) service, 
then wouldn't the GUI show only one line for the service, containing 
details for only the most recent tag/file that failed the check?

Thanks,

DR


--
Enter the BlackBerry Developer Challenge  
This is your chance to win up to $100,000 in prizes! For a limited time, 
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize  
details at: http://p.sf.net/sfu/Challenge
___
Nagios-users mailing list
Nagios-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when reporting 
any issue. 
::: Messages without supporting info will risk being sent to /dev/null

Re: [Nagios-users] Dynamically generated service checks

[Kevin Keane]

Marc Powell wrote:
> On Jul 15, 2009, at 11:13 AM, David Rosenstrauch wrote:
>
>   
>> IMO, the ideal solution here would be if I could just submit passive
>> check results for services that aren't explicitly configured in  
>> Nagios.
>>  But alas, that's not allowed and it fails with messages like  
>> "Warning:
>>  Passive check result was received for service 'foo' on host
>> 'mysql-dev', but the service could not be found!"
>> 
>
> That's correct but this would be a lot like submitting SNMP traps to  
> nagios. You might glance over that documentation to get ideas. You  
> could create a generic service, set it as passive and volatile and  
> then just start submitting passive results to it with descriptive/ 
> useful output. Each non-ok result would generate an alert so you  
> wouldn't miss anything but if you don't build in some protections, you  
> could repeat notify about the same problem.
>   
I am using a script to look for these error messages. In my situation, 
the list of services changes only rarely, so all this script does is 
submit another check result to Nagios to alert me that I need to 
manually add a service. You could also use such a script to 
automatically add any new services, and then restart Nagios.

-- 
Kevin Keane
Owner
The NetTech
Find the Uncommon: Expert Solutions for a Network You Never Have to Think About

Office: 866-642-7116
http://www.4nettech.com

This e-mail and attachments, if any, may contain confidential and/or 
proprietary information. Please be advised that the unauthorized use or 
disclosure of the information is strictly prohibited. The information herein is 
intended only for use by the intended recipient(s) named above. If you have 
received this transmission in error, please notify the sender immediately and 
permanently delete the e-mail and any copies, printouts or attachments thereof.


--
Enter the BlackBerry Developer Challenge  
This is your chance to win up to $100,000 in prizes! For a limited time, 
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize  
details at: http://p.sf.net/sfu/Challenge
___
Nagios-users mailing list
Nagios-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when reporting 
any issue. 
::: Messages without supporting info will risk being sent to /dev/null

Re: [Nagios-users] Dynamically generated service checks

[Marc Powell]

On Jul 15, 2009, at 11:13 AM, David Rosenstrauch wrote:

> IMO, the ideal solution here would be if I could just submit passive
> check results for services that aren't explicitly configured in  
> Nagios.
>  But alas, that's not allowed and it fails with messages like  
> "Warning:
>  Passive check result was received for service 'foo' on host
> 'mysql-dev', but the service could not be found!"

That's correct but this would be a lot like submitting SNMP traps to  
nagios. You might glance over that documentation to get ideas. You  
could create a generic service, set it as passive and volatile and  
then just start submitting passive results to it with descriptive/ 
useful output. Each non-ok result would generate an alert so you  
wouldn't miss anything but if you don't build in some protections, you  
could repeat notify about the same problem.

--
Marc


--
Enter the BlackBerry Developer Challenge  
This is your chance to win up to $100,000 in prizes! For a limited time, 
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize  
details at: http://p.sf.net/sfu/Challenge
___
Nagios-users mailing list
Nagios-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when reporting 
any issue. 
::: Messages without supporting info will risk being sent to /dev/null

Re: [Nagios-users] Dynamically generated service checks

[Max]

On Wed, Jul 15, 2009 at 12:17 PM, Mark Gius wrote:
> Max wrote:
>
> As long as you are not using ePN, you can just send a HUP signal to
> the parent Nagios process; this will cause it to re-read the
> configuration file(s) from disk.
>
>
> - Max
>
>
> Slight OT, but why can't you do this with ePN?

Actually, you can, but you will leak a LOT of memory if it works and
it also has a good chance of causing Nagios to crash (I have seen both
situations in the past).

--
Enter the BlackBerry Developer Challenge  
This is your chance to win up to $100,000 in prizes! For a limited time, 
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize  
details at: http://p.sf.net/sfu/Challenge
___
Nagios-users mailing list
Nagios-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when reporting 
any issue. 
::: Messages without supporting info will risk being sent to /dev/null

Re: [Nagios-users] Dynamically generated service checks

[Mark Gius]

Max wrote:


As long as you are not using ePN, you can just send a HUP signal to
the parent Nagios process; this will cause it to re-read the
configuration file(s) from disk.


- Max
  


Slight OT, but why can't you do this with ePN?

-Gius
--
Enter the BlackBerry Developer Challenge  
This is your chance to win up to $100,000 in prizes! For a limited time, 
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize  
details at: http://p.sf.net/sfu/Challenge___
Nagios-users mailing list
Nagios-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when reporting 
any issue. 
::: Messages without supporting info will risk being sent to /dev/null

Re: [Nagios-users] Dynamically generated service checks

[David Rosenstrauch]

Marc Powell wrote:
> On Jul 15, 2009, at 10:19 AM, David Rosenstrauch wrote:
> 
>> Matthew Jurgens wrote:
> 
>> 2) If I understand correctly, Nagios loads its config files at startup
>> time and does not re-read them after that.  So if I'm regenerating a
>> config file, then that means I'd need to restart the Nagios service
>> afterwards, which is a bit of an onerous imposition.
> 
> Using the init script, restart stops and starts the daemon, reload  
> sends a HUP signal to the running process to tell it to re-read it's  
> config files. The init script verifies config before doing either. You  
> could manually HUP the running process if you're sure the config files  
> are syntactically correct.
> 
>>  And although
>> again, I could in theory do this in a cron job, I'm not sure I'm
>> comfortable with that.  There's the potential for the Nagios service  
>> to
>> not start up again successfully, and I don't like taking the risk that
>> this dynamic update procedure could potentially bring down the entire
>> Nagios system.
> 
> I've been doing this exact thing (nagios reload) hourly for several  
> years quite successfully. If you wanted to be paranoid about it, you  
> could script a run of '/path/to/nagios -v /path/to/nagios.cfg' and  
> only reload if that exits 0 else send yourself an e-mail with the bad  
> output.
> 
> --
> Marc

Thanks much for these pointers (and the quick response).  Didn't know 
about the SIGHUP thing.

Well, it's good to know that this is a viable option.  Still, re-writing 
a config file seems a bit of a kludgey way to handle this.

I'm mulling over another idea, which I'm calling a "rotating" service 
check (for lack of a better word).  Basic idea is:  I set up one single 
service (rather than a service for each tag/file), and each time the 
service runs it automatically "rotates" to check the next tag/file.  If 
the check fails, then it stops rotating and continues checking that 
tag/file until the problem gets fixed.

Still working out the details, though, and not sure if this is a viable 
solution for us or not.  On the one hand, assuming we run the check 
every 5 minutes, then the service check would rotate through and thereby 
check all of the tags within some reasonable period of time.  On the 
other hand, if it hits a tag that fails the check, we'd only get an 
alert on that particular tag and wouldn't know if other tags were 
failing the check too.


IMO, the ideal solution here would be if I could just submit passive 
check results for services that aren't explicitly configured in Nagios. 
  But alas, that's not allowed and it fails with messages like "Warning: 
  Passive check result was received for service 'foo' on host 
'mysql-dev', but the service could not be found!"


I'll have to mull this over some more ...

Thanks,


--
Enter the BlackBerry Developer Challenge  
This is your chance to win up to $100,000 in prizes! For a limited time, 
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize  
details at: http://p.sf.net/sfu/Challenge
___
Nagios-users mailing list
Nagios-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when reporting 
any issue. 
::: Messages without supporting info will risk being sent to /dev/null

Re: [Nagios-users] Dynamically generated service checks

[Max]

On Wed, Jul 15, 2009 at 11:19 AM, David Rosenstrauch wrote:
> 2) If I understand correctly, Nagios loads its config files at startup
> time and does not re-read them after that.  So if I'm regenerating a
> config file, then that means I'd need to restart the Nagios service
> afterwards, which is a bit of an onerous imposition.  And although
> again, I could in theory do this in a cron job, I'm not sure I'm
> comfortable with that.

As long as you are not using ePN, you can just send a HUP signal to
the parent Nagios process; this will cause it to re-read the
configuration file(s) from disk.

> There's the potential for the Nagios service to
> not start up again successfully, and I don't like taking the risk that
> this dynamic update procedure could potentially bring down the entire
> Nagios system.

We let our users make changes via SVN using a branch per project, tag
their code, then we merge their sub-trees into our master configs, tag
our configs, and push them out to our production system; we have a
test instance that  validates the configs with checkconfig before we
push to production.  would be easy enough to have your automated push
out do something similar and send Nagios a trap or an NSCA passive
service to alert you to a bad config and not push to production

Because we do a lot of trending, we restart Nagios every night to
minimize scheduling skew; 20 minutes or so before the auto-restart a
cron job runs that does a checkconfig on the Nagios configuration ..
if the checkconfig fails, it touches a flag file .. the restart script
will then look for this flag and not restart if the flag file is
present.

In other words, there are many programmatic ways to ensure that a bad
restart does not happen, all are pretty straight-forward :).

- Max

--
Enter the BlackBerry Developer Challenge  
This is your chance to win up to $100,000 in prizes! For a limited time, 
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize  
details at: http://p.sf.net/sfu/Challenge
___
Nagios-users mailing list
Nagios-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when reporting 
any issue. 
::: Messages without supporting info will risk being sent to /dev/null

Re: [Nagios-users] Dynamically generated service checks

[Marc Powell]

On Jul 15, 2009, at 10:19 AM, David Rosenstrauch wrote:

> Matthew Jurgens wrote:

> 2) If I understand correctly, Nagios loads its config files at startup
> time and does not re-read them after that.  So if I'm regenerating a
> config file, then that means I'd need to restart the Nagios service
> afterwards, which is a bit of an onerous imposition.

Using the init script, restart stops and starts the daemon, reload  
sends a HUP signal to the running process to tell it to re-read it's  
config files. The init script verifies config before doing either. You  
could manually HUP the running process if you're sure the config files  
are syntactically correct.

>  And although
> again, I could in theory do this in a cron job, I'm not sure I'm
> comfortable with that.  There's the potential for the Nagios service  
> to
> not start up again successfully, and I don't like taking the risk that
> this dynamic update procedure could potentially bring down the entire
> Nagios system.

I've been doing this exact thing (nagios reload) hourly for several  
years quite successfully. If you wanted to be paranoid about it, you  
could script a run of '/path/to/nagios -v /path/to/nagios.cfg' and  
only reload if that exits 0 else send yourself an e-mail with the bad  
output.

--
Marc

--
Enter the BlackBerry Developer Challenge  
This is your chance to win up to $100,000 in prizes! For a limited time, 
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize  
details at: http://p.sf.net/sfu/Challenge
___
Nagios-users mailing list
Nagios-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when reporting 
any issue. 
::: Messages without supporting info will risk being sent to /dev/null

Re: [Nagios-users] Dynamically generated service checks

[David Rosenstrauch]

Matthew Jurgens wrote:
> I do a similar thing where I dynamically change the service check 
> definitions configured into Nagios. You've almost got to this solution 
> but say "and couldn't just ". Not sure why you say that but I'll briefly 
> cover what I did. You will need some programming/scripting.
> 
> First up in the nagios.cfg file configure either a specific 
> configuration file or directory where you will define your dynamic 
> service checks
> eg cfg_dir=/etc/nagios/dynamic_services or
> cfg_file=/etc/nagios/dynamic_services.cfg
> 
> Using a file/directory will be dependent on exactly how you need to 
> generate your checks.
> 
> Then at the point where you know what the service checks will be 
> generate the appropriate nagios config file based on what is in the 
> database into either your single config file or into one file each in 
> the dynamic directory.
> 
> You will then need to reload the nagios config and make sure it syntaxes 
> ok.
> 
> We do this quite sucessfully. When customers add/delete servers we just 
> regenerate the config to match and reload Nagios.


Thanks much for the response Matthew.  Very helpful!

I understand what you're suggesting here.  Sounds like a couple of 
potential issues with it though.  By doing this via regenerating a 
config file, this means that:

1) We would need to manually kick off a "config file regenerate" process 
each time a user adds/changes a tag/file in our system.  It would be 
much better if there were some way to have nagios automatically pick up 
the change without any manual intervention.  (Though I suppose we could 
work around this by using a cron job that runs periodically, checks for 
updates, and then regenerates the config if needed.)

2) If I understand correctly, Nagios loads its config files at startup 
time and does not re-read them after that.  So if I'm regenerating a 
config file, then that means I'd need to restart the Nagios service 
afterwards, which is a bit of an onerous imposition.  And although 
again, I could in theory do this in a cron job, I'm not sure I'm 
comfortable with that.  There's the potential for the Nagios service to 
not start up again successfully, and I don't like taking the risk that 
this dynamic update procedure could potentially bring down the entire 
Nagios system.


H  there's got to be *some* other way to accomplish this, no?

Thanks,

DR

--
Enter the BlackBerry Developer Challenge  
This is your chance to win up to $100,000 in prizes! For a limited time, 
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize  
details at: http://p.sf.net/sfu/Challenge
___
Nagios-users mailing list
Nagios-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when reporting 
any issue. 
::: Messages without supporting info will risk being sent to /dev/null

Re: [Nagios-users] Dynamically generated service checks

[Matthew Jurgens]

I do a similar thing where I dynamically change the service check 
definitions configured into Nagios. You've almost got to this solution 
but say "and couldn't just ". Not sure why you say that but I'll briefly 
cover what I did. You will need some programming/scripting.


First up in the nagios.cfg file configure either a specific 
configuration file or directory where you will define your dynamic 
service checks

eg cfg_dir=/etc/nagios/dynamic_services or
cfg_file=/etc/nagios/dynamic_services.cfg

Using a file/directory will be dependent on exactly how you need to 
generate your checks.


Then at the point where you know what the service checks will be 
generate the appropriate nagios config file based on what is in the 
database into either your single config file or into one file each in 
the dynamic directory.


You will then need to reload the nagios config and make sure it syntaxes ok.

We do this quite sucessfully. When customers add/delete servers we just 
regenerate the config to match and reload Nagios.


David Rosenstrauch wrote:
Scratching my head as to how it might be possible to accomplish a 
monitoring task I need to do:


We have a database table of tags.  For each tag we have a process that 
generates a Javascript file and pushes it out to a CDN.  I've 
successfully created a service check to monitor that the process runs 
successfully (by using check_mysql_query to verify that a "pushed" date 
gets updated on the database table).  But we also need to monitor that 
each of the files exists - and in the correct version - over at the CDN, 
and here's where I'm hitting a wall.


What I'm envisioning is a way whereby I could dynamically generate a 
whole bunch of service checks - one for each tag in the database table. 
  Each service check would then go and download its corresponding file 
from the CDN, parse it, and verify the version number.  In this way, if 
any particular file was missing or the wrong version, we would get a 
Nagios alert on that tag/file.


However, I don't see any clear way to approach this with Nagios.  Near 
as I can tell there doesn't seem to be any way to dynamically generate 
service checks.  From my reading of the docs, every service need to be 
explicitly listed in the config files beforehand.


I was thinking that perhaps having a cron job that generates passive 
checks might be a solution here, but again it looks like I would still 
need to have the service checks defined beforehand in the nagios config 
files, and couldn't just generate them on-the-fly based on the contents 
of the database table.



Anyone have any ideas on how I could accomplish this file monitoring in 
a dynamic fashion?  (Note that alerting on a file-by-file basis is 
needed here; it would not be sufficient to have a single service check 
that monitors and alerts on all of the files.)


TIA,

DR

--
Enter the BlackBerry Developer Challenge  
This is your chance to win up to $100,000 in prizes! For a limited time, 
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize  
details at: http://p.sf.net/sfu/Challenge

___
Nagios-users mailing list
Nagios-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when reporting any issue. 
::: Messages without supporting info will risk being sent to /dev/null


  


--
Smartmon System Monitoring 
www.smartmon.com.au
--
Enter the BlackBerry Developer Challenge  
This is your chance to win up to $100,000 in prizes! For a limited time, 
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize  
details at: http://p.sf.net/sfu/Challenge___
Nagios-users mailing list
Nagios-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when reporting 
any issue. 
::: Messages without supporting info will risk being sent to /dev/null