Hi Jason, this sounds like a corrupted rrd file. Could you remove the host from the jffnms system, delete the associated rrd files (or move them), re-create the host in jffnms and see if you still get segfaults?
Otherwise you could try and manually run the poller on the affected hosts and use strace, that may give you some clue as to where the problems lie. Cheers Martin ... wrote: > Hi > When JFFNMS fails to poll a host, all the threads die; > > 10:45:42 ITEMS Added 252 items > 10:45:42 Poller 4969 finished working on host 293 in 00:01:10, Items > 203/203. > 10:45:42 Poller failed when polling host 247: READY PID 5003. > > Any idea what this failure is? Thanks > > Jason > > On Thu, Feb 28, 2008 at 9:24 AM, ... <[EMAIL PROTECTED] > <mailto:[EMAIL PROTECTED]>> wrote: > > Here is the last few lines of a host I manually polled and its > resulting seg fault; > > 09:21:54 : H 71 : I 9377 : P 80 : no_poller(): 0 -> rrd(*): > input:131172149814 - output:100346604674 - inputerrors:0 - > outputerrors:0 - rtt:1 - packetloss:0 - inpackets:303521404 - > outpackets:735576203 - drops:0 - bandwidthin:125000000 - > bandwidthout:125000000 (time P: 0.58 | B: 7.89) > 09:21:54 : H 71 : I 9378 : P 80 : no_poller(): 0 -> rrd(*): > input:5557021246 - output:26536350173 - inputerrors:0 - > outputerrors:0 - rtt:1 - packetloss:0 - inpackets:26260420 - > outpackets:286573032 - drops:0 - bandwidthin:125000000 - > bandwidthout:125000000 (time P: 0.58 | B: 7.89) > 09:21:54 : H 71 : I 9379 : P 80 : no_poller(): 0 -> rrd(*): > input:504445049133 - output:101535188966 - inputerrors:0 - > outputerrors:0 - rtt:1 - packetloss:0 - inpackets:473996682 - > outpackets:726357874 - drops:0 - bandwidthin:125000000 - > bandwidthout:125000000 (time P: 0.58 | B: 7.82) > Segmentation fault > > This is one of the same hosts that is hanging up the poller2.php > master 10...it seems that those 10 slots are all probably seg > faulting and thus no new hosts are added and the RRDs never > populate. Hrmmmmmmm... > > Jason > > > On Thu, Feb 28, 2008 at 9:20 AM, ... <[EMAIL PROTECTED] > <mailto:[EMAIL PROTECTED]>> wrote: > > So when running that command, I find that after a few minutes, > there are no new hosts finishing and if I look up to see which > hosts were last added and then run the poller manually against > them, I find quite a few crash with segmentation faults. Is > this a hardware problem on this box, or is it software? Thanks > > Jason > > > On Thu, Feb 28, 2008 at 8:50 AM, ... <[EMAIL PROTECTED] > <mailto:[EMAIL PROTECTED]>> wrote: > > And now this; > > 08:49:09 STATUS Remaining/Pending/Done/Bad=Total items: > 262/10/4/0=258, Childs: 10:WWWWWWWWWW, Time 129 > 08:49:12 STATUS Remaining/Pending/Done/Bad=Total items: > 262/10/4/0=258, Childs: 10:WWWWWWWWWW, Time 132 > > Warning: mysql_connect(): Lost connection to MySQL server at > 'reading authorization packet', system error: 0 in > /opt/jffnms/lib/api.db.inc.php on line 150 > > DB(mysql): Could not connect to the Database. > > Jason > > > On Thu, Feb 28, 2008 at 8:42 AM, ... <[EMAIL PROTECTED] > <mailto:[EMAIL PROTECTED]>> wrote: > > Hi > And after running that command for a few minutes, this > is what I've seen; > > 08:39:57 STATUS Remaining/Pending/Done/Bad=Total > items: 262/5/8/0=254, Childs: 5:WWWWW, Time 250 > 08:40:01 STOP Stopping child with PID 4336, reason: > timeout and it had to be Killed! > Killed > 08:40:13 START Started child with PID > 08:40:13 STATUS Remaining/Pending/Done/Bad=Total > items: 262/4/8/0=254, Childs: 5:WWWWR, Time 266 > 08:40:13 ITEMS Added 254 items > 08:40:16 WORK Child was ready, putting it > to work on item 213, Try: 1 > 08:40:16 STATUS Remaining/Pending/Done/Bad=Total > items: 261/5/8/0=254, Childs: 5:WWWWW, Time 269 > 08:40:19 STATUS Remaining/Pending/Done/Bad=Total > items: 261/5/8/0=254, Childs: 5:WWWWW, Time 272 > 08:40:22 STATUS Remaining/Pending/Done/Bad=Total > items: 261/5/8/0=254, Childs: 5:WWWWW, Time 275 > 08:40:25 STATUS Remaining/Pending/Done/Bad=Total > items: 261/5/8/0=254, Childs: 5:WWWWW, Time 278 > 08:40:28 STATUS Remaining/Pending/Done/Bad=Total > items: 261/5/8/0=254, Childs: 5:WWWWW, Time 281 > 08:40:31 STATUS Remaining/Pending/Done/Bad=Total > items: 261/5/8/0=254, Childs: 5:WWWWW, Time 284 > 08:40:34 STATUS Remaining/Pending/Done/Bad=Total > items: 261/5/8/0=254, Childs: 5:WWWWW, Time 287 > 08:40:34 ITEMS Added 254 items > 08:40:37 STATUS Remaining/Pending/Done/Bad=Total > items: 262/5/8/0=254, Childs: 5:WWWWW, Time 290 > 08:40:38 Poller 4342 finished working on host 306 in > 00:02:40, Items 84/84. > 08:40:39 STATUS Remaining/Pending/Done/Bad=Total > items: 262/4/9/0=254, Childs: 5:WWWIW, Time 292 > 08:40:41 STATUS Remaining/Pending/Done/Bad=Total > items: 262/4/9/0=254, Childs: 5:WWWIW, Time 294 > 08:40:41 Poller failed when polling host 213: READY PID > 4382. > > > There are some failed hosts there. Any idea why? > > Thanks > > Jason > > > On Thu, Feb 28, 2008 at 8:41 AM, ... > <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>> wrote: > > Hi again > So from the above msg, I found my database had maxed > out the connection limit (not sure why the > connections were not closing out), but I stopped and > restarted mysql and reran that command, and here is > the output; > > > $ cd /opt/jffnms/engine && php -q poller2.php master 5 > 08:35:47 Launcher Starting. Parameters: childs: 5, > Timeout: 180, item Retries: 2, Rest Time: 3 secs, > Read Timeout: 3, Child Hearbeat every 5 secs. > $ 08:35:50 STATUS Remaining/Pending/Done/Bad > =Total items: 0/0/0/0=0, Childs: 0:, Time 3 > 08:35:50 ITEMS Added 262 items > 08:35:54 START Started child with PID 4334 > 08:35:55 START Started child with PID 4336 > 08:35:56 START Started child with PID 4338 > 08:35:57 START Started child with PID 4340 > 08:35:58 START Started child with PID 4342 > 08:35:58 STATUS Remaining/Pending/Done/Bad=Total > items: 262/0/0/0=262, Childs: 5:RRRRR, Time 11 > 08:36:01 WORK Child 4334 was ready, putting > it to work on item 381, Try: 1 > 08:36:01 WORK Child 4336 was ready, putting > it to work on item 379, Try: 1 > 08:36:01 WORK Child 4338 was ready, putting > it to work on item 282, Try: 1 > 08:36:01 WORK Child 4340 was ready, putting > it to work on item 233, Try: 1 > 08:36:01 WORK Child 4342 was ready, putting > it to work on item 428, Try: 1 > 08:36:01 STATUS Remaining/Pending/Done/Bad=Total > items: 257/5/0/0=262, Childs: 5:WWWWW, Time 14 > 08:36:03 STATUS Remaining/Pending/Done/Bad=Total > items: 257/5/0/0=262, Childs: 5:WWWWW, Time 16 > 08:36:05 STATUS Remaining/Pending/Done/Bad=Total > items: 257/5/0/0=262, Childs: 5:WWWWW, Time 18 > 08:36:07 STATUS Remaining/Pending/Done/Bad=Total > items: 257/5/0/0=262, Childs: 5:WWWWW, Time 20 > 08:36:09 STATUS Remaining/Pending/Done/Bad=Total > items: 257/5/0/0=262, Childs: 5:WWWWW, Time 22 > 08:36:11 STATUS Remaining/Pending/Done/Bad=Total > items: 257/5/0/0=262, Childs: 5:WWWWW, Time 24 > 08:36:11 ITEMS Added 262 items > 08:36:13 STATUS Remaining/Pending/Done/Bad=Total > items: 262/5/0/0=262, Childs: 5:WWWWW, Time 26 > 08:36:16 STATUS Remaining/Pending/Done/Bad=Total > items: 262/5/0/0=262, Childs: 5:WWWWW, Time 29 > 08:36:17 Poller 4338 finished working on host 282 in > 00:00:16, Items 84/84. > 08:36:17 Poller 4342 finished working on host 428 in > 00:00:16, Items 8/8. > 08:36:17 Poller 4340 finished working on host 233 in > 00:00:16, Items 32/32. > 08:36:20 STATUS Remaining/Pending/Done/Bad=Total > items: 262/2/3/0=262, Childs: 5:WWIII, Time 33 > 08:36:23 STATUS Remaining/Pending/Done/Bad=Total > items: 262/2/3/0=262, Childs: 5:WWIII, Time 36 > 08:36:26 WORK Child 4338 was ready, putting > it to work on item 204, Try: 1 > 08:36:26 WORK Child 4340 was ready, putting > it to work on item 407, Try: 1 > 08:36:26 WORK Child 4342 was ready, putting > it to work on item 235, Try: 1 > 08:36:26 STATUS Remaining/Pending/Done/Bad=Total > items: 259/5/3/0=262, Childs: 5:WWWWW, Time 39 > 08:36:28 STATUS Remaining/Pending/Done/Bad=Total > items: 259/5/3/0=262, Childs: 5:WWWWW, Time 41 > 08:36:30 STATUS Remaining/Pending/Done/Bad=Total > items: 259/5/3/0=262, Childs: 5:WWWWW, Time 43 > 08:36:32 STATUS Remaining/Pending/Done/Bad=Total > items: 259/5/3/0=262, Childs: 5:WWWWW, Time 45 > 08:36:32 ITEMS Added 259 items > 08:36:34 STATUS Remaining/Pending/Done/Bad=Total > items: 262/5/3/0=259, Childs: 5:WWWWW, Time 47 > 08:36:36 STATUS Remaining/Pending/Done/Bad=Total > items: 262/5/3/0=259, Childs: 5:WWWWW, Time 49 > 08:36:38 STATUS Remaining/Pending/Done/Bad=Total > items: 262/5/3/0=259, Childs: 5:WWWWW, Time 51 > 08:36:40 STATUS Remaining/Pending/Done/Bad=Total > items: 262/5/3/0=259, Childs: 5:WWWWW, Time 53 > 08:36:42 STATUS Remaining/Pending/Done/Bad=Total > items: 262/5/3/0=259, Childs: 5:WWWWW, Time 55 > 08:36:42 Poller 4338 finished working on host 204 in > 00:00:16, Items 41/41. > 08:36:43 Poller 4342 finished working on host 235 in > 00:00:16, Items 32/32. > > > Looks much better, does this look correct? > > Thanks again > > Jason > > > On Thu, Feb 28, 2008 at 8:40 AM, ... > <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>> > wrote: > > Hi > So I ran the poller as you advised above and I > think we've got something here; > > [EMAIL PROTECTED]:~# su jffnms > $ cd /opt/jffnms/engine && php -q poller2.php > master 5 > 08:28:12 Launcher Starting. Parameters: childs: > 5, Timeout: 180, item Retries: 2, Rest Time: 3 > secs, Read Timeout: 3, Child Hearbeat every 5 secs. > $ 08:28:15 STATUS Remaining/Pending/Done/Bad > =Total items: 0/0/0/0=0, Childs: 0:, Time 3 > 08:28:15 ITEMS Added 262 items > 08:28:19 START Started child with PID 4313 > 08:28:20 START Started child with PID 4317 > 08:28:21 START Started child with PID 4319 > 08:28:22 START Started child with PID 4321 > 08:28:23 START Started child with PID 4323 > 08:28:23 STATUS > Remaining/Pending/Done/Bad=Total items: > 262/0/0/0=262, Childs: 5:RRRRR, Time 11 > 08:28:26 WORK Child 4313 was ready, > putting it to work on item 381, Try: 1 > 08:28:26 WORK Child 4317 was ready, > putting it to work on item 379, Try: 1 > 08:28:26 WORK Child 4319 was ready, > putting it to work on item 282, Try: 1 > 08:28:26 WORK Child 4321 was ready, > putting it to work on item 233, Try: 1 > 08:28:26 WORK Child 4323 was ready, > putting it to work on item 428, Try: 1 > 08:28:26 STATUS > Remaining/Pending/Done/Bad=Total items: > 257/5/0/0=262, Childs: 5:WWWWW, Time 14 > 08:28:26 Poller 4321 failed when polling host > 233: Warning: mysql_connect(): Too many > connections in /opt/jffnms/lib/apidbincphp on > line 150. > 08:28:26 STOP Stopping child with PID 4321, > reason: and it exited normally saying: > DB(mysql): Could not connect to the Database > 08:28:27 Poller 4319 failed when polling host > 282: Warning: mysql_connect(): Too many > connections in /opt/jffnms/lib/apidbincphp on > line 150. > 08:28:27 STOP Stopping child with PID 4319, > reason: and it exited normally saying: > DB(mysql): Could not connect to the Database > 08:28:28 Poller 4323 failed when polling host > 428: Warning: mysql_connect(): Too many > connections in /opt/jffnms/lib/apidbincphp on > line 150. > 08:28:28 STOP Stopping child with PID 4323, > reason: and it exited normally saying: > DB(mysql): Could not connect to the Database > 08:28:29 STATUS > Remaining/Pending/Done/Bad=Total items: > 260/2/0/0=262, Childs: 5:WWIII, Time 17 > 08:28:30 WORK Child 4321 was ready, > putting it to work on item 204, Try: 1 > > Warning: fputs(): 45 is not a valid stream > resource in /opt/jffnms/engine/launcher.inc.php > on line 36 > 08:28:31 STATUS > Remaining/Pending/Done/Bad=Total items: > 259/3/0/0=262, Childs: 5:WWIWI, Time 19 > 08:28:31 WORK Child 4319 was ready, > putting it to work on item 407, Try: 1 > > Warning: fputs(): 41 is not a valid stream > resource in /opt/jffnms/engine/launcher.inc.php > on line 36 > 08:28:31 WORK Child 4323 was ready, > putting it to work on item 235, Try: 1 > > Warning: fputs(): 49 is not a valid stream > resource in /opt/jffnms/engine/launcher.inc.php > on line 36 > 08:28:33 STATUS > Remaining/Pending/Done/Bad=Total items: > 257/5/0/0=262, Childs: 5:WWWWW, Time 21 > 08:28:35 STATUS > Remaining/Pending/Done/Bad=Total items: > 257/5/0/0=262, Childs: 5:WWWWW, Time 23 > > Warning: mysql_connect(): Too many connections > in /opt/jffnms/lib/api.db.inc.php on line 150 > DB(mysql): Could not connect to the Database. > > > At that point I stopped the process...but it > looks like a problem connecting to the > database? Any ideas? Thanks a ton for all your > help! > > Jason > > > 2008/2/28 Martin Beecroft <[EMAIL PROTECTED] > <mailto:[EMAIL PROTECTED]>>: > > Shouldn't it be "sudo -u jffnms php -q > poller2.php master X"? > > LIMA David wrote: > > Hi, > > > > Can you double check the rights on the > rrd files? > > Is the field "poll" ticked on the > ADMIN/HOST screen ? > > Are you sure that doing sudo jffnms... is > the same as su - jffnms; php > > -q poller2.php ? > > If you use poller2 the correct syntax is > php -q poller2.php master X > > where X specify the number of poller > process to run. > > > > David LIMA > > > > -------- Message d'origine-------- > > *De:* > [EMAIL PROTECTED] > <mailto:[EMAIL PROTECTED]> > de la part de ... > > *Date:* mer. 27/02/2008 22:10 > > *À:* Torfinn Ingolfsen; > [email protected] > <mailto:[email protected]> > > *Cc:* > > *Objet:* Re: [jffnms-users] poller works > when run manually,but NaN from > > cron?? > > > > Hi > > So I've fully updated my backend > mysql database to 0.8.3 to match > > the running frontend. Yet, when > running from cron, the pollers > > never populate the RRD files, yet > when run manually, as the jffnms > > user (sudo jffnms, php -q poller2.php > HOSTID), the RRD files show > > data. Is there something else that > could be wrong? Thanks > > > > Jason > > > > On Wed, Feb 27, 2008 at 2:14 PM, > Torfinn Ingolfsen <[EMAIL PROTECTED] > <mailto:[EMAIL PROTECTED]> > > > > <https://portailsch.sch-groupe.fr/CitrixFEI/[EMAIL PROTECTED]>> > > wrote: > > > > Hello, > > > > Just to complete this part of the > info. > > > > On Tue, Feb 26, 2008 at 8:32 PM, > ... <[EMAIL PROTECTED] > <mailto:[EMAIL PROTECTED]> > > > > <https://portailsch.sch-groupe.fr/CitrixFEI/[EMAIL PROTECTED]>> > > wrote: > > > Hi > > > So I've edited my cron file to > replace the variables $JFFNMS > > and $PHP with > > > the actual paths in each command; > > > > You don't have to do that. Here > are the first few lines of my > > crontab > > file for jffnms: > > # jffnms crontab file > > # Created by Sergey Akifyev > <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]> > > > > <https://portailsch.sch-groupe.fr/CitrixFEI/[EMAIL PROTECTED]>> > > JFFNMS=/usr/local/share/jffnms/engine > > > > PATH=${PATH}:/bin:/usr/bin:/usr/local/bin:/sbin:/usr/sbin://usr/local/sbin > > # JFFNMS > > */1 * * * * cd $JFFNMS && php -q > consolidate.php >/dev/null 2>&1 > > */5 * * * * cd $JFFNMS && php -q > poller.php >/dev/null 2>&1 > > > > You see, it is erfextly possible > to define env variables in the > > crontab file. > > > > As I have read elsewhere in this > thread, your problem seems to be > > something else. > > -- > > Regards, > > Torfinn Ingolfsen > > > > > > ------------------------------------------------------------------------- > > This SF.net email is sponsored > by: Microsoft > > Defy all challenges. Microsoft(R) > Visual Studio 2008. > > > > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > > > _______________________________________________ > > jffnms-users mailing list > > > [email protected] > <mailto:[email protected]> > > > > <https://portailsch.sch-groupe.fr/CitrixFEI/[EMAIL PROTECTED]> > > > > https://lists.sourceforge.net/lists/listinfo/jffnms-users > > > > > > > > > > ______________________________________________________________________ > > Ce message contient des informations dont > le contenu est susceptible > > d'etre confidentiel. > > Il est destine au(x) destinataire(s) > indique(s) exclusivement. > > > > A moins que vous ne fassiez partie de la > liste des destinataires, ou que > > vous soyez habilite a recevoir le mail a > leur place, il vous est > > interdit de le copier, de l'utiliser ou > de devoiler son contenu a un tiers. > > > > Si vous avez recu cet email par erreur, > merci de prendre contact avec > > l'emetteur. > > > > Les opinions exprimees dans cet e-mail > sont celles de l'emetteur et ne > > refletent pas necessairement celles de > l'entreprise. > > > > Ce e-mail peut contenir des pieces > jointes dont certaines pourraient > > contenir des virus qui pourraient > endommager votre systeme informatique. > > > > La compagnie a pris toutes dispositions > afin de minimiser ce risque et > > decline toute responsabilite pour toute > perte ou dommage resultant > > directement ou indirectement de > l'utilisation de cet email ou de son > > contenu. > > > > Il vous appartient d'effectuer vos > propres controles anti-virus avant > > d'ouvrir la ou les pieces jointes. > > > > ______________________________________________________________________ > > > > > > > > ------------------------------------------------------------------------ > > > > > > ------------------------------------------------------------------------- > > This SF.net email is sponsored by: Microsoft > > Defy all challenges. Microsoft(R) Visual > Studio 2008. > > > > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > > > > > > > > ------------------------------------------------------------------------ > > > > > _______________________________________________ > > jffnms-users mailing list > > [email protected] > <mailto:[email protected]> > > > > https://lists.sourceforge.net/lists/listinfo/jffnms-users > > > > > > > > > > ------------------------------------------------------------------------ > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > > > ------------------------------------------------------------------------ > > _______________________________________________ > jffnms-users mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/jffnms-users ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ jffnms-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/jffnms-users
