Ok. Thank you so much! -Jason
Rainer Gerhards wrote: >> -----Original Message----- >> From: [email protected] [mailto:rsyslog- >> [email protected]] On Behalf Of Jason Antman >> Sent: Friday, November 05, 2010 6:13 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] segfault with dynamic filenames >> >> Ok... >> >> in output from configure script: >> checking whether the compiler provides atomic builtins... yes >> checking whether the compiler provides atomic builtins for 64 bit data >> types... yes >> >> in omfile.c, lines 89-102: >> #if HAVE_ATOMIC_BUILTINS_64BIT >> static uint64 clockFileAccess = 0; >> #else >> static unsigned clockFileAccess = 0; >> #endif >> /* and the "tick" function */ >> #ifndef HAVE_ATOMIC_BUILTINS >> static pthread_mutex_t mutClock; >> #endif >> static inline uint64 >> getClockFileAccess(void) >> { >> return ATOMIC_INC_AND_FETCH(&clockFileAccess, &mutClock); >> } >> >> Once again, my C skills are pretty poor... but if HAVE_ATOMIC_BUILTINS >> is defined.... what is mutClock?? >> > > A mutex name for a mutex that is not even created in this case ;) > > >> Output, strace log, and valgrind output are being sent to rainer. >> > > I'll probably not manage to have more than a peek at it today, but will look > ASAP (it's getting a bit late over here ;)). > > Rainer > >> Jason Antman wrote: >> >>> I haven't run it with valgrind yet... honestly I don't think I've >>> >> used >> >>> valgrind more than once or twice, I'd need to do a bit of reading up >>> >> to >> >>> remember how to use it. >>> >>> This isn't a matter of *if* the issue reappears. Every time rsyslogd >>> tries to write out a message matching a dynafile output, it dies >>> >> (within >> >>> a few ms of starting, or whenever it actually tries to write the >>> >> first >> >>> line out to disk). >>> >>> At the moment, I have the issue localized to the call to >>> getClockFileAccess() on line 517 of tools/omfile.c, and am continuing >>> >> to >> >>> trace it through the code. I'll see if one of the more c-skilled guys >>> can give me a hand with valgrind. >>> >>> -Jason >>> >>> Rainer Gerhards wrote: >>> >>> >>>> An you run it under valgrind control? If the issue reappears, it >>>> >> should give >> >>>> us *very* good hints of where to look at... >>>> >>>> RAiner >>>> >>>> >>>> >>>> >>>>> -----Original Message----- >>>>> From: [email protected] [mailto:rsyslog- >>>>> [email protected]] On Behalf Of Jason Antman >>>>> Sent: Friday, November 05, 2010 5:26 PM >>>>> To: rsyslog-users >>>>> Subject: Re: [rsyslog] segfault with dynamic filenames >>>>> >>>>> Thanks for the info! >>>>> >>>>> Just sent this to Tom, forgot to CC the list.... >>>>> >>>>> I just rebuilt the RPM with all debugging features enabled and am >>>>> experiencing the same issue. >>>>> >>>>> I have: >>>>> - debug log output (rsyslogd -c 5 -dn, with RSYSLOG_DEBUG env >>>>> >> variable >> >>>>> set to "LogFuncFlow FileTrace=omfile.c Debug") >>>>> - strace output >>>>> - RPM spec file including configure options, etc. >>>>> - the Fedora patch that is being applied >>>>> - the actual RPMs, if they're of interest, and a source rpm >>>>> >>>>> Which of this is of use? Should I email it? Open a ticket and >>>>> >> attach? >> >>>>> Here are the last few lines before the crash: >>>>> 2954.318300688:b7f06b90: Processing next action >>>>> 2954.318308532:b7f06b90: Called action(NotAllMark), logging to >>>>> >> builtin- >> >>>>> file >>>>> 2954.318315864:b7f06b90: Called action(Batch), logging to builtin- >>>>> >> file >> >>>>> 2954.318382400:b7f06b90: omfile.c:682: beginTransaction: enter >>>>> 2954.318390551:b7f06b90: omfile.c:682: beginTransaction: exit: 0 >>>>> 2954.318397068:b7f06b90: Action 0x9fbb3a0 transitioned to state: >>>>> >> itx >> >>>>> 2954.318404578:b7f06b90: entering actionCalldoAction(), state: itx >>>>> 2954.318411997:b7f06b90: omfile.c:697: doAction: enter >>>>> 2954.318418085:b7f06b90: file to log to: LocalHost >>>>> 2954.318424740:b7f06b90: omfile.c:634: writeFile: enter >>>>> 2954.318431685:b7f06b90: omfile.c:493: prepareDynFile: enter >>>>> (and then it dies, in the middle of a select() call) >>>>> >>>>> I'm going to start looking through omfile.c and perhaps adding some >>>>> >> of >> >>>>> my own instrumentation just so I can localize it a bit more. >>>>> >>>>> Thanks for any advice you can provide, >>>>> Jason Antman >>>>> >>>>> Tom Bergfeld wrote: >>>>> >>>>> >>>>> >>>>>> Hi Jason, >>>>>> >>>>>> unfortunately we were not able to reproduce your issue in Fedora >>>>>> >> 13. >> >>>>>> We will create an environment with centos and try it again. >>>>>> >>>>>> Tom >>>>>> >>>>>> >>>>>> -----Ursprüngliche Nachricht----- >>>>>> Von: [email protected] >>>>>> [mailto:[email protected]] Im Auftrag von Jason >>>>>> >>>>>> >>>>>> >>>>> Antman >>>>> >>>>> >>>>> >>>>>> Gesendet: Donnerstag, 4. November 2010 22:47 >>>>>> An: rsyslog-users >>>>>> Betreff: [rsyslog] segfault with dynamic filenames (was: Rsyslog >>>>>> evaluationquestions) >>>>>> >>>>>> Thanks to both of you guys for your answers. FYI, our days have 24 >>>>>> >>>>>> >>>>>> >>>>> hours >>>>> >>>>> >>>>> >>>>>> in them :) >>>>>> >>>>>> At this point I just got a config file worked up. I got the >>>>>> >> regexes >> >>>>>> working to do all of the matching I need into separate fields (the >>>>>> online regex tester was a big help), and upgraded to 5.6.0 (CentOS >>>>>> >>>>>> >>>>>> >>>>> 5.5, >>>>> >>>>> >>>>> >>>>>> in-house RPM build) in order to of bind imudp to a ruleset (so we >>>>>> >> can >> >>>>>> have one ruleset for localhost/unix socket, and one ruleset for >>>>>> >> all >> >>>>>> remote hosts). >>>>>> >>>>>> Unfortunately, I'm getting a segfault almost immediately on >>>>>> >> startup >> >>>>>> >>>>> due >>>>> >>>>> >>>>> >>>>>> to the following lines: >>>>>> $template >>>>>> >>>>>> >>>>>> >>>>>> >> RemoteHost,"/var/log/HOSTS/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/%syslogfa >> >>>>> cility >>>>> >>>>> >>>>> >>>>>> -text%.log" >>>>>> *.* ?RemoteHost >>>>>> >>>>>> I believe I've narrowed the problem down to any templated >>>>>> >> (dynamic) >> >>>>>> filename... even if I dump everything external directly to >>>>>> /var/log/remote and add the following in my local ruleset: >>>>>> $template LocalHost,"/var/log/LOCAL/%syslogfacility-text%.log" >>>>>> *.* ?LocalHost >>>>>> >>>>>> I still get a segfault. The last line in the output running with - >>>>>> >> dn >> >>>>>> >>>>> is >>>>> >>>>> >>>>> >>>>>> always: >>>>>> "file to log to: RemoteHost" (or LocalHost, when I used that >>>>>> >> config). >> >>>>>> Has anyone seen anything like this? I have rsyslog 2.0.6 running >>>>>> >> at >> >>>>>> another site and dynamic filenames work fine... sort of troubling >>>>>> >>>>>> >>>>>> >>>>> that >>>>> >>>>> >>>>> >>>>>> 5.6.0 is having issues with them... >>>>>> >>>>>> Thanks, >>>>>> Jason Antman >>>>>> >>>>>> Aaron Wiebe wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>> 3) Assuming an even distribution over time (not quite accurate), >>>>>>>> >>>>>>>> >>>>>>>> >>>>> any >>>>> >>>>> >>>>> >>>>>>>> thoughts on how dumping ~2M lines/day of syslog to MySQL on a VM >>>>>>>> >>>>>>>> >>>>>>>> >>>>> (Xen) >>>>> >>>>> >>>>> >>>>>>>> with a single 2.8GHz CPU and 512MB RAM would go? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> 2 million lines a day is 70 lines per second assuming an 8 hour >>>>>>> >> day. >> >>>>>>> MySQL, properly configured, will eat that. Rsyslog won't even >>>>>>> >>>>>>> >>>>>>> >>>>> notice >>>>> >>>>> >>>>> >>>>>>> that log level. >>>>>>> >>>>>>> Your table sizes may be problematic over time, and your ability >>>>>>> >> to >> >>>>>>> query may impact things. Regardless, your concern should be >>>>>>> >> mysql, >> >>>>>>> not rsyslog. >>>>>>> >>>>>>> -Aaron >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>>> >>>>> >>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>>> >>>> >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
