Ok...
in output from configure script:
checking whether the compiler provides atomic builtins... yes
checking whether the compiler provides atomic builtins for 64 bit data
types... yes
in omfile.c, lines 89-102:
#if HAVE_ATOMIC_BUILTINS_64BIT
static uint64 clockFileAccess = 0;
#else
static unsigned clockFileAccess = 0;
#endif
/* and the "tick" function */
#ifndef HAVE_ATOMIC_BUILTINS
static pthread_mutex_t mutClock;
#endif
static inline uint64
getClockFileAccess(void)
{
return ATOMIC_INC_AND_FETCH(&clockFileAccess, &mutClock);
}
Once again, my C skills are pretty poor... but if HAVE_ATOMIC_BUILTINS
is defined.... what is mutClock??
Output, strace log, and valgrind output are being sent to rainer.
Jason Antman wrote:
> I haven't run it with valgrind yet... honestly I don't think I've used
> valgrind more than once or twice, I'd need to do a bit of reading up to
> remember how to use it.
>
> This isn't a matter of *if* the issue reappears. Every time rsyslogd
> tries to write out a message matching a dynafile output, it dies (within
> a few ms of starting, or whenever it actually tries to write the first
> line out to disk).
>
> At the moment, I have the issue localized to the call to
> getClockFileAccess() on line 517 of tools/omfile.c, and am continuing to
> trace it through the code. I'll see if one of the more c-skilled guys
> can give me a hand with valgrind.
>
> -Jason
>
> Rainer Gerhards wrote:
>
>> An you run it under valgrind control? If the issue reappears, it should give
>> us *very* good hints of where to look at...
>>
>> RAiner
>>
>>
>>
>>> -----Original Message-----
>>> From: [email protected] [mailto:rsyslog-
>>> [email protected]] On Behalf Of Jason Antman
>>> Sent: Friday, November 05, 2010 5:26 PM
>>> To: rsyslog-users
>>> Subject: Re: [rsyslog] segfault with dynamic filenames
>>>
>>> Thanks for the info!
>>>
>>> Just sent this to Tom, forgot to CC the list....
>>>
>>> I just rebuilt the RPM with all debugging features enabled and am
>>> experiencing the same issue.
>>>
>>> I have:
>>> - debug log output (rsyslogd -c 5 -dn, with RSYSLOG_DEBUG env variable
>>> set to "LogFuncFlow FileTrace=omfile.c Debug")
>>> - strace output
>>> - RPM spec file including configure options, etc.
>>> - the Fedora patch that is being applied
>>> - the actual RPMs, if they're of interest, and a source rpm
>>>
>>> Which of this is of use? Should I email it? Open a ticket and attach?
>>>
>>> Here are the last few lines before the crash:
>>> 2954.318300688:b7f06b90: Processing next action
>>> 2954.318308532:b7f06b90: Called action(NotAllMark), logging to builtin-
>>> file
>>> 2954.318315864:b7f06b90: Called action(Batch), logging to builtin-file
>>> 2954.318382400:b7f06b90: omfile.c:682: beginTransaction: enter
>>> 2954.318390551:b7f06b90: omfile.c:682: beginTransaction: exit: 0
>>> 2954.318397068:b7f06b90: Action 0x9fbb3a0 transitioned to state: itx
>>> 2954.318404578:b7f06b90: entering actionCalldoAction(), state: itx
>>> 2954.318411997:b7f06b90: omfile.c:697: doAction: enter
>>> 2954.318418085:b7f06b90: file to log to: LocalHost
>>> 2954.318424740:b7f06b90: omfile.c:634: writeFile: enter
>>> 2954.318431685:b7f06b90: omfile.c:493: prepareDynFile: enter
>>> (and then it dies, in the middle of a select() call)
>>>
>>> I'm going to start looking through omfile.c and perhaps adding some of
>>> my own instrumentation just so I can localize it a bit more.
>>>
>>> Thanks for any advice you can provide,
>>> Jason Antman
>>>
>>> Tom Bergfeld wrote:
>>>
>>>
>>>> Hi Jason,
>>>>
>>>> unfortunately we were not able to reproduce your issue in Fedora 13.
>>>> We will create an environment with centos and try it again.
>>>>
>>>> Tom
>>>>
>>>>
>>>> -----Ursprüngliche Nachricht-----
>>>> Von: [email protected]
>>>> [mailto:[email protected]] Im Auftrag von Jason
>>>>
>>>>
>>> Antman
>>>
>>>
>>>> Gesendet: Donnerstag, 4. November 2010 22:47
>>>> An: rsyslog-users
>>>> Betreff: [rsyslog] segfault with dynamic filenames (was: Rsyslog
>>>> evaluationquestions)
>>>>
>>>> Thanks to both of you guys for your answers. FYI, our days have 24
>>>>
>>>>
>>> hours
>>>
>>>
>>>> in them :)
>>>>
>>>> At this point I just got a config file worked up. I got the regexes
>>>> working to do all of the matching I need into separate fields (the
>>>> online regex tester was a big help), and upgraded to 5.6.0 (CentOS
>>>>
>>>>
>>> 5.5,
>>>
>>>
>>>> in-house RPM build) in order to of bind imudp to a ruleset (so we can
>>>> have one ruleset for localhost/unix socket, and one ruleset for all
>>>> remote hosts).
>>>>
>>>> Unfortunately, I'm getting a segfault almost immediately on startup
>>>>
>>>>
>>> due
>>>
>>>
>>>> to the following lines:
>>>> $template
>>>>
>>>>
>>>>
>>> RemoteHost,"/var/log/HOSTS/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/%syslogfa
>>> cility
>>>
>>>
>>>> -text%.log"
>>>> *.* ?RemoteHost
>>>>
>>>> I believe I've narrowed the problem down to any templated (dynamic)
>>>> filename... even if I dump everything external directly to
>>>> /var/log/remote and add the following in my local ruleset:
>>>> $template LocalHost,"/var/log/LOCAL/%syslogfacility-text%.log"
>>>> *.* ?LocalHost
>>>>
>>>> I still get a segfault. The last line in the output running with -dn
>>>>
>>>>
>>> is
>>>
>>>
>>>> always:
>>>> "file to log to: RemoteHost" (or LocalHost, when I used that config).
>>>>
>>>> Has anyone seen anything like this? I have rsyslog 2.0.6 running at
>>>> another site and dynamic filenames work fine... sort of troubling
>>>>
>>>>
>>> that
>>>
>>>
>>>> 5.6.0 is having issues with them...
>>>>
>>>> Thanks,
>>>> Jason Antman
>>>>
>>>> Aaron Wiebe wrote:
>>>>
>>>>
>>>>
>>>>>> 3) Assuming an even distribution over time (not quite accurate),
>>>>>>
>>>>>>
>>> any
>>>
>>>
>>>>>> thoughts on how dumping ~2M lines/day of syslog to MySQL on a VM
>>>>>>
>>>>>>
>>> (Xen)
>>>
>>>
>>>>>> with a single 2.8GHz CPU and 512MB RAM would go?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> 2 million lines a day is 70 lines per second assuming an 8 hour day.
>>>>> MySQL, properly configured, will eat that. Rsyslog won't even
>>>>>
>>>>>
>>> notice
>>>
>>>
>>>>> that log level.
>>>>>
>>>>> Your table sizes may be problematic over time, and your ability to
>>>>> query may impact things. Regardless, your concern should be mysql,
>>>>> not rsyslog.
>>>>>
>>>>> -Aaron
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>>>
>>>>
>>>>
>>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
>>
>>
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
