Hi Rainer, Thanks for the information. I didn't know that.
Scott On Fri, Jul 4, 2008 at 3:35 AM, Rainer Gerhards <[EMAIL PROTECTED]> wrote: > Hi Scott, > > just for your understanding: once a release is stable, it will only > receive fixes. So the fix we are looking it needs to become part of the > devel, be migrated to beta and then be migrated to stable. In general, > one migration step takes between 4 and 12 weeks, depending on what was > done. So far, the average seems on the average, that is two month. So > two cycles mean roughly four month. HOWEVER, I intend to put it into the > next beta, which will bring us to just one cycle. So it would be > available in roughly two month. > > Rainer > >> -----Original Message----- >> From: [EMAIL PROTECTED] [mailto:rsyslog- >> [EMAIL PROTECTED] On Behalf Of Scott Phuong >> Sent: Friday, July 04, 2008 12:32 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] rsyslog threads questions >> >> Hi Rainer, >> >> Thanks!! I'll give this a try next week and let you know the results. >> If it works, I think it is okay not to include in the upcoming v3 >> stable since it is due to release very soon. But if it does work, it >> would be great to have it in the following stable release. >> >> Scott >> >> On Fri, Jul 4, 2008 at 1:31 AM, Rainer Gerhards >> <[EMAIL PROTECTED]> wrote: >> > Scott, >> > >> > I have rewritten the omusrmsg plugin. This is currently part of the >> > development tree, but will become part of the (new) beta when we do > a >> > version number shift next week. I'd appreciate if you could give it > a >> > try. It is available here: >> > >> > http://download.rsyslog.com/download/rsyslog-3.19.9-pre1.tar.gz >> > >> > I am hesitant to put this in the upcoming new v3-stable. There were >> lots >> > of changes and one rule for a stable is that there should be no non- >> fix >> > type of changes for at least 4 weeks before it turns into a stable. >> It >> > of course is debatable if the change I made is a fix or not. In some >> > sense it is, but on the other hand it is a rewrite that changed a >> lot. >> > So given the fact that nobody on a "regular" machine saw an issue > the >> > past year, I would not like to bring the new code directly into the >> > stable. If that causes problems for you, you may want to try simply >> > using the omusrmsg.c code together with the v3-stable. I haven't >> tried >> > it, but it should work well. All changes were just to that file, and >> > there are no dependencies to anything external. >> > >> > OK... but now let's see first if it fixes the issue ;) >> > >> > Rainer >> > >> >> -----Original Message----- >> >> From: [EMAIL PROTECTED] [mailto:rsyslog- >> >> [EMAIL PROTECTED] On Behalf Of Scott Phuong >> >> Sent: Thursday, July 03, 2008 2:58 AM >> >> To: [email protected] >> >> Subject: Re: [rsyslog] rsyslog threads questions >> >> >> >> Hi, >> >> >> >> I've attached four files. Two of which are debug dumps, one is the >> >> conf file and the last one is a test case scenario that constantly >> >> fails on my end. I hope this gives a little more information. >> >> Furthermore, the dumps are from 3.17.5 which is the "closest" >> version >> >> to 3.18.0 that I was able to find. >> >> >> >> Both failed scenarios occur when lots of messages were being > flooded >> >> to rsyslogd at a very fast rate (look at logtest.c) The >> >> my_arm_rsyslog_suicide_debug.txt received a sigsegv fault while >> >> my_arm_rsyslog_sh_cannot_fork caused so many "Z [rsyslogd]" >> threads >> >> that it took up so much memory that executing any command as simple >> as >> >> 'ls -l' would not work from the command line. I think the number of >> >> threads grew as much as the number of messages. In the latter >> >> scenario, after killing logtest.c, it didn't look like the those >> >> zombies threads went away until I did a CTRL+C to the rsyslogd > which >> >> was running in the foreground since I use the "-dn" option. >> >> >> >> This is on an embedded system that runs significantly slower than a >> >> desktop or laptop so maybe it would be harder to reproduce on a >> >> regular computer. I looked at all the parameters that I believe >> could >> >> affect this and believe for the most part the defaults are more > than >> >> adequate. The main message queue never looked like it hit the high >> >> water mark but it did hit the lower one. So, I don't think messages >> >> were being dropped (not sure) or an overflow condition occurred. >> >> >> >> The processor is ARM-based and it is using Linux kernel 2.6.16.12 >> and >> >> compiled using GCC and the standard GNU C libraries version 3.4.5. >> >> Rsyslog source code is cross-compiled using the following configure >> >> line: >> >> >> >> ./configure --disable-zlib --disable-largefile >> >> --enable-share=yes >> >> --prefix=/ >> >> --host=arm-unknown-linux-gnu >> >> ac_cv_func_malloc_0_nonnull=yes >> >> ac_cv_func_realloc_0_nonnull=yes >> >> >> >> ac_cv_func_lstat_dereferences_slashed_symlink=yes >> >> ac_cv_func_stat_empty_string_bug=no >> >> enable_debug=no >> >> enable_rtinst=no >> >> >> >> Lastly, the logtest was executed with just the "-s" parameter. It > is >> a >> >> simple C file that I came up with. >> >> >> >> I took a look at the debug messages and it does not appear that new >> >> threads are created via calls to wtpStartWrkr in wtp.c. >> >> >> >> Any help I can bring to solve this issue, please let me know. I > hope >> I >> >> am not doing anything wrong here. >> >> >> >> Thanks, >> >> >> >> Scott >> >> > >> >> > On Wed, Jul 2, 2008 at 12:04 PM, Scott Phuong >> > <[EMAIL PROTECTED]> >> >> wrote: >> >> >> Hi Rainer, >> >> >> >> >> >> Thanks for your reply. Looking at the default settings (from > the >> >> >> online help's configuration page), they are what I wanted. The >> main >> >> >> messages queue is set to fix sized array with 1 worker thread >> >> created >> >> >> at maximum and action queues are direct mode which according to >> the >> >> >> queue document page, means that there will not be a worker > thread >> >> >> created. Is my understanding correct? If yes, how do I quickly >> >> check >> >> >> without using the -d option if the defaults are set correctly? > Or >> >> what >> >> >> do I look for in the debug messages that gets printed out to >> ensure >> >> >> this? >> >> >> >> >> >> You also mentioned that version 3.18.0 is probably going to be >> >> >> released as the stable version next week. I see on the webpage >> > there >> >> >> is a 3.17.4 and 3.17.5. Are these two versions similiar to >> 3.18.0? >> >> >> >> >> >> Also, how come I did not get your reply in my email inbox? My >> >> account >> >> >> settings look correct. >> >> >> >> >> >> Thanks, >> >> >> >> >> >> Scott Phuong >> >> >> >> >> >> As for the syslog buffer size, that applies to syslogd and does >> not >> >> >> apply to rsyslog. >> >> >> >> >> >> >> >> >> >> >> >> My configuration files do not change the Action queue or Worker >> >> queue >> >> >> parameters at all. Looking at >> >> >> On Wed, 2008-07-02 at 01:15 -0700, Scott Phuong wrote: >> >> >>> Hi, >> >> >>> >> >> >>> I have 3.16.2 which was recently released. I see that under >> > certain >> >> >>> conditions rsyslogd spawns a lot of threads: >> >> >>> 5949 root 11216 S rsyslogd >> >> >>> 5950 root 11216 S rsyslogd >> >> >>> 5951 root 11216 S rsyslogd >> >> >>> 5952 root 11216 S rsyslogd >> >> >>> 5953 root 11216 S rsyslogd >> >> >>> 5954 root 11216 S rsyslogd >> >> >>> 5985 root Z [rsyslogd] >> >> >>> 6445 root Z [rsyslogd] >> >> >>> >> >> >>> I had to kill the rsyslogd and restart it. The first invocation >> > had >> >> a >> >> >>> pid of 219 before it had to be killed. The second invocation of >> > pid >> >> >>> which you see above starts with 5949. The difference is the >> amount >> >> of >> >> >>> zombie threads that were invoked by rsyslogd before I had to >> kill >> >> the >> >> >>> first invocation of it. >> >> >> >> >> >> I have no explanation yet for the zombies. They should not > happen >> >> and so >> >> >> far I have never seen them. We may need to go through a debug > log >> >> (which >> >> >> will become very large) to find out what's going on. >> >> >> >> >> >>> The question is under what conditions does rsyslogd spawn a new >> >> >>> thread/process and why was it a zombie? >> >> >> >> >> >> Unfortunately, there is no quick answer. A quick one may be: > when >> > it >> >> >> needs them, based on queue watermark settings and based on you >> >> >> configuration. But to really understand it, you need to read > this >> >> doc: >> >> >> >> >> >> http://www.rsyslog.com/doc-queues.html >> >> >> >> >> >> The doc also describes all the knobs that you can use to control >> >> thread >> >> >> creation. There are many ;) >> >> >> >> >> >>> I am running rsyslogd in an >> >> >>> embedded environment and not a regular laptop/desktop. >> >> >> >> >> >> Interesting use case... >> >> >> >> >> >>> In addition, I >> >> >>> am using busybox and I believe the syslog buffer size is set to >> >> >> >> >> >> what do yo mean by "syslog buffer size"? The length of a receive >> >> buffer? >> >> >> It is 2K, thus single messages up to 2K are supported. It can be >> >> changed >> >> >> by modifying the MAXLINE define. Note that stock syslogd (and >> >> RFC3164) >> >> >> support only up to 1K. >> >> >> >> >> >>> something very low or perhaps none at all. Would this be a >> factor? >> >> >>> Furthermore, I ran rsyslogd with -c3 and also without -c3 and >> both >> >> >>> cases happen. >> >> >> >> >> >> The compatibility modes do not affect queue operation. >> >> >> >> >> >>> Are these issues already known and fixed in a later version? >> > Sorry, >> >> if >> >> >>> I am asking the same questions or have the same issues as >> previous >> >> >>> people but without the ability to search (or at least, I don't >> > know >> >> >>> how to) the archive, I don't know if my problem/questions has >> >> already >> >> >>> been seen and/or resolved. >> >> >> >> >> >> If we need to find out about the zombies, we need to move on to >> the >> >> >> current devel version. So I would give that a try in any case. >> >> 3.16.2 >> >> >> will (most probably) be replaced by 3.18.0 (based on the current >> >> beta) >> >> >> next week. So I won't touch it any longer. >> >> >> >> >> >> Looking forward to your feedback, >> >> >> Rainer >> >> >> >> >> >>> >> >> >>> Thank you very much for your support. >> >> >>> >> >> >>> Scott >> >> >>> _______________________________________________ >> >> >>> rsyslog mailing list >> >> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> >> >> >> > >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog
