Sorry fort he sluggish replies -- I will leave for xmas break in about a week and have a couple of things to finish... Some quicke notes below:
> -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Dražen Kacar > Sent: Wednesday, December 15, 2010 3:14 PM > To: [email protected] > Subject: [rsyslog] Losing UDP packages > > I have rsyslog 5.6.2 which is a destination for a lot of traffic which > is > currently using UDP as a transport protocol. We intend to switch some > sources to TCP, but UDP will still be used for a large chunk of data > and I > can't do anything about that. > > Current tests show that we lose 100-400 lines in every 100000 sent. > That > isn't much and we could live with it, but I'd like to do something > about > it, nevertheless. > > My guess is that the kernel buffer fills up, rsyslog doesn't manage to > make room fast enough and then the kernel drops packages. I'm on CentOS > 5.5. Kernel log shows lines like: > > Dec 8 14:44:22 ssm1 kernel: Initalizing network drop monitor service > Dec 13 15:09:26 ssm1 kernel: Initalizing network drop monitor service > > Curiously, the times do not match the ones for which I'm certain that > the > package loss occured. But even so, the log shows that packet drop is > occuring on the system. > > As far as I know, two things can be done. > > 1. Increase UDP receive buffer size. That should allow the system to > handle the bigger spike load than it's handling now, but that's about > it. > There will be a spike load which is too large for any given buffer > size. > > I didn't manage to find a configuration option for this. I can increase > the system default, but I dislike doing that because it affects all > sockets on the system. If rsyslog really doesn't have this > configuration > option, I'm willing to write a patch. I doesn't do that right now. You need to look at imudp. > > 2. Put a thread that handles UDP socket in real-time class (or the > closest > equivalent that a given OS has). The POSIX way for this is to use > sched_setscheduler() with SCHED_FIFO. There could be other non-portable > interfaces of interest (like priocntl() on Solaris), but I won't > discuss > them right now. > > The patch should add two configuration parameters: > * one to define scheduling policy (SCHED_FIFO, SCHED_RR, etc.), let's > call it $<object>QueueThreadSchedulingPolicy (although I'd like a > shorter name) > * and another to define sched_priority, let's call it > $<object>QueueThreadSchedulingPriority > > In case of SCHED_SPORADIC (and maybe other non-portable policies on > some > operating systems) there are additional parameters that can be set, so > maybe the second parameter would turn out to be the string with several > numbers that needs to be parsed. Or there could be additional > configuration options introduced. I don't have a position about that > and > would like to follow the existing rsyslog policy regarding > configuration > options. Except that I don't know what that policy is, so I'd > appreciate > if someone can enlighten me. UDP is not handled at the queue level. It is soley the thread that imudp runs on. Each input has its own thread, so you can focus on imudp. > > Anyway, the idea is to have a high priority thread (or a small number > of > threads) sitting on a UDP socket whose sole purpose in life is to > transfer the messages from UDP socket to another queue(s), as fast as > possible. That's what imudp does. I suggest to also have a look at: http://www.gerhards.net/download/LinuxKongress2010rsyslog.pdf RAiner > default (if $<object>QueueThreadSchedulingPolicy is > defined), > the number of threads put in that scheduling policy can be > $<object>QueueWorkerThreads, ie. the minimum number of worker threads. > There could be additional parameter to define the number of such > threads, > but I'm not sure how to go about it. > > There could be no imposed limit, so if an administator puts 128 in the > config file, 128 threads it is. But this is potentialy dangerous. Or > there > could be rsyslog imposed restriction that the number of such threads > cannot > exceed the number defined in $<object>QueueWorkerThreads, which is less > dangerous. I'm personally leaning to the second option. > > > Now, this whole thing is dangerous and must be implemented and used > with > care and understanding. Currently I don't understand rsyslog queues > well > enough, which is why I'm writing this mail. The UDP thread in FIFO > class > should be used just to pick the packet from the socket and place it on > another action queue. Let's say we have the following configuration: > > $Ruleset indata > $RulesetCreateMainQueue on > > $UDPServerRun 514 > $InputUDPServerBindRuleset indata > > $MainMsgQueueThreadSchedulingPolicy fifo > $MainMsgQueueThreadSchedulingPriority 10 > > $ActionName local1 > $ActionQueueType LinkedList > $ActionQueueFileName local1 > $ActionResumeRetryCount -1 > $ActionQueueSaveOnShutdown on > $ActionQueueDiscardSeverity 8 > > local1.* |/path/to/fifo > & /path/to/file > > > If I understand things correctly, there are two queues here. The first > one > is ruleset indata queue with the UDP input module and the second one is > the action queue for the destinations (FIFO and file). I'm not sure > about > the following: > > 1. After a $Ruleset directive followed by "$RulesetCreateMainQueue on", > are $MainMsgQueue* directives applied to that ruleset queue? It's not > very > intuitive. > > 2. Say we add a TCP listener (or any other input module) to the above > configuration and we want UDP processing thread(s) to have SCHED_FIFO > priority, but other input handling threads to stay at the default > priority. I'm not sure if that can be specified with one ruleset. If > two > rulesets must be specified, then I suppose the action directives must > be > duplicated for each ruleset, which isn't very readable and > maintenance-friendly. (This is without taking omruleset plugin into > consideration). > > 3. The action queue is disk assisted. Which thread writes to disk when > the > need arises? The one from the ruleset indata queue or the one from the > action queue? > > -- > .-. .-. Yes, I am an agent of Satan, but my duties are largely > (_ \ / _) ceremonial. > | > | [email protected] > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
