On Wed, Oct 9, 2013 at 11:01 PM, David Lang <[email protected]> wrote: > set timerequery to the a multiple of batchsize (1000 for each is > reasonable), it makes no sense to try and pull 1000 messages from the > kernel in one command, and then do time lookups for every 100, they all > arrived to you at exactly the same time. > > That message made me lookup the actual algo, as I thought the requery is done based on OS calls. And it is! I have updated the doc:
http://www.rsyslog.com/doc/imudp.html Rainer > David Lang > > > On Wed, 9 Oct 2013, David Lang wrote: > > Date: Wed, 9 Oct 2013 11:49:40 -0700 (PDT) >> From: David Lang <[email protected]> >> To: Robert <[email protected]> >> >> Cc: Rainer Gerhards <[email protected]>, rmkml <[email protected]> >> Subject: Re: [rsyslog] Tr : Re: perfomance tweaking (fwd) >> >> thanks for including the config, it's hard to keep track of everything. >> >> the 'unexpected end of file' error is because rsyslog still has the file >> open at the time you are doing the zgrep, if you rotate the files (mv the >> file then hit rsyslog with kill -HUP) rsyslog will write the remainder of >> the file and you will not get that error. >> >> The fact that all the per second numbers are eactly 200000 or 300000 is >> very odd >> >> what did top look like during the test? >> >> and just to confirm, this is rsyslog 7.5 right? >> >> if you remove ziplevel, it will disable gzip entirely. I expect that when >> you do that you will find iostat going to 100%, but you can give it a try. >> >> could you do a zgrep for a time frame when the queue was full (this will >> give us an idea of the throughput), say around 14:12 or so >> >> David Lang >> >> On Wed, 9 Oct 2013, Robert wrote: >> >> Date: Wed, 09 Oct 2013 14:40:13 -0400 >>> From: Robert <[email protected]> >>> To: David Lang <[email protected]> >>> Cc: Rainer Gerhards <[email protected]>, rmkml <[email protected]> >>> Subject: Re: [rsyslog] Tr : Re: perfomance tweaking (fwd) >>> >>> Hello Everyone, >>> >>> I have been away for business, but I'm back in action, attached you will >>> find the stats for the rsyslog server, [we are currently throwing 395k/eps >>> to the server ~ 985Mb] and below you will find the current config (just to >>> refreshen the memory) >>> >>> >>> #### MODULES #### >>> >>> module(load="imuxsock") # needs to be done just once Roberto 8-9-13 >>> module(load="imklog") >>> module(load="impstats" interval="30" log.syslog="off" >>> log.file="/var/log/pstats") >>> >>> # Provides UDP syslog reception >>> module(load="imudp" >>> batchSize="1000" >>> TimeRequery="100" >>> SchedulingPolicy="fifo" >>> SchedulingPriority="10") >>> input(type="imudp" port="514" ) >>> >>> #### GLOBAL DIRECTIVES #### >>> >>> # Use imprecise-old-style timestamp format >>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat >>> >>> $MainMsgQueueType FixedArray >>> $MainMsgQueueSize 20000000 >>> $MainMsgQueueWorkerThreads 2 >>> $MainMsgQueueDequeueBatchSize 1000 >>> $InputUDPMaxSessions 40000000 >>> >>> #### RULES #### >>> >>> >>> >>> action (type="omfile" >>> File="/var/log/test/f_all" >>> ziplevel="1" >>> ASyncWriting="on") >>> >>> >>> Lastly these are the result from the zgrep command: >>> >>> >>> >>> [rcortiz@simon test]$ sudo zgrep 14:07:53 f_all | wc -l >>> >>> gzip: f_all: unexpected end of file >>> 200001 >>> [rcortiz@simon test]$ sudo zgrep 14:07:54 f_all | wc -l >>> >>> gzip: f_all: unexpected end of file >>> 200000 >>> [rcortiz@simon test]$ sudo zgrep 14:07:55 f_all | wc -l >>> >>> gzip: f_all: unexpected end of file >>> 200000 >>> [rcortiz@simon test]$ sudo zgrep 14:08:00 f_all | wc -l >>> >>> gzip: f_all: unexpected end of file >>> 300000 >>> [rcortiz@simon test]$ sudo zgrep 14:08:10 f_all | wc -l >>> >>> gzip: f_all: unexpected end of file >>> 200000 >>> [rcortiz@simon test]$ sudo zgrep 14:08:20 f_all | wc -l >>> >>> gzip: f_all: unexpected end of file >>> 200000 >>> [rcortiz@simon test]$ sudo zgrep 14:08:30 f_all | wc -l >>> >>> gzip: f_all: unexpected end of file >>> 200000 >>> [rcortiz@simon test]$ sudo zgrep 14:08:40 f_all | wc -l >>> >>> gzip: f_all: unexpected end of file >>> 200001 >>> [rcortiz@simon test]$ sudo zgrep 14:08:50 f_all | wc -l >>> >>> gzip: f_all: unexpected end of file >>> 200000 >>> [rcortiz@simon test]$ >>> >>> >>> I am curious if we need to remove the zipping part of the action also >>> maybe adding the recvbuffer size? >>> >>> Robert. >>> >>> >> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
