Re: [rsyslog] ISO timestamp, mmnormalize, and omelasticsearch for LogStash

Ahh, Thanks Pavel! I hadn't noticed the JSON formatting error, I corrected the issue and solved that issue completely shortly after. :) I managed to come up with, thanks to everyone's help, and some continued tinkering still, with this: http://linux-help.org/wiki/logging/advanced-rsyslog -- E

Re: [rsyslog] Rsyslog with RELP not sending/receiving messages for long intervals

On 11/25/2013 02:45 PM, David Lang wrote: On Mon, 25 Nov 2013, Erik Steffl wrote: On 11/25/2013 12:18 AM, David Lang wrote: On Mon, 25 Nov 2013, Pavel Levshin wrote: Date: Mon, 25 Nov 2013 12:05:15 +0400 From: Pavel Levshin Reply-To: rsyslog-users To: rsyslog@lists.adiscon.com Subject: Re:

Re: [rsyslog] Rsyslog with RELP not sending/receiving messages for long intervals

On Mon, 25 Nov 2013, Erik Steffl wrote: On 11/25/2013 12:18 AM, David Lang wrote: On Mon, 25 Nov 2013, Pavel Levshin wrote: Date: Mon, 25 Nov 2013 12:05:15 +0400 From: Pavel Levshin Reply-To: rsyslog-users To: rsyslog@lists.adiscon.com Subject: Re: [rsyslog] Rsyslog with RELP not sending/re

Re: [rsyslog] hostfile

On Mon, 25 Nov 2013, robert s wrote: Hello all, I was curious if you disable dns lookup when starting rsyslog does it look at the /etc/hosts to determine which hosts are sending logs ? no, if you disable the DNS lookup it doesn't do any name lookup at all. If the sending host put a hostname

Re: [rsyslog] Rsyslog with RELP not sending/receiving messages for long intervals

On 11/25/2013 03:20 AM, Pavel Levshin wrote: 25.11.2013 15:06, Pavel Levshin: - 6 hosts that were sending syslog messages to the log collector STOP sending anything (as verified by stracing rsyslogd, tcpdump and in amazon AWS console metric for network in) - after this nothing is ever writte

Re: [rsyslog] Rsyslog with RELP not sending/receiving messages for long intervals

On 11/25/2013 03:06 AM, Pavel Levshin wrote: 25.11.2013 12:35, David Lang пишет: On Mon, 25 Nov 2013, Pavel Levshin wrote: 25.11.2013 12:18, David Lang:\ Another possibly interesting message is: 7975.038523942:7fe2064cf700: main Q: doEnqSingleObject: LightDelay mark reached for light delay

[rsyslog] hostfile

Hello all, I was curious if you disable dns lookup when starting rsyslog does it look at the /etc/hosts to determine which hosts are sending logs ? Robert ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/

Re: [rsyslog] Rsyslog with RELP not sending/receiving messages for long intervals

On 11/25/2013 12:18 AM, David Lang wrote: On Mon, 25 Nov 2013, Pavel Levshin wrote: Date: Mon, 25 Nov 2013 12:05:15 +0400 From: Pavel Levshin Reply-To: rsyslog-users To: rsyslog@lists.adiscon.com Subject: Re: [rsyslog] Rsyslog with RELP not sending/receiving messages for long intervals 2

Re: [rsyslog] mitigating periodic statistics perf impact

Thanks to both of you for the quick response. I can see how it's a fair warning in the doc, and I could have just ran some tests...but this saves me a lot of time. -Original Message- From: Rainer Gerhards Reply-To: rsyslog-users Date: Monday, November 25, 2013 2:41 PM To: rsyslog-users

Re: [rsyslog] mitigating periodic statistics perf impact

I need to re-word that. Its from the initial release. Practice has shown its usually less than 1% on very busy servers (e.g 300,000 mps vs 297,000mps). If its not high end, you don't notice it... Sent from phone, thus brief. Am 25.11.2013 19:41 schrieb "Robert McIntyre" : > I've been using impsta

Re: [rsyslog] mitigating periodic statistics perf impact

I've been using impstats on a modest (60k+ mps) server, and haven't noticed any difference with stats enabled/disabled. --Robert From: Mike Hoskins (michoski) Sent: ‎11/‎25/‎2013 10:22 AM To: rsyslog@lists.adiscon.com

[rsyslog] mitigating periodic statistics perf impact

Hi folks, I've recently moved several environments from syslog-ng to rsyslog, and still working to fully integrate with various other tools like graphite and elasticsearch. As part of that, one of the things I was looking at was enabling impstats for shoving metrics into graphite. That said, the

Re: [rsyslog] v8 patches

25.11.2013 18:38, Rainer Gerhards: In the prototype, I currently pass in the strlen as it made things easy to change. If we think NUL is useful, we can keep it that way -- but I've never seen NUL in real world log data EXCEPT for attacks. There it may be useful [and that was a prime reason fo

[rsyslog] Sorting with LogAnalyzer

Hi all, Now my central logger setup is nearly complete, I'm workign with LogAnalyzer which is quite efficient even though it could be far better (especially on the user/groups handling which is really bad). I began sending my logs from my remote servers to this central logger, however, I have

Re: [rsyslog] Missing logs while polling logfile

what does the config on the receiving machine look like? David Lang On Mon, 25 Nov 2013, Bijohn Bijohn wrote: Date: Mon, 25 Nov 2013 17:53:47 +0530 From: Bijohn Bijohn Reply-To: rsyslog-users To: rsyslog-users Subject: Re: [rsyslog] Missing logs while polling logfile Thanks David. Followi

Re: [rsyslog] v8 patches

side note: in the longer term (not a prioprity), I want to remove libestr from rsyslog as well. Rainer On Mon, Nov 25, 2013 at 3:38 PM, Rainer Gerhards wrote: > On Mon, Nov 25, 2013 at 3:10 PM, Pavel Levshin wrote: > >> >> One thought: while we are right now changing the interface, we may thin

Re: [rsyslog] v8 patches

On Mon, Nov 25, 2013 at 3:10 PM, Pavel Levshin wrote: > > One thought: while we are right now changing the interface, we may think >> about going a bit further: libestr's primary purpose was also CEE, which >> originally required the \0 bytes must be valid chars (even though later >> folks that

Re: [rsyslog] v8 patches

On Mon, Nov 25, 2013 at 3:17 PM, Pavel Levshin wrote: > 25.11.2013 17:55, Rainer Gerhards: > > I have given the idea a quick try. The goal was to make the API use >> traditional c strings, NOT to (yet) to remove libestr from the inner >> workings. This should notably remove CPU requirements of m

Re: [rsyslog] v8 patches

25.11.2013 17:55, Rainer Gerhards: I have given the idea a quick try. The goal was to make the API use traditional c strings, NOT to (yet) to remove libestr from the inner workings. This should notably remove CPU requirements of mmnormalize. The experimental versions are currently available via

Re: [rsyslog] v8 patches

25.11.2013 16:57, Rainer Gerhards: I have merged these now, currently into the master-lognorm1 branch. I have also reviewed the liblognorm changes, great work, thx! You are welcome. And thanks to you, too, for a very good software. One thought: while we are right now changing the interface,

Re: [rsyslog] v8 patches

Hello! I'm the original author of omhiredis. I have not had the proper time to devote to supporting it after the initial release. I'd love to get time again to go back and improve it. That being said concentration on it hasn't been a priority lately and I certainly would not be offended if s

Re: [rsyslog] v8 patches

On Mon, Nov 25, 2013 at 1:57 PM, Rainer Gerhards wrote: > On Mon, Nov 18, 2013 at 8:29 AM, Pavel Levshin wrote: > >> Hello. >> >> Rsyslog v8 is running on my testbed (about 80krps right now), with >> mmnormalize and omhiredis. At first glance, it is much more scalable than >> previous versions. >>

Re: [rsyslog] v8 patches

On Mon, Nov 18, 2013 at 8:29 AM, Pavel Levshin wrote: > Hello. > > Rsyslog v8 is running on my testbed (about 80krps right now), with > mmnormalize and omhiredis. At first glance, it is much more scalable than > previous versions. > > Patch for omhiredis with v8 interface is attached. I'm not sur

Re: [rsyslog] Missing logs while polling logfile

If you are able to conduct that synthetic test, maybe you can collect debug log from rsyslog as well? It will most probably show what is wrong here. -- Pavel Levshin 25.11.2013 16:23, Bijohn Bijohn: Thanks David. Following are the configs I used. *syslogd 7.5.4* module(load="imfile" P

Re: [rsyslog] Missing logs while polling logfile

Thanks David. Following are the configs I used. *syslogd 7.5.4* module(load="imfile" PollingInterval="60") input(type="imfile" File="/tmp/ne.original.log" Tag="test.ne.ne-error" StateFile="test.ne.ne-error" ReadMode="1" escapeLF="on") if $syslogtag == 'test.ne.ne-error'

[rsyslog] rsyslog 7.5.7 (v7-devel) released

Hi all, this is primarily a bug-fixing release, but offers some improvements in worker thread handling (thanks to Pavel Levshin!) as well as usability improvements when working changing queue sizes. More detailed information is available in the changelog. ChangeLog: http://www.rsyslog.com/chang

Re: [rsyslog] Rsyslog with RELP not sending/receiving messages for long intervals

25.11.2013 15:06, Pavel Levshin: - 6 hosts that were sending syslog messages to the log collector STOP sending anything (as verified by stracing rsyslogd, tcpdump and in amazon AWS console metric for network in) - after this nothing is ever written into /path/2013-10-10/03/00/log.json -

Re: [rsyslog] Rsyslog with RELP not sending/receiving messages for long intervals

25.11.2013 12:35, David Lang пишет: On Mon, 25 Nov 2013, Pavel Levshin wrote: 25.11.2013 12:18, David Lang:\ Another possibly interesting message is: 7975.038523942:7fe2064cf700: main Q: doEnqSingleObject: LightDelay mark reached for light delayable message - blocking a bit. which was re

Re: [rsyslog] Missing logs while polling logfile

what is your config? David Lang On Mon, 25 Nov 2013, Bijohn Bijohn wrote: Date: Mon, 25 Nov 2013 14:48:22 +0530 From: Bijohn Bijohn Reply-To: rsyslog-users To: rsyslog@lists.adiscon.com Subject: [rsyslog] Missing logs while polling logfile I am facing an issue while writing lines of logs in

[rsyslog] Missing logs while polling logfile

I am facing an issue while writing lines of logs in a log file to another using rsyslog. rsyslog is* missing few lines of logs when polling the log file.* If the polling interval is 10 sec, it misses few lines in every 10 sec. SO I changed it to 60 sec. But it still misses some lines once in a min

Re: [rsyslog] Rsyslog with RELP not sending/receiving messages for long intervals

On Mon, 25 Nov 2013, Pavel Levshin wrote: 25.11.2013 12:18, David Lang:\ Another possibly interesting message is: 7975.038523942:7fe2064cf700: main Q: doEnqSingleObject: LightDelay mark reached for light delayable message - blocking a bit. which was received approximately once per secon

Re: [rsyslog] Rsyslog with RELP not sending/receiving messages for long intervals

25.11.2013 12:18, David Lang:\ Another possibly interesting message is: 7975.038523942:7fe2064cf700: main Q: doEnqSingleObject: LightDelay mark reached for light delayable message - blocking a bit. which was received approximately once per second during following interval (this is also w

Re: [rsyslog] Rsyslog with RELP not sending/receiving messages for long intervals

On Mon, 25 Nov 2013, Pavel Levshin wrote: Date: Mon, 25 Nov 2013 12:05:15 +0400 From: Pavel Levshin Reply-To: rsyslog-users To: rsyslog@lists.adiscon.com Subject: Re: [rsyslog] Rsyslog with RELP not sending/receiving messages for long intervals 23.11.2013 7:01, Erik Steffl: Another poss

Re: [rsyslog] Rsyslog with RELP not sending/receiving messages for long intervals

On Mon, 25 Nov 2013, Pavel Levshin wrote: 23.11.2013 7:01, Erik Steffl: Another possibly interesting message is: 7975.038523942:7fe2064cf700: main Q: doEnqSingleObject: LightDelay mark reached for light delayable message - blocking a bit. which was received approximately once per second

Re: [rsyslog] Rsyslog with RELP not sending/receiving messages for long intervals

23.11.2013 7:01, Erik Steffl: Another possibly interesting message is: 7975.038523942:7fe2064cf700: main Q: doEnqSingleObject: LightDelay mark reached for light delayable message - blocking a bit. which was received approximately once per second during following interval (this is also whe