Re: [rsyslog] InputTCPMaxSessions and MaxOpenFiles relation
Agree it's not simple, but once providing the possibility to define these type of limits the developer should take care of some basic set of checks at least. Would like to propose to project owners to take care of making some notices in documentation till/if the checks will not be implemented. Some admins could assume increasing of InputTCPMaxSessions is enough, but it's not. Other approach could be to report once the rsyslog process is reaching those limits. That will lead rsyslog admin to think about those messages within investigation of rsyslog crash. Anyway the InputTCPMaxSessions should never be equal nor larger than MaxOpenFiles. And this could be somehow taken care of at least. On Tue, Dec 9, 2014 at 12:33 AM, David Lang wrote: > On Mon, 8 Dec 2014, Peter Viskup wrote: > > Hi all, >> we observed issues once setting InputTCPMaxSessions above the default >> MaxOpenFiles process limits. >> The rsyslog reached MaxOpenFiles limit and just crashed. >> Setting the value of InputTCPMaxSessions should increase MaxOpenFiles or >> rsyslog should not start or complain about it at least. There are more >> configuration parameters which need some additional pre-start >> checks/modifications (MaxListeners, ...). >> > > It's not quite that simple. > > I agree that it would be useful to detect that maxsessions is > > maxopenfiles, but if it is, figuring out what maxopenfiles needs to be > increased to requires knowing a lot more about what's going on. How many > files is rsyslog going to be writing to? and do you have dynafiles > configured which makes this variable based on the contents of the log > messages? Are you using any modules that will require filehandles? if so, > how many will they need. > > I think the best you can do is to complain and tell the admin that it > looks bad. > > David Lang > ___ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] InputTCPMaxSessions and MaxOpenFiles relation
2014-12-10 10:11 GMT+01:00 Peter Viskup : > Agree it's not simple, but once providing the possibility to define these > type of limits the developer should take care of some basic set of checks > at least. > I agree we should try to find at least the easy config errors. I'll open a github feature request soon. > Would like to propose to project owners to take care of making some notices > in documentation till/if the checks will not be implemented. Some admins > could assume increasing of InputTCPMaxSessions is enough, but it's not. > Other approach could be to report once the rsyslog process is reaching > those limits. That will lead rsyslog admin to think about those messages > within investigation of rsyslog crash. > > It would be great if you could just update the rsyslog-doc project: https://github.com/rsyslog/rsyslog-doc/blob/master/source/configuration/modules/imtcp.rst contributions are always very welcome. Rainer > Anyway the InputTCPMaxSessions should never be equal nor larger than > MaxOpenFiles. And this could be somehow taken care of at least. > > On Tue, Dec 9, 2014 at 12:33 AM, David Lang wrote: > > > On Mon, 8 Dec 2014, Peter Viskup wrote: > > > > Hi all, > >> we observed issues once setting InputTCPMaxSessions above the default > >> MaxOpenFiles process limits. > >> The rsyslog reached MaxOpenFiles limit and just crashed. > >> Setting the value of InputTCPMaxSessions should increase MaxOpenFiles or > >> rsyslog should not start or complain about it at least. There are more > >> configuration parameters which need some additional pre-start > >> checks/modifications (MaxListeners, ...). > >> > > > > It's not quite that simple. > > > > I agree that it would be useful to detect that maxsessions is > > > maxopenfiles, but if it is, figuring out what maxopenfiles needs to be > > increased to requires knowing a lot more about what's going on. How many > > files is rsyslog going to be writing to? and do you have dynafiles > > configured which makes this variable based on the contents of the log > > messages? Are you using any modules that will require filehandles? if so, > > how many will they need. > > > > I think the best you can do is to complain and tell the admin that it > > looks bad. > > > > David Lang > > ___ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > ___ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
[rsyslog] mmnormalize improvements
Hi all, I have now merged Janmejay's awesome mmnormalize improvements. It is currently available in the master-mmnormalize branch. It would be great if some folks could try out that branch and provide feedack. Note that in order to compile, the git master version of liblognorm is required. Right now, one of the new testbench tests fails for me. If we can get this fixed, and nobody reports bad things, I plan to merge it around Friday into the master-candidate branch, from where it is expected to migrate into master early next week. Thanks, Rainer ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
[rsyslog] Stuck omfwd connections
Hi, I have a pair of Linux/RHEL servers (RHEL 6.x), A and B, that forward logs to multiple destinations: - one copy to Splunk syslog listener - one copy to local flume process over TCP - one copy to a remote RSyslog receiver, X and Y (RHEL 6.x) Forwarding copies to Splunk and Flume works fine. However, forwarding to the remote Syslog receivers gets stuck in a strange way. The forwarding is setup as: RSyslog-Server-A -> RSyslog-Server-X RSyslog-Server-B -> RSyslog-Server-Y All four - A,B, X and Y are running exactly the same version of RSyslog - 8.6.2-2, from the adiscon repo: rsyslog-8.6.0-2.el6.x86_64 What happens is A/B stop sending logs to X/Y. Looking at the send/receive TCP queues at both ends, the receive queue on X/Y is clear but the sendQ on A/B gets stuck. As an example, this connection lingers forever (extracted with netstat -an | grep EST): tcp0 103660 10.24.62.9:47081 10.2.1.2:514 ESTABLISHED Observations: == - The connection remains established with the same number of bytes in the sendQ - No data is transferred over the "stuck" connection, looking at tcpdump - Re-starting the receive end, X/Y, does not help - I don't see an action suspended error in the rsyslog logs - Running the send side in debug doesn't help - I easily ended up with 100+ Gigs of debug logs without the issue manifesting itself. The A/B pair handle lots of traffic and running rsyslogd in debug mode reduces their throughput - perhaps the issue does not manifest at lower EPS. - Only re-starting the send side, A/B, resolves the issue. I tweaked omfwd action to change TCP_Framing from default to octet-based. Here is the send side omfwd config on A/B: action (name="it_tcp_X" type="omfwd" Target="X.abc.com" Port="514" Protocol="tcp" TCP_Framing="octet-counted" queue.filename="it_tcp_X" queue.maxdiskspace="10G" queue.Size="864" queue.dequeuebatchsize="4096" queue.type="LinkedList" queue.timeoutenqueue="0" queue.maxfilesize="1G" queue.saveonshutdown="on" queue.workerThreads="4" RebindInterval="1000" template="fwdformat" ) The receive side, X/Y, config: module(load="imptcp" threads="16") # needs to be done just once global ( workdirectory="/data/rsyslog/queues" maxmessagesize="64K" debug.logfile="/data/rsyslog/debug/debug.log" net.enabledns="off" ) $DebugLevel 0 main_queue ( queue.FileName="globalqueue" queue.Type="LinkedList" queue.MaxDiskSpace="250g" queue.maxfilesize="5g" queue.Size="86400" queue.dequeuebatchsize="1000" queue.TimeoutEnqueue="0" queue.workerThreads="4" queue.SaveOnShutdown="on" ) ruleset(name="aggregate") { action (name="to_flume" type="omfwd" Target="localhost" Port="5614" Protocol="tcp" queue.filename="to_flume" queue.size="36000" queue.maxdiskspace="360G" queue.highwatermark="21600" # 60% of queue.size queue.discardmark="28800" # 80% of queue.size queue.type="LinkedList" queue.dequeuebatchsize="4096" queue.timeoutenqueue="0" queue.maxfilesize="4G" queue.saveonshutdown="on" queue.workerThreads="4" RebindInterval="1000" template="rawfwd" ) stop } input(type="imptcp" port="514" ruleset="aggregate") Any pointers to troubleshoot and smoke out the bug will be highly appreciated :) Thanks ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Stuck omfwd connections
As I was typing out the email, it occurred to me that the issue is OS related: Looking at a sending server, A, I saw these messages in dmesg: TCP: Peer 10.2.1.2:514/47081 unexpectedly shrunk window 861404336:861405796 (repaired) The local TCP port, 47081 is the same one that is part of the stuck connection. Now, I know what the problem is :) However, cannot seem to find a fix :( On Wed, Dec 10, 2014 at 8:46 PM, Tim Smith wrote: > Hi, > > I have a pair of Linux/RHEL servers (RHEL 6.x), A and B, that forward logs > to multiple destinations: > - one copy to Splunk syslog listener > - one copy to local flume process over TCP > - one copy to a remote RSyslog receiver, X and Y (RHEL 6.x) > > Forwarding copies to Splunk and Flume works fine. However, forwarding to > the remote Syslog receivers gets stuck in a strange way. The forwarding is > setup as: > RSyslog-Server-A -> RSyslog-Server-X > RSyslog-Server-B -> RSyslog-Server-Y > > All four - A,B, X and Y are running exactly the same version of RSyslog - > 8.6.2-2, from the adiscon repo: > rsyslog-8.6.0-2.el6.x86_64 > > What happens is A/B stop sending logs to X/Y. Looking at the send/receive > TCP queues at both ends, the receive queue on X/Y is clear but the sendQ on > A/B gets stuck. As an example, this connection lingers forever (extracted > with netstat -an | grep EST): > tcp0 103660 10.24.62.9:47081 10.2.1.2:514 > ESTABLISHED > > Observations: > == > - The connection remains established with the same number of bytes in the > sendQ > - No data is transferred over the "stuck" connection, looking at tcpdump > - Re-starting the receive end, X/Y, does not help > - I don't see an action suspended error in the rsyslog logs > - Running the send side in debug doesn't help - I easily ended up with > 100+ Gigs of debug logs without the issue manifesting itself. The A/B pair > handle lots of traffic and running rsyslogd in debug mode reduces their > throughput - perhaps the issue does not manifest at lower EPS. > - Only re-starting the send side, A/B, resolves the issue. > > I tweaked omfwd action to change TCP_Framing from default to octet-based. > Here is the send side omfwd config on A/B: > > action (name="it_tcp_X" type="omfwd" Target="X.abc.com" Port="514" > Protocol="tcp" TCP_Framing="octet-counted" queue.filename="it_tcp_X" > queue.maxdiskspace="10G" queue.Size="864" > queue.dequeuebatchsize="4096" queue.type="LinkedList" > queue.timeoutenqueue="0" queue.maxfilesize="1G" queue.saveonshutdown="on" > queue.workerThreads="4" RebindInterval="1000" template="fwdformat" ) > > > > The receive side, X/Y, config: > > module(load="imptcp" threads="16") # needs to be done just once > > global ( > workdirectory="/data/rsyslog/queues" > maxmessagesize="64K" > debug.logfile="/data/rsyslog/debug/debug.log" > net.enabledns="off" > ) > > $DebugLevel 0 > > main_queue ( > queue.FileName="globalqueue" > queue.Type="LinkedList" > queue.MaxDiskSpace="250g" > queue.maxfilesize="5g" > queue.Size="86400" > queue.dequeuebatchsize="1000" > queue.TimeoutEnqueue="0" > queue.workerThreads="4" > queue.SaveOnShutdown="on" > ) > > ruleset(name="aggregate") { > action (name="to_flume" > type="omfwd" > Target="localhost" > Port="5614" > Protocol="tcp" > queue.filename="to_flume" > queue.size="36000" > queue.maxdiskspace="360G" > queue.highwatermark="21600" # 60% of queue.size > queue.discardmark="28800" # 80% of queue.size > queue.type="LinkedList" > queue.dequeuebatchsize="4096" > queue.timeoutenqueue="0" > queue.maxfilesize="4G" > queue.saveonshutdown="on" > queue.workerThreads="4" > RebindInterval="1000" > template="rawfwd" > ) stop > } > > input(type="imptcp" port="514" ruleset="aggregate") > > > Any pointers to troubleshoot and smoke out the bug will be highly > appreciated :) > > Thanks > > > > ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] mmnormalize improvements
Rainer - do you have a link to a quick summary of the changes? Would love to know about them and haven't been paying attention due to other work priorities. Thanks! Brian On Wed, Dec 10, 2014 at 10:01 AM, Rainer Gerhards wrote: > Hi all, > > I have now merged Janmejay's awesome mmnormalize improvements. It is > currently available in the master-mmnormalize branch. It would be great if > some folks could try out that branch and provide feedack. Note that in > order to compile, the git master version of liblognorm is required. > > Right now, one of the new testbench tests fails for me. If we can get this > fixed, and nobody reports bad things, I plan to merge it around Friday into > the master-candidate branch, from where it is expected to migrate into > master early next week. > > Thanks, > Rainer > ___ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
[rsyslog] How to split syslog from syslog server into separate files?
Quick setup: Have five log sources that send syslog to a syslog server (running rsyslog), which then sends syslog to Splunk server (also running rsyslog) Trying to split syslog coming from syslog server to Splunk into separate files. Here's the rsyslog on Splunk server: # send all messages from XXX Manager and YYY to a specific files if $hostname contains 'xxx' then /syslog/xxx/%$YEAR%/%$MONTH%/xxx-%$YEAR%%$MONTH%%$DAY%.log else /syslog/yyy/%$YEAR%/%$MONTH%/yyy-%$YEAR%%$MONTH%%$DAY%.log & ~ Here's snippets of rsyslog on syslog server: # Provides TCP syslog reception $ModLoad imtcp.so $InputTCPServerBindRuleset remote #$InputTCPServerRun 514 $RuleSet remote $template DynaFile,"/syslog/%HOSTNAME%/%HOSTNAME%" *.* -?DynaFile # Forwarding rule for remote to Splunk server *.* @@x.x.x.x:514 $InputUDPServerBindRuleset remote $UDPServerRun 514 My question lies in the syntax in Splunk's rsyslog. Is the if|then|else statement correct? Basically want any message related to xxx going to file for xxx, and any messages from yyy going to file for yyy. Running rsyslogd 5.8.10 on both servers Thx, Jeff ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] How to split syslog from syslog server into separate files?
2014-12-10 19:00 GMT+01:00 Walzer, Jeff R : > Quick setup: Have five log sources that send syslog to a syslog server > (running rsyslog), which then sends syslog to Splunk server (also running > rsyslog) > > Trying to split syslog coming from syslog server to Splunk into separate > files. Here's the rsyslog on Splunk server: > > # send all messages from XXX Manager and YYY to a specific files > if $hostname contains 'xxx' > then /syslog/xxx/%$YEAR%/%$MONTH%/xxx-%$YEAR%%$MONTH%%$DAY%.log > else /syslog/yyy/%$YEAR%/%$MONTH%/yyy-%$YEAR%%$MONTH%%$DAY%.log > & ~ > > Here's snippets of rsyslog on syslog server: > > # Provides TCP syslog reception > $ModLoad imtcp.so > $InputTCPServerBindRuleset remote > #$InputTCPServerRun 514 > > $RuleSet remote > $template DynaFile,"/syslog/%HOSTNAME%/%HOSTNAME%" > *.* -?DynaFile > > # Forwarding rule for remote to Splunk server > *.* @@x.x.x.x:514 > > $InputUDPServerBindRuleset remote > $UDPServerRun 514 > > My question lies in the syntax in Splunk's rsyslog. Is the if|then|else > statement correct? Basically want any message related to xxx going to file > for xxx, and any messages from yyy going to file for yyy. > > Running rsyslogd 5.8.10 on both servers > I think you need at least v7, probably 8 for "else". Make sure you record "syslog.*" messages so that you can see startup error messages (I am sure rsyslog spits out quite a lot of them ;)). HTH Rainer > > Thx, > Jeff > > ___ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
[rsyslog] Safe to upgrade to 8.6.0
Hey all, After seeing all the emails back and forth over 8.6, I thought I'd better ask here. I'm currently on 8.4.2: rsyslog [8.4.2.ad1-0adiscon1precise1] via apt I would be upgrading to: (8.6.0.r1-0adiscon2precise1 RSyslog V8-Stable:12.04/precise [amd64]) I have nothing special in my rsyslog setup save the below: module(load="imfile" PollingInterval="1") # File 1 input(type="imfile" File="/media/backup/bro/conn.log" Tag="bro_conn" StateFile="stat-bro_conn" Severity="info" Facility="local7") local7.* @x.x.x.x:6514 Anyone see any issues with this? Thanks much...just wanting to make sure I don't have a bad evening here ;) James ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Safe to upgrade to 8.6.0
The only problems with 8.6 that we know of are related to running the tests. David Lang On Wed, 10 Dec 2014, James Lay wrote: Hey all, After seeing all the emails back and forth over 8.6, I thought I'd better ask here. I'm currently on 8.4.2: rsyslog [8.4.2.ad1-0adiscon1precise1] via apt I would be upgrading to: (8.6.0.r1-0adiscon2precise1 RSyslog V8-Stable:12.04/precise [amd64]) I have nothing special in my rsyslog setup save the below: module(load="imfile" PollingInterval="1") # File 1 input(type="imfile" File="/media/backup/bro/conn.log" Tag="bro_conn" StateFile="stat-bro_conn" Severity="info" Facility="local7") local7.* @x.x.x.x:6514 Anyone see any issues with this? Thanks much...just wanting to make sure I don't have a bad evening here ;) James ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Safe to upgrade to 8.6.0
On 2014-12-10 04:33 PM, David Lang wrote: The only problems with 8.6 that we know of are related to running the tests. David Lang On Wed, 10 Dec 2014, James Lay wrote: Hey all, After seeing all the emails back and forth over 8.6, I thought I'd better ask here. I'm currently on 8.4.2: rsyslog [8.4.2.ad1-0adiscon1precise1] via apt I would be upgrading to: (8.6.0.r1-0adiscon2precise1 RSyslog V8-Stable:12.04/precise [amd64]) I have nothing special in my rsyslog setup save the below: module(load="imfile" PollingInterval="1") # File 1 input(type="imfile" File="/media/backup/bro/conn.log" Tag="bro_conn" StateFile="stat-bro_conn" Severity="info" Facility="local7") local7.* @x.x.x.x:6514 Anyone see any issues with this? Thanks much...just wanting to make sure I don't have a bad evening here ;) James Awesome...thanks for the quick response David. James ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.