Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

2017-10-24 Thread deoren
On 10/19/2017 6:58 PM, deoren wrote: On 10/19/2017 3:12 PM, Rainer Gerhards wrote: Am 19.10.2017 21:55 schrieb "David Lang" : RELP has it's place, but most of the time I'm willing to loose some logs under rare failure conditions and so haven't bothered to use it. large maxmessagesize leads to

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

2017-10-19 Thread deoren
On 10/19/2017 3:12 PM, Rainer Gerhards wrote: Am 19.10.2017 21:55 schrieb "David Lang" : RELP has it's place, but most of the time I'm willing to loose some logs under rare failure conditions and so haven't bothered to use it. large maxmessagesize leads to wasted memory in rsyslog, but nothing

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

2017-10-19 Thread Rainer Gerhards
Am 19.10.2017 21:55 schrieb "David Lang" : On Thu, 19 Oct 2017, deoren wrote: On 10/18/2017 8:10 PM, David Lang wrote: > >> On Wed, 18 Oct 2017, deoren wrote: >> >> On 10/18/2017 3:15 PM, David Lang wrote: >>> On Wed, 18 Oct 2017, deoren wrote: On 10/18/2017 1:36 PM, David Lang wro

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

2017-10-19 Thread David Lang
On Thu, 19 Oct 2017, deoren wrote: On 10/18/2017 8:10 PM, David Lang wrote: On Wed, 18 Oct 2017, deoren wrote: On 10/18/2017 3:15 PM, David Lang wrote: On Wed, 18 Oct 2017, deoren wrote: On 10/18/2017 1:36 PM, David Lang wrote: On Wed, 18 Oct 2017, deoren wrote: Since the sender and recei

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

2017-10-19 Thread deoren
On 10/18/2017 8:10 PM, David Lang wrote: On Wed, 18 Oct 2017, deoren wrote: On 10/18/2017 3:15 PM, David Lang wrote: On Wed, 18 Oct 2017, deoren wrote: On 10/18/2017 1:36 PM, David Lang wrote: On Wed, 18 Oct 2017, deoren wrote: Since the sender and receiver in this are both the latest versi

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

2017-10-18 Thread David Lang
On Wed, 18 Oct 2017, deoren wrote: On 10/18/2017 3:15 PM, David Lang wrote: On Wed, 18 Oct 2017, deoren wrote: On 10/18/2017 1:36 PM, David Lang wrote: On Wed, 18 Oct 2017, deoren wrote: Since the sender and receiver in this are both the latest versions of rsyslog (with the plan for the set

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

2017-10-18 Thread deoren
On 10/18/2017 3:15 PM, David Lang wrote: On Wed, 18 Oct 2017, deoren wrote: On 10/18/2017 1:36 PM, David Lang wrote: On Wed, 18 Oct 2017, deoren wrote: Since the sender and receiver in this are both the latest versions of rsyslog (with the plan for the setup to remain that way), can I scale

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

2017-10-18 Thread deoren
On 10/18/2017 5:02 PM, deoren wrote: On 10/18/2017 3:22 PM, Rainer Gerhards wrote: The queue errors are bad. Anything else in regard to that queue? After discussing it on this thread, I stopped rsyslog yesterday and moved all content from /var/spool/rsyslog to a different directory, hoping t

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

2017-10-18 Thread deoren
On 10/18/2017 12:34 PM, Rainer Gerhards wrote: 2017-10-18 1:14 GMT+02:00 deoren : On 10/17/2017 3:45 PM, Rainer Gerhards wrote: Errno 11 seems to be EAGAIN, more a status than a warning. The full Debug log may reveal details. Is the debug on demand log file sufficient or should enabling deb

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

2017-10-18 Thread deoren
On 10/18/2017 3:22 PM, Rainer Gerhards wrote: The queue errors are bad. Anything else in regard to that queue? After discussing it on this thread, I stopped rsyslog yesterday and moved all content from /var/spool/rsyslog to a different directory, hoping to have rsyslog come back online with a

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

2017-10-18 Thread Rainer Gerhards
The queue errors are bad. Anything else in regard to that queue? The thread messages are informational and part of the better status indicators. They say that the load was so high an additional worker thread was started... And later shut down as it was no longer needed. Rainer Sent from phone, t

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

2017-10-18 Thread David Lang
On Wed, 18 Oct 2017, deoren wrote: On 10/18/2017 1:36 PM, David Lang wrote: On Wed, 18 Oct 2017, deoren wrote: Since the sender and receiver in this are both the latest versions of rsyslog (with the plan for the setup to remain that way), can I scale the accepted message size values to proper

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

2017-10-18 Thread deoren
On 10/18/2017 1:36 PM, David Lang wrote: On Wed, 18 Oct 2017, deoren wrote: I checked and sawmill1 is having trouble sending the messages on to the "downstream" receivers (sawmill2, sawmill3). Based on the "... at least 232 byte larger than max msg size ..." log entries, I'd guess that mess

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

2017-10-18 Thread David Lang
On Wed, 18 Oct 2017, deoren wrote: I checked and sawmill1 is having trouble sending the messages on to the "downstream" receivers (sawmill2, sawmill3). Based on the "... at least 232 byte larger than max msg size ..." log entries, I'd guess that message size is causing the issue here. I'm

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

2017-10-18 Thread Rainer Gerhards
2017-10-18 1:14 GMT+02:00 deoren : > On 10/17/2017 3:45 PM, Rainer Gerhards wrote: >> >> Errno 11 seems to be EAGAIN, more a status than a warning. The full Debug >> log may reveal details. > > > Is the debug on demand log file sufficient or should enabling debug mode at > startup the better route?

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

2017-10-18 Thread deoren
On 10/18/2017 12:02 PM, deoren wrote: On 10/18/2017 11:51 AM, deoren wrote: On 10/17/2017 6:57 PM, David Lang wrote: Yes, rsyslog will accept messages it can't deliver, the accepting of messages is decoupled from the delivery. if a message is too long, it will get ttruncated, even if it's jso

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

2017-10-18 Thread deoren
On 10/18/2017 11:51 AM, deoren wrote: On 10/17/2017 6:57 PM, David Lang wrote: Yes, rsyslog will accept messages it can't deliver, the accepting of messages is decoupled from the delivery. if a message is too long, it will get ttruncated, even if it's json (at that point it's a string of byte

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

2017-10-18 Thread deoren
On 10/17/2017 6:57 PM, David Lang wrote: Yes, rsyslog will accept messages it can't deliver, the accepting of messages is decoupled from the delivery. if a message is too long, it will get ttruncated, even if it's json (at that point it's a string of bytes, rsyslog has no way of knowing that

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

2017-10-17 Thread David Lang
Yes, rsyslog will accept messages it can't deliver, the accepting of messages is decoupled from the delivery. if a message is too long, it will get ttruncated, even if it's json (at that point it's a string of bytes, rsyslog has no way of knowing that it's json) David Lang ___

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

2017-10-17 Thread deoren
On 10/17/2017 3:45 PM, Rainer Gerhards wrote: Errno 11 seems to be EAGAIN, more a status than a warning. The full Debug log may reveal details. Is the debug on demand log file sufficient or should enabling debug mode at startup the better route? On a different note, will rsyslog accept a mes

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

2017-10-17 Thread deoren
On 10/17/2017 3:36 PM, David Lang wrote: you can copy the queue files somewhere else (best done with rsyslog stopped), and then configure a copy of rsyslog,conf to not have any inputs, but have the queue files and the rules for what to do with them. You can then run a second copy of rsyslog (di

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

2017-10-17 Thread Rainer Gerhards
-0500 >> From: deoren >> Reply-To: rsyslog-users >> To: rsyslog-users >> Subject: [rsyslog] If messages are stuck in a queue, >> do you have any option other than nuking the queue file(s)? >> >> Refs: >> >> https://github.com/rsyslog/rs

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

2017-10-17 Thread David Lang
ly-To: rsyslog-users To: rsyslog-users Subject: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)? Refs: https://github.com/rsyslog/rsyslog/issues/1782 Scenario: * rsyslog v8.30.0 (Ubuntu PPA) * Ubuntu 16.04 * rsyslog sender setup t

[rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

2017-10-17 Thread deoren
Refs: https://github.com/rsyslog/rsyslog/issues/1782 Scenario: * rsyslog v8.30.0 (Ubuntu PPA) * Ubuntu 16.04 * rsyslog sender setup to forward via omrelp (with a DA queue) to a remote receiver * nearly 1 GB of held message content in /var/spool/rsyslog There are 1272152 messages currently