Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx
yes, as said reject_unknown_client_hostname is relatively strict, one needs to set an exception now and then if the server is legit in the end... but it's not legit requests more often... https://www.postfix.org/postconf.5.html#reject_unknown_reverse_client_hostname On 21/09/2023 21:06, David Lang wrote: hmm, dlang@dlang-mobile:~$ nslookup 66.167.227.145 8.8.8.8 145.227.167.66.in-addr.arpa name = mail.lang.hm. Authoritative answers can be found from: dlang@dlang-mobile:~$ nslookup mail.lang.hm 8.8.8.8 Server: 8.8.8.8 Address: 8.8.8.8#53 Non-authoritative answer: Name: mail.lang.hm Address: 66.167.227.134 On Thu, 21 Sep 2023, TG Servers wrote: this is a standard reject_unknown_client_hostname from postfix, this is relatively strict, but ususally no problem on clean servers, I have only a handful exceptions this is not the IP you sent from...the mail came from 66.167.227.145 cannot find your hostname, [66.167.227.145]; from= to= proto=ESMTP helo= Thomas On 21/09/2023 11:00, David Lang wrote: On Thu, 21 Sep 2023, TG Servers wrote: I did not get a single message from you David regarding that, that confused me quite a bit as Rainer mentioned you already before, now I know why : 450 4.7.25 Client host rejected: cannot find your hostname, [66.167.xxx.xxx]; from= to= proto=ESMTP helo= made an exception now hmm, I haven't made any DNS changes or IP changes for a couple of years (since the last time my ISP broke reverse DNS) lookups against google DNS servers seem to work dlang@dlang-mobile:~$ nslookup mail.lang.hm 8.8.8.8 Server: 8.8.8.8 Address: 8.8.8.8#53 Non-authoritative answer: Name: mail.lang.hm Address: 66.167.227.134 dlang@dlang-mobile:~$ nslookup 66.167.227.134 134.227.167.66.in-addr.arpa name = gw.lang.hm. Authoritative answers can be found from: what is the check that you are doing? I'm not running into problems with anyone else that I know of. David Lang ___ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx
yes that was in the first message, I reimplemented everything, it is
working without any issues now. that was not the problem btw. that was
not even there at the time of the first message, this is actually the
reimplementatiom I did on the rsyslog side. as said, it works without
any issues now. but thanks
On 21/09/2023 21:09, Joan Sala wrote:
(Disclaimer: I have not read all this thread in depth.)
This flag "confirmMessages=on" sounds suspicious. With this flag,
omprog waits for the script to confirm each received log line (if I
recall well), but your script (looking at your first message) doesn't
seem to do this (?) (to be sure we would need to see the python part).
This could cause the queue to stall...
On Thu, Sep 21, 2023, 11:26 TG Servers via rsyslog
wrote:
I don't think we are talking same things anymore.
I told you several times journald is not involved in this. you cannot
find 1 line of these logs in journald, so I am not using journald as
queue, because I am not using journald at all for this process.
"you did not configure rsyslog to use a separate queue for the
logs from
this socket..."?
I told you that I implemented a separate queue... you tell me I didn't
what are you talking about? about the implementation I sent in my
original message? Yes at that point I didn't.
But I wrote several mails in the meanwhile and wrote that I
completely
reimplemented this, with a dedicated socket, with a dedicated queue.
The socket does nothing else than receive messages with tag "app".
And
these messages do not ever touch journald.
This is a separate queue, isn't it?
if $programname == "app" then {
action(type="omprog"
name="app_log"
binary="/usr/local/script/app_log.sh"
template="app"
confirmMessages="on"
confirmTimeout="5000"
queue.type="LinkedList"
queue.size="1"
closeTimeout="1"
queue.workerThreads="2"
action.resumeInterval="5"
killUnresponsive="on"
)
also that nginx had no access to the socket after a rsyslog
restart had
nothing to do with a full queue, the socket was simply not listening.
since it is now handled by systemd this is not a problem anymore, too
On 21/09/2023 10:55, David Lang wrote:
> if you are sending logs to journald and having journald send
logs to
> syslog, you are using journald as a queue for the delivery
>
> when you were delivering directly to rsyslog, what was probably
> happening (we don't know because you never enabled impstats to
see) is
> that the logs were arriving, but because your script takes so
long to
> process each log message, the queue was filling up, and when the
queue
> is full, rsyslog cannot accept another message, and that results in
> the error that you are reporting.
>
> you did not configure rsyslog to use a separate queue for the logs
> from this socket, so as they arrived they got added to the main
queue
> along with all other logs.
>
> rsyslog has options to tell it to throw away logs when it's too
busy
> (and even can prioritize which ones it throws away). you can also
> configure it to write logs to disk when the memory queue gets too
> full. But eventually you will run out of disk space if you keep
> getting logs faster than you can process them.
>
> David Lang
>
>
>
> On Thu, 21 Sep 2023, TG Servers wrote:
>
>> I did not get a single message from you David regarding that,
that
>> confused me quite a bit as Rainer mentioned you already before,
now I
>> know why :
>> 450 4.7.25 Client host rejected: cannot find your hostname,
>> [66.167.xxx.xxx]; from= to=
>> proto=ESMTP helo=http://mail.lang.hm>>
>> made an exception now
>>
>> But to the point, I sadly don't get what both of you are
telling me
>> now. This has nothing to do with journald. It just did not work
when
>> the socket was created by rsyslog. If this is/was a rsyslog or a
>> nginx problem does not matter in the end to me, as this had to
be fixed.
>>
>> I am using a dedicated socket, completely aside from sysSock,
and a
>> dedicated queue. sysSock ist not involved, nginx does not even
log a
>> single line to journald, so how should journald act as a queue
here,
>> or being negatively affected when it does not even receive a
single
>> message of the process involved? It can't throw something away it
>> does not have.
>> This socket is only used by and for this process and nothing else.
>> I also won't likely run out of queue space because this is not a
>> process that is 24/7 under a full load scenario. That might happen
>> under an
Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx
now we are drifting away completely I am afraid. Please don't talk to me
like I am slow, I know what journald is
"when you say you aren't using journald"... I don't, for this use case I
don't
"and are just sending the logs to systemd"...where do I send a log to
systemd? Or say I send a log to systemd?
journald is systemd's logging system, yes. and?
I told you now several times that this process doesn't do it's logging
via journald
journald is doing its thing, and I let it, I don't care, journald writes
to /run/systemd/journal/syslog -> SysSock
nginx access log is NOT logging to journald, the nginx service unit
output itself, yes, but I don't care about that, the access logfile
content is written directly to a complete different socket in /run...and
this socket is a direct input socket for rsyslog...and I said this
socket is handled by systemd now, in terms of creation and socket
handling...I said nothing else...not in terms of logging...
this socket in /run is used only for this one log process, and for
nothing else...
yea it is going into the main queue, and then it is going into the
output queue where it is staying until it is handled...
Thomas
On 21/09/2023 21:05, David Lang wrote:
That is a queue on the output, but the incoming message still goes to
the main queue.
create a ruleset for the input and put a queue on that ruleset to
avoid the message going into the main queue.
when you say you aren't useing journald, and are just sending the logs
to systemd, you aren't understanding what you are doing. The systemd
logging system is named journald
David Lang
On Thu, 21 Sep 2023, TG Servers wrote:
Date: Thu, 21 Sep 2023 11:26:50 +0200
From: TG Servers
To: David Lang
Cc: Rainer Gerhards ,
TG Servers via rsyslog
Subject: Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx
I don't think we are talking same things anymore.
I told you several times journald is not involved in this. you cannot
find 1 line of these logs in journald, so I am not using journald as
queue, because I am not using journald at all for this process.
"you did not configure rsyslog to use a separate queue for the logs
from this socket..."?
I told you that I implemented a separate queue... you tell me I didn't
what are you talking about? about the implementation I sent in my
original message? Yes at that point I didn't.
But I wrote several mails in the meanwhile and wrote that I
completely reimplemented this, with a dedicated socket, with a
dedicated queue.
The socket does nothing else than receive messages with tag "app".
And these messages do not ever touch journald.
This is a separate queue, isn't it?
if $programname == "app" then {
action(type="omprog"
name="app_log"
binary="/usr/local/script/app_log.sh"
template="app"
confirmMessages="on"
confirmTimeout="5000"
queue.type="LinkedList"
queue.size="1"
closeTimeout="1"
queue.workerThreads="2"
action.resumeInterval="5"
killUnresponsive="on"
)
also that nginx had no access to the socket after a rsyslog restart
had nothing to do with a full queue, the socket was simply not
listening. since it is now handled by systemd this is not a problem
anymore, too
On 21/09/2023 10:55, David Lang wrote:
if you are sending logs to journald and having journald send logs to
syslog, you are using journald as a queue for the delivery
when you were delivering directly to rsyslog, what was probably
happening (we don't know because you never enabled impstats to see)
is that the logs were arriving, but because your script takes so
long to process each log message, the queue was filling up, and when
the queue is full, rsyslog cannot accept another message, and that
results in the error that you are reporting.
you did not configure rsyslog to use a separate queue for the logs
from this socket, so as they arrived they got added to the main
queue along with all other logs.
rsyslog has options to tell it to throw away logs when it's too busy
(and even can prioritize which ones it throws away). you can also
configure it to write logs to disk when the memory queue gets too
full. But eventually you will run out of disk space if you keep
getting logs faster than you can process them.
David Lang
On Thu, 21 Sep 2023, TG Servers wrote:
I did not get a single message from you David regarding that, that
confused me quite a bit as Rainer mentioned you already before, now
I know why :
450 4.7.25 Client host rejected: cannot find your hostname,
[66.167.xxx.xxx]; from= to=
proto=ESMTP helo=
made an exception now
But to the point, I sadly don't get what both of you are telling me
now. This has nothing to do with journald. It just did not wo
Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx
(Disclaimer: I have not read all this thread in depth.)
This flag "confirmMessages=on" sounds suspicious. With this flag, omprog
waits for the script to confirm each received log line (if I recall well),
but your script (looking at your first message) doesn't seem to do this (?)
(to be sure we would need to see the python part). This could cause the
queue to stall...
On Thu, Sep 21, 2023, 11:26 TG Servers via rsyslog <
rsyslog@lists.adiscon.com> wrote:
> I don't think we are talking same things anymore.
> I told you several times journald is not involved in this. you cannot
> find 1 line of these logs in journald, so I am not using journald as
> queue, because I am not using journald at all for this process.
>
> "you did not configure rsyslog to use a separate queue for the logs from
> this socket..."?
> I told you that I implemented a separate queue... you tell me I didn't
>
> what are you talking about? about the implementation I sent in my
> original message? Yes at that point I didn't.
> But I wrote several mails in the meanwhile and wrote that I completely
> reimplemented this, with a dedicated socket, with a dedicated queue.
>
> The socket does nothing else than receive messages with tag "app". And
> these messages do not ever touch journald.
>
> This is a separate queue, isn't it?
>
> if $programname == "app" then {
> action(type="omprog"
>name="app_log"
>binary="/usr/local/script/app_log.sh"
>template="app"
>confirmMessages="on"
>confirmTimeout="5000"
>queue.type="LinkedList"
>queue.size="1"
>closeTimeout="1"
>queue.workerThreads="2"
>action.resumeInterval="5"
>killUnresponsive="on"
>)
>
> also that nginx had no access to the socket after a rsyslog restart had
> nothing to do with a full queue, the socket was simply not listening.
> since it is now handled by systemd this is not a problem anymore, too
>
>
> On 21/09/2023 10:55, David Lang wrote:
> > if you are sending logs to journald and having journald send logs to
> > syslog, you are using journald as a queue for the delivery
> >
> > when you were delivering directly to rsyslog, what was probably
> > happening (we don't know because you never enabled impstats to see) is
> > that the logs were arriving, but because your script takes so long to
> > process each log message, the queue was filling up, and when the queue
> > is full, rsyslog cannot accept another message, and that results in
> > the error that you are reporting.
> >
> > you did not configure rsyslog to use a separate queue for the logs
> > from this socket, so as they arrived they got added to the main queue
> > along with all other logs.
> >
> > rsyslog has options to tell it to throw away logs when it's too busy
> > (and even can prioritize which ones it throws away). you can also
> > configure it to write logs to disk when the memory queue gets too
> > full. But eventually you will run out of disk space if you keep
> > getting logs faster than you can process them.
> >
> > David Lang
> >
> >
> >
> > On Thu, 21 Sep 2023, TG Servers wrote:
> >
> >> I did not get a single message from you David regarding that, that
> >> confused me quite a bit as Rainer mentioned you already before, now I
> >> know why :
> >> 450 4.7.25 Client host rejected: cannot find your hostname,
> >> [66.167.xxx.xxx]; from= to=
> >> proto=ESMTP helo=
> >> made an exception now
> >>
> >> But to the point, I sadly don't get what both of you are telling me
> >> now. This has nothing to do with journald. It just did not work when
> >> the socket was created by rsyslog. If this is/was a rsyslog or a
> >> nginx problem does not matter in the end to me, as this had to be fixed.
> >>
> >> I am using a dedicated socket, completely aside from sysSock, and a
> >> dedicated queue. sysSock ist not involved, nginx does not even log a
> >> single line to journald, so how should journald act as a queue here,
> >> or being negatively affected when it does not even receive a single
> >> message of the process involved? It can't throw something away it
> >> does not have.
> >> This socket is only used by and for this process and nothing else.
> >> I also won't likely run out of queue space because this is not a
> >> process that is 24/7 under a full load scenario. That might happen
> >> under an attack maybe, but otherwise I don't see that happening
> >> If I am seeing things wrong then I would be happy if it could be made
> >> clear to me because as of now I do not see the problem.
> >>
> >> Thanks,
> >> Thomas
> >>
> >>
> >> On 21/09/2023 08:34, Rainer Gerhards wrote:
> >>> I guess it works because journal always throws messages away if it
> >>> cannot deliver them quickly. Luke a very short timeout+drop queue
> >>> config in rsyslog.
> >>>
> >>> Rainer
> >>>
> >>> Sent from phone, thus brief.
> >>>
> >>> David Lang schrieb am Do., 21. Sept. 2023, 08:23:
> >>>
> >>>
Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx
hmm, dlang@dlang-mobile:~$ nslookup 66.167.227.145 8.8.8.8 145.227.167.66.in-addr.arpa name = mail.lang.hm. Authoritative answers can be found from: dlang@dlang-mobile:~$ nslookup mail.lang.hm 8.8.8.8 Server: 8.8.8.8 Address:8.8.8.8#53 Non-authoritative answer: Name: mail.lang.hm Address: 66.167.227.134 On Thu, 21 Sep 2023, TG Servers wrote: this is a standard reject_unknown_client_hostname from postfix, this is relatively strict, but ususally no problem on clean servers, I have only a handful exceptions this is not the IP you sent from...the mail came from 66.167.227.145 cannot find your hostname, [66.167.227.145]; from= to= proto=ESMTP helo= Thomas On 21/09/2023 11:00, David Lang wrote: On Thu, 21 Sep 2023, TG Servers wrote: I did not get a single message from you David regarding that, that confused me quite a bit as Rainer mentioned you already before, now I know why : 450 4.7.25 Client host rejected: cannot find your hostname, [66.167.xxx.xxx]; from= to= proto=ESMTP helo= made an exception now hmm, I haven't made any DNS changes or IP changes for a couple of years (since the last time my ISP broke reverse DNS) lookups against google DNS servers seem to work dlang@dlang-mobile:~$ nslookup mail.lang.hm 8.8.8.8 Server: 8.8.8.8 Address: 8.8.8.8#53 Non-authoritative answer: Name: mail.lang.hm Address: 66.167.227.134 dlang@dlang-mobile:~$ nslookup 66.167.227.134 134.227.167.66.in-addr.arpa name = gw.lang.hm. Authoritative answers can be found from: what is the check that you are doing? I'm not running into problems with anyone else that I know of. David Lang ___ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx
That is a queue on the output, but the incoming message still goes to the main
queue.
create a ruleset for the input and put a queue on that ruleset to avoid the
message going into the main queue.
when you say you aren't useing journald, and are just sending the logs to
systemd, you aren't understanding what you are doing. The systemd logging system
is named journald
David Lang
On Thu, 21 Sep 2023, TG Servers wrote:
Date: Thu, 21 Sep 2023 11:26:50 +0200
From: TG Servers
To: David Lang
Cc: Rainer Gerhards ,
TG Servers via rsyslog
Subject: Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx
I don't think we are talking same things anymore.
I told you several times journald is not involved in this. you cannot find 1
line of these logs in journald, so I am not using journald as queue, because
I am not using journald at all for this process.
"you did not configure rsyslog to use a separate queue for the logs from this
socket..."?
I told you that I implemented a separate queue... you tell me I didn't
what are you talking about? about the implementation I sent in my original
message? Yes at that point I didn't.
But I wrote several mails in the meanwhile and wrote that I completely
reimplemented this, with a dedicated socket, with a dedicated queue.
The socket does nothing else than receive messages with tag "app". And these
messages do not ever touch journald.
This is a separate queue, isn't it?
if $programname == "app" then {
action(type="omprog"
name="app_log"
binary="/usr/local/script/app_log.sh"
template="app"
confirmMessages="on"
confirmTimeout="5000"
queue.type="LinkedList"
queue.size="1"
closeTimeout="1"
queue.workerThreads="2"
action.resumeInterval="5"
killUnresponsive="on"
)
also that nginx had no access to the socket after a rsyslog restart had
nothing to do with a full queue, the socket was simply not listening. since
it is now handled by systemd this is not a problem anymore, too
On 21/09/2023 10:55, David Lang wrote:
if you are sending logs to journald and having journald send logs to
syslog, you are using journald as a queue for the delivery
when you were delivering directly to rsyslog, what was probably happening
(we don't know because you never enabled impstats to see) is that the logs
were arriving, but because your script takes so long to process each log
message, the queue was filling up, and when the queue is full, rsyslog
cannot accept another message, and that results in the error that you are
reporting.
you did not configure rsyslog to use a separate queue for the logs from
this socket, so as they arrived they got added to the main queue along with
all other logs.
rsyslog has options to tell it to throw away logs when it's too busy (and
even can prioritize which ones it throws away). you can also configure it
to write logs to disk when the memory queue gets too full. But eventually
you will run out of disk space if you keep getting logs faster than you can
process them.
David Lang
On Thu, 21 Sep 2023, TG Servers wrote:
I did not get a single message from you David regarding that, that
confused me quite a bit as Rainer mentioned you already before, now I know
why :
450 4.7.25 Client host rejected: cannot find your hostname,
[66.167.xxx.xxx]; from= to= proto=ESMTP
helo=
made an exception now
But to the point, I sadly don't get what both of you are telling me now.
This has nothing to do with journald. It just did not work when the socket
was created by rsyslog. If this is/was a rsyslog or a nginx problem does
not matter in the end to me, as this had to be fixed.
I am using a dedicated socket, completely aside from sysSock, and a
dedicated queue. sysSock ist not involved, nginx does not even log a
single line to journald, so how should journald act as a queue here, or
being negatively affected when it does not even receive a single message
of the process involved? It can't throw something away it does not have.
This socket is only used by and for this process and nothing else.
I also won't likely run out of queue space because this is not a process
that is 24/7 under a full load scenario. That might happen under an attack
maybe, but otherwise I don't see that happening
If I am seeing things wrong then I would be happy if it could be made
clear to me because as of now I do not see the problem.
Thanks,
Thomas
On 21/09/2023 08:34, Rainer Gerhards wrote:
I guess it works because journal always throws messages away if it cannot
deliver them quickly. Luke a very short timeout+drop queue config in
rsyslog.
Rainer
Sent from phone, thus brief.
David Lang schrieb am Do., 21. Sept. 2023, 08:23:
now you have journald acting as
Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx
I don't think we are talking same things anymore.
I told you several times journald is not involved in this. you cannot
find 1 line of these logs in journald, so I am not using journald as
queue, because I am not using journald at all for this process.
"you did not configure rsyslog to use a separate queue for the logs from
this socket..."?
I told you that I implemented a separate queue... you tell me I didn't
what are you talking about? about the implementation I sent in my
original message? Yes at that point I didn't.
But I wrote several mails in the meanwhile and wrote that I completely
reimplemented this, with a dedicated socket, with a dedicated queue.
The socket does nothing else than receive messages with tag "app". And
these messages do not ever touch journald.
This is a separate queue, isn't it?
if $programname == "app" then {
action(type="omprog"
name="app_log"
binary="/usr/local/script/app_log.sh"
template="app"
confirmMessages="on"
confirmTimeout="5000"
queue.type="LinkedList"
queue.size="1"
closeTimeout="1"
queue.workerThreads="2"
action.resumeInterval="5"
killUnresponsive="on"
)
also that nginx had no access to the socket after a rsyslog restart had
nothing to do with a full queue, the socket was simply not listening.
since it is now handled by systemd this is not a problem anymore, too
On 21/09/2023 10:55, David Lang wrote:
if you are sending logs to journald and having journald send logs to
syslog, you are using journald as a queue for the delivery
when you were delivering directly to rsyslog, what was probably
happening (we don't know because you never enabled impstats to see) is
that the logs were arriving, but because your script takes so long to
process each log message, the queue was filling up, and when the queue
is full, rsyslog cannot accept another message, and that results in
the error that you are reporting.
you did not configure rsyslog to use a separate queue for the logs
from this socket, so as they arrived they got added to the main queue
along with all other logs.
rsyslog has options to tell it to throw away logs when it's too busy
(and even can prioritize which ones it throws away). you can also
configure it to write logs to disk when the memory queue gets too
full. But eventually you will run out of disk space if you keep
getting logs faster than you can process them.
David Lang
On Thu, 21 Sep 2023, TG Servers wrote:
I did not get a single message from you David regarding that, that
confused me quite a bit as Rainer mentioned you already before, now I
know why :
450 4.7.25 Client host rejected: cannot find your hostname,
[66.167.xxx.xxx]; from= to=
proto=ESMTP helo=
made an exception now
But to the point, I sadly don't get what both of you are telling me
now. This has nothing to do with journald. It just did not work when
the socket was created by rsyslog. If this is/was a rsyslog or a
nginx problem does not matter in the end to me, as this had to be fixed.
I am using a dedicated socket, completely aside from sysSock, and a
dedicated queue. sysSock ist not involved, nginx does not even log a
single line to journald, so how should journald act as a queue here,
or being negatively affected when it does not even receive a single
message of the process involved? It can't throw something away it
does not have.
This socket is only used by and for this process and nothing else.
I also won't likely run out of queue space because this is not a
process that is 24/7 under a full load scenario. That might happen
under an attack maybe, but otherwise I don't see that happening
If I am seeing things wrong then I would be happy if it could be made
clear to me because as of now I do not see the problem.
Thanks,
Thomas
On 21/09/2023 08:34, Rainer Gerhards wrote:
I guess it works because journal always throws messages away if it
cannot deliver them quickly. Luke a very short timeout+drop queue
config in rsyslog.
Rainer
Sent from phone, thus brief.
David Lang schrieb am Do., 21. Sept. 2023, 08:23:
now you have journald acting as a queue, so all messages from
journald will end
up delayed when your script cannot keep up. You haven't solved the
problem of
the slow script, you've just added another layer of buffer to fill
up before you
notice.
with rsyslog you can set the queue size to whatever you want, and
you can spill
logs to disk when your queue fills up.
but no matter what you do, if you have something that is
processing logs slower
than they are being generated, eventually you will run out of
queue space (in
memory or on disk) and have to stop accepting new messages, or
start throwing
away messages you haven't processed yet
David Lang
On Thu, 21 Sep 2023, TG Servers via rsyslog wrote:
> the only way
Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx
this is a standard reject_unknown_client_hostname from postfix, this is relatively strict, but ususally no problem on clean servers, I have only a handful exceptions this is not the IP you sent from...the mail came from 66.167.227.145 cannot find your hostname, [66.167.227.145]; from= to= proto=ESMTP helo= Thomas On 21/09/2023 11:00, David Lang wrote: On Thu, 21 Sep 2023, TG Servers wrote: I did not get a single message from you David regarding that, that confused me quite a bit as Rainer mentioned you already before, now I know why : 450 4.7.25 Client host rejected: cannot find your hostname, [66.167.xxx.xxx]; from= to= proto=ESMTP helo= made an exception now hmm, I haven't made any DNS changes or IP changes for a couple of years (since the last time my ISP broke reverse DNS) lookups against google DNS servers seem to work dlang@dlang-mobile:~$ nslookup mail.lang.hm 8.8.8.8 Server: 8.8.8.8 Address: 8.8.8.8#53 Non-authoritative answer: Name: mail.lang.hm Address: 66.167.227.134 dlang@dlang-mobile:~$ nslookup 66.167.227.134 134.227.167.66.in-addr.arpa name = gw.lang.hm. Authoritative answers can be found from: what is the check that you are doing? I'm not running into problems with anyone else that I know of. David Lang ___ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx
___ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx
On Thu, 21 Sep 2023, TG Servers wrote: I did not get a single message from you David regarding that, that confused me quite a bit as Rainer mentioned you already before, now I know why : 450 4.7.25 Client host rejected: cannot find your hostname, [66.167.xxx.xxx]; from= to= proto=ESMTP helo= made an exception now hmm, I haven't made any DNS changes or IP changes for a couple of years (since the last time my ISP broke reverse DNS) lookups against google DNS servers seem to work dlang@dlang-mobile:~$ nslookup mail.lang.hm 8.8.8.8 Server: 8.8.8.8 Address:8.8.8.8#53 Non-authoritative answer: Name: mail.lang.hm Address: 66.167.227.134 dlang@dlang-mobile:~$ nslookup 66.167.227.134 134.227.167.66.in-addr.arpa name = gw.lang.hm. Authoritative answers can be found from: what is the check that you are doing? I'm not running into problems with anyone else that I know of. David Lang ___ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx
if you are sending logs to journald and having journald send logs to syslog, you are using journald as a queue for the delivery when you were delivering directly to rsyslog, what was probably happening (we don't know because you never enabled impstats to see) is that the logs were arriving, but because your script takes so long to process each log message, the queue was filling up, and when the queue is full, rsyslog cannot accept another message, and that results in the error that you are reporting. you did not configure rsyslog to use a separate queue for the logs from this socket, so as they arrived they got added to the main queue along with all other logs. rsyslog has options to tell it to throw away logs when it's too busy (and even can prioritize which ones it throws away). you can also configure it to write logs to disk when the memory queue gets too full. But eventually you will run out of disk space if you keep getting logs faster than you can process them. David Lang On Thu, 21 Sep 2023, TG Servers wrote: I did not get a single message from you David regarding that, that confused me quite a bit as Rainer mentioned you already before, now I know why : 450 4.7.25 Client host rejected: cannot find your hostname, [66.167.xxx.xxx]; from= to= proto=ESMTP helo= made an exception now But to the point, I sadly don't get what both of you are telling me now. This has nothing to do with journald. It just did not work when the socket was created by rsyslog. If this is/was a rsyslog or a nginx problem does not matter in the end to me, as this had to be fixed. I am using a dedicated socket, completely aside from sysSock, and a dedicated queue. sysSock ist not involved, nginx does not even log a single line to journald, so how should journald act as a queue here, or being negatively affected when it does not even receive a single message of the process involved? It can't throw something away it does not have. This socket is only used by and for this process and nothing else. I also won't likely run out of queue space because this is not a process that is 24/7 under a full load scenario. That might happen under an attack maybe, but otherwise I don't see that happening If I am seeing things wrong then I would be happy if it could be made clear to me because as of now I do not see the problem. Thanks, Thomas On 21/09/2023 08:34, Rainer Gerhards wrote: I guess it works because journal always throws messages away if it cannot deliver them quickly. Luke a very short timeout+drop queue config in rsyslog. Rainer Sent from phone, thus brief. David Lang schrieb am Do., 21. Sept. 2023, 08:23: now you have journald acting as a queue, so all messages from journald will end up delayed when your script cannot keep up. You haven't solved the problem of the slow script, you've just added another layer of buffer to fill up before you notice. with rsyslog you can set the queue size to whatever you want, and you can spill logs to disk when your queue fills up. but no matter what you do, if you have something that is processing logs slower than they are being generated, eventually you will run out of queue space (in memory or on disk) and have to stop accepting new messages, or start throwing away messages you haven't processed yet David Lang On Thu, 21 Sep 2023, TG Servers via rsyslog wrote: > the only way I was able to fix this was to use a dedicated socket > created via systemd and passed via systemd to rsyslog > since then it is working without any issues. > although I implemented a queue, too, this did not fix the problem as > long as the socket was handled by rsyslog itself > so this is "fixed" from my point of view, I know for the future now > > On 18/09/2023 21:53, TG Servers via rsyslog wrote: >> I don't know what this is... I implemented a complete queue solution >> and it occasionally happens when there is no request but one in sight, >> and this one gets a 111 then, nothing in nginx debug log, no error to >> be seen in rsyslog log >> but one thing I realized, after a restart the first log message >> always, reproducable gets a 111 >> the socket is not connected, nor listening, only after the first >> request is logged/or not logged (which is logged with 111 in nginx) >> the socket is connected and listening, so restarting rsyslog via >> systemd does not connect/listen to/on the socket >> >> the rsyslog debug log just tells us this : >> 6289.088037540:main thread : imuxsock.c: imuxsock: Opened UNIX >> socket '/run/logmat' (fd 6). >> >> [root@xxx rsyslog.d]# systemctl restart rsyslog >> [root@xxx rsyslog.d]# ss -x | grep logmat >> [root@xxx rsyslog.d]# lsof /run/logmat >> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME >> rsyslogd
Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx
I did not get a single message from you David regarding that, that confused me quite a bit as Rainer mentioned you already before, now I know why : 450 4.7.25 Client host rejected: cannot find your hostname, [66.167.xxx.xxx]; from= to= proto=ESMTP helo= made an exception now But to the point, I sadly don't get what both of you are telling me now. This has nothing to do with journald. It just did not work when the socket was created by rsyslog. If this is/was a rsyslog or a nginx problem does not matter in the end to me, as this had to be fixed. I am using a dedicated socket, completely aside from sysSock, and a dedicated queue. sysSock ist not involved, nginx does not even log a single line to journald, so how should journald act as a queue here, or being negatively affected when it does not even receive a single message of the process involved? It can't throw something away it does not have. This socket is only used by and for this process and nothing else. I also won't likely run out of queue space because this is not a process that is 24/7 under a full load scenario. That might happen under an attack maybe, but otherwise I don't see that happening If I am seeing things wrong then I would be happy if it could be made clear to me because as of now I do not see the problem. Thanks, Thomas On 21/09/2023 08:34, Rainer Gerhards wrote: I guess it works because journal always throws messages away if it cannot deliver them quickly. Luke a very short timeout+drop queue config in rsyslog. Rainer Sent from phone, thus brief. David Lang schrieb am Do., 21. Sept. 2023, 08:23: now you have journald acting as a queue, so all messages from journald will end up delayed when your script cannot keep up. You haven't solved the problem of the slow script, you've just added another layer of buffer to fill up before you notice. with rsyslog you can set the queue size to whatever you want, and you can spill logs to disk when your queue fills up. but no matter what you do, if you have something that is processing logs slower than they are being generated, eventually you will run out of queue space (in memory or on disk) and have to stop accepting new messages, or start throwing away messages you haven't processed yet David Lang On Thu, 21 Sep 2023, TG Servers via rsyslog wrote: > the only way I was able to fix this was to use a dedicated socket > created via systemd and passed via systemd to rsyslog > since then it is working without any issues. > although I implemented a queue, too, this did not fix the problem as > long as the socket was handled by rsyslog itself > so this is "fixed" from my point of view, I know for the future now > > On 18/09/2023 21:53, TG Servers via rsyslog wrote: >> I don't know what this is... I implemented a complete queue solution >> and it occasionally happens when there is no request but one in sight, >> and this one gets a 111 then, nothing in nginx debug log, no error to >> be seen in rsyslog log >> but one thing I realized, after a restart the first log message >> always, reproducable gets a 111 >> the socket is not connected, nor listening, only after the first >> request is logged/or not logged (which is logged with 111 in nginx) >> the socket is connected and listening, so restarting rsyslog via >> systemd does not connect/listen to/on the socket >> >> the rsyslog debug log just tells us this : >> 6289.088037540:main thread : imuxsock.c: imuxsock: Opened UNIX >> socket '/run/logmat' (fd 6). >> >> [root@xxx rsyslog.d]# systemctl restart rsyslog >> [root@xxx rsyslog.d]# ss -x | grep logmat >> [root@xxx rsyslog.d]# lsof /run/logmat >> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME >> rsyslogd 2097140 root 6u unix 0x 0t0 25300317 >> /run/logmat type=DGRAM (UNCONNECTED) >> >> make a request from browser or curl >> >> [root@xxx rsyslog.d]# lsof /run/logmat >> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME >> rsyslogd 2097140 root 6u unix 0x 0t0 25300317 >> /run/logmat type=DGRAM (CONNECTED) >> [root@xxx rsyslog.d]# ss -x | grep logmat >> u_dgr ESTAB 0 0 /run/logmat 25300317 * 0 >> >> On 18/09/2023 16:34, TG Servers via rsyslog wrote: >>> I just wanted to add that in a further message as it came to my mind. >>> you were faster... >>> the script is definitely "slow", this is what I know for sure as it >>> does quite a lot of processing/analytics in the background, so even >>> if you trigger it from command line it can take half a sec or so >>> I can't change that, it needs to do what it does, I didn't write it >>> though it can handle
Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx
depends on the journald config. It can be configured to queue to disk, with limits on disk size. David Lang On Thu, 21 Sep 2023, Rainer Gerhards wrote: I guess it works because journal always throws messages away if it cannot deliver them quickly. Luke a very short timeout+drop queue config in rsyslog. Rainer Sent from phone, thus brief. David Lang schrieb am Do., 21. Sept. 2023, 08:23: now you have journald acting as a queue, so all messages from journald will end up delayed when your script cannot keep up. You haven't solved the problem of the slow script, you've just added another layer of buffer to fill up before you notice. with rsyslog you can set the queue size to whatever you want, and you can spill logs to disk when your queue fills up. but no matter what you do, if you have something that is processing logs slower than they are being generated, eventually you will run out of queue space (in memory or on disk) and have to stop accepting new messages, or start throwing away messages you haven't processed yet David Lang On Thu, 21 Sep 2023, TG Servers via rsyslog wrote: the only way I was able to fix this was to use a dedicated socket created via systemd and passed via systemd to rsyslog since then it is working without any issues. although I implemented a queue, too, this did not fix the problem as long as the socket was handled by rsyslog itself so this is "fixed" from my point of view, I know for the future now On 18/09/2023 21:53, TG Servers via rsyslog wrote: I don't know what this is... I implemented a complete queue solution and it occasionally happens when there is no request but one in sight, and this one gets a 111 then, nothing in nginx debug log, no error to be seen in rsyslog log but one thing I realized, after a restart the first log message always, reproducable gets a 111 the socket is not connected, nor listening, only after the first request is logged/or not logged (which is logged with 111 in nginx) the socket is connected and listening, so restarting rsyslog via systemd does not connect/listen to/on the socket the rsyslog debug log just tells us this : 6289.088037540:main thread: imuxsock.c: imuxsock: Opened UNIX socket '/run/logmat' (fd 6). [root@xxx rsyslog.d]# systemctl restart rsyslog [root@xxx rsyslog.d]# ss -x | grep logmat [root@xxx rsyslog.d]# lsof /run/logmat COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME rsyslogd 2097140 root6u unix 0x 0t0 25300317 /run/logmat type=DGRAM (UNCONNECTED) make a request from browser or curl [root@xxx rsyslog.d]# lsof /run/logmat COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME rsyslogd 2097140 root6u unix 0x 0t0 25300317 /run/logmat type=DGRAM (CONNECTED) [root@xxx rsyslog.d]# ss -x | grep logmat u_dgr ESTAB 0 0 /run/logmat 25300317* 0 On 18/09/2023 16:34, TG Servers via rsyslog wrote: I just wanted to add that in a further message as it came to my mind. you were faster... the script is definitely "slow", this is what I know for sure as it does quite a lot of processing/analytics in the background, so even if you trigger it from command line it can take half a sec or so I can't change that, it needs to do what it does, I didn't write it though it can handle manual fast F5 triggers in the browser without issue and then it 111s when there are 2 requests incoming... I thought rsyslog might handle that just well via the queue... but then this might eventually really be the issue, and if it is, is there anything to mitigate this from rsyslog side (in terms of own queue for that socket or something in that direction)? ok, will enable impstats, too when I switch back Thanks, Tom On 18/09/2023 16:17, Rainer Gerhards wrote: so far not a single 111 today, I let this run the until late evening, and if there is stil no 111 I will put back the python script in order because right now there are 2 possibilities, I moved the socket as said, and I skipped the script and just appended the message to a file if either of the 2 things are responsible in the end I won't understand it either :) I don't know what the script does. But if it is slow, it may push back to the main queue, making rsyslog unresponsive. This is David's concern. Tomorrow, if you re-enable, you should also enable impstats as David suggested. Rainer ___ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow
Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx
I guess it works because journal always throws messages away if it cannot deliver them quickly. Luke a very short timeout+drop queue config in rsyslog. Rainer Sent from phone, thus brief. David Lang schrieb am Do., 21. Sept. 2023, 08:23: > now you have journald acting as a queue, so all messages from journald > will end > up delayed when your script cannot keep up. You haven't solved the problem > of > the slow script, you've just added another layer of buffer to fill up > before you > notice. > > with rsyslog you can set the queue size to whatever you want, and you can > spill > logs to disk when your queue fills up. > > but no matter what you do, if you have something that is processing logs > slower > than they are being generated, eventually you will run out of queue space > (in > memory or on disk) and have to stop accepting new messages, or start > throwing > away messages you haven't processed yet > > David Lang > > On Thu, 21 Sep 2023, TG Servers via rsyslog wrote: > > > the only way I was able to fix this was to use a dedicated socket > > created via systemd and passed via systemd to rsyslog > > since then it is working without any issues. > > although I implemented a queue, too, this did not fix the problem as > > long as the socket was handled by rsyslog itself > > so this is "fixed" from my point of view, I know for the future now > > > > On 18/09/2023 21:53, TG Servers via rsyslog wrote: > >> I don't know what this is... I implemented a complete queue solution > >> and it occasionally happens when there is no request but one in sight, > >> and this one gets a 111 then, nothing in nginx debug log, no error to > >> be seen in rsyslog log > >> but one thing I realized, after a restart the first log message > >> always, reproducable gets a 111 > >> the socket is not connected, nor listening, only after the first > >> request is logged/or not logged (which is logged with 111 in nginx) > >> the socket is connected and listening, so restarting rsyslog via > >> systemd does not connect/listen to/on the socket > >> > >> the rsyslog debug log just tells us this : > >> 6289.088037540:main thread: imuxsock.c: imuxsock: Opened UNIX > >> socket '/run/logmat' (fd 6). > >> > >> [root@xxx rsyslog.d]# systemctl restart rsyslog > >> [root@xxx rsyslog.d]# ss -x | grep logmat > >> [root@xxx rsyslog.d]# lsof /run/logmat > >> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME > >> rsyslogd 2097140 root6u unix 0x 0t0 25300317 > >> /run/logmat type=DGRAM (UNCONNECTED) > >> > >> make a request from browser or curl > >> > >> [root@xxx rsyslog.d]# lsof /run/logmat > >> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME > >> rsyslogd 2097140 root6u unix 0x 0t0 25300317 > >> /run/logmat type=DGRAM (CONNECTED) > >> [root@xxx rsyslog.d]# ss -x | grep logmat > >> u_dgr ESTAB 0 0 /run/logmat 25300317* 0 > >> > >> On 18/09/2023 16:34, TG Servers via rsyslog wrote: > >>> I just wanted to add that in a further message as it came to my mind. > >>> you were faster... > >>> the script is definitely "slow", this is what I know for sure as it > >>> does quite a lot of processing/analytics in the background, so even > >>> if you trigger it from command line it can take half a sec or so > >>> I can't change that, it needs to do what it does, I didn't write it > >>> though it can handle manual fast F5 triggers in the browser without > >>> issue and then it 111s when there are 2 requests incoming... > >>> I thought rsyslog might handle that just well via the queue... > >>> but then this might eventually really be the issue, and if it is, is > >>> there anything to mitigate this from rsyslog side (in terms of own > >>> queue for that socket or something in that direction)? > >>> ok, will enable impstats, too when I switch back > >>> > >>> Thanks, > >>> Tom > >>> > >>> On 18/09/2023 16:17, Rainer Gerhards wrote: > > so far not a single 111 today, I let this run the until late evening, > > and if there is stil no 111 I will put back the python script in > order > > because right now there are 2 possibilities, I moved the socket as > > said, > > and I skipped the script and just appended the message to a file > > if either of the 2 things are responsible in the end I won't > > understand > > it either :) > I don't know what the script does. But if it is slow, it may push back > to the main queue, making rsyslog unresponsive. > > This is David's concern. Tomorrow, if you re-enable, you should also > enable impstats as David suggested. > > Rainer > >>> > >>> ___ > >>> rsyslog mailing list > >>> https://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com/professional-services/ > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>> NOTE WELL: This is a PUBLIC mailing list, posts are
Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx
now you have journald acting as a queue, so all messages from journald will end up delayed when your script cannot keep up. You haven't solved the problem of the slow script, you've just added another layer of buffer to fill up before you notice. with rsyslog you can set the queue size to whatever you want, and you can spill logs to disk when your queue fills up. but no matter what you do, if you have something that is processing logs slower than they are being generated, eventually you will run out of queue space (in memory or on disk) and have to stop accepting new messages, or start throwing away messages you haven't processed yet David Lang On Thu, 21 Sep 2023, TG Servers via rsyslog wrote: the only way I was able to fix this was to use a dedicated socket created via systemd and passed via systemd to rsyslog since then it is working without any issues. although I implemented a queue, too, this did not fix the problem as long as the socket was handled by rsyslog itself so this is "fixed" from my point of view, I know for the future now On 18/09/2023 21:53, TG Servers via rsyslog wrote: I don't know what this is... I implemented a complete queue solution and it occasionally happens when there is no request but one in sight, and this one gets a 111 then, nothing in nginx debug log, no error to be seen in rsyslog log but one thing I realized, after a restart the first log message always, reproducable gets a 111 the socket is not connected, nor listening, only after the first request is logged/or not logged (which is logged with 111 in nginx) the socket is connected and listening, so restarting rsyslog via systemd does not connect/listen to/on the socket the rsyslog debug log just tells us this : 6289.088037540:main thread : imuxsock.c: imuxsock: Opened UNIX socket '/run/logmat' (fd 6). [root@xxx rsyslog.d]# systemctl restart rsyslog [root@xxx rsyslog.d]# ss -x | grep logmat [root@xxx rsyslog.d]# lsof /run/logmat COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME rsyslogd 2097140 root 6u unix 0x 0t0 25300317 /run/logmat type=DGRAM (UNCONNECTED) make a request from browser or curl [root@xxx rsyslog.d]# lsof /run/logmat COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME rsyslogd 2097140 root 6u unix 0x 0t0 25300317 /run/logmat type=DGRAM (CONNECTED) [root@xxx rsyslog.d]# ss -x | grep logmat u_dgr ESTAB 0 0 /run/logmat 25300317 * 0 On 18/09/2023 16:34, TG Servers via rsyslog wrote: I just wanted to add that in a further message as it came to my mind. you were faster... the script is definitely "slow", this is what I know for sure as it does quite a lot of processing/analytics in the background, so even if you trigger it from command line it can take half a sec or so I can't change that, it needs to do what it does, I didn't write it though it can handle manual fast F5 triggers in the browser without issue and then it 111s when there are 2 requests incoming... I thought rsyslog might handle that just well via the queue... but then this might eventually really be the issue, and if it is, is there anything to mitigate this from rsyslog side (in terms of own queue for that socket or something in that direction)? ok, will enable impstats, too when I switch back Thanks, Tom On 18/09/2023 16:17, Rainer Gerhards wrote: so far not a single 111 today, I let this run the until late evening, and if there is stil no 111 I will put back the python script in order because right now there are 2 possibilities, I moved the socket as said, and I skipped the script and just appended the message to a file if either of the 2 things are responsible in the end I won't understand it either :) I don't know what the script does. But if it is slow, it may push back to the main queue, making rsyslog unresponsive. This is David's concern. Tomorrow, if you re-enable, you should also enable impstats as David suggested. Rainer ___ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with
Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx
the only way I was able to fix this was to use a dedicated socket created via systemd and passed via systemd to rsyslog since then it is working without any issues. although I implemented a queue, too, this did not fix the problem as long as the socket was handled by rsyslog itself so this is "fixed" from my point of view, I know for the future now On 18/09/2023 21:53, TG Servers via rsyslog wrote: I don't know what this is... I implemented a complete queue solution and it occasionally happens when there is no request but one in sight, and this one gets a 111 then, nothing in nginx debug log, no error to be seen in rsyslog log but one thing I realized, after a restart the first log message always, reproducable gets a 111 the socket is not connected, nor listening, only after the first request is logged/or not logged (which is logged with 111 in nginx) the socket is connected and listening, so restarting rsyslog via systemd does not connect/listen to/on the socket the rsyslog debug log just tells us this : 6289.088037540:main thread : imuxsock.c: imuxsock: Opened UNIX socket '/run/logmat' (fd 6). [root@xxx rsyslog.d]# systemctl restart rsyslog [root@xxx rsyslog.d]# ss -x | grep logmat [root@xxx rsyslog.d]# lsof /run/logmat COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME rsyslogd 2097140 root 6u unix 0x 0t0 25300317 /run/logmat type=DGRAM (UNCONNECTED) make a request from browser or curl [root@xxx rsyslog.d]# lsof /run/logmat COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME rsyslogd 2097140 root 6u unix 0x 0t0 25300317 /run/logmat type=DGRAM (CONNECTED) [root@xxx rsyslog.d]# ss -x | grep logmat u_dgr ESTAB 0 0 /run/logmat 25300317 * 0 On 18/09/2023 16:34, TG Servers via rsyslog wrote: I just wanted to add that in a further message as it came to my mind. you were faster... the script is definitely "slow", this is what I know for sure as it does quite a lot of processing/analytics in the background, so even if you trigger it from command line it can take half a sec or so I can't change that, it needs to do what it does, I didn't write it though it can handle manual fast F5 triggers in the browser without issue and then it 111s when there are 2 requests incoming... I thought rsyslog might handle that just well via the queue... but then this might eventually really be the issue, and if it is, is there anything to mitigate this from rsyslog side (in terms of own queue for that socket or something in that direction)? ok, will enable impstats, too when I switch back Thanks, Tom On 18/09/2023 16:17, Rainer Gerhards wrote: so far not a single 111 today, I let this run the until late evening, and if there is stil no 111 I will put back the python script in order because right now there are 2 possibilities, I moved the socket as said, and I skipped the script and just appended the message to a file if either of the 2 things are responsible in the end I won't understand it either :) I don't know what the script does. But if it is slow, it may push back to the main queue, making rsyslog unresponsive. This is David's concern. Tomorrow, if you re-enable, you should also enable impstats as David suggested. Rainer ___ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx
when rsyslog starts up, it generates various log messages, are they being sent to the script? it would really help to see the queue data from impstats David Lang On Mon, 18 Sep 2023, TG Servers via rsyslog wrote: I don't know what this is... I implemented a complete queue solution and it occasionally happens when there is no request but one in sight, and this one gets a 111 then, nothing in nginx debug log, no error to be seen in rsyslog log but one thing I realized, after a restart the first log message always, reproducable gets a 111 the socket is not connected, nor listening, only after the first request is logged/or not logged (which is logged with 111 in nginx) the socket is connected and listening, so restarting rsyslog via systemd does not connect/listen to/on the socket the rsyslog debug log just tells us this : 6289.088037540:main thread : imuxsock.c: imuxsock: Opened UNIX socket '/run/logmat' (fd 6). [root@xxx rsyslog.d]# systemctl restart rsyslog [root@xxx rsyslog.d]# ss -x | grep logmat [root@xxx rsyslog.d]# lsof /run/logmat COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME rsyslogd 2097140 root 6u unix 0x 0t0 25300317 /run/logmat type=DGRAM (UNCONNECTED) make a request from browser or curl [root@xxx rsyslog.d]# lsof /run/logmat COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME rsyslogd 2097140 root 6u unix 0x 0t0 25300317 /run/logmat type=DGRAM (CONNECTED) [root@xxx rsyslog.d]# ss -x | grep logmat u_dgr ESTAB 0 0 /run/logmat 25300317 * 0 On 18/09/2023 16:34, TG Servers via rsyslog wrote: I just wanted to add that in a further message as it came to my mind. you were faster... the script is definitely "slow", this is what I know for sure as it does quite a lot of processing/analytics in the background, so even if you trigger it from command line it can take half a sec or so I can't change that, it needs to do what it does, I didn't write it though it can handle manual fast F5 triggers in the browser without issue and then it 111s when there are 2 requests incoming... I thought rsyslog might handle that just well via the queue... but then this might eventually really be the issue, and if it is, is there anything to mitigate this from rsyslog side (in terms of own queue for that socket or something in that direction)? ok, will enable impstats, too when I switch back Thanks, Tom On 18/09/2023 16:17, Rainer Gerhards wrote: so far not a single 111 today, I let this run the until late evening, and if there is stil no 111 I will put back the python script in order because right now there are 2 possibilities, I moved the socket as said, and I skipped the script and just appended the message to a file if either of the 2 things are responsible in the end I won't understand it either :) I don't know what the script does. But if it is slow, it may push back to the main queue, making rsyslog unresponsive. This is David's concern. Tomorrow, if you re-enable, you should also enable impstats as David suggested. Rainer ___ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx
I don't know what this is... I implemented a complete queue solution and it occasionally happens when there is no request but one in sight, and this one gets a 111 then, nothing in nginx debug log, no error to be seen in rsyslog log but one thing I realized, after a restart the first log message always, reproducable gets a 111 the socket is not connected, nor listening, only after the first request is logged/or not logged (which is logged with 111 in nginx) the socket is connected and listening, so restarting rsyslog via systemd does not connect/listen to/on the socket the rsyslog debug log just tells us this : 6289.088037540:main thread : imuxsock.c: imuxsock: Opened UNIX socket '/run/logmat' (fd 6). [root@xxx rsyslog.d]# systemctl restart rsyslog [root@xxx rsyslog.d]# ss -x | grep logmat [root@xxx rsyslog.d]# lsof /run/logmat COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME rsyslogd 2097140 root 6u unix 0x 0t0 25300317 /run/logmat type=DGRAM (UNCONNECTED) make a request from browser or curl [root@xxx rsyslog.d]# lsof /run/logmat COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME rsyslogd 2097140 root 6u unix 0x 0t0 25300317 /run/logmat type=DGRAM (CONNECTED) [root@xxx rsyslog.d]# ss -x | grep logmat u_dgr ESTAB 0 0 /run/logmat 25300317 * 0 On 18/09/2023 16:34, TG Servers via rsyslog wrote: I just wanted to add that in a further message as it came to my mind. you were faster... the script is definitely "slow", this is what I know for sure as it does quite a lot of processing/analytics in the background, so even if you trigger it from command line it can take half a sec or so I can't change that, it needs to do what it does, I didn't write it though it can handle manual fast F5 triggers in the browser without issue and then it 111s when there are 2 requests incoming... I thought rsyslog might handle that just well via the queue... but then this might eventually really be the issue, and if it is, is there anything to mitigate this from rsyslog side (in terms of own queue for that socket or something in that direction)? ok, will enable impstats, too when I switch back Thanks, Tom On 18/09/2023 16:17, Rainer Gerhards wrote: so far not a single 111 today, I let this run the until late evening, and if there is stil no 111 I will put back the python script in order because right now there are 2 possibilities, I moved the socket as said, and I skipped the script and just appended the message to a file if either of the 2 things are responsible in the end I won't understand it either :) I don't know what the script does. But if it is slow, it may push back to the main queue, making rsyslog unresponsive. This is David's concern. Tomorrow, if you re-enable, you should also enable impstats as David suggested. Rainer ___ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx
rsyslog does not just pipe the socket to the script. It reads a message from the socket, adds it to a queue (by default the main queue), then a separate thread reads from the queue and sends the line to the script. If the script takes too long to process the line, then a backlog of messages build up in the queue. When the queue is full, rsyslog stops reading the input (note that the OS can have a buffer as well, depending on the input when that buffer fills, it will stall the writer or throw away data). unix sockets block if the reader isn't able to accept a message, which would result in the failure you are seeing There are all sorts of options for the queue, you can make a dedicated queue for that input, you can configure that queue to spill logs to disk when it gets too full (reading them back later, which is a separate thread), throwing away logs when it gets too full (depending on the log severity), etc. David Lang On Mon, 18 Sep 2023, TG Servers via rsyslog wrote: so far not a single 111 today, I let this run the until late evening, and if there is stil no 111 I will put back the python script in order because right now there are 2 possibilities, I moved the socket as said, and I skipped the script and just appended the message to a file if either of the 2 things are responsible in the end I won't understand it either :) because the read socket is just piped to the script, so the script should have absoletely no influence on the socket, but we will see what happens instead of speculating now, nginx is also running on debug just for the case On 18/09/2023 09:24, TG Servers via rsyslog wrote: what I did today in the morning was to put the socket in /run input(type="imuxsock" socket="/run/logmat") until now no errors so far but that does not mean a lot as there is not much traffic right now. you mean remove the python script or the whole script call? what I could do is to echo the message to a file instead of piping it to the python script, yes. I will try it an let it run for today, or until I have an error again On 18/09/2023 09:18, Rainer Gerhards wrote: Maybe a debug logs helps, but if rsyslog does not emit an error message, it does not sound like it has some issue. I also don't see a relation to the script. But to be sure, would it be possible to temporarily remove it and see if that changes anything? Rainer El lun, 18 sept 2023 a las 9:09, TG Servers () escribió: Hi Rainer, this is from nginx error log, yes. No I cannot find any other errors, thats my problem But it happens every single day, regularly... as just written in another message re the question if it occurs with rsyslog restart or logrotate : no absolutely not, I cannot see any relation to things like that, that is what leaves me a bit baffled here. You can see this is all from one day, and if there is a 111 on 2:52:19 on 2:52:22 there is everything ok (just as an example) Rsyslog restarts run between 0:10 and 0:15, depending on finish of nightly dnf-update, logrotate runs at 0:00 via systemd timer, nginx is not restarted automatically There is no information in any log except in the nginx logs, and the entries that are shown as failed are clearly missing in the target analytics log I cannot see any pattern... On 18/09/2023 08:53, Rainer Gerhards wrote: Is this from a nginx text log? Any errors infos from rsyslog itself? Rainer PS: I do not see how this can be related to rsyslog, but you never know. I do not yet understand the fault scenario TBH. El dom, 17 sept 2023 a las 18:39, TG Servers via rsyslog () escribió: Hi, ever since I started logging to a UDS from my nginx I get the occasional 111 in my nginx error logs. As I literally don't have any other information or log entries I honestly do not know how to debug this. The thing is requests one second, or a few seconds later are processed totally fine, so it cannot be a general problem, nor a access problem or a permission problem. I would be glad if anyone could help on fixing this Examples: error.log:2023/09/17 02:41:59 [error] 346192#346192: send() failed (111: Connection refused) while logging to syslog, server: unix:/var/cache/nginx/rsyslog.socket error.log:2023/09/17 02:41:59 [error] 346191#346191: send() failed (111: Connection refused) while logging to syslog, server: unix:/var/cache/nginx/rsyslog.socket error.log:2023/09/17 02:52:19 [error] 346192#346192: send() failed (111: Connection refused) while logging to syslog, server: unix:/var/cache/nginx/rsyslog.socket error.log:2023/09/17 04:09:44 [error] 346193#346193: send() failed (111: Connection refused) while logging to syslog, server: unix:/var/cache/nginx/rsyslog.socket error.log:2023/09/17 04:09:45 [error] 346185#346185: send() failed (111: Connection refused) while logging to syslog, server: unix:/var/cache/nginx/rsyslog.socket error.log:2023/09/17 04:20:20 [error] 346186#346186: send() failed (111: Connection refused) while logging to syslog,
Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx
I just wanted to add that in a further message as it came to my mind. you were faster... the script is definitely "slow", this is what I know for sure as it does quite a lot of processing/analytics in the background, so even if you trigger it from command line it can take half a sec or so I can't change that, it needs to do what it does, I didn't write it though it can handle manual fast F5 triggers in the browser without issue and then it 111s when there are 2 requests incoming... I thought rsyslog might handle that just well via the queue... but then this might eventually really be the issue, and if it is, is there anything to mitigate this from rsyslog side (in terms of own queue for that socket or something in that direction)? ok, will enable impstats, too when I switch back Thanks, Tom On 18/09/2023 16:17, Rainer Gerhards wrote: so far not a single 111 today, I let this run the until late evening, and if there is stil no 111 I will put back the python script in order because right now there are 2 possibilities, I moved the socket as said, and I skipped the script and just appended the message to a file if either of the 2 things are responsible in the end I won't understand it either :) I don't know what the script does. But if it is slow, it may push back to the main queue, making rsyslog unresponsive. This is David's concern. Tomorrow, if you re-enable, you should also enable impstats as David suggested. Rainer ___ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx
> so far not a single 111 today, I let this run the until late evening, > and if there is stil no 111 I will put back the python script in order > because right now there are 2 possibilities, I moved the socket as said, > and I skipped the script and just appended the message to a file > if either of the 2 things are responsible in the end I won't understand > it either :) I don't know what the script does. But if it is slow, it may push back to the main queue, making rsyslog unresponsive. This is David's concern. Tomorrow, if you re-enable, you should also enable impstats as David suggested. Rainer ___ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx
so far not a single 111 today, I let this run the until late evening, and if there is stil no 111 I will put back the python script in order because right now there are 2 possibilities, I moved the socket as said, and I skipped the script and just appended the message to a file if either of the 2 things are responsible in the end I won't understand it either :) because the read socket is just piped to the script, so the script should have absoletely no influence on the socket, but we will see what happens instead of speculating now, nginx is also running on debug just for the case On 18/09/2023 09:24, TG Servers via rsyslog wrote: what I did today in the morning was to put the socket in /run input(type="imuxsock" socket="/run/logmat") until now no errors so far but that does not mean a lot as there is not much traffic right now. you mean remove the python script or the whole script call? what I could do is to echo the message to a file instead of piping it to the python script, yes. I will try it an let it run for today, or until I have an error again On 18/09/2023 09:18, Rainer Gerhards wrote: Maybe a debug logs helps, but if rsyslog does not emit an error message, it does not sound like it has some issue. I also don't see a relation to the script. But to be sure, would it be possible to temporarily remove it and see if that changes anything? Rainer El lun, 18 sept 2023 a las 9:09, TG Servers () escribió: Hi Rainer, this is from nginx error log, yes. No I cannot find any other errors, thats my problem But it happens every single day, regularly... as just written in another message re the question if it occurs with rsyslog restart or logrotate : no absolutely not, I cannot see any relation to things like that, that is what leaves me a bit baffled here. You can see this is all from one day, and if there is a 111 on 2:52:19 on 2:52:22 there is everything ok (just as an example) Rsyslog restarts run between 0:10 and 0:15, depending on finish of nightly dnf-update, logrotate runs at 0:00 via systemd timer, nginx is not restarted automatically There is no information in any log except in the nginx logs, and the entries that are shown as failed are clearly missing in the target analytics log I cannot see any pattern... On 18/09/2023 08:53, Rainer Gerhards wrote: Is this from a nginx text log? Any errors infos from rsyslog itself? Rainer PS: I do not see how this can be related to rsyslog, but you never know. I do not yet understand the fault scenario TBH. El dom, 17 sept 2023 a las 18:39, TG Servers via rsyslog () escribió: Hi, ever since I started logging to a UDS from my nginx I get the occasional 111 in my nginx error logs. As I literally don't have any other information or log entries I honestly do not know how to debug this. The thing is requests one second, or a few seconds later are processed totally fine, so it cannot be a general problem, nor a access problem or a permission problem. I would be glad if anyone could help on fixing this Examples: error.log:2023/09/17 02:41:59 [error] 346192#346192: send() failed (111: Connection refused) while logging to syslog, server: unix:/var/cache/nginx/rsyslog.socket error.log:2023/09/17 02:41:59 [error] 346191#346191: send() failed (111: Connection refused) while logging to syslog, server: unix:/var/cache/nginx/rsyslog.socket error.log:2023/09/17 02:52:19 [error] 346192#346192: send() failed (111: Connection refused) while logging to syslog, server: unix:/var/cache/nginx/rsyslog.socket error.log:2023/09/17 04:09:44 [error] 346193#346193: send() failed (111: Connection refused) while logging to syslog, server: unix:/var/cache/nginx/rsyslog.socket error.log:2023/09/17 04:09:45 [error] 346185#346185: send() failed (111: Connection refused) while logging to syslog, server: unix:/var/cache/nginx/rsyslog.socket error.log:2023/09/17 04:20:20 [error] 346186#346186: send() failed (111: Connection refused) while logging to syslog, server: unix:/var/cache/nginx/rsyslog.socket error.log:2023/09/17 06:20:01 [error] 346182#346182: send() failed (111: Connection refused) while logging to syslog, server: unix:/var/cache/nginx/rsyslog.socket error.log:2023/09/17 08:32:35 [error] 346182#346182: send() failed (111: Connection refused) while logging to syslog, server: unix:/var/cache/nginx/rsyslog.socket error.log:2023/09/17 08:32:35 [error] 346188#346188: send() failed (111: Connection refused) while logging to syslog, server: unix:/var/cache/nginx/rsyslog.socket error.log:2023/09/17 09:34:34 [error] 346183#346183: send() failed (111: Connection refused) while logging to syslog, server: unix:/var/cache/nginx/rsyslog.socket error.log:2023/09/17 09:34:35 [error] 346183#346183: send() failed (111: Connection refused) while logging to syslog, server: unix:/var/cache/nginx/rsyslog.socket error.log:2023/09/17 16:00:25 [error] 346187#346187: send() failed (111: Connection refused) while logging to syslog, server:
Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx
This email may contain proprietary information of BAE Systems and/or third parties. Sorry, replied to the wrong message. -Original Message- From: rsyslog On Behalf Of Lennon, Sean (UK) via rsyslog Sent: 18 September 2023 14:09 To: rsyslog-users Cc: Lennon, Sean (UK) Subject: Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx - PHISHING ALERT - This email has been sent from an account outside of the BAE Systems network. Please treat the email with caution, especially if you are requested to click on a link or open an attachment. For further information on how to spot and report a phishing email please access the Global Intranet, then select / . This email may contain proprietary information of BAE Systems and/or third parties. Hi, Has anyone got any thoughts/suggestions on this? Cheers, Sean. -Original Message- From: rsyslog On Behalf Of David Lang via rsyslog Sent: 18 September 2023 08:22 To: TG Servers via rsyslog Cc: David Lang Subject: Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx - PHISHING ALERT - This email has been sent from an account outside of the BAE Systems network. Please treat the email with caution, especially if you are requested to click on a link or open an attachment. For further information on how to spot and report a phishing email please access the Global Intranet, then select / . my thought is that if rsyslog is getting blocked (queues full) then it will stop accepting new logs via unix sockets. can you enable impstats and see if you have any reports of the main queue being full during the times that this happens? full configs for rsyslog could identify if there is anything you have there that is likely to block for extended times. David Lang On Sun, 17 Sep 2023, TG Servers via rsyslog wrote: > Hi, > > ever since I started logging to a UDS from my nginx I get the > occasional > 111 in my nginx error logs. > As I literally don't have any other information or log entries I > honestly do not know how to debug this. > The thing is requests one second, or a few seconds later are processed > totally fine, so it cannot be a general problem, nor a access problem > or a permission problem. > I would be glad if anyone could help on fixing this > > Examples: > error.log:2023/09/17 02:41:59 [error] 346192#346192: send() failed (111: > Connection refused) while logging to syslog, server: > unix:/var/cache/nginx/rsyslog.socket > error.log:2023/09/17 02:41:59 [error] 346191#346191: send() failed (111: > Connection refused) while logging to syslog, server: > unix:/var/cache/nginx/rsyslog.socket > error.log:2023/09/17 02:52:19 [error] 346192#346192: send() failed (111: > Connection refused) while logging to syslog, server: > unix:/var/cache/nginx/rsyslog.socket > error.log:2023/09/17 04:09:44 [error] 346193#346193: send() failed (111: > Connection refused) while logging to syslog, server: > unix:/var/cache/nginx/rsyslog.socket > error.log:2023/09/17 04:09:45 [error] 346185#346185: send() failed (111: > Connection refused) while logging to syslog, server: > unix:/var/cache/nginx/rsyslog.socket > error.log:2023/09/17 04:20:20 [error] 346186#346186: send() failed (111: > Connection refused) while logging to syslog, server: > unix:/var/cache/nginx/rsyslog.socket > error.log:2023/09/17 06:20:01 [error] 346182#346182: send() failed (111: > Connection refused) while logging to syslog, server: > unix:/var/cache/nginx/rsyslog.socket > error.log:2023/09/17 08:32:35 [error] 346182#346182: send() failed (111: > Connection refused) while logging to syslog, server: > unix:/var/cache/nginx/rsyslog.socket > error.log:2023/09/17 08:32:35 [error] 346188#346188: send() failed (111: > Connection refused) while logging to syslog, server: > unix:/var/cache/nginx/rsyslog.socket > error.log:2023/09/17 09:34:34 [error] 346183#346183: send() failed (111: > Connection refused) while logging to syslog, server: > unix:/var/cache/nginx/rsyslog.socket > error.log:2023/09/17 09:34:35 [error] 346183#346183: send() failed (111: > Connection refused) while logging to syslog, server: > unix:/var/cache/nginx/rsyslog.socket > error.log:2023/09/17 16:00:25 [error] 346187#346187: send() failed (111: > Connection refused) while logging to syslog, server: > unix:/var/cache/nginx/rsyslog.socket > > configuration for this single use case, of course there is a lot more > in rsyslog.conf $AddUnixListenSocket /var/cache/nginx/rsyslog.socket > > $template app,"%msg:2:$%" >
Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx
This email may contain proprietary information of BAE Systems and/or third
parties.
Hi,
Has anyone got any thoughts/suggestions on this?
Cheers,
Sean.
-Original Message-
From: rsyslog On Behalf Of David Lang via
rsyslog
Sent: 18 September 2023 08:22
To: TG Servers via rsyslog
Cc: David Lang
Subject: Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx
- PHISHING ALERT -
This email has been sent from an account outside of the BAE Systems network.
Please treat the email with caution, especially if you are requested to click
on a link or open an attachment.
For further information on how to spot and report a phishing email please
access the Global Intranet, then select / .
my thought is that if rsyslog is getting blocked (queues full) then it will
stop
accepting new logs via unix sockets.
can you enable impstats and see if you have any reports of the main queue being
full during the times that this happens?
full configs for rsyslog could identify if there is anything you have there
that
is likely to block for extended times.
David Lang
On Sun, 17 Sep 2023, TG Servers via rsyslog wrote:
> Hi,
>
> ever since I started logging to a UDS from my nginx I get the occasional
> 111 in my nginx error logs.
> As I literally don't have any other information or log entries I
> honestly do not know how to debug this.
> The thing is requests one second, or a few seconds later are processed
> totally fine, so it cannot be a general problem, nor a access problem or
> a permission problem.
> I would be glad if anyone could help on fixing this
>
> Examples:
> error.log:2023/09/17 02:41:59 [error] 346192#346192: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 02:41:59 [error] 346191#346191: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 02:52:19 [error] 346192#346192: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 04:09:44 [error] 346193#346193: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 04:09:45 [error] 346185#346185: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 04:20:20 [error] 346186#346186: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 06:20:01 [error] 346182#346182: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 08:32:35 [error] 346182#346182: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 08:32:35 [error] 346188#346188: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 09:34:34 [error] 346183#346183: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 09:34:35 [error] 346183#346183: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 16:00:25 [error] 346187#346187: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
>
> configuration for this single use case, of course there is a lot more in
> rsyslog.conf
> $AddUnixListenSocket /var/cache/nginx/rsyslog.socket
>
> $template app,"%msg:2:$%"
>
> if $programname == "app" then {
> ^/usr/local/script/app_log.sh;app
> stop
> }
>
> The script app_log.sh does simply
> echo "${@}" | /usr/bin/python
>
> Many thanks
>
> ___
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rs
Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx
what I did today in the morning was to put the socket in /run
input(type="imuxsock" socket="/run/logmat")
until now no errors so far but that does not mean a lot as there is not
much traffic right now.
you mean remove the python script or the whole script call? what I could
do is to echo the message to a file instead of piping it to the python
script, yes.
I will try it an let it run for today, or until I have an error again
On 18/09/2023 09:18, Rainer Gerhards wrote:
Maybe a debug logs helps, but if rsyslog does not emit an error
message, it does not sound like it has some issue. I also don't see a
relation to the script. But to be sure, would it be possible to
temporarily remove it and see if that changes anything?
Rainer
El lun, 18 sept 2023 a las 9:09, TG Servers () escribió:
Hi Rainer,
this is from nginx error log, yes.
No I cannot find any other errors, thats my problem
But it happens every single day, regularly...
as just written in another message re the question if it occurs with rsyslog
restart or logrotate :
no absolutely not, I cannot see any relation to things like that, that is what
leaves me a bit baffled here.
You can see this is all from one day, and if there is a 111 on 2:52:19 on
2:52:22 there is everything ok (just as an example)
Rsyslog restarts run between 0:10 and 0:15, depending on finish of nightly
dnf-update, logrotate runs at 0:00 via systemd timer, nginx is not restarted
automatically
There is no information in any log except in the nginx logs, and the entries
that are shown as failed are clearly missing in the target analytics log
I cannot see any pattern...
On 18/09/2023 08:53, Rainer Gerhards wrote:
Is this from a nginx text log? Any errors infos from rsyslog itself?
Rainer
PS: I do not see how this can be related to rsyslog, but you never
know. I do not yet understand the fault scenario TBH.
El dom, 17 sept 2023 a las 18:39, TG Servers via rsyslog
() escribió:
Hi,
ever since I started logging to a UDS from my nginx I get the occasional
111 in my nginx error logs.
As I literally don't have any other information or log entries I
honestly do not know how to debug this.
The thing is requests one second, or a few seconds later are processed
totally fine, so it cannot be a general problem, nor a access problem or
a permission problem.
I would be glad if anyone could help on fixing this
Examples:
error.log:2023/09/17 02:41:59 [error] 346192#346192: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 02:41:59 [error] 346191#346191: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 02:52:19 [error] 346192#346192: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 04:09:44 [error] 346193#346193: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 04:09:45 [error] 346185#346185: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 04:20:20 [error] 346186#346186: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 06:20:01 [error] 346182#346182: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 08:32:35 [error] 346182#346182: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 08:32:35 [error] 346188#346188: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 09:34:34 [error] 346183#346183: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 09:34:35 [error] 346183#346183: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 16:00:25 [error] 346187#346187: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
configuration for this single use case, of course there is a lot more in
rsyslog.conf
$AddUnixListenSocket /var/cache/nginx/rsyslog.socket
$template app,"%msg:2:$%"
if $programname == "app" then {
^/usr/local/script/app_log.sh;app
stop
}
The script app_log.sh does simply
echo "${@}" | /usr/bin/python
Many thanks
___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our
Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx
my thought is that if rsyslog is getting blocked (queues full) then it will stop
accepting new logs via unix sockets.
can you enable impstats and see if you have any reports of the main queue being
full during the times that this happens?
full configs for rsyslog could identify if there is anything you have there that
is likely to block for extended times.
David Lang
On Sun, 17 Sep 2023, TG Servers via rsyslog wrote:
Hi,
ever since I started logging to a UDS from my nginx I get the occasional
111 in my nginx error logs.
As I literally don't have any other information or log entries I
honestly do not know how to debug this.
The thing is requests one second, or a few seconds later are processed
totally fine, so it cannot be a general problem, nor a access problem or
a permission problem.
I would be glad if anyone could help on fixing this
Examples:
error.log:2023/09/17 02:41:59 [error] 346192#346192: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 02:41:59 [error] 346191#346191: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 02:52:19 [error] 346192#346192: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 04:09:44 [error] 346193#346193: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 04:09:45 [error] 346185#346185: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 04:20:20 [error] 346186#346186: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 06:20:01 [error] 346182#346182: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 08:32:35 [error] 346182#346182: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 08:32:35 [error] 346188#346188: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 09:34:34 [error] 346183#346183: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 09:34:35 [error] 346183#346183: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 16:00:25 [error] 346187#346187: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
configuration for this single use case, of course there is a lot more in
rsyslog.conf
$AddUnixListenSocket /var/cache/nginx/rsyslog.socket
$template app,"%msg:2:$%"
if $programname == "app" then {
^/usr/local/script/app_log.sh;app
stop
}
The script app_log.sh does simply
echo "${@}" | /usr/bin/python
Many thanks
___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx
Maybe a debug logs helps, but if rsyslog does not emit an error
message, it does not sound like it has some issue. I also don't see a
relation to the script. But to be sure, would it be possible to
temporarily remove it and see if that changes anything?
Rainer
El lun, 18 sept 2023 a las 9:09, TG Servers () escribió:
>
> Hi Rainer,
>
> this is from nginx error log, yes.
> No I cannot find any other errors, thats my problem
> But it happens every single day, regularly...
> as just written in another message re the question if it occurs with rsyslog
> restart or logrotate :
>
> no absolutely not, I cannot see any relation to things like that, that is
> what leaves me a bit baffled here.
> You can see this is all from one day, and if there is a 111 on 2:52:19 on
> 2:52:22 there is everything ok (just as an example)
> Rsyslog restarts run between 0:10 and 0:15, depending on finish of nightly
> dnf-update, logrotate runs at 0:00 via systemd timer, nginx is not restarted
> automatically
> There is no information in any log except in the nginx logs, and the entries
> that are shown as failed are clearly missing in the target analytics log
> I cannot see any pattern...
>
>
> On 18/09/2023 08:53, Rainer Gerhards wrote:
>
> Is this from a nginx text log? Any errors infos from rsyslog itself?
>
> Rainer
> PS: I do not see how this can be related to rsyslog, but you never
> know. I do not yet understand the fault scenario TBH.
>
> El dom, 17 sept 2023 a las 18:39, TG Servers via rsyslog
> () escribió:
>
> Hi,
>
> ever since I started logging to a UDS from my nginx I get the occasional
> 111 in my nginx error logs.
> As I literally don't have any other information or log entries I
> honestly do not know how to debug this.
> The thing is requests one second, or a few seconds later are processed
> totally fine, so it cannot be a general problem, nor a access problem or
> a permission problem.
> I would be glad if anyone could help on fixing this
>
> Examples:
> error.log:2023/09/17 02:41:59 [error] 346192#346192: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 02:41:59 [error] 346191#346191: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 02:52:19 [error] 346192#346192: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 04:09:44 [error] 346193#346193: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 04:09:45 [error] 346185#346185: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 04:20:20 [error] 346186#346186: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 06:20:01 [error] 346182#346182: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 08:32:35 [error] 346182#346182: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 08:32:35 [error] 346188#346188: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 09:34:34 [error] 346183#346183: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 09:34:35 [error] 346183#346183: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 16:00:25 [error] 346187#346187: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
>
> configuration for this single use case, of course there is a lot more in
> rsyslog.conf
> $AddUnixListenSocket /var/cache/nginx/rsyslog.socket
>
> $template app,"%msg:2:$%"
>
> if $programname == "app" then {
> ^/usr/local/script/app_log.sh;app
> stop
> }
>
> The script app_log.sh does simply
> echo "${@}" | /usr/bin/python
>
> Many thanks
>
> ___
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
>
>
___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow
Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx
Hi Rainer,
this is from nginx error log, yes.
No I cannot find any other errors, thats my problem
But it happens every single day, regularly...
as just written in another message re the question if it occurs with
rsyslog restart or logrotate :
no absolutely not, I cannot see any relation to things like that, that
is what leaves me a bit baffled here.
You can see this is all from one day, and if there is a 111 on 2:52:19
on 2:52:22 there is everything ok (just as an example)
Rsyslog restarts run between 0:10 and 0:15, depending on finish of
nightly dnf-update, logrotate runs at 0:00 via systemd timer, nginx is
not restarted automatically
There is no information in any log except in the nginx logs, and the
entries that are shown as failed are clearly missing in the target
analytics log
I cannot see any pattern...
On 18/09/2023 08:53, Rainer Gerhards wrote:
Is this from a nginx text log? Any errors infos from rsyslog itself?
Rainer
PS: I do not see how this can be related to rsyslog, but you never
know. I do not yet understand the fault scenario TBH.
El dom, 17 sept 2023 a las 18:39, TG Servers via rsyslog
() escribió:
Hi,
ever since I started logging to a UDS from my nginx I get the occasional
111 in my nginx error logs.
As I literally don't have any other information or log entries I
honestly do not know how to debug this.
The thing is requests one second, or a few seconds later are processed
totally fine, so it cannot be a general problem, nor a access problem or
a permission problem.
I would be glad if anyone could help on fixing this
Examples:
error.log:2023/09/17 02:41:59 [error] 346192#346192: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 02:41:59 [error] 346191#346191: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 02:52:19 [error] 346192#346192: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 04:09:44 [error] 346193#346193: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 04:09:45 [error] 346185#346185: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 04:20:20 [error] 346186#346186: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 06:20:01 [error] 346182#346182: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 08:32:35 [error] 346182#346182: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 08:32:35 [error] 346188#346188: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 09:34:34 [error] 346183#346183: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 09:34:35 [error] 346183#346183: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 16:00:25 [error] 346187#346187: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
configuration for this single use case, of course there is a lot more in
rsyslog.conf
$AddUnixListenSocket /var/cache/nginx/rsyslog.socket
$template app,"%msg:2:$%"
if $programname == "app" then {
^/usr/local/script/app_log.sh;app
stop
}
The script app_log.sh does simply
echo "${@}" | /usr/bin/python
Many thanks
___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx
___ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx
Hi Yury,
no absolutely not, I cannot see any relation to things like that, that
is what leaves me a bit baffled here.
You can see this is all from one day, and if there is a 111 on 2:52:19
on 2:52:22 there is everything ok (just as an example)
Rsyslog restarts run between 0:10 and 0:15, depending on finish of
nightly dnf-update, logrotate runs at 0:00 via systemd timer, nginx is
not restarted automatically
There is no information in any log except in the nginx logs, and the
entries that are shown as failed are clearly missing in the target
analytics log
I cannot see any pattern...
On 18/09/2023 08:44, Yury Bushmelev via rsyslog wrote:
Hello!
Does the timing match with rsyslog restarts (manual or logrotate-initiated)?
On Mon, 18 Sept 2023 at 00:39, TG Servers via rsyslog <
rsyslog@lists.adiscon.com> wrote:
Hi,
ever since I started logging to a UDS from my nginx I get the occasional
111 in my nginx error logs.
As I literally don't have any other information or log entries I
honestly do not know how to debug this.
The thing is requests one second, or a few seconds later are processed
totally fine, so it cannot be a general problem, nor a access problem or
a permission problem.
I would be glad if anyone could help on fixing this
Examples:
error.log:2023/09/17 02:41:59 [error] 346192#346192: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 02:41:59 [error] 346191#346191: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 02:52:19 [error] 346192#346192: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 04:09:44 [error] 346193#346193: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 04:09:45 [error] 346185#346185: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 04:20:20 [error] 346186#346186: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 06:20:01 [error] 346182#346182: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 08:32:35 [error] 346182#346182: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 08:32:35 [error] 346188#346188: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 09:34:34 [error] 346183#346183: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 09:34:35 [error] 346183#346183: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 16:00:25 [error] 346187#346187: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
configuration for this single use case, of course there is a lot more in
rsyslog.conf
$AddUnixListenSocket /var/cache/nginx/rsyslog.socket
$template app,"%msg:2:$%"
if $programname == "app" then {
^/usr/local/script/app_log.sh;app
stop
}
The script app_log.sh does simply
echo "${@}" | /usr/bin/python
Many thanks
___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx
Is this from a nginx text log? Any errors infos from rsyslog itself?
Rainer
PS: I do not see how this can be related to rsyslog, but you never
know. I do not yet understand the fault scenario TBH.
El dom, 17 sept 2023 a las 18:39, TG Servers via rsyslog
() escribió:
>
> Hi,
>
> ever since I started logging to a UDS from my nginx I get the occasional
> 111 in my nginx error logs.
> As I literally don't have any other information or log entries I
> honestly do not know how to debug this.
> The thing is requests one second, or a few seconds later are processed
> totally fine, so it cannot be a general problem, nor a access problem or
> a permission problem.
> I would be glad if anyone could help on fixing this
>
> Examples:
> error.log:2023/09/17 02:41:59 [error] 346192#346192: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 02:41:59 [error] 346191#346191: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 02:52:19 [error] 346192#346192: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 04:09:44 [error] 346193#346193: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 04:09:45 [error] 346185#346185: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 04:20:20 [error] 346186#346186: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 06:20:01 [error] 346182#346182: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 08:32:35 [error] 346182#346182: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 08:32:35 [error] 346188#346188: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 09:34:34 [error] 346183#346183: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 09:34:35 [error] 346183#346183: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 16:00:25 [error] 346187#346187: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
>
> configuration for this single use case, of course there is a lot more in
> rsyslog.conf
> $AddUnixListenSocket /var/cache/nginx/rsyslog.socket
>
> $template app,"%msg:2:$%"
>
> if $programname == "app" then {
> ^/usr/local/script/app_log.sh;app
> stop
> }
>
> The script app_log.sh does simply
> echo "${@}" | /usr/bin/python
>
> Many thanks
>
> ___
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx
Hello!
Does the timing match with rsyslog restarts (manual or logrotate-initiated)?
On Mon, 18 Sept 2023 at 00:39, TG Servers via rsyslog <
rsyslog@lists.adiscon.com> wrote:
> Hi,
>
> ever since I started logging to a UDS from my nginx I get the occasional
> 111 in my nginx error logs.
> As I literally don't have any other information or log entries I
> honestly do not know how to debug this.
> The thing is requests one second, or a few seconds later are processed
> totally fine, so it cannot be a general problem, nor a access problem or
> a permission problem.
> I would be glad if anyone could help on fixing this
>
> Examples:
> error.log:2023/09/17 02:41:59 [error] 346192#346192: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 02:41:59 [error] 346191#346191: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 02:52:19 [error] 346192#346192: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 04:09:44 [error] 346193#346193: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 04:09:45 [error] 346185#346185: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 04:20:20 [error] 346186#346186: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 06:20:01 [error] 346182#346182: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 08:32:35 [error] 346182#346182: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 08:32:35 [error] 346188#346188: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 09:34:34 [error] 346183#346183: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 09:34:35 [error] 346183#346183: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 16:00:25 [error] 346187#346187: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
>
> configuration for this single use case, of course there is a lot more in
> rsyslog.conf
> $AddUnixListenSocket /var/cache/nginx/rsyslog.socket
>
> $template app,"%msg:2:$%"
>
> if $programname == "app" then {
> ^/usr/local/script/app_log.sh;app
> stop
> }
>
> The script app_log.sh does simply
> echo "${@}" | /usr/bin/python
>
> Many thanks
>
> ___
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
--
Yury Bushmelev
___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
[rsyslog] Repeated 111 to rsyslog UDS from nginx
Hi,
ever since I started logging to a UDS from my nginx I get the occasional
111 in my nginx error logs.
As I literally don't have any other information or log entries I
honestly do not know how to debug this.
The thing is requests one second, or a few seconds later are processed
totally fine, so it cannot be a general problem, nor a access problem or
a permission problem.
I would be glad if anyone could help on fixing this
Examples:
error.log:2023/09/17 02:41:59 [error] 346192#346192: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 02:41:59 [error] 346191#346191: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 02:52:19 [error] 346192#346192: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 04:09:44 [error] 346193#346193: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 04:09:45 [error] 346185#346185: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 04:20:20 [error] 346186#346186: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 06:20:01 [error] 346182#346182: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 08:32:35 [error] 346182#346182: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 08:32:35 [error] 346188#346188: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 09:34:34 [error] 346183#346183: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 09:34:35 [error] 346183#346183: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 16:00:25 [error] 346187#346187: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
configuration for this single use case, of course there is a lot more in
rsyslog.conf
$AddUnixListenSocket /var/cache/nginx/rsyslog.socket
$template app,"%msg:2:$%"
if $programname == "app" then {
^/usr/local/script/app_log.sh;app
stop
}
The script app_log.sh does simply
echo "${@}" | /usr/bin/python
Many thanks
___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
