[jira] [Commented] (LOG4J2-3425) Syslog appender lacks the SocketOptions setting
[ https://issues.apache.org/jira/browse/LOG4J2-3425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17503148#comment-17503148 ] Jiří Smolík commented on LOG4J2-3425: - Alright, I will close this ticket. A new one has just been created: https://issues.apache.org/jira/browse/LOG4J2-3430 > Syslog appender lacks the SocketOptions setting > --- > > Key: LOG4J2-3425 > URL: https://issues.apache.org/jira/browse/LOG4J2-3425 > Project: Log4j 2 > Issue Type: Improvement > Components: Appenders >Affects Versions: 2.12.1 >Reporter: Jiří Smolík >Priority: Major > Fix For: 2.17.3 > > > Recently, we've had an issue with our syslog server (collector), where it was > suddenly unable to forward messages sent to it. This resulted in flooding the > server and ultimately, exhausting its resources (memory and allocated > temporary storage). > However, more interestingly, Log4j2 appender behaved as follows: > # At the moment of exhaustion, one thread (T1) entered the synchronized > block in TcpSocketManager (see > https://github.com/apache/logging-log4j2/blob/e2ee2f4dd9327eb1841023c3269ba12f5673e694/log4j-core/src/main/java/org/apache/logging/log4j/core/net/TcpSocketManager.java#L217) > and tried to write the message (see > [https://github.com/apache/logging-log4j2/blob/e2ee2f4dd9327eb1841023c3269ba12f5673e694/log4j-core/src/main/java/org/apache/logging/log4j/core/net/TcpSocketManager.java#L253]). > # The server didn't respond immediately - actually, it blocked T1 for hours > (it was most likely waiting for resources to be free). > # In the meantime, all threads attempting to write a message to the syslog > appender hanged at the synchronized block (they were waiting for T1 to > finish). > Eventually, the connection to the server broke apart or some resources were > freed, and T1 finished. Interestingly, it didn't throw any exception, but > that is beside the point in this issue. > So, what happened is that the server got flooded and blocked one thread in > our application, but due to the way TcpSocketManager is implemented, our > application suffered a DoS. > As a precaution to such dreadful scenario, I would like to request adding a > read timeout setting to the TcpSocketManager (also syslog appender), which > would in case of the above described situation prevent T1 from blocking other > threads for a long time. At least, that's my idea of an easy fix. Connect and > reconnect timeout settings have a different purpose and were useless in this > case. > I am aware that the issue is also a matter of the syslog server > configuration, but application developers do not always have it under their > control. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (LOG4J2-3425) Syslog appender lacks the SocketOptions setting
[ https://issues.apache.org/jira/browse/LOG4J2-3425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17502006#comment-17502006 ] Gary D. Gregory commented on LOG4J2-3425: - I think we should close this ticket and open a new one titled something like "provide a socker appender using NIO" > Syslog appender lacks the SocketOptions setting > --- > > Key: LOG4J2-3425 > URL: https://issues.apache.org/jira/browse/LOG4J2-3425 > Project: Log4j 2 > Issue Type: Improvement > Components: Appenders >Affects Versions: 2.12.1 >Reporter: Jiří Smolík >Priority: Major > Fix For: 2.17.3 > > > Recently, we've had an issue with our syslog server (collector), where it was > suddenly unable to forward messages sent to it. This resulted in flooding the > server and ultimately, exhausting its resources (memory and allocated > temporary storage). > However, more interestingly, Log4j2 appender behaved as follows: > # At the moment of exhaustion, one thread (T1) entered the synchronized > block in TcpSocketManager (see > https://github.com/apache/logging-log4j2/blob/e2ee2f4dd9327eb1841023c3269ba12f5673e694/log4j-core/src/main/java/org/apache/logging/log4j/core/net/TcpSocketManager.java#L217) > and tried to write the message (see > [https://github.com/apache/logging-log4j2/blob/e2ee2f4dd9327eb1841023c3269ba12f5673e694/log4j-core/src/main/java/org/apache/logging/log4j/core/net/TcpSocketManager.java#L253]). > # The server didn't respond immediately - actually, it blocked T1 for hours > (it was most likely waiting for resources to be free). > # In the meantime, all threads attempting to write a message to the syslog > appender hanged at the synchronized block (they were waiting for T1 to > finish). > Eventually, the connection to the server broke apart or some resources were > freed, and T1 finished. Interestingly, it didn't throw any exception, but > that is beside the point in this issue. > So, what happened is that the server got flooded and blocked one thread in > our application, but due to the way TcpSocketManager is implemented, our > application suffered a DoS. > As a precaution to such dreadful scenario, I would like to request adding a > read timeout setting to the TcpSocketManager (also syslog appender), which > would in case of the above described situation prevent T1 from blocking other > threads for a long time. At least, that's my idea of an easy fix. Connect and > reconnect timeout settings have a different purpose and were useless in this > case. > I am aware that the issue is also a matter of the syslog server > configuration, but application developers do not always have it under their > control. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (LOG4J2-3425) Syslog appender lacks the SocketOptions setting
[ https://issues.apache.org/jira/browse/LOG4J2-3425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1750#comment-1750 ] Ralph Goers commented on LOG4J2-3425: - If the appender configuration doesn't have to change I would just modify the implementation. To be honest, I have thought about creating a Netty version of the appender for a while but just haven't had the time. I suspect that like the FlumeAppender we could implement multiple Managers and choose one based on configuration options. > Syslog appender lacks the SocketOptions setting > --- > > Key: LOG4J2-3425 > URL: https://issues.apache.org/jira/browse/LOG4J2-3425 > Project: Log4j 2 > Issue Type: Improvement > Components: Appenders >Affects Versions: 2.12.1 >Reporter: Jiří Smolík >Priority: Major > Fix For: 2.17.3 > > > Recently, we've had an issue with our syslog server (collector), where it was > suddenly unable to forward messages sent to it. This resulted in flooding the > server and ultimately, exhausting its resources (memory and allocated > temporary storage). > However, more interestingly, Log4j2 appender behaved as follows: > # At the moment of exhaustion, one thread (T1) entered the synchronized > block in TcpSocketManager (see > https://github.com/apache/logging-log4j2/blob/e2ee2f4dd9327eb1841023c3269ba12f5673e694/log4j-core/src/main/java/org/apache/logging/log4j/core/net/TcpSocketManager.java#L217) > and tried to write the message (see > [https://github.com/apache/logging-log4j2/blob/e2ee2f4dd9327eb1841023c3269ba12f5673e694/log4j-core/src/main/java/org/apache/logging/log4j/core/net/TcpSocketManager.java#L253]). > # The server didn't respond immediately - actually, it blocked T1 for hours > (it was most likely waiting for resources to be free). > # In the meantime, all threads attempting to write a message to the syslog > appender hanged at the synchronized block (they were waiting for T1 to > finish). > Eventually, the connection to the server broke apart or some resources were > freed, and T1 finished. Interestingly, it didn't throw any exception, but > that is beside the point in this issue. > So, what happened is that the server got flooded and blocked one thread in > our application, but due to the way TcpSocketManager is implemented, our > application suffered a DoS. > As a precaution to such dreadful scenario, I would like to request adding a > read timeout setting to the TcpSocketManager (also syslog appender), which > would in case of the above described situation prevent T1 from blocking other > threads for a long time. At least, that's my idea of an easy fix. Connect and > reconnect timeout settings have a different purpose and were useless in this > case. > I am aware that the issue is also a matter of the syslog server > configuration, but application developers do not always have it under their > control. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (LOG4J2-3425) Syslog appender lacks the SocketOptions setting
[ https://issues.apache.org/jira/browse/LOG4J2-3425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17500984#comment-17500984 ] Jiří Smolík commented on LOG4J2-3425: - [~ggregory] When I find some time, I will try to test your changes in a mocked-up situation of network or server congestion, but just like [~rgoers] , I don't think that soTimeout is the answer after all. I will also reopen the issue now. Could you perhaps change the title again to reflect the hidden Socket/NIO caveat? > Syslog appender lacks the SocketOptions setting > --- > > Key: LOG4J2-3425 > URL: https://issues.apache.org/jira/browse/LOG4J2-3425 > Project: Log4j 2 > Issue Type: Improvement > Components: Appenders >Affects Versions: 2.12.1 >Reporter: Jiří Smolík >Priority: Major > Fix For: 2.17.3 > > > Recently, we've had an issue with our syslog server (collector), where it was > suddenly unable to forward messages sent to it. This resulted in flooding the > server and ultimately, exhausting its resources (memory and allocated > temporary storage). > However, more interestingly, Log4j2 appender behaved as follows: > # At the moment of exhaustion, one thread (T1) entered the synchronized > block in TcpSocketManager (see > https://github.com/apache/logging-log4j2/blob/e2ee2f4dd9327eb1841023c3269ba12f5673e694/log4j-core/src/main/java/org/apache/logging/log4j/core/net/TcpSocketManager.java#L217) > and tried to write the message (see > [https://github.com/apache/logging-log4j2/blob/e2ee2f4dd9327eb1841023c3269ba12f5673e694/log4j-core/src/main/java/org/apache/logging/log4j/core/net/TcpSocketManager.java#L253]). > # The server didn't respond immediately - actually, it blocked T1 for hours > (it was most likely waiting for resources to be free). > # In the meantime, all threads attempting to write a message to the syslog > appender hanged at the synchronized block (they were waiting for T1 to > finish). > Eventually, the connection to the server broke apart or some resources were > freed, and T1 finished. Interestingly, it didn't throw any exception, but > that is beside the point in this issue. > So, what happened is that the server got flooded and blocked one thread in > our application, but due to the way TcpSocketManager is implemented, our > application suffered a DoS. > As a precaution to such dreadful scenario, I would like to request adding a > read timeout setting to the TcpSocketManager (also syslog appender), which > would in case of the above described situation prevent T1 from blocking other > threads for a long time. At least, that's my idea of an easy fix. Connect and > reconnect timeout settings have a different purpose and were useless in this > case. > I am aware that the issue is also a matter of the syslog server > configuration, but application developers do not always have it under their > control. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (LOG4J2-3425) Syslog appender lacks the SocketOptions setting
[ https://issues.apache.org/jira/browse/LOG4J2-3425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17500956#comment-17500956 ] Jiří Smolík commented on LOG4J2-3425: - In the short term, SocketChannelAppender is likely a better choice but once it is stable and battle-tested, the big question is whether to remove SocketAppender altogether. If not removed, developers of many applications could unknowingly encounter the same problem at times. > Syslog appender lacks the SocketOptions setting > --- > > Key: LOG4J2-3425 > URL: https://issues.apache.org/jira/browse/LOG4J2-3425 > Project: Log4j 2 > Issue Type: Improvement > Components: Appenders >Affects Versions: 2.12.1 >Reporter: Jiří Smolík >Priority: Major > Fix For: 2.17.3 > > > Recently, we've had an issue with our syslog server (collector), where it was > suddenly unable to forward messages sent to it. This resulted in flooding the > server and ultimately, exhausting its resources (memory and allocated > temporary storage). > However, more interestingly, Log4j2 appender behaved as follows: > # At the moment of exhaustion, one thread (T1) entered the synchronized > block in TcpSocketManager (see > https://github.com/apache/logging-log4j2/blob/e2ee2f4dd9327eb1841023c3269ba12f5673e694/log4j-core/src/main/java/org/apache/logging/log4j/core/net/TcpSocketManager.java#L217) > and tried to write the message (see > [https://github.com/apache/logging-log4j2/blob/e2ee2f4dd9327eb1841023c3269ba12f5673e694/log4j-core/src/main/java/org/apache/logging/log4j/core/net/TcpSocketManager.java#L253]). > # The server didn't respond immediately - actually, it blocked T1 for hours > (it was most likely waiting for resources to be free). > # In the meantime, all threads attempting to write a message to the syslog > appender hanged at the synchronized block (they were waiting for T1 to > finish). > Eventually, the connection to the server broke apart or some resources were > freed, and T1 finished. Interestingly, it didn't throw any exception, but > that is beside the point in this issue. > So, what happened is that the server got flooded and blocked one thread in > our application, but due to the way TcpSocketManager is implemented, our > application suffered a DoS. > As a precaution to such dreadful scenario, I would like to request adding a > read timeout setting to the TcpSocketManager (also syslog appender), which > would in case of the above described situation prevent T1 from blocking other > threads for a long time. At least, that's my idea of an easy fix. Connect and > reconnect timeout settings have a different purpose and were useless in this > case. > I am aware that the issue is also a matter of the syslog server > configuration, but application developers do not always have it under their > control. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (LOG4J2-3425) Syslog appender lacks the SocketOptions setting
[
https://issues.apache.org/jira/browse/LOG4J2-3425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17500951#comment-17500951
]
Gary D. Gregory commented on LOG4J2-3425:
-
Hm, do we want a new {{SocketChannelAppender}} so we can keep
{{SocketAppender}} for maximum compatibility or do we want to reimplement
{{SocketAppender}} using channels?
> Syslog appender lacks the SocketOptions setting
> ---
>
> Key: LOG4J2-3425
> URL: https://issues.apache.org/jira/browse/LOG4J2-3425
> Project: Log4j 2
> Issue Type: Improvement
> Components: Appenders
>Affects Versions: 2.12.1
>Reporter: Jiří Smolík
>Priority: Major
> Fix For: 2.17.3
>
>
> Recently, we've had an issue with our syslog server (collector), where it was
> suddenly unable to forward messages sent to it. This resulted in flooding the
> server and ultimately, exhausting its resources (memory and allocated
> temporary storage).
> However, more interestingly, Log4j2 appender behaved as follows:
> # At the moment of exhaustion, one thread (T1) entered the synchronized
> block in TcpSocketManager (see
> https://github.com/apache/logging-log4j2/blob/e2ee2f4dd9327eb1841023c3269ba12f5673e694/log4j-core/src/main/java/org/apache/logging/log4j/core/net/TcpSocketManager.java#L217)
> and tried to write the message (see
> [https://github.com/apache/logging-log4j2/blob/e2ee2f4dd9327eb1841023c3269ba12f5673e694/log4j-core/src/main/java/org/apache/logging/log4j/core/net/TcpSocketManager.java#L253]).
> # The server didn't respond immediately - actually, it blocked T1 for hours
> (it was most likely waiting for resources to be free).
> # In the meantime, all threads attempting to write a message to the syslog
> appender hanged at the synchronized block (they were waiting for T1 to
> finish).
> Eventually, the connection to the server broke apart or some resources were
> freed, and T1 finished. Interestingly, it didn't throw any exception, but
> that is beside the point in this issue.
> So, what happened is that the server got flooded and blocked one thread in
> our application, but due to the way TcpSocketManager is implemented, our
> application suffered a DoS.
> As a precaution to such dreadful scenario, I would like to request adding a
> read timeout setting to the TcpSocketManager (also syslog appender), which
> would in case of the above described situation prevent T1 from blocking other
> threads for a long time. At least, that's my idea of an easy fix. Connect and
> reconnect timeout settings have a different purpose and were useless in this
> case.
> I am aware that the issue is also a matter of the syslog server
> configuration, but application developers do not always have it under their
> control.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (LOG4J2-3425) Syslog appender lacks the SocketOptions setting
[ https://issues.apache.org/jira/browse/LOG4J2-3425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17500937#comment-17500937 ] Ralph Goers commented on LOG4J2-3425: - I actually experienced this very problem with the SocketAppender. See LOG4J2-3130. As I investigated this I realized that the none of the socket options actually cover the case of a write hanging. According to [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=4031100] NIO must be used to handle this. > Syslog appender lacks the SocketOptions setting > --- > > Key: LOG4J2-3425 > URL: https://issues.apache.org/jira/browse/LOG4J2-3425 > Project: Log4j 2 > Issue Type: Improvement > Components: Appenders >Affects Versions: 2.12.1 >Reporter: Jiří Smolík >Priority: Major > Fix For: 2.17.3 > > > Recently, we've had an issue with our syslog server (collector), where it was > suddenly unable to forward messages sent to it. This resulted in flooding the > server and ultimately, exhausting its resources (memory and allocated > temporary storage). > However, more interestingly, Log4j2 appender behaved as follows: > # At the moment of exhaustion, one thread (T1) entered the synchronized > block in TcpSocketManager (see > https://github.com/apache/logging-log4j2/blob/e2ee2f4dd9327eb1841023c3269ba12f5673e694/log4j-core/src/main/java/org/apache/logging/log4j/core/net/TcpSocketManager.java#L217) > and tried to write the message (see > [https://github.com/apache/logging-log4j2/blob/e2ee2f4dd9327eb1841023c3269ba12f5673e694/log4j-core/src/main/java/org/apache/logging/log4j/core/net/TcpSocketManager.java#L253]). > # The server didn't respond immediately - actually, it blocked T1 for hours > (it was most likely waiting for resources to be free). > # In the meantime, all threads attempting to write a message to the syslog > appender hanged at the synchronized block (they were waiting for T1 to > finish). > Eventually, the connection to the server broke apart or some resources were > freed, and T1 finished. Interestingly, it didn't throw any exception, but > that is beside the point in this issue. > So, what happened is that the server got flooded and blocked one thread in > our application, but due to the way TcpSocketManager is implemented, our > application suffered a DoS. > As a precaution to such dreadful scenario, I would like to request adding a > read timeout setting to the TcpSocketManager (also syslog appender), which > would in case of the above described situation prevent T1 from blocking other > threads for a long time. At least, that's my idea of an easy fix. Connect and > reconnect timeout settings have a different purpose and were useless in this > case. > I am aware that the issue is also a matter of the syslog server > configuration, but application developers do not always have it under their > control. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (LOG4J2-3425) Syslog appender lacks the SocketOptions setting
[
https://issues.apache.org/jira/browse/LOG4J2-3425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17500899#comment-17500899
]
Gary D. Gregory commented on LOG4J2-3425:
-
[~SkyCrawl]
Dang it, well that's lame, sorry about that. I fixed this in the git branch
{{release-2.x}} and our Maven snapshot repository
https://repository.apache.org/content/repositories/snapshots for version
*2.17.3-SNAPSHOT*
Please give it a try.
> Syslog appender lacks the SocketOptions setting
> ---
>
> Key: LOG4J2-3425
> URL: https://issues.apache.org/jira/browse/LOG4J2-3425
> Project: Log4j 2
> Issue Type: Improvement
> Components: Appenders
>Affects Versions: 2.12.1
>Reporter: Jiří Smolík
>Priority: Major
>
> Recently, we've had an issue with our syslog server (collector), where it was
> suddenly unable to forward messages sent to it. This resulted in flooding the
> server and ultimately, exhausting its resources (memory and allocated
> temporary storage).
> However, more interestingly, Log4j2 appender behaved as follows:
> # At the moment of exhaustion, one thread (T1) entered the synchronized
> block in TcpSocketManager (see
> https://github.com/apache/logging-log4j2/blob/e2ee2f4dd9327eb1841023c3269ba12f5673e694/log4j-core/src/main/java/org/apache/logging/log4j/core/net/TcpSocketManager.java#L217)
> and tried to write the message (see
> [https://github.com/apache/logging-log4j2/blob/e2ee2f4dd9327eb1841023c3269ba12f5673e694/log4j-core/src/main/java/org/apache/logging/log4j/core/net/TcpSocketManager.java#L253]).
> # The server didn't respond immediately - actually, it blocked T1 for hours
> (it was most likely waiting for resources to be free).
> # In the meantime, all threads attempting to write a message to the syslog
> appender hanged at the synchronized block (they were waiting for T1 to
> finish).
> Eventually, the connection to the server broke apart or some resources were
> freed, and T1 finished. Interestingly, it didn't throw any exception, but
> that is beside the point in this issue.
> So, what happened is that the server got flooded and blocked one thread in
> our application, but due to the way TcpSocketManager is implemented, our
> application suffered a DoS.
> As a precaution to such dreadful scenario, I would like to request adding a
> read timeout setting to the TcpSocketManager (also syslog appender), which
> would in case of the above described situation prevent T1 from blocking other
> threads for a long time. At least, that's my idea of an easy fix. Connect and
> reconnect timeout settings have a different purpose and were useless in this
> case.
> I am aware that the issue is also a matter of the syslog server
> configuration, but application developers do not always have it under their
> control.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (LOG4J2-3425) Syslog appender lacks the SocketOptions setting
[ https://issues.apache.org/jira/browse/LOG4J2-3425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17500895#comment-17500895 ] ASF subversion and git services commented on LOG4J2-3425: - Commit dc1742c0ffb51cf85ae690e91f7661af525e4c86 in logging-log4j2's branch refs/heads/release-2.x from Gary Gregory [ https://gitbox.apache.org/repos/asf?p=logging-log4j2.git;h=dc1742c ] [LOG4J2-3425] Syslog appender lacks the SocketOptions setting. > Syslog appender lacks the SocketOptions setting > --- > > Key: LOG4J2-3425 > URL: https://issues.apache.org/jira/browse/LOG4J2-3425 > Project: Log4j 2 > Issue Type: Improvement > Components: Appenders >Affects Versions: 2.12.1 >Reporter: Jiří Smolík >Priority: Major > > Recently, we've had an issue with our syslog server (collector), where it was > suddenly unable to forward messages sent to it. This resulted in flooding the > server and ultimately, exhausting its resources (memory and allocated > temporary storage). > However, more interestingly, Log4j2 appender behaved as follows: > # At the moment of exhaustion, one thread (T1) entered the synchronized > block in TcpSocketManager (see > https://github.com/apache/logging-log4j2/blob/e2ee2f4dd9327eb1841023c3269ba12f5673e694/log4j-core/src/main/java/org/apache/logging/log4j/core/net/TcpSocketManager.java#L217) > and tried to write the message (see > [https://github.com/apache/logging-log4j2/blob/e2ee2f4dd9327eb1841023c3269ba12f5673e694/log4j-core/src/main/java/org/apache/logging/log4j/core/net/TcpSocketManager.java#L253]). > # The server didn't respond immediately - actually, it blocked T1 for hours > (it was most likely waiting for resources to be free). > # In the meantime, all threads attempting to write a message to the syslog > appender hanged at the synchronized block (they were waiting for T1 to > finish). > Eventually, the connection to the server broke apart or some resources were > freed, and T1 finished. Interestingly, it didn't throw any exception, but > that is beside the point in this issue. > So, what happened is that the server got flooded and blocked one thread in > our application, but due to the way TcpSocketManager is implemented, our > application suffered a DoS. > As a precaution to such dreadful scenario, I would like to request adding a > read timeout setting to the TcpSocketManager (also syslog appender), which > would in case of the above described situation prevent T1 from blocking other > threads for a long time. At least, that's my idea of an easy fix. Connect and > reconnect timeout settings have a different purpose and were useless in this > case. > I am aware that the issue is also a matter of the syslog server > configuration, but application developers do not always have it under their > control. -- This message was sent by Atlassian Jira (v8.20.1#820001)
