[jira] [Comment Edited] (SPARK-6305) Add support for log4j 2.x to Spark
[ https://issues.apache.org/jira/browse/SPARK-6305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17466876#comment-17466876 ] Steve Loughran edited comment on SPARK-6305 at 12/30/21, 6:44 PM: -- If anyone wants a version of a log4j 1.17 without the known (and never used) CVE, you could grab the Cloudera patched JAR. ASF projects are not allowed to release their own builds of other projects, so I'm afraid you are not allowed to include this in Apache releases. https://mvnrepository.com/artifact/log4j/log4j/1.2.17-cloudera1 You don't need to move to log4j 2, and if you are, now is a good time to look at alternatives e.g. logback. I suspect this is what Hadoop will pick up in the new year, was (Author: ste...@apache.org): If anyone wants a version of a log4j 1.17 without the known (and never used) CVE, you could grab the Cloudera patched JAR. ASF projects are not allowed to release their own builds of other projects, so I'm afraid you were not allowed her to include this in Apache releases. https://mvnrepository.com/artifact/log4j/log4j/1.2.17-cloudera1 You don't need to move to log4j 2, and if you are, now is a good time to look at alternatives e.g. logback. I suspect this is what Hadoop will pick up in the new year, > Add support for log4j 2.x to Spark > -- > > Key: SPARK-6305 > URL: https://issues.apache.org/jira/browse/SPARK-6305 > Project: Spark > Issue Type: Improvement > Components: Build >Reporter: Tal Sliwowicz >Assignee: L. C. Hsieh >Priority: Minor > Fix For: 3.3.0 > > > log4j 2 requires replacing the slf4j binding and adding the log4j jars in the > classpath. Since there are shaded jars, it must be done during the build. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-6305) Add support for log4j 2.x to Spark
[ https://issues.apache.org/jira/browse/SPARK-6305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17042761#comment-17042761 ] Ralph Goers edited comment on SPARK-6305 at 2/23/20 12:01 AM: -- [~ste...@apache.org] Regarding your comment on Sept 17, 2019 - [CVE-2019-17571|https://nvd.nist.gov/vuln/detail/CVE-2019-17571] was created recently against Log4j 1. It is essentially the same as the CVE you noted above against Log4j 2. It was created specifically because people were confused in thinking that CVE-2017-5645 did not apply to Log4j 1. CVE-2019-17571 has been mitigated in third party distributions of Log4j but will never be fixed in an ASF distribution, so any use of Log4j 1 will now permanently show up in security scans, although some projects (ZOOKEEPER-3677) are choosing to suppress the security failure. Also note that Log4j 2 now offers [experimental support|http://logging.apache.org/log4j/2.x/manual/compatibility.html] for Log4j 1 configuration files. was (Author: ralph.go...@dslextreme.com): [~ste...@apache.org] Regarding your comment on Sept 17, 2019 - [CVE-2019-17571|https://nvd.nist.gov/vuln/detail/CVE-2019-17571] was created recently against Log4j 1. It is essentially the same as the CVE you noted above against Log4j 2. It was created specifically because people were confused in thinking that CVE-2017-5645 did not apply to Log4j 1. CVE-2019-17571 has been mitigated in third party distributions of Log4j but will never be fixedin an ASF distribution, so any use of Log4j 1 will now permanently show up in security scans, although some projects (ZOOKEEPER-3677) are choosing to suppress the security failure. Also note that Log4j 2 now offers [experimental support|http://logging.apache.org/log4j/2.x/manual/compatibility.html] for Log4j 1 configuration files. > Add support for log4j 2.x to Spark > -- > > Key: SPARK-6305 > URL: https://issues.apache.org/jira/browse/SPARK-6305 > Project: Spark > Issue Type: Improvement > Components: Build >Reporter: Tal Sliwowicz >Priority: Minor > > log4j 2 requires replacing the slf4j binding and adding the log4j jars in the > classpath. Since there are shaded jars, it must be done during the build. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-6305) Add support for log4j 2.x to Spark
[ https://issues.apache.org/jira/browse/SPARK-6305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16938467#comment-16938467 ] Rajiv Bandi edited comment on SPARK-6305 at 9/26/19 10:20 AM: -- Steve, apart from the security issue concern, we also have the logs not rolling properly. In any case, my question is "Is Spark going to decouple itself from log4j 1.x and provide a way to configure logging with an alternative implementation?". If yes, what would be the tentative timeline? was (Author: rajivbandi): Steve, apart from the security issue concern, we also have the logs not rolling properly. In any case, my question is "Is Spark going to decouple itself from log4j 1.x and provided a way to configure logging with an alternative implementation?". If yes, what would be the tentative timeline? > Add support for log4j 2.x to Spark > -- > > Key: SPARK-6305 > URL: https://issues.apache.org/jira/browse/SPARK-6305 > Project: Spark > Issue Type: Improvement > Components: Build >Reporter: Tal Sliwowicz >Priority: Minor > > log4j 2 requires replacing the slf4j binding and adding the log4j jars in the > classpath. Since there are shaded jars, it must be done during the build. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-6305) Add support for log4j 2.x to Spark
[ https://issues.apache.org/jira/browse/SPARK-6305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523888#comment-16523888 ] Hari Sekhon edited comment on SPARK-6305 at 6/26/18 3:47 PM: - Log4j 2.x would really help with Spark logging integration to ELK as there are a lot of things that just don't work properly in Log4j 1.x like layout.ConversionPattern for constructing JSON enriched logs, such as logging user and app names to distinguish jobs and provide much needed search usability. This is simply ignored in the SocketAppender in Log4j 1.x, while SyslogAppender respects ConversionPattern but then splits all Java Exceptions in to multiple syslog logs so the JSON no longer parses and routes to the right indices for the Yarn queue, nor can you reassemble the exception logs using multiline codec at the other end as you'd end up with corrupted input streams from multiple loggers) :-/ Running Filebeats everywhere instead seems like overkill compared to being able to enable logging for debugging jobs on an ad-hoc basis to a Logstash sink that works using much better Log4j 2.x output appenders. I hope someone finally manages to sort this out as it's years overdue given Log4j 1.x was end of life 3 years ago and there is a big jump in capabilities between Log4j 1.x and 2.x, both in the number of appenders as well as completeness of even the old appenders such as the SocketAppender as mentioned above. was (Author: harisekhon): Log4j 2.x would really help with Spark logging integration to ELK as there are a lot of things that just don't work properly in Log4j 1.x like layout.ConversionPattern for constructing JSON enriched logs, such as logging user and app names to distinguish jobs and provide much needed search usability. This is simply ignored in the SocketAppender in Log4j 1.x :-/ while SyslogAppender respects ConversionPattern but then splits all Java Exceptions in to multiple syslog logs so the JSON no longer parses and routes to the right indices for the Yarn queue, nor can you reassemble the exception logs using multiline codec at the other end as you'd end up with corrupted input streams from multiple loggers) :-/ Running Filebeats everywhere instead seems like overkill compared to being able to enable logging for debugging jobs on an ad-hoc basis to a Logstash sink that works using much better Log4j 2.x output appenders. I hope someone finally manages to sort this out as it's years overdue given Log4j 1.x was end of life 3 years ago and there is a big jump in capabilities between Log4j 1.x and 2.x, both in the number of appenders as well as completeness of even the old appenders such as the SocketAppender as mentioned above. > Add support for log4j 2.x to Spark > -- > > Key: SPARK-6305 > URL: https://issues.apache.org/jira/browse/SPARK-6305 > Project: Spark > Issue Type: Improvement > Components: Build >Reporter: Tal Sliwowicz >Priority: Minor > > log4j 2 requires replacing the slf4j binding and adding the log4j jars in the > classpath. Since there are shaded jars, it must be done during the build. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-6305) Add support for log4j 2.x to Spark
[ https://issues.apache.org/jira/browse/SPARK-6305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964554#comment-15964554 ] Dan Dutrow edited comment on SPARK-6305 at 4/11/17 4:18 PM: Any update on the current status of this ticket? It would be particularly beneficial to our program, and presumably many others, if the KafkaAppender (available in log4j 2.x) could be provided to spark workers for real-time log aggregation. https://logging.apache.org/log4j/2.x/manual/appenders.html#KafkaAppender This is not an easy thing to override in a spark application due to native log4j loading with the CoarseGrainedExecutorBackend before the application. was (Author: dutrow): Please provide an update on the current status of this ticket? It would be particularly beneficial to our program, and presumably many others, if the KafkaAppender (available in log4j 2.x) could be provided to spark workers for real-time log aggregation. https://logging.apache.org/log4j/2.x/manual/appenders.html#KafkaAppender This is not an easy thing to override in a spark application due to native log4j loading with the CoarseGrainedExecutorBackend before the application. > Add support for log4j 2.x to Spark > -- > > Key: SPARK-6305 > URL: https://issues.apache.org/jira/browse/SPARK-6305 > Project: Spark > Issue Type: Improvement > Components: Build >Reporter: Tal Sliwowicz >Priority: Minor > > log4j 2 requires replacing the slf4j binding and adding the log4j jars in the > classpath. Since there are shaded jars, it must be done during the build. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
