[
https://issues.apache.org/jira/browse/SPARK-44646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17758552#comment-17758552
]
Franziska Iltzsche commented on SPARK-44646:
--------------------------------------------
Hi all,
we are running into the same problem, and I believe the problem is not in the
forwarding library but in Spark itself.
When initializing the logging in
[Logging.scala|[https://github.com/apache/spark/blob/v3.3.1/core/src/main/scala/org/apache/spark/internal/Logging.scala#L232C22-L232C22]],
the log4j implementation (dependency
org.apache.logging.log4j.log4j-slf4j-impl) is required.
However, this dependency also needs to be excluded in order to successfully use
logging via logback.
I was able to run our program with logback logging in debug mode, by skipping
the code line liked above. Wrapping the call to
“StaticLoggerBinder.getSingleton.getLoggerFactoryClassStr” in a try block and
return an empty string if the class is not known would probably solve the
problem.
It would be great if this issue could be fixed soon, as it blocks us from
upgrading to Spark version 3.3.0 and newer.
Thanks.
> Migrate Log4j 2.x in Spark 3.4.1 to Logback
> -------------------------------------------
>
> Key: SPARK-44646
> URL: https://issues.apache.org/jira/browse/SPARK-44646
> Project: Spark
> Issue Type: Brainstorming
> Components: Build
> Affects Versions: 3.4.1
> Reporter: Yu Tian
> Priority: Major
> Attachments: Screenshot 2023-08-09 at 2.40.12 PM.png
>
>
> Hi,
> We are working on the spark 3.4.1 upgrade from spark 3.1.3, in our logging
> system, we are using logback framework, it is working with spark 3.1.3 since
> it is using log4j 1.x. However, when we upgrade spark to 3.4.1, based on the
> [release
> notes|https://spark.apache.org/docs/latest/core-migration-guide.html], spark
> is migrating from log4j 2.x from log4j 1.x, the way we are replacing the
> log4j with logback is causing build failures in spark master start process.
> Error: Unable to initialize main class org.apache.spark.deploy.master.Master
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/logging/log4j/core/Filter
> In our current approach, we are using log4j-over-slf4j to replace the
> log4j-core, it is only applicable to log4j 1.x library. And there is no
> log4j-over-slf4j for log4j 2.x out there yet. (please correct me if I am
> wrong).
> I am also curious that why spark choose to use log4j 2.x instead of using
> SPI, which gives the users less flexibility to choose whatever logger
> implementation they want to use.
> I want to share this issue and see if anyone else has been reported this and
> if there is any work-around or alternative solutions for it. Any suggestions
> are appreciated, thanks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
