[
https://issues.apache.org/jira/browse/SPARK-59059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated SPARK-59059:
-------------------------------
Description:
SPARK-43782 added `spark.log.level`, which sets the level of the *root* logger
at startup, as a static-conf equivalent of `SparkContext.setLogLevel()`. There
is still no static-conf way to set the level of an individual logger.
Today, turning on DEBUG for one class means shipping a `log4j2.properties` and
pointing both the driver and the executors at it:
{code:java}
spark-submit \
--files /tmp/log4j2.properties#log4j2.properties \
--conf
spark.driver.extraJavaOptions=-Dlog4j.configurationFile=log4j2.properties \
--conf
spark.executor.extraJavaOptions=-Dlog4j.configurationFile=log4j2.properties \
...
{code}
That is awkward in cluster mode and impossible for a Spark Connect client,
which has no `SparkContext` and often cannot change the log4j configuration of
the platform it connects to, the same motivation that was accepted for
SPARK-43782.
Proposal: support *spark.log.level.<logger>*, applied on the driver and on the
executors after the root level from `spark.log.level`, so it takes precedence
for the named logger only.
{code:java}
--conf spark.log.level=WARN
--conf spark.log.level.org.apache.spark.sql.execution=DEBUG
{code}
Note: SPARK-16784 asked for a version of this and was bulk-closed as Incomplete
in 2021 without a fix.
was:
SPARK-43782 added `spark.log.level`, which sets the level of the *root* logger
at startup, as a static-conf equivalent of `SparkContext.setLogLevel()`. There
is still no static-conf way to set the level of an individual logger.
Today, turning on DEBUG for one class means shipping a `log4j2.properties` and
pointing both the driver and the executors at it:
{code:java}
spark-submit \
--files /tmp/log4j2.properties#log4j2.properties \
--conf
spark.driver.extraJavaOptions=-Dlog4j.configurationFile=log4j2.properties \
--conf
spark.executor.extraJavaOptions=-Dlog4j.configurationFile=log4j2.properties \
...
{code}
That is awkward in cluster mode and impossible for a Spark Connect client,
which has no `SparkContext` and often cannot change the log4j configuration of
the platform it connects to, the same motivation that was accepted for
SPARK-43782.
Proposal: support `spark.log.level.<logger>`, applied on the driver and on the
executors after the root level from `spark.log.level`, so it takes precedence
for the named logger only.
{code:java}
--conf spark.log.level=WARN
--conf spark.log.level.org.apache.spark.sql.execution=DEBUG
{code}
Note: SPARK-16784 asked for a version of this and was bulk-closed as Incomplete
in 2021 without a fix.
> Support per-logger log level configuration with static Spark conf
> -----------------------------------------------------------------
>
> Key: SPARK-59059
> URL: https://issues.apache.org/jira/browse/SPARK-59059
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 4.2.0
> Reporter: John Zhuge
> Priority: Trivial
>
> SPARK-43782 added `spark.log.level`, which sets the level of the *root*
> logger at startup, as a static-conf equivalent of
> `SparkContext.setLogLevel()`. There is still no static-conf way to set the
> level of an individual logger.
> Today, turning on DEBUG for one class means shipping a `log4j2.properties`
> and pointing both the driver and the executors at it:
> {code:java}
> spark-submit \
> --files /tmp/log4j2.properties#log4j2.properties \
> --conf
> spark.driver.extraJavaOptions=-Dlog4j.configurationFile=log4j2.properties \
> --conf
> spark.executor.extraJavaOptions=-Dlog4j.configurationFile=log4j2.properties \
> ...
> {code}
> That is awkward in cluster mode and impossible for a Spark Connect client,
> which has no `SparkContext` and often cannot change the log4j configuration
> of the platform it connects to, the same motivation that was accepted for
> SPARK-43782.
> Proposal: support *spark.log.level.<logger>*, applied on the driver and on
> the executors after the root level from `spark.log.level`, so it takes
> precedence for the named logger only.
> {code:java}
> --conf spark.log.level=WARN
> --conf spark.log.level.org.apache.spark.sql.execution=DEBUG
> {code}
>
> Note: SPARK-16784 asked for a version of this and was bulk-closed as
> Incomplete in 2021 without a fix.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
