Yan-Lin (Jared) Wang created SPARK-43782:
--------------------------------------------
Summary: Support log level configuration with static Spark conf
Key: SPARK-43782
URL: https://issues.apache.org/jira/browse/SPARK-43782
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 3.4.0
Reporter: Yan-Lin (Jared) Wang
Today users are using {{sc.setLogLevel}} to configure the current log level
using {{SparkContext}} accessible from within the {{{}SparkSession{}}}. It is a
convenient alternative to directly depending on the Log4j mechanism like
statically changing the {{log4j.xml}} or dynamically invoke the Log4j API.
It would be great to allow the configuration of the log level using a static
Spark conf that has the same kind of simplicity, which comes in handy when we
don't want to manually change the {{{}log4j.xml{}}}. This is particularly
helpful in the Spark Connect scenario where there’s no way for the client to
call {{setLogLevel}} because {{SparkContext}} is not yet supported in its API,
and when it connects to a platform where it can't easily change the Log4j
config file.
./bin/pyspark --conf spark.log.level ERROR
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
