This is an automated email from the ASF dual-hosted git repository.
kabhwan pushed a commit to branch branch-3.5
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.5 by this push:
new 19c8001dacb [SPARK-44683][SS] Logging level isn't passed to RocksDB
state store provider correctly
19c8001dacb is described below
commit 19c8001dacb2933ef0a052babdd976fa886b8b4d
Author: Siying Dong <siying.d...@databricks.com>
AuthorDate: Tue Aug 8 11:11:56 2023 +0900
[SPARK-44683][SS] Logging level isn't passed to RocksDB state store
provider correctly
### What changes were proposed in this pull request?
The logging level is passed into RocksDB in a correct way.
### Why are the changes needed?
We pass log4j's log level to RocksDB so that RocksDB debug log can go to
log4j. However, we pass in log level after we create the logger. However, the
way it is set isn't effective. This has two impacts: (1) setting DEBUG level
don't make RocksDB generate DEBUG level logs; (2) setting WARN or ERROR level
does prevent INFO level logging, but RocksDB still makes JNI calls to Scala,
which is an unnecessary overhead.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manually change the log level and observe the log lines in unit tests.
Closes #42354 from siying/rocks_log_level.
Authored-by: Siying Dong <siying.d...@databricks.com>
Signed-off-by: Jungtaek Lim <kabhwan.opensou...@gmail.com>
(cherry picked from commit 630b1777904f15c7ac05c3cd61c0006cd692bc93)
Signed-off-by: Jungtaek Lim <kabhwan.opensou...@gmail.com>
---
.../org/apache/spark/sql/execution/streaming/state/RocksDB.scala | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
index 386df61a9e0..2398b778072 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
@@ -613,8 +613,11 @@ class RocksDB(
if (log.isWarnEnabled) dbLogLevel = InfoLogLevel.WARN_LEVEL
if (log.isInfoEnabled) dbLogLevel = InfoLogLevel.INFO_LEVEL
if (log.isDebugEnabled) dbLogLevel = InfoLogLevel.DEBUG_LEVEL
- dbOptions.setLogger(dbLogger)
+ dbLogger.setInfoLogLevel(dbLogLevel)
+ // The log level set in dbLogger is effective and the one to dbOptions
isn't applied to
+ // customized logger. We still set it as it might show up in RocksDB
config file or logging.
dbOptions.setInfoLogLevel(dbLogLevel)
+ dbOptions.setLogger(dbLogger)
logInfo(s"Set RocksDB native logging level to $dbLogLevel")
dbLogger
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
