[
https://issues.apache.org/jira/browse/SPARK-44683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siying Dong updated SPARK-44683:
--------------------------------
Description: We pass log4j's log level to RocksDB so that RocksDB debug log
can go to log4j. However, we pass in log level after we create the logger.
However, the way it is set isn't effective. This has two impacts: (1) setting
DEBUG level don't make RocksDB generate DEBUG level logs; (2) setting WARN or
ERROR level does prevent INFO level logging, but RocksDB still makes JNI calls
to Scala, which is an unnecessary overhead. (was: We pass log4j's log level to
RocksDB so that RocksDB debug log can go to log4j. However, we pass in log
level after we create the logger. However, RocksDB only takes log level when a
logger is created, so it never changes. This has two impacts: (1) setting DEBUG
level don't make RocksDB generate DEBUG level logs; (2) setting WARN or ERROR
level does prevent INFO level logging, but RocksDB still makes JNI calls to
Scala, which is an unnecessary overhead.)
> Logging level isn't passed to RocksDB state store provider correctly
> --------------------------------------------------------------------
>
> Key: SPARK-44683
> URL: https://issues.apache.org/jira/browse/SPARK-44683
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 3.4.1
> Reporter: Siying Dong
> Priority: Minor
>
> We pass log4j's log level to RocksDB so that RocksDB debug log can go to
> log4j. However, we pass in log level after we create the logger. However, the
> way it is set isn't effective. This has two impacts: (1) setting DEBUG level
> don't make RocksDB generate DEBUG level logs; (2) setting WARN or ERROR level
> does prevent INFO level logging, but RocksDB still makes JNI calls to Scala,
> which is an unnecessary overhead.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
