baohe-zhang commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r440489973
##########
File path:
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -1167,6 +1172,17 @@ private[history] class FsHistoryProvider(conf:
SparkConf, clock: Clock)
// At this point the disk data either does not exist or was deleted
because it failed to
// load, so the event log needs to be replayed.
+ // If hybrid store is enabled, try it first.
+ if (hybridStoreEnabled) {
+ try {
+ return createHybridStore(dm, appId, attempt, metadata)
+ } catch {
+ case e: Exception =>
+ logInfo(s"Failed to create HybridStore for
$appId/${attempt.info.attemptId}." +
+ " Using LevelDB.", e)
+ }
+ }
+
Review comment:
Addressed. Split the logic below to a new function createLevelDBStore().
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -195,4 +195,20 @@ private[spark] object History {
.version("3.0.0")
.booleanConf
.createWithDefault(true)
+
+ val HYBRID_STORE_ENABLED =
ConfigBuilder("spark.history.store.hybridStore.enabled")
+ .doc("Whether to use HybridStore as the store when parsing event logs. " +
+ "HybridStore will first write data to an in-memory store and having a
background thread " +
+ "that dumps data to a disk store after the writing to in-memory store is
completed.")
+ .version("3.1.0")
+ .booleanConf
+ .createWithDefault(false)
+
+ val MAX_IN_MEMORY_STORE_USAGE =
ConfigBuilder("spark.history.store.hybridStore.maxMemoryUsage")
Review comment:
monitoring.md updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
