[
https://issues.apache.org/jira/browse/SPARK-48575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arnaud Nauwynck updated SPARK-48575:
------------------------------------
Attachment: EventLogFileReaders.patch
> spark.history.fs.update.interval calling too many directory pollings when
> spark log dir contains many sparkEvent apps
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-48575
> URL: https://issues.apache.org/jira/browse/SPARK-48575
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.3.3, 4.0.0, 3.5.1, 3.4.3
> Reporter: Arnaud Nauwynck
> Priority: Critical
> Attachments: EventLogFileReaders.patch
>
>
> In the case of a sparkLog dir containing "lot" of spark eventLogs sub-dirs
> (example 1000),
> running a supposedly "iddle" Sparkhistory server is causing millions of
> directory listing calls each hour.
> see code :
> [https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala#L283|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala#L283]
> example: with ~1000 apps, every 10 seconds (default of
> "spark.history.fs.update.interval") SparkHistory is performing
> - 1x VirtualFileSystem.listStatus(path) with path=sparkLog dir
> - then 2x foreach each appSubDirPath (corresponding to a sparkApp eventLogs)
> => 2 x 1000 x VirtualFileSystem.listStatus(appSubDirPath)
> On a cloud provider (example Azure), this cost a lot per month :
> because "List FileSystem" calls ~$0.065 per 10000 ops for Tier "Hot" or
> $0.0228 for "Premium" (cf
> https://azure.microsoft.com/en-us/pricing/details/storage/data-lake/ )
> Let's do the multiplications:
> 30 (days per month) * 86400 (sec per day) / 10 (interval second) = 259 000
> update times
> ... * 2001 (listings ops per update) = 518 millions listing calls per month
> ... * 0.0228 / 10000 = 1182 USD/month !!!!
> Admitedly, the retention conf "spark.history.fs.cleaner.maxAge" (default
> =7d) for spark eventLog is too much for workflows than run many short spark
> apps, and it would be possible to reduce it.
> It is extremely important to reduce these recurrent costs
> Here are several whishes
> 1/ fix "bug" in Spark History that is calling twice the
> VirtualFileSystem.listStatus(appSubDirPath).
> cf source code: [first call
> EventLogFileReaders.scala#L123|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/history/EventLogFileReaders.scala#L123]
> : it only test that the dir contains a child file with name prefix
> "eventLog" and an appStatus file, but then the list is unused.
> It creates an instance of RollingEventLogFilesFileReader, and shortly after,
> the listing is called again:
> cf [second call (lazy field)
> EventLogFileReaders.scala#L224|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/history/EventLogFileReaders.scala#L224]
> the lazy field "files" is immediatly evaluated after object creation from
> here:
> [second called from
> EventLogFileReaders.scala#L252|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/history/EventLogFileReaders.scala#L252]
> .. [called from FsHistoryProvider.scala#L506
> |https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala#L506]
> Indeed, it is easilly possible to perform only 1 listing per sub-dir
> (cf attach patch, changing ~5 lines of code)
> This would divide cost by x2.
> 2/ in addition to conf "spark.history.fs.cleaner.maxAge", add another conf
> param ""spark.history.fs.cleaner.maxCount" to limit the number of spark apps.
> This could be defaulted to ~50.
> This would additionaly divide cost by x10 (in case you have 1000 apps).
> 3/ change the code in SparkHistory to check lazily for update only on demand
> when someone click in Spark History web UI. For example, if the last cached
> update time is less than "spark.history.fs.update.interval" then no update is
> needed, else update is immediatly performed and cached before returning
> response.
> 4/ change the code in SparkHistory to avoid doing a listing on each app
> sub-dir.
> It is possible to perform a single listing on "sparkLog" top level dir, to
> discover new apps.
> Then for each app subdir, most of them are already finished, and already
> recompacted by SparkHistory itself. This info is already stored in spark
> history Keystore db.
> Allmost all the listing sub-dirs can thefore be completly avoided.
> see [KVStore declaration
> |https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala#L141],
> [KVStore
> ApplicationInfo|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala#L1194]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
