[
https://issues.apache.org/jira/browse/SPARK-52737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-52737:
----------------------------------
Parent: SPARK-54249
Issue Type: Sub-task (was: Bug)
> SHS performance regression when visiting homepage
> -------------------------------------------------
>
> Key: SPARK-52737
> URL: https://issues.apache.org/jira/browse/SPARK-52737
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 3.4.2, 3.4.0, 3.4.1, 3.5.0, 3.5.1, 3.5.2, 3.4.3, 3.4.4,
> 3.5.3, 3.5.4, 3.5.5, 4.1.0, 3.5.6, 4.0.0
> Reporter: Shardul Mahadik
> Assignee: Shardul Mahadik
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.1.0, 4.0.1, 3.5.7
>
>
> SPARK-38896 modified how applications are listed from the KVStore to close
> the KVStore iterator eagerly
> [Link|https://github.com/apache/spark/pull/36237/files#diff-128a6af0d78f4a6180774faedb335d6168dfc4defff58f5aa3021fc1bd767bc0R328].
> This meant that FsHistoryProvider.getListing now eagerly goes through every
> application in the KVStore before returning an iterator to the caller. In a
> couple of contexts where FsHistoryProvider.getListing is used, this is very
> detrimental. e.g.
> [here|https://github.com/apache/spark/blame/589e93a02725939c266f9ee97f96fdc6d3db33cd/core/src/main/scala/org/apache/spark/deploy/history/HistoryPage.scala#L112],
> due to .exists(), we would previously only need to go through a handful of
> applications before the condition is satisfied. This causes significant perf
> regression for the SHS homepage in our environment which contains ~10000s
> Spark apps in a single history server.
> To fix the issue, while preserving the original intent of closing the
> iterator early, I propose pushing down filter predicates and number of
> applications required to FsHistoryProvider.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
