Shardul Mahadik created SPARK-52737:
---------------------------------------
Summary: SHS performance regression when visiting homepage
Key: SPARK-52737
URL: https://issues.apache.org/jira/browse/SPARK-52737
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 4.0.0, 3.5.0, 4.1.0
Reporter: Shardul Mahadik
SPARK-38896 modified how applications are listed from the KVStore to close the
KVStore iterator eagerly
[Link|https://github.com/apache/spark/pull/36237/files#diff-128a6af0d78f4a6180774faedb335d6168dfc4defff58f5aa3021fc1bd767bc0R328].
This meant that FsHistoryProvider.getListing now eagerly goes through every
application in the KVStore before returning an iterator to the caller. In a
couple of contexts where FsHistoryProvider.getListing is used, this is very
detrimental. e.g.
[here|https://github.com/apache/spark/blame/589e93a02725939c266f9ee97f96fdc6d3db33cd/core/src/main/scala/org/apache/spark/deploy/history/HistoryPage.scala#L112],
due to .exists(), we would previously only need to go through a handful of
applications before the condition is satisfied. This causes significant perf
regression for the SHS homepage in our environment which contains ~10000s Spark
apps in a single history server.
To fix the issue, while preserving the original intent of closing the iterator
early, I propose pushing down filter predicates and number of applications
required to FsHistoryProvider.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
