[
https://issues.apache.org/jira/browse/SPARK-60759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuqin Luo updated SPARK-60759:
------------------------------
Issue Type: Improvement (was: Bug)
> SQLAppStatusListener.aggregateMetrics has O(N^2) cost when merging driver
> accumulator updates
> ---------------------------------------------------------------------------------------------
>
> Key: SPARK-60759
> URL: https://issues.apache.org/jira/browse/SPARK-60759
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.0.6
> Reporter: Yuqin Luo
> Priority: Major
>
> During event-log replay, Spark's {{SQLAppStatusListener.aggregateMetrics()}}
> repeatedly copies the accumulated metric arrays, which becomes extremely
> expensive for these apps.
> A similar O(N²) issue exists in LiveExecutionData.driverAccumUpdates, where
> repeated List ++ rebuilds the collection on every event.
> h2.
> *FIX:* Commit 1 — Group updates by ID and merge them once, avoiding repeated
> array copies, reducing aggregation from O(N²) to O(N).
> Commit 2 — Change {{LiveExecutionData.driverAccumUpdates}} from *List* to
> *Vector* so new updates can be appended more efficiently.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
