Yuming Wang created SPARK-35251:
-----------------------------------
Summary: Improve LiveEntityHelpers.newAccumulatorInfos performace
Key: SPARK-35251
URL: https://issues.apache.org/jira/browse/SPARK-35251
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 3.2.0
Reporter: Yuming Wang
[It|https://github.com/apache/spark/blob/3854ad87c78f2a331f9c9c1a34f9ec281900f8fe/core/src/main/scala/org/apache/spark/status/LiveEntity.scala#L646-L661]
will impact performance if there are lot of {{AccumulableInfo}} instances.
{noformat}
num #instances #bytes class name
----------------------------------------------
1: 331897850 50708371592 [C
2: 795358 24591381896 [J
3: 330836234 10586759488 java.lang.String
4: 139726297 8942483008
org.apache.spark.sql.execution.metric.SQLMetric
5: 249040156 5976963744 scala.Some
6: 146278719 5851148760 org.apache.spark.util.AccumulatorMetadata
7: 38448113 5536528272 java.net.URI
8: 37540162 3603855552 org.apache.hadoop.fs.FileStatus
9: 69724130 3346758240 java.util.Hashtable$Entry
10: 61521559 2953034832 java.util.concurrent.ConcurrentHashMap$Node
11: 50421974 2823630544 scala.collection.mutable.LinkedEntry
12: 43349222 2774350208 org.apache.spark.scheduler.AccumulableInfo
{noformat}
{noformat}
--- 15430388364 ns (2.03%), 1543 samples
[ 0] scala.collection.TraversableLike.noneIn$1
[ 1] scala.collection.TraversableLike.filterImpl
[ 2] scala.collection.TraversableLike.filterImpl$
[ 3] scala.collection.AbstractTraversable.filterImpl
[ 4] scala.collection.TraversableLike.filter
[ 5] scala.collection.TraversableLike.filter$
[ 6] scala.collection.AbstractTraversable.filter
[ 7] org.apache.spark.status.LiveEntityHelpers$.newAccumulatorInfos
[ 8] org.apache.spark.status.LiveTask.doUpdate
[ 9] org.apache.spark.status.LiveEntity.write
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
