Sital Kedia created SPARK-15569:
-----------------------------------
Summary: Executors spending significant time in
DiskObjectWriter.updateBytesWritten function
Key: SPARK-15569
URL: https://issues.apache.org/jira/browse/SPARK-15569
Project: Spark
Issue Type: Bug
Components: Shuffle
Reporter: Sital Kedia
Profiling a Spark job spilling large amount of intermediate data we found that
significant portion of time is being spent in
DiskObjectWriter.updateBytesWritten function. Looking at the code
(https://github.com/sitalkedia/spark/blob/master/core/src/main/scala/org/apache/spark/storage/DiskBlockObjectWriter.scala#L206),
we see that the function is being called too frequently to update the number
of bytes written to disk. We should reduce the frequency to avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
