[ https://issues.apache.org/jira/browse/SPARK-15569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-15569: ------------------------------------ Assignee: Apache Spark > Executors spending significant time in DiskObjectWriter.updateBytesWritten > function > ----------------------------------------------------------------------------------- > > Key: SPARK-15569 > URL: https://issues.apache.org/jira/browse/SPARK-15569 > Project: Spark > Issue Type: Bug > Components: Shuffle > Reporter: Sital Kedia > Assignee: Apache Spark > > Profiling a Spark job spilling large amount of intermediate data we found > that significant portion of time is being spent in > DiskObjectWriter.updateBytesWritten function. Looking at the code > (https://github.com/sitalkedia/spark/blob/master/core/src/main/scala/org/apache/spark/storage/DiskBlockObjectWriter.scala#L206), > we see that the function is being called too frequently to update the number > of bytes written to disk. We should reduce the frequency to avoid this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org