[ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15508076#comment-15508076 ]
Josh Rosen commented on SPARK-17562: ------------------------------------ I'm not sure that "zero objects written" necessarily implies "no bytes written to the disk output stream" because it's possible that the act of opening the serialization stream could have written a "start-of-stream" header. Given this, I'm not comfortable removing this check. > I think a little code is unnecessary to exist in > ExternalSorter.spillMemoryIteratorToDisk > ----------------------------------------------------------------------------------------- > > Key: SPARK-17562 > URL: https://issues.apache.org/jira/browse/SPARK-17562 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 2.0.0 > Reporter: Jianfei Wang > Priority: Trivial > Labels: easyfix, performance > > In ExternalSorter.spillMemoryIteratorToDisk, I think the code below will > never be executed, so we can remove them > {code} > else { > writer.revertPartialWritersAndClose() > } > {code} > the source code is as below: > {code} > try { > while (inMemoryIterator.hasNext) { > val partitionId = inMemoryIterator.nextPartition() > require(partitionId >= 0 && partitionId < numPartitions, > s"partition Id: ${partitionId} should be in the range [0, > ${numPartitions})") > inMemoryIterator.writeNext(writer) > elementsPerPartition(partitionId) += 1 > objectsWritten += 1 > if (objectsWritten == serializerBatchSize) { > flush() > } > } > if (objectsWritten > 0) { > flush() > } else { > writer.revertPartialWritesAndClose() > } > success = true > } finally { > if (success) { > writer.close() > } else { > // This code path only happens if an exception was thrown above > before we set success; > // close our stuff and let the exception be thrown further > writer.revertPartialWritesAndClose() > if (file.exists()) { > if (!file.delete()) { > logWarning(s"Error deleting ${file}") > } > } > } > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org