Jianfei Wang created SPARK-17562:
------------------------------------
Summary: I think a little code is unnecessary to exist in
ExternalSorter.spillMemoryIteratorToDisk
Key: SPARK-17562
URL: https://issues.apache.org/jira/browse/SPARK-17562
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 2.0.0
Reporter: Jianfei Wang
Priority: Trivial
In ExternalSorter.spillMemoryIteratorToDisk, I think the code below will never
be executed, so we can remove them
else {
writer.revertPartialWritersAndClose()
}
the source code is as below:
{code}
try {
while (inMemoryIterator.hasNext) {
val partitionId = inMemoryIterator.nextPartition()
require(partitionId >= 0 && partitionId < numPartitions,
s"partition Id: ${partitionId} should be in the range [0,
${numPartitions})")
inMemoryIterator.writeNext(writer)
elementsPerPartition(partitionId) += 1
objectsWritten += 1
if (objectsWritten == serializerBatchSize) {
flush()
}
}
if (objectsWritten > 0) {
flush()
} else {
writer.revertPartialWritesAndClose()
}
success = true
} finally {
if (success) {
writer.close()
} else {
// This code path only happens if an exception was thrown above before
we set success;
// close our stuff and let the exception be thrown further
writer.revertPartialWritesAndClose()
if (file.exists()) {
if (!file.delete()) {
logWarning(s"Error deleting ${file}")
}
}
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
