[
https://issues.apache.org/jira/browse/SPARK-36324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17388539#comment-17388539
]
Apache Spark commented on SPARK-36324:
--------------------------------------
User 'LuciferYang' has created a pull request for this issue:
https://github.com/apache/spark/pull/33556
> Replace revertPartialWritesAndClose with close in ExternalSorter.spill and
> ExternalAppendOnlyMap.spill
> ------------------------------------------------------------------------------------------------------
>
> Key: SPARK-36324
> URL: https://issues.apache.org/jira/browse/SPARK-36324
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.3.0
> Reporter: Yang Jie
> Priority: Minor
>
> ExternalAppendOnlyMap.spill method call `revertPartialWritesAndClose` method
> when `objectsWritten == 0` as follows:
> {code:java}
> try {
> while (inMemoryIterator.hasNext) {
> ...
> if (objectsWritten == serializerBatchSize) {
> flush()
> }
> }
> if (objectsWritten > 0) {
> flush()
> writer.close()
> } else {
> writer.revertPartialWritesAndClose()
> }
> success = true
> } finally {
> ...
> }{code}
>
> writer.revertPartialWritesAndClose() can replace with writer.close to reduce
> a set of file operations includes open, truncate and close.
>
> A similar situation exists for ExternalSorter
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
