[
https://issues.apache.org/jira/browse/SPARK-23253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599714#comment-16599714
]
Imran Rashid commented on SPARK-23253:
--------------------------------------
I think I see the issue you are referring to [~cloud_fan], but I'm not sure
this change is actually the responsible one. Isn't it really from here
https://github.com/apache/spark/pull/9610 ? the change here just changed
whether we bother to write {{lengths}} to a file, but doesn't actually change
whether we use that file at all.
There is more history discussing that change (and non-determinism etc.) here
https://github.com/apache/spark/pull/9214 and
https://github.com/apache/spark/pull/6648
> Only write shuffle temporary index file when there is not an existing one
> -------------------------------------------------------------------------
>
> Key: SPARK-23253
> URL: https://issues.apache.org/jira/browse/SPARK-23253
> Project: Spark
> Issue Type: Improvement
> Components: Shuffle, Spark Core
> Affects Versions: 2.2.1
> Reporter: Kent Yao
> Assignee: Kent Yao
> Priority: Major
> Fix For: 2.4.0
>
>
> Shuffle Index temporay file is used for atomic creating shuffle index file,
> it is not needed when the index file already exists after another attempts of
> same task had it done.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
