GitHub user ConeyLiu opened a pull request:
https://github.com/apache/spark/pull/22371
[SPARK-25386][CORE] Don't need to synchronize the IndexShuffleBlockResolver
for each writeIndexFileAndCommit
## What changes were proposed in this pull request?
Now, we need synchronize the instance of IndexShuffleBlockResolver in order
to make the commit check and tmp file rename atomically. This can be improved.
We could synchronize a lock which is different for each `shuffleId + mapId`
instead of synchronize the indexShuffleBlockResolver for each
writeIndexFileAndCommit.
This should be an optimization with space for time, but it doesn't take up
a lot of space.
## How was this patch tested?
Existing UT.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ConeyLiu/spark indexShuffleBlockResolver
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22371.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22371
----
commit 92c2e07171f60b977c62661ea6475486a1599b19
Author: Xianyang Liu <xianyang.liu@...>
Date: 2018-09-09T10:44:23Z
don't need synchronized the IndexShuffleBlockResolver for each
writeIndexFileAndCommit
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
