GitHub user jerryshao opened a pull request: https://github.com/apache/spark/pull/20144
[SPARK-21475][CORE][2nd attempt] Change to use NIO's Files API for external shuffle service ## What changes were proposed in this pull request? This PR is the second attempt of #18684 , NIO's Files API doesn't override `skip` method for `InputStream`, so it will bring in performance issue (mentioned in #20119). But using `FileInputStream`/`FileOutputStream` will also bring in memory issue (https://dzone.com/articles/fileinputstream-fileoutputstream-considered-harmful), which is severe for long running external shuffle service. So here in this proposal, only fixing the external shuffle service related code. ## How was this patch tested? Existing tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jerryshao/apache-spark SPARK-21475-v2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/20144.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #20144 ---- commit 277b801b3ccee95d8f30c02b1ba8693ab39f302a Author: jerryshao <sshao@...> Date: 2018-01-04T01:48:55Z Change to NIO's file API for external shuffle service ---- --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org