GitHub user themodernlife opened a pull request: https://github.com/apache/spark/pull/2450
[SPARK-3595] Respect configured OutputCommitters when calling saveAsHadoopFile Addresses the issue in https://issues.apache.org/jira/browse/SPARK-3595, namely saveAsHadoopFile hardcoding the OutputCommitter. This is not ideal when running Spark jobs that write to S3, especially when running them from an EMR cluster where the default OutputCommitter is a DirectOutputCommitter. You can merge this pull request into a Git repository by running: $ git pull https://github.com/themodernlife/spark spark-3595 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2450.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2450 ---- commit 8b6be94801ca33bca32aa574b1a8f6a76760869d Author: Ian Hummel <i...@themodernlife.net> Date: 2014-09-15T15:38:59Z Add ability to specify OutputCommitter, espcially useful when writing to an S3 bucket from an EMR cluster commit 4359664b1d557d55b0579023df809542386d5b8c Author: Ian Hummel <i...@themodernlife.net> Date: 2014-09-18T20:18:57Z Add an example showing usage commit a11d9f3806e6a8d06d13417af9f27bfd3795334b Author: Ian Hummel <i...@themodernlife.net> Date: 2014-09-18T20:52:17Z Fix formatting ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org