Hi All, I am using spark-sql 1.3.1 with hadoop 2.4.0 version. I am running sql query against parquet files and wanted to save result on s3 but looks like https://issues.apache.org/jira/browse/SPARK-2984 problem still coming while saving data to s3.
Hence Now i am saving result on hdfs and with the help of JavaSparkListener, copying file from hdfs to s3 with hadoop fileUtil in onApplicationEnd method. But my job is getting failed with OOM in spark driver. *5/09/02 04:17:57 INFO cluster.YarnClusterSchedulerBackend: Asking each executor to shut down* *15/09/02 04:17:59 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorActor: OutputCommitCoordinator stopped!* *Exception in thread "Reporter" * *Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Reporter"* *Exception in thread "SparkListenerBus" * *Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "SparkListenerBus"* *Exception in thread "Driver" * *Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Driver"* Strage part is, result is getting saved on HDFS but while copying file job is getting failed. size of file is under 1MB. Any help or leads would be appreciated.
