Or you can increase the driver heap space (export _JAVA_OPTIONS="-Xmx5g")
Thanks Best Regards On Wed, Sep 2, 2015 at 11:57 PM, Mike Hynes <91m...@gmail.com> wrote: > Just a thought; this has worked for me before on standalone client > with a similar OOM error in a driver thread. Try setting: > export SPARK_DAEMON_MEMORY=4G #or whatever size you can afford on your > machine > in your environment/spark-env.sh before running spark-submit. > Mike > > On 9/2/15, ankit tyagi <ankittyagi.mn...@gmail.com> wrote: > > Hi All, > > > > I am using spark-sql 1.3.1 with hadoop 2.4.0 version. I am running sql > > query against parquet files and wanted to save result on s3 but looks > like > > https://issues.apache.org/jira/browse/SPARK-2984 problem still coming > while > > saving data to s3. > > > > Hence Now i am saving result on hdfs and with the help > > of JavaSparkListener, copying file from hdfs to s3 with hadoop fileUtil > > in onApplicationEnd method. But my job is getting failed with OOM in > spark > > driver. > > > > *5/09/02 04:17:57 INFO cluster.YarnClusterSchedulerBackend: Asking each > > executor to shut down* > > *15/09/02 04:17:59 INFO > > scheduler.OutputCommitCoordinator$OutputCommitCoordinatorActor: > > OutputCommitCoordinator stopped!* > > *Exception in thread "Reporter" * > > *Exception: java.lang.OutOfMemoryError thrown from the > > UncaughtExceptionHandler in thread "Reporter"* > > *Exception in thread "SparkListenerBus" * > > *Exception: java.lang.OutOfMemoryError thrown from the > > UncaughtExceptionHandler in thread "SparkListenerBus"* > > *Exception in thread "Driver" * > > *Exception: java.lang.OutOfMemoryError thrown from the > > UncaughtExceptionHandler in thread "Driver"* > > > > > > Strage part is, result is getting saved on HDFS but while copying file > job > > is getting failed. size of file is under 1MB. > > > > Any help or leads would be appreciated. > > > > > -- > Thanks, > Mike > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > For additional commands, e-mail: dev-h...@spark.apache.org > >
