Have you considered posting on vendor forum ? FYI
On Mon, Sep 14, 2015 at 6:09 AM, Renu Yadav <yren...@gmail.com> wrote: > > ---------- Forwarded message ---------- > From: Renu Yadav <yren...@gmail.com> > Date: Mon, Sep 14, 2015 at 4:51 PM > Subject: Spark job failed > To: d...@spark.apache.org > > > I am getting below error while running spark job: > > storage.DiskBlockObjectWriter: Uncaught exception while reverting partial > writes to file > /data/vol5/hadoop/yarn/local/usercache/renu_yadav/appcache/application_1438196554863_31545/spark-4686a622-82be-418e-a8b0-1653458bc8cb/22/temp_shuffle_8c437ba7-55d2-4520-80ec-adcfe932b3bd > java.io.FileNotFoundException: > /data/vol5/hadoop/yarn/local/usercache/renu_yadav/appcache/application_1438196554863_31545/spark-4686a622-82be-418e-a8b0-1653458bc8cb/22/temp_shuffle_8c437ba7-55d2-4520-80ec-adcfe932b3bd > (No such file or directory > > > > I am running 1.3TB data > following are the transformation > > read from hadoop->map(key/value).coalease(2000).groupByKey. > then sorting each record by server_ts and select most recent > > saving data into parquet. > > > Following is the command > spark-submit --class com.test.Myapp--master yarn-cluster --driver-memory > 16g --executor-memory 20g --executor-cores 5 --num-executors 150 > --files /home/renu_yadav/fmyapp/hive-site.xml --conf > spark.yarn.preserve.staging.files=true --conf > spark.shuffle.memoryFraction=0.6 --conf spark.storage.memoryFraction=0.1 > --conf SPARK_SUBMIT_OPTS="-XX:MaxPermSize=768m" --conf > SPARK_SUBMIT_OPTS="-XX:MaxPermSize=768m" --conf > spark.akka.timeout=400000 --conf spark.locality.wait=10 --conf > spark.yarn.executor.memoryOverhead=8000 --conf > SPARK_JAVA_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps" > --conf spark.reducer.maxMbInFlight=96 --conf > spark.shuffle.file.buffer.kb=64 --conf > spark.core.connection.ack.wait.timeout=120 --jars > /usr/hdp/2.2.6.0-2800/hive/lib/datanucleus-api-jdo-3.2.6.jar,/usr/hdp/2.2.6.0-2800/hive/lib/datanucleus-core-3.2.10.jar,/usr/hdp/2.2.6.0-2800/hive/lib/datanucleus-rdbms-3.2.9.jar > myapp_2.10-1.0.jar > > > > > > > > Cluster configuration > > 20 Nodes > 32 cores per node > 125 GB ram per node > > > Please Help. > > Thanks & Regards, > Renu Yadav > >
