Hi,
I am running spark jobs using apache oozie in yarn-client mode.
My job.properties has sparkConf which gets used in workflow.xml.
I have tried increasing MaxPermSize using sparkConf in job.properties
but that is not resolving the issue.
*sparkConf*=--verbose --driver-java-options '-XX:MaxPermSize=8192M' --conf
spark.speculation=false --conf
spark.hadoop.spark.sql.parquet.output.committer.class=\
"org.apache.spark.sql.parquet.DirectParquetOutputCommitter" --conf
spark.hadoop.mapred.output.committer.class="org.apache.hadoop.mapred.DirectFileOutputCommit\
ter.class" --conf spark.hadoop.mapreduce.use.directfileoutputcommitter=true
Am I missing anything ?
I am seeing following errors.
2016-08-03 22:33:43,318 WARN SparkActionExecutor:523 -
SERVER[ip-10-0-0-161.ec2.internal] USER[hadoop] GROUP[-] TOKEN[]
APP[ApprouteOozie] JOB[0000031-160803180548580-oozie-oozi-W]
ACTION[0000031-160803180548580-oozie-oozi-W@spark-approute] Launcher ERROR,
reason: Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw
exception, PermGen space
2016-08-03 22:33:43,319 WARN SparkActionExecutor:523 -
SERVER[ip-10-0-0-161.ec2.internal] USER[hadoop] GROUP[-] TOKEN[]
APP[ApprouteOozie] JOB[0000031-160803180548580-oozie-oozi-W]
ACTION[0000031-160803180548580-oozie-oozi-W@spark-approute] Launcher
exception: PermGen space
java.lang.OutOfMemoryError: PermGen space
oozie-oozi-W@spark-approute] Launcher exception: PermGen space
java.lang.OutOfMemoryError: PermGen space
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2595)
at java.lang.Class.getConstructor0(Class.java:2895)
at java.lang.Class.newInstance(Class.java:354)
at
sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:399)
at
sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:396)
at java.security.AccessController.doPrivileged(Native Method)
at
sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:395)
at
sun.reflect.MethodAccessorGenerator.generateSerializationConstructor(MethodAccessorGenerator.java:113)
at
sun.reflect.ReflectionFactory.newConstructorForSerialization(ReflectionFactory.java:331)
at
java.io.ObjectStreamClass.getSerializableConstructor(ObjectStreamClass.java:1420)
at java.io.ObjectStreamClass.access$1500(ObjectStreamClass.java:72)
at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:497)
at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472)
at java.security.AccessController.doPrivileged(Native Method)
at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:472)
at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1133)
at
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
at
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
at
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
at
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
at
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
at
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1377)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1173)
--
-$iddhi.
