Re: Spark jobs failing due to java.lang.OutOfMemoryError: PermGen space
Hi Deepak, My files are always > 50MB. I would think there would be a small config to overcome this. Tried almost everything i could after searching online. Any help from the mailing list would be appreciated. On Thu, Aug 4, 2016 at 7:43 AM, Deepak Sharma <deepakmc...@gmail.com> wrote: > I am facing the same issue with spark 1.5.2 > If the file size that's being processed by spark , is of size 10-12 MB , > it throws out of memory . > But if the same file is within 5 MB limit , it runs fine. > I am using spark configuration with 7GB of memory and 3 cores for > executors in the cluster of 8 executor. > > Thanks > Deepak > > On 4 Aug 2016 8:04 pm, "$iddhe$h Divekar" <siddhesh.dive...@gmail.com> > wrote: > >> Hi, >> >> I am running spark jobs using apache oozie in yarn-client mode. >> My job.properties has sparkConf which gets used in workflow.xml. >> >> I have tried increasing MaxPermSize using sparkConf in job.properties >> but that is not resolving the issue. >> >> *sparkConf*=--verbose --driver-java-options '-XX:MaxPermSize=8192M' >> --conf spark.speculation=false --conf >> spark.hadoop.spark.sql.parquet.output.committer.class=\ >> "org.apache.spark.sql.parquet.DirectParquetOutputCommitter" --conf >> spark.hadoop.mapred.output.committer.class="org.apache.hadoop.mapred.DirectFileOutputCommit\ >> ter.class" --conf >> spark.hadoop.mapreduce.use.directfileoutputcommitter=true >> >> Am I missing anything ? >> >> I am seeing following errors. >> >> 2016-08-03 22:33:43,318 WARN SparkActionExecutor:523 - >> SERVER[ip-10-0-0-161.ec2.internal] USER[hadoop] GROUP[-] TOKEN[] >> APP[ApprouteOozie] JOB[031-160803180548580-oozie-oozi-W] >> ACTION[031-160803180548580-oozie-oozi-W@spark-approute] Launcher >> ERROR, reason: Main class [org.apache.oozie.action.hadoop.SparkMain], >> main() threw exception, PermGen space >> 2016-08-03 22:33:43,319 WARN SparkActionExecutor:523 - >> SERVER[ip-10-0-0-161.ec2.internal] USER[hadoop] GROUP[-] TOKEN[] >> APP[ApprouteOozie] JOB[031-160803180548580-oozie-oozi-W] >> ACTION[031-160803180548580-oozie-oozi-W@spark-approute] Launcher >> exception: PermGen space >> java.lang.OutOfMemoryError: PermGen space >> >> oozie-oozi-W@spark-approute] Launcher exception: PermGen space >> java.lang.OutOfMemoryError: PermGen space >> at java.lang.Class.getDeclaredConstructors0(Native Method) >> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2595) >> at java.lang.Class.getConstructor0(Class.java:2895) >> at java.lang.Class.newInstance(Class.java:354) >> at >> sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:399) >> at >> sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:396) >> at java.security.AccessController.doPrivileged(Native Method) >> at >> sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:395) >> at >> sun.reflect.MethodAccessorGenerator.generateSerializationConstructor(MethodAccessorGenerator.java:113) >> at >> sun.reflect.ReflectionFactory.newConstructorForSerialization(ReflectionFactory.java:331) >> at >> java.io.ObjectStreamClass.getSerializableConstructor(ObjectStreamClass.java:1420) >> at java.io.ObjectStreamClass.access$1500(ObjectStreamClass.java:72) >> at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:497) >> at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.io.ObjectStreamClass.(ObjectStreamClass.java:472) >> at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369) >> at >> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1133) >> at >> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) >> at >> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) >> at >> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) >> at >> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) >> at >> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) >> at >> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) >> at >> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) >> at >> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) >> at >> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) >> at >> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) >> at >> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) >> at >> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) >> at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1377) >> at >> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1173) >> >> -- >> -$iddhi. >> > -- -$iddhi.
Spark jobs failing due to java.lang.OutOfMemoryError: PermGen space
Hi, I am running spark jobs using apache oozie in yarn-client mode. My job.properties has sparkConf which gets used in workflow.xml. I have tried increasing MaxPermSize using sparkConf in job.properties but that is not resolving the issue. *sparkConf*=--verbose --driver-java-options '-XX:MaxPermSize=8192M' --conf spark.speculation=false --conf spark.hadoop.spark.sql.parquet.output.committer.class=\ "org.apache.spark.sql.parquet.DirectParquetOutputCommitter" --conf spark.hadoop.mapred.output.committer.class="org.apache.hadoop.mapred.DirectFileOutputCommit\ ter.class" --conf spark.hadoop.mapreduce.use.directfileoutputcommitter=true Am I missing anything ? I am seeing following errors. 2016-08-03 22:33:43,318 WARN SparkActionExecutor:523 - SERVER[ip-10-0-0-161.ec2.internal] USER[hadoop] GROUP[-] TOKEN[] APP[ApprouteOozie] JOB[031-160803180548580-oozie-oozi-W] ACTION[031-160803180548580-oozie-oozi-W@spark-approute] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, PermGen space 2016-08-03 22:33:43,319 WARN SparkActionExecutor:523 - SERVER[ip-10-0-0-161.ec2.internal] USER[hadoop] GROUP[-] TOKEN[] APP[ApprouteOozie] JOB[031-160803180548580-oozie-oozi-W] ACTION[031-160803180548580-oozie-oozi-W@spark-approute] Launcher exception: PermGen space java.lang.OutOfMemoryError: PermGen space oozie-oozi-W@spark-approute] Launcher exception: PermGen space java.lang.OutOfMemoryError: PermGen space at java.lang.Class.getDeclaredConstructors0(Native Method) at java.lang.Class.privateGetDeclaredConstructors(Class.java:2595) at java.lang.Class.getConstructor0(Class.java:2895) at java.lang.Class.newInstance(Class.java:354) at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:399) at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:396) at java.security.AccessController.doPrivileged(Native Method) at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:395) at sun.reflect.MethodAccessorGenerator.generateSerializationConstructor(MethodAccessorGenerator.java:113) at sun.reflect.ReflectionFactory.newConstructorForSerialization(ReflectionFactory.java:331) at java.io.ObjectStreamClass.getSerializableConstructor(ObjectStreamClass.java:1420) at java.io.ObjectStreamClass.access$1500(ObjectStreamClass.java:72) at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:497) at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472) at java.security.AccessController.doPrivileged(Native Method) at java.io.ObjectStreamClass.(ObjectStreamClass.java:472) at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1133) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1377) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1173) -- -$iddhi.
Re: PermGen space Error
I am running the spark job in yarn-client mode. On Wed, Aug 3, 2016 at 8:14 PM, $iddhe$h Divekar <siddhesh.dive...@gmail.com > wrote: > Hi, > > I am running spark jobs using apache oozie. > My job.properties has sparkConf which gets used in workflow.xml. > > I have tried increasing MaxPermSize using sparkConf in job.properties > but that is not resolving the issue. > > *sparkConf*=--verbose --driver-java-options '-XX:MaxPermSize=8192M' > --conf spark.speculation=false --conf > spark.hadoop.spark.sql.parquet.output.committer.class=\ > "org.apache.spark.sql.parquet.DirectParquetOutputCommitter" --conf > spark.hadoop.mapred.output.committer.class="org.apache.hadoop.mapred.DirectFileOutputCommit\ > ter.class" --conf spark.hadoop.mapreduce.use.directfileoutputcommitter=true > > Am I missing anything ? > > I am seeing following errors. > > 2016-08-03 22:33:43,318 WARN SparkActionExecutor:523 - > SERVER[ip-10-0-0-161.ec2.internal] USER[hadoop] GROUP[-] TOKEN[] > APP[ApprouteOozie] JOB[031-160803180548580-oozie-oozi-W] > ACTION[031-160803180548580-oozie-oozi-W@spark-approute] Launcher > ERROR, reason: Main class [org.apache.oozie.action.hadoop.SparkMain], > main() threw exception, PermGen space > 2016-08-03 22:33:43,319 WARN SparkActionExecutor:523 - > SERVER[ip-10-0-0-161.ec2.internal] USER[hadoop] GROUP[-] TOKEN[] > APP[ApprouteOozie] JOB[031-160803180548580-oozie-oozi-W] > ACTION[031-160803180548580-oozie-oozi-W@spark-approute] Launcher > exception: PermGen space > java.lang.OutOfMemoryError: PermGen space > > oozie-oozi-W@spark-approute] Launcher exception: PermGen space > java.lang.OutOfMemoryError: PermGen space > at java.lang.Class.getDeclaredConstructors0(Native Method) > at java.lang.Class.privateGetDeclaredConstructors(Class.java:2595) > at java.lang.Class.getConstructor0(Class.java:2895) > at java.lang.Class.newInstance(Class.java:354) > at > sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:399) > at > sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:396) > at java.security.AccessController.doPrivileged(Native Method) > at > sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:395) > at > sun.reflect.MethodAccessorGenerator.generateSerializationConstructor(MethodAccessorGenerator.java:113) > at > sun.reflect.ReflectionFactory.newConstructorForSerialization(ReflectionFactory.java:331) > at > java.io.ObjectStreamClass.getSerializableConstructor(ObjectStreamClass.java:1420) > at java.io.ObjectStreamClass.access$1500(ObjectStreamClass.java:72) > at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:497) > at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472) > at java.security.AccessController.doPrivileged(Native Method) > at java.io.ObjectStreamClass.(ObjectStreamClass.java:472) > at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369) > at > java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1133) > at > java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) > at > java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) > at > java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) > at > java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) > at > java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) > at > java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) > at > java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) > at > java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) > at > java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) > at > java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) > at > java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) > at > java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) > at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1377) > at > java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1173) > > -- > -$iddhi. > -- -$iddhi.
PermGen space Error
Hi, I am running spark jobs using apache oozie. My job.properties has sparkConf which gets used in workflow.xml. I have tried increasing MaxPermSize using sparkConf in job.properties but that is not resolving the issue. *sparkConf*=--verbose --driver-java-options '-XX:MaxPermSize=8192M' --conf spark.speculation=false --conf spark.hadoop.spark.sql.parquet.output.committer.class=\ "org.apache.spark.sql.parquet.DirectParquetOutputCommitter" --conf spark.hadoop.mapred.output.committer.class="org.apache.hadoop.mapred.DirectFileOutputCommit\ ter.class" --conf spark.hadoop.mapreduce.use.directfileoutputcommitter=true Am I missing anything ? I am seeing following errors. 2016-08-03 22:33:43,318 WARN SparkActionExecutor:523 - SERVER[ip-10-0-0-161.ec2.internal] USER[hadoop] GROUP[-] TOKEN[] APP[ApprouteOozie] JOB[031-160803180548580-oozie-oozi-W] ACTION[031-160803180548580-oozie-oozi-W@spark-approute] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, PermGen space 2016-08-03 22:33:43,319 WARN SparkActionExecutor:523 - SERVER[ip-10-0-0-161.ec2.internal] USER[hadoop] GROUP[-] TOKEN[] APP[ApprouteOozie] JOB[031-160803180548580-oozie-oozi-W] ACTION[031-160803180548580-oozie-oozi-W@spark-approute] Launcher exception: PermGen space java.lang.OutOfMemoryError: PermGen space oozie-oozi-W@spark-approute] Launcher exception: PermGen space java.lang.OutOfMemoryError: PermGen space at java.lang.Class.getDeclaredConstructors0(Native Method) at java.lang.Class.privateGetDeclaredConstructors(Class.java:2595) at java.lang.Class.getConstructor0(Class.java:2895) at java.lang.Class.newInstance(Class.java:354) at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:399) at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:396) at java.security.AccessController.doPrivileged(Native Method) at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:395) at sun.reflect.MethodAccessorGenerator.generateSerializationConstructor(MethodAccessorGenerator.java:113) at sun.reflect.ReflectionFactory.newConstructorForSerialization(ReflectionFactory.java:331) at java.io.ObjectStreamClass.getSerializableConstructor(ObjectStreamClass.java:1420) at java.io.ObjectStreamClass.access$1500(ObjectStreamClass.java:72) at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:497) at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472) at java.security.AccessController.doPrivileged(Native Method) at java.io.ObjectStreamClass.(ObjectStreamClass.java:472) at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1133) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1377) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1173) -- -$iddhi.
