Yes agreed.It seems to be issue with mapping the text file contents to case classes, not sure though.
On Thu, Aug 4, 2016 at 8:17 PM, $iddhe$h Divekar <siddhesh.dive...@gmail.com > wrote: > Hi Deepak, > > My files are always > 50MB. > I would think there would be a small config to overcome this. > Tried almost everything i could after searching online. > > Any help from the mailing list would be appreciated. > > On Thu, Aug 4, 2016 at 7:43 AM, Deepak Sharma <deepakmc...@gmail.com> > wrote: > >> I am facing the same issue with spark 1.5.2 >> If the file size that's being processed by spark , is of size 10-12 MB , >> it throws out of memory . >> But if the same file is within 5 MB limit , it runs fine. >> I am using spark configuration with 7GB of memory and 3 cores for >> executors in the cluster of 8 executor. >> >> Thanks >> Deepak >> >> On 4 Aug 2016 8:04 pm, "$iddhe$h Divekar" <siddhesh.dive...@gmail.com> >> wrote: >> >>> Hi, >>> >>> I am running spark jobs using apache oozie in yarn-client mode. >>> My job.properties has sparkConf which gets used in workflow.xml. >>> >>> I have tried increasing MaxPermSize using sparkConf in job.properties >>> but that is not resolving the issue. >>> >>> *sparkConf*=--verbose --driver-java-options '-XX:MaxPermSize=8192M' >>> --conf spark.speculation=false --conf spark.hadoop.spark.sql. >>> parquet.output.committer.class=\ >>> "org.apache.spark.sql.parquet.DirectParquetOutputCommitter" --conf >>> spark.hadoop.mapred.output.committer.class="org.apache.hadoop.mapred. >>> DirectFileOutputCommit\ >>> ter.class" --conf spark.hadoop.mapreduce.use. >>> directfileoutputcommitter=true >>> >>> Am I missing anything ? >>> >>> I am seeing following errors. >>> >>> 2016-08-03 22:33:43,318 WARN SparkActionExecutor:523 - >>> SERVER[ip-10-0-0-161.ec2.internal] USER[hadoop] GROUP[-] TOKEN[] >>> APP[ApprouteOozie] JOB[0000031-160803180548580-oozie-oozi-W] >>> ACTION[0000031-160803180548580-oozie-oozi-W@spark-approute] Launcher >>> ERROR, reason: Main class [org.apache.oozie.action.hadoop.SparkMain], >>> main() threw exception, PermGen space >>> 2016-08-03 22:33:43,319 WARN SparkActionExecutor:523 - >>> SERVER[ip-10-0-0-161.ec2.internal] USER[hadoop] GROUP[-] TOKEN[] >>> APP[ApprouteOozie] JOB[0000031-160803180548580-oozie-oozi-W] >>> ACTION[0000031-160803180548580-oozie-oozi-W@spark-approute] Launcher >>> exception: PermGen space >>> java.lang.OutOfMemoryError: PermGen space >>> >>> oozie-oozi-W@spark-approute] Launcher exception: PermGen space >>> java.lang.OutOfMemoryError: PermGen space >>> at java.lang.Class.getDeclaredConstructors0(Native Method) >>> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2595) >>> at java.lang.Class.getConstructor0(Class.java:2895) >>> at java.lang.Class.newInstance(Class.java:354) >>> at sun.reflect.MethodAccessorGenerator$1.run( >>> MethodAccessorGenerator.java:399) >>> at sun.reflect.MethodAccessorGenerator$1.run( >>> MethodAccessorGenerator.java:396) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at sun.reflect.MethodAccessorGenerator.generate( >>> MethodAccessorGenerator.java:395) >>> at sun.reflect.MethodAccessorGenerator. >>> generateSerializationConstructor(MethodAccessorGenerator.java:113) >>> at sun.reflect.ReflectionFactory.newConstructorForSerialization >>> (ReflectionFactory.java:331) >>> at java.io.ObjectStreamClass.getSerializableConstructor( >>> ObjectStreamClass.java:1420) >>> at java.io.ObjectStreamClass.access$1500(ObjectStreamClass.java:72) >>> at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:497) >>> at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:472) >>> at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369) >>> at java.io.ObjectOutputStream.writeObject0( >>> ObjectOutputStream.java:1133) >>> at java.io.ObjectOutputStream.defaultWriteFields( >>> ObjectOutputStream.java:1547) >>> at java.io.ObjectOutputStream.writeSerialData( >>> ObjectOutputStream.java:1508) >>> at java.io.ObjectOutputStream.writeOrdinaryObject( >>> ObjectOutputStream.java:1431) >>> at java.io.ObjectOutputStream.writeObject0( >>> ObjectOutputStream.java:1177) >>> at java.io.ObjectOutputStream.defaultWriteFields( >>> ObjectOutputStream.java:1547) >>> at java.io.ObjectOutputStream.writeSerialData( >>> ObjectOutputStream.java:1508) >>> at java.io.ObjectOutputStream.writeOrdinaryObject( >>> ObjectOutputStream.java:1431) >>> at java.io.ObjectOutputStream.writeObject0( >>> ObjectOutputStream.java:1177) >>> at java.io.ObjectOutputStream.defaultWriteFields( >>> ObjectOutputStream.java:1547) >>> at java.io.ObjectOutputStream.writeSerialData( >>> ObjectOutputStream.java:1508) >>> at java.io.ObjectOutputStream.writeOrdinaryObject( >>> ObjectOutputStream.java:1431) >>> at java.io.ObjectOutputStream.writeObject0( >>> ObjectOutputStream.java:1177) >>> at java.io.ObjectOutputStream.writeArray(ObjectOutputStream. >>> java:1377) >>> at java.io.ObjectOutputStream.writeObject0( >>> ObjectOutputStream.java:1173) >>> >>> -- >>> -$iddhi. >>> >> > > > -- > -$iddhi. > -- Thanks Deepak www.bigdatabig.com www.keosha.net
