You should not disable the GC overhead limit. How does increasing executor total memory cause you to not have enough memory? Do you mean something else? On Jan 30, 2015 1:16 AM, "ey-chih chow" <eyc...@hotmail.com> wrote:
> I use the default value, which I think is 512MB. If I change to 1024MB, > Spark submit will fail due to not enough memory for rdd. > > Ey-Chih Chow > > ------------------------------ > From: moham...@glassbeam.com > To: eyc...@hotmail.com; user@spark.apache.org > Subject: RE: unknown issue in submitting a spark job > Date: Fri, 30 Jan 2015 00:32:57 +0000 > > How much memory are you assigning to the Spark executor on the worker > node? > > > > Mohammed > > > > *From:* ey-chih chow [mailto:eyc...@hotmail.com] > *Sent:* Thursday, January 29, 2015 3:35 PM > *To:* Mohammed Guller; user@spark.apache.org > *Subject:* RE: unknown issue in submitting a spark job > > > > The worker node has 15G memory, 1x32 GB SSD, and 2 core. The data file is > from S3. If I don't set mapred.max.split.size, it is fine with only one > partition. Otherwise, it will generate OOME. > > > > Ey-Chih Chow > > > > > From: moham...@glassbeam.com > > > To: eyc...@hotmail.com; user@spark.apache.org > > Subject: RE: unknown issue in submitting a spark job > > Date: Thu, 29 Jan 2015 21:16:13 +0000 > > > > Looks like the application is using a lot more memory than available. > Could be a bug somewhere in the code or just underpowered machine. Hard to > say without looking at the code. > > > > Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded > > > > Mohammed > > > > > > -----Original Message----- > > From: ey-chih chow [mailto:eyc...@hotmail.com <eyc...@hotmail.com>] > > Sent: Thursday, January 29, 2015 1:06 AM > > To: user@spark.apache.org > > Subject: unknown issue in submitting a spark job > > > > Hi, > > > > I submitted a job using spark-submit and got the following exception. > > Anybody knows how to fix this? Thanks. > > > > Ey-Chih Chow > > > > ============================================ > > > > 15/01/29 08:53:10 INFO storage.BlockManagerMasterActor: Registering > block manager ip-10-10-8-191.us-west-2.compute.internal:47722 with 6.6 GB > RAM Exception in thread "main" java.lang.reflect.InvocationTargetException > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) > > at > > > org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:40) > > at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala) > > Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded > > at > > > org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:265) > > at org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:94) > > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204) > > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202) > > at scala.Option.getOrElse(Option.scala:120) > > at org.apache.spark.rdd.RDD.partitions(RDD.scala:202) > > at > > > org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32) > > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204) > > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202) > > at scala.Option.getOrElse(Option.scala:120) > > at org.apache.spark.rdd.RDD.partitions(RDD.scala:202) > > at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28) > > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204) > > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202) > > at scala.Option.getOrElse(Option.scala:120) > > at org.apache.spark.rdd.RDD.partitions(RDD.scala:202) > > at org.apache.spark.SparkContext.runJob(SparkContext.scala:1128) > > at > > > org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:935) > > at > > > org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:832) > > at com.crowdstar.etl.ParseAndClean$.main(ParseAndClean.scala:109) > > at com.crowdstar.etl.ParseAndClean.main(ParseAndClean.scala) > > ... 6 more > > 15/01/29 08:54:33 INFO storage.BlockManager: Removing RDD 1 > > 15/01/29 08:54:33 ERROR actor.ActorSystemImpl: exception on LARS’ timer > thread > > java.lang.OutOfMemoryError: GC overhead limit exceeded > > at > > > akka.actor.LightArrayRevolverScheduler$$anon$12.nextTick(Scheduler.scala:397) > > at > akka.actor.LightArrayRevolverScheduler$$anon$12.run(Scheduler.scala:363) > > at java.lang.Thread.run(Thread.java:745) > > 15/01/29 08:54:33 ERROR actor.ActorSystemImpl: Uncaught fatal error from > thread [sparkDriver-scheduler-1] shutting down ActorSystem [sparkDriver] > > java.lang.OutOfMemoryError: GC overhead limit exceeded > > at > > > akka.actor.LightArrayRevolverScheduler$$anon$12.nextTick(Scheduler.scala:397) > > at > akka.actor.LightArrayRevolverScheduler$$anon$12.run(Scheduler.scala:363) > > at java.lang.Thread.run(Thread.java:745) > > 15/01/29 08:54:33 ERROR actor.ActorSystemImpl: exception on LARS’ timer > thread > > java.lang.OutOfMemoryError: GC overhead limit exceeded > > at akka.dispatch.AbstractNodeQueue.<init>(AbstractNodeQueue.java:19) > > at > > > akka.actor.LightArrayRevolverScheduler$TaskQueue.<init>(Scheduler.scala:431) > > at > > > akka.actor.LightArrayRevolverScheduler$$anon$12.nextTick(Scheduler.scala:397) > > at > akka.actor.LightArrayRevolverScheduler$$anon$12.run(Scheduler.scala:363) > > at java.lang.Thread.run(Thread.java:745) > > 15/01/29 08:54:33 ERROR actor.ActorSystemImpl: Uncaught fatal error from > thread [Driver-scheduler-1] shutting down ActorSystem [Driver] > > java.lang.OutOfMemoryError: GC overhead limit exceeded > > at akka.dispatch.AbstractNodeQueue.<init>(AbstractNodeQueue.java:19) > > at > > > akka.actor.LightArrayRevolverScheduler$TaskQueue.<init>(Scheduler.scala:431) > > at > > > akka.actor.LightArrayRevolverScheduler$$anon$12.nextTick(Scheduler.scala:397) > > at > akka.actor.LightArrayRevolverScheduler$$anon$12.run(Scheduler.scala:363) > > at java.lang.Thread.run(Thread.java:745) > > 15/01/29 08:54:33 WARN storage.BlockManagerMasterActor: Removing > BlockManager BlockManagerId(0, ip-10-10-8-191.us-west-2.compute.internal, > > 47722, 0) with no recent heart beats: 82575ms exceeds 45000ms > > 15/01/29 08:54:33 INFO spark.ContextCleaner: Cleaned RDD 1 > > 15/01/29 08:54:33 WARN util.AkkaUtils: Error sending message in 1 > attempts > > akka.pattern.AskTimeoutException: > > Recipient[Actor[akka://sparkDriver/user/BlockManagerMaster#-538003375]] > had already been terminated. > > at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134) > > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:175) > > at > > > org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:218) > > at > > > org.apache.spark.storage.BlockManagerMaster.removeBroadcast(BlockManagerMaster.scala:126) > > > > > > > > > > > > -- > > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/unknown-issue-in-submitting-a-spark-job-tp21418.html > > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For > additional commands, e-mail: user-h...@spark.apache.org > > >
