All stated symptoms are consistent with GC pressure (other nodes timeout trying to connect because of a long stop-the-world), quite possibly due to groupByKey. groupByKey is a very expensive operation as it may bring all the data for a particular partition into memory (in particular, it cannot spill values for a single key, so if you have a single very skewed key you can get behavior like this).
On Sat, Feb 28, 2015 at 11:33 AM, Paweł Szulc <[email protected]> wrote: > But groupbykey will repartition according to numer of keys as I understand > how it works. How do you know that you haven't reached the groupbykey > phase? Are you using a profiler or do yoi base that assumption only on logs? > > sob., 28 lut 2015, 8:12 PM Arun Luthra użytkownik <[email protected]> > napisał: > > A correction to my first post: >> >> There is also a repartition right before groupByKey to help avoid >> too-many-open-files error: >> >> >> rdd2.union(rdd1).map(...).filter(...).repartition(15000).groupByKey().map(...).flatMap(...).saveAsTextFile() >> >> On Sat, Feb 28, 2015 at 11:10 AM, Arun Luthra <[email protected]> >> wrote: >> >>> The job fails before getting to groupByKey. >>> >>> I see a lot of timeout errors in the yarn logs, like: >>> >>> 15/02/28 12:47:16 WARN util.AkkaUtils: Error sending message in 1 >>> attempts >>> akka.pattern.AskTimeoutException: Timed out >>> >>> and >>> >>> 15/02/28 12:47:49 WARN util.AkkaUtils: Error sending message in 2 >>> attempts >>> java.util.concurrent.TimeoutException: Futures timed out after [30 >>> seconds] >>> >>> and some of these are followed by: >>> >>> 15/02/28 12:48:02 ERROR executor.CoarseGrainedExecutorBackend: Driver >>> Disassociated [akka.tcp://sparkExecutor@...] -> [akka.tcp://sparkDriver@...] >>> disassociated! Shutting down. >>> 15/02/28 12:48:02 ERROR executor.Executor: Exception in task 421027.0 in >>> stage 1.0 (TID 336601) >>> java.io.FileNotFoundException: >>> ..../hadoop/yarn/local/......../spark-local-20150228123450-3a71/36/shuffle_0_421027_0 >>> (No such file or directory) >>> >>> >>> >>> >>> On Sat, Feb 28, 2015 at 9:33 AM, Paweł Szulc <[email protected]> >>> wrote: >>> >>>> I would first check whether there is any possibility that after doing >>>> groupbykey one of the groups does not fit in one of the executors' memory. >>>> >>>> To back up my theory, instead of doing groupbykey + map try reducebykey >>>> + mapvalues. >>>> >>>> Let me know if that helped. >>>> >>>> Pawel Szulc >>>> http://rabbitonweb.com >>>> >>>> sob., 28 lut 2015, 6:22 PM Arun Luthra użytkownik < >>>> [email protected]> napisał: >>>> >>>> So, actually I am removing the persist for now, because there is >>>>> significant filtering that happens after calling textFile()... but I will >>>>> keep that option in mind. >>>>> >>>>> I just tried a few different combinations of number of executors, >>>>> executor memory, and more importantly, number of tasks... *all three >>>>> times it failed when approximately 75.1% of the tasks were completed (no >>>>> matter how many tasks resulted from repartitioning the data in >>>>> textfile(..., N))*. Surely this is a strong clue to something? >>>>> >>>>> >>>>> >>>>> On Fri, Feb 27, 2015 at 1:07 PM, Burak Yavuz <[email protected]> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> Not sure if it can help, but `StorageLevel.MEMORY_AND_DISK_SER` >>>>>> generates many small objects that lead to very long GC time, causing the >>>>>> executor losts, heartbeat not received, and GC overhead limit exceeded >>>>>> messages. >>>>>> Could you try using `StorageLevel.MEMORY_AND_DISK` instead? You can >>>>>> also try `OFF_HEAP` (and use Tachyon). >>>>>> >>>>>> Burak >>>>>> >>>>>> On Fri, Feb 27, 2015 at 11:39 AM, Arun Luthra <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> My program in pseudocode looks like this: >>>>>>> >>>>>>> val conf = new SparkConf().setAppName("Test") >>>>>>> .set("spark.storage.memoryFraction","0.2") // default 0.6 >>>>>>> .set("spark.shuffle.memoryFraction","0.12") // default 0.2 >>>>>>> .set("spark.shuffle.manager","SORT") // preferred setting for >>>>>>> optimized joins >>>>>>> .set("spark.shuffle.consolidateFiles","true") // helpful for >>>>>>> "too many files open" >>>>>>> .set("spark.mesos.coarse", "true") // helpful for >>>>>>> MapOutputTracker errors? >>>>>>> .set("spark.akka.frameSize","500") // helpful when using >>>>>>> consildateFiles=true >>>>>>> .set("spark.akka.askTimeout", "30") >>>>>>> .set("spark.shuffle.compress","false") // >>>>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Fetch-Failure-tp20787p20811.html >>>>>>> .set("spark.file.transferTo","false") // >>>>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Fetch-Failure-tp20787p20811.html >>>>>>> .set("spark.core.connection.ack.wait.timeout","600") // >>>>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Fetch-Failure-tp20787p20811.html >>>>>>> .set("spark.speculation","true") >>>>>>> .set("spark.worker.timeout","600") // >>>>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Heartbeat-exceeds-td3798.html >>>>>>> .set("spark.akka.timeout","300") // >>>>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Heartbeat-exceeds-td3798.html >>>>>>> .set("spark.storage.blockManagerSlaveTimeoutMs","120000") >>>>>>> .set("spark.driver.maxResultSize","2048") // in response to >>>>>>> error: Total size of serialized results of 39901 tasks (1024.0 MB) is >>>>>>> bigger than spark.driver.maxResultSize (1024.0 MB) >>>>>>> .set("spark.serializer", >>>>>>> "org.apache.spark.serializer.KryoSerializer") >>>>>>> >>>>>>> .set("spark.kryo.registrator","com.att.bdcoe.cip.ooh.MyRegistrator") >>>>>>> .set("spark.kryo.registrationRequired", "true") >>>>>>> >>>>>>> val rdd1 = sc.textFile(file1).persist(StorageLevel >>>>>>> .MEMORY_AND_DISK_SER).map(_.split("\\|", -1)...filter(...) >>>>>>> >>>>>>> val rdd2 = >>>>>>> sc.textFile(file2).persist(StorageLevel.MEMORY_AND_DISK_SER).map(_.split("\\|", >>>>>>> -1)...filter(...) >>>>>>> >>>>>>> >>>>>>> rdd2.union(rdd1).map(...).filter(...).groupByKey().map(...).flatMap(...).saveAsTextFile() >>>>>>> >>>>>>> >>>>>>> I run the code with: >>>>>>> --num-executors 500 \ >>>>>>> --driver-memory 20g \ >>>>>>> --executor-memory 20g \ >>>>>>> --executor-cores 32 \ >>>>>>> >>>>>>> >>>>>>> I'm using kryo serialization on everything, including broadcast >>>>>>> variables. >>>>>>> >>>>>>> Spark creates 145k tasks, and the first stage includes everything >>>>>>> before groupByKey(). It fails before getting to groupByKey. I have tried >>>>>>> doubling and tripling the number of partitions when calling textFile, >>>>>>> with >>>>>>> no success. >>>>>>> >>>>>>> Very similar code (trivial changes, to accomodate different input) >>>>>>> worked on a smaller input (~8TB)... Not that it was easy to get that >>>>>>> working. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Errors vary, here is what I am getting right now: >>>>>>> >>>>>>> ERROR SendingConnection: Exception while reading SendingConnection >>>>>>> ... java.nio.channels.ClosedChannelException >>>>>>> (^ guessing that is symptom of something else) >>>>>>> >>>>>>> WARN BlockManagerMasterActor: Removing BlockManager >>>>>>> BlockManagerId(...) with no recent heart beats: 120030ms exceeds >>>>>>> 120000ms >>>>>>> (^ guessing that is symptom of something else) >>>>>>> >>>>>>> ERROR ActorSystemImpl: Uncaught fatal error from thread (...) >>>>>>> shutting down ActorSystem [sparkDriver] >>>>>>> *java.lang.OutOfMemoryError: GC overhead limit exceeded* >>>>>>> >>>>>>> >>>>>>> >>>>>>> Other times I will get messages about "executor lost..." about 1 >>>>>>> message per second, after ~~50k tasks complete, until there are almost >>>>>>> no >>>>>>> executors left and progress slows to nothing. >>>>>>> >>>>>>> I ran with verbose GC info; I do see failing yarn containers that >>>>>>> have multiple (like 30) "Full GC" messages but I don't know how to >>>>>>> interpret if that is the problem. Typical Full GC time taken seems >>>>>>> ok: [Times: user=23.30 sys=0.06, real=1.94 secs] >>>>>>> >>>>>>> >>>>>>> >>>>>>> Suggestions, please? >>>>>>> >>>>>>> Huge thanks for useful suggestions, >>>>>>> Arun >>>>>>> >>>>>> >>>>>> >>>>> >>> >>
