Re: Spark timeout issue
You need to look more deep into your worker logs, you may find GC error, IO exceptions etc if you look closely which is triggering the timeout. Thanks Best Regards On Mon, Apr 27, 2015 at 3:18 AM, Deepak Gopalakrishnan wrote: > Hello Patrick, > > Sure. I've posted this on user as well. Will be cool to get a response. > > Thanks > Deepak > > On Mon, Apr 27, 2015 at 2:58 AM, Patrick Wendell > wrote: > >> Hi Deepak - please direct this to the user@ list. This list is for >> development of Spark itself. >> >> On Sun, Apr 26, 2015 at 12:42 PM, Deepak Gopalakrishnan >> wrote: >> > Hello All, >> > >> > I'm trying to process a 3.5GB file on standalone mode using spark. I >> could >> > run my spark job succesfully on a 100MB file and it works as expected. >> But, >> > when I try to run it on the 3.5GB file, I run into the below error : >> > >> > >> > 15/04/26 12:45:50 INFO BlockManagerMaster: Updated info of block >> taskresult_83 >> > 15/04/26 12:46:46 WARN AkkaUtils: Error sending message [message = >> > Heartbeat(2,[Lscala.Tuple2;@790223d3,BlockManagerId(2, >> > master.spark.com, 39143))] in 1 attempts >> > java.util.concurrent.TimeoutException: Futures timed out after [30 >> seconds] >> > at >> scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) >> > at >> scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) >> > at >> scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) >> > at >> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) >> > at scala.concurrent.Await$.result(package.scala:107) >> > at >> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:195) >> > at >> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427) >> > 15/04/26 12:47:15 INFO MemoryStore: ensureFreeSpace(26227673) called >> > with curMem=265897, maxMem=5556991426 >> > 15/04/26 12:47:15 INFO MemoryStore: Block taskresult_92 stored as >> > bytes in memory (estimated size 25.0 MB, free 5.2 GB) >> > 15/04/26 12:47:16 INFO MemoryStore: ensureFreeSpace(26272879) called >> > with curMem=26493570, maxMem=5556991426 >> > 15/04/26 12:47:16 INFO MemoryStore: Block taskresult_94 stored as >> > bytes in memory (estimated size 25.1 MB, free 5.1 GB) >> > 15/04/26 12:47:18 INFO MemoryStore: ensureFreeSpace(26285327) called >> > with curMem=52766449, maxMem=5556991426 >> > >> > >> > and the job fails. >> > >> > >> > I'm on AWS and have opened all ports. Also, since the 100MB file works, >> it >> > should not be a connection issue. I've a r3 xlarge and 2 m3 large. >> > >> > Can anyone suggest a way to fix this? >> > > > > -- > Regards, > *Deepak Gopalakrishnan* > *Mobile*:+918891509774 > *Skype* : deepakgk87 > http://myexps.blogspot.com > >
Re: Spark timeout issue
Hello Patrick, Sure. I've posted this on user as well. Will be cool to get a response. Thanks Deepak On Mon, Apr 27, 2015 at 2:58 AM, Patrick Wendell wrote: > Hi Deepak - please direct this to the user@ list. This list is for > development of Spark itself. > > On Sun, Apr 26, 2015 at 12:42 PM, Deepak Gopalakrishnan > wrote: > > Hello All, > > > > I'm trying to process a 3.5GB file on standalone mode using spark. I > could > > run my spark job succesfully on a 100MB file and it works as expected. > But, > > when I try to run it on the 3.5GB file, I run into the below error : > > > > > > 15/04/26 12:45:50 INFO BlockManagerMaster: Updated info of block > taskresult_83 > > 15/04/26 12:46:46 WARN AkkaUtils: Error sending message [message = > > Heartbeat(2,[Lscala.Tuple2;@790223d3,BlockManagerId(2, > > master.spark.com, 39143))] in 1 attempts > > java.util.concurrent.TimeoutException: Futures timed out after [30 > seconds] > > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > > at > scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > > at scala.concurrent.Await$.result(package.scala:107) > > at > org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:195) > > at > org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427) > > 15/04/26 12:47:15 INFO MemoryStore: ensureFreeSpace(26227673) called > > with curMem=265897, maxMem=5556991426 > > 15/04/26 12:47:15 INFO MemoryStore: Block taskresult_92 stored as > > bytes in memory (estimated size 25.0 MB, free 5.2 GB) > > 15/04/26 12:47:16 INFO MemoryStore: ensureFreeSpace(26272879) called > > with curMem=26493570, maxMem=5556991426 > > 15/04/26 12:47:16 INFO MemoryStore: Block taskresult_94 stored as > > bytes in memory (estimated size 25.1 MB, free 5.1 GB) > > 15/04/26 12:47:18 INFO MemoryStore: ensureFreeSpace(26285327) called > > with curMem=52766449, maxMem=5556991426 > > > > > > and the job fails. > > > > > > I'm on AWS and have opened all ports. Also, since the 100MB file works, > it > > should not be a connection issue. I've a r3 xlarge and 2 m3 large. > > > > Can anyone suggest a way to fix this? > -- Regards, *Deepak Gopalakrishnan* *Mobile*:+918891509774 *Skype* : deepakgk87 http://myexps.blogspot.com
Re: Spark timeout issue
Hi Deepak - please direct this to the user@ list. This list is for development of Spark itself. On Sun, Apr 26, 2015 at 12:42 PM, Deepak Gopalakrishnan wrote: > Hello All, > > I'm trying to process a 3.5GB file on standalone mode using spark. I could > run my spark job succesfully on a 100MB file and it works as expected. But, > when I try to run it on the 3.5GB file, I run into the below error : > > > 15/04/26 12:45:50 INFO BlockManagerMaster: Updated info of block taskresult_83 > 15/04/26 12:46:46 WARN AkkaUtils: Error sending message [message = > Heartbeat(2,[Lscala.Tuple2;@790223d3,BlockManagerId(2, > master.spark.com, 39143))] in 1 attempts > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:195) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427) > 15/04/26 12:47:15 INFO MemoryStore: ensureFreeSpace(26227673) called > with curMem=265897, maxMem=5556991426 > 15/04/26 12:47:15 INFO MemoryStore: Block taskresult_92 stored as > bytes in memory (estimated size 25.0 MB, free 5.2 GB) > 15/04/26 12:47:16 INFO MemoryStore: ensureFreeSpace(26272879) called > with curMem=26493570, maxMem=5556991426 > 15/04/26 12:47:16 INFO MemoryStore: Block taskresult_94 stored as > bytes in memory (estimated size 25.1 MB, free 5.1 GB) > 15/04/26 12:47:18 INFO MemoryStore: ensureFreeSpace(26285327) called > with curMem=52766449, maxMem=5556991426 > > > and the job fails. > > > I'm on AWS and have opened all ports. Also, since the 100MB file works, it > should not be a connection issue. I've a r3 xlarge and 2 m3 large. > > Can anyone suggest a way to fix this? - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
