Seems like the exexutor memory is not enough for your job and it is writing objects to disk On Jun 17, 2016 2:25 AM, "Cassa L" <lcas...@gmail.com> wrote:
> > > On Thu, Jun 16, 2016 at 5:27 AM, Deepak Goel <deic...@gmail.com> wrote: > >> What is your hardware configuration like which you are running Spark on? >> >> It is 24core, 128GB RAM > >> Hey >> >> Namaskara~Nalama~Guten Tag~Bonjour >> >> >> -- >> Keigu >> >> Deepak >> 73500 12833 >> www.simtree.net, dee...@simtree.net >> deic...@gmail.com >> >> LinkedIn: www.linkedin.com/in/deicool >> Skype: thumsupdeicool >> Google talk: deicool >> Blog: http://loveandfearless.wordpress.com >> Facebook: http://www.facebook.com/deicool >> >> "Contribute to the world, environment and more : >> http://www.gridrepublic.org >> " >> >> On Thu, Jun 16, 2016 at 5:33 PM, Jacek Laskowski <ja...@japila.pl> wrote: >> >>> Hi, >>> >>> What do you see under Executors and Details for Stage (for the >>> affected stages)? Anything weird memory-related? >>> >>> How does your "I am reading data from Kafka into Spark and writing it >>> into Cassandra after processing it." pipeline look like? >>> >>> Pozdrawiam, >>> Jacek Laskowski >>> ---- >>> https://medium.com/@jaceklaskowski/ >>> Mastering Apache Spark http://bit.ly/mastering-apache-spark >>> Follow me at https://twitter.com/jaceklaskowski >>> >>> >>> On Mon, Jun 13, 2016 at 11:56 PM, Cassa L <lcas...@gmail.com> wrote: >>> > Hi, >>> > >>> > I'm using spark 1.5.1 version. I am reading data from Kafka into Spark >>> and >>> > writing it into Cassandra after processing it. Spark job starts fine >>> and >>> > runs all good for some time until I start getting below errors. Once >>> these >>> > errors come, job start to lag behind and I see that job has scheduling >>> and >>> > processing delays in streaming UI. >>> > >>> > Worker memory is 6GB, executor-memory is 5GB, I also tried to tweak >>> > memoryFraction parameters. Nothing works. >>> > >>> > >>> > 16/06/13 21:26:02 INFO MemoryStore: ensureFreeSpace(4044) called with >>> > curMem=565394, maxMem=2778495713 >>> > 16/06/13 21:26:02 INFO MemoryStore: Block broadcast_69652_piece0 >>> stored as >>> > bytes in memory (estimated size 3.9 KB, free 2.6 GB) >>> > 16/06/13 21:26:02 INFO TorrentBroadcast: Reading broadcast variable >>> 69652 >>> > took 2 ms >>> > 16/06/13 21:26:02 WARN MemoryStore: Failed to reserve initial memory >>> > threshold of 1024.0 KB for computing block broadcast_69652 in memory. >>> > 16/06/13 21:26:02 WARN MemoryStore: Not enough space to cache >>> > broadcast_69652 in memory! (computed 496.0 B so far) >>> > 16/06/13 21:26:02 INFO MemoryStore: Memory use = 556.1 KB (blocks) + >>> 2.6 GB >>> > (scratch space shared across 0 tasks(s)) = 2.6 GB. Storage limit = 2.6 >>> GB. >>> > 16/06/13 21:26:02 WARN MemoryStore: Persisting block broadcast_69652 >>> to disk >>> > instead. >>> > 16/06/13 21:26:02 INFO BlockManager: Found block rdd_100761_1 locally >>> > 16/06/13 21:26:02 INFO Executor: Finished task 0.0 in stage 71577.0 >>> (TID >>> > 452316). 2043 bytes result sent to driver >>> > >>> > >>> > Thanks, >>> > >>> > L >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>> For additional commands, e-mail: user-h...@spark.apache.org >>> >>> >> >
