What is your hardware configuration like which you are running Spark on? Hey
Namaskara~Nalama~Guten Tag~Bonjour -- Keigu Deepak 73500 12833 www.simtree.net, dee...@simtree.net deic...@gmail.com LinkedIn: www.linkedin.com/in/deicool Skype: thumsupdeicool Google talk: deicool Blog: http://loveandfearless.wordpress.com Facebook: http://www.facebook.com/deicool "Contribute to the world, environment and more : http://www.gridrepublic.org " On Thu, Jun 16, 2016 at 5:33 PM, Jacek Laskowski <ja...@japila.pl> wrote: > Hi, > > What do you see under Executors and Details for Stage (for the > affected stages)? Anything weird memory-related? > > How does your "I am reading data from Kafka into Spark and writing it > into Cassandra after processing it." pipeline look like? > > Pozdrawiam, > Jacek Laskowski > ---- > https://medium.com/@jaceklaskowski/ > Mastering Apache Spark http://bit.ly/mastering-apache-spark > Follow me at https://twitter.com/jaceklaskowski > > > On Mon, Jun 13, 2016 at 11:56 PM, Cassa L <lcas...@gmail.com> wrote: > > Hi, > > > > I'm using spark 1.5.1 version. I am reading data from Kafka into Spark > and > > writing it into Cassandra after processing it. Spark job starts fine and > > runs all good for some time until I start getting below errors. Once > these > > errors come, job start to lag behind and I see that job has scheduling > and > > processing delays in streaming UI. > > > > Worker memory is 6GB, executor-memory is 5GB, I also tried to tweak > > memoryFraction parameters. Nothing works. > > > > > > 16/06/13 21:26:02 INFO MemoryStore: ensureFreeSpace(4044) called with > > curMem=565394, maxMem=2778495713 > > 16/06/13 21:26:02 INFO MemoryStore: Block broadcast_69652_piece0 stored > as > > bytes in memory (estimated size 3.9 KB, free 2.6 GB) > > 16/06/13 21:26:02 INFO TorrentBroadcast: Reading broadcast variable 69652 > > took 2 ms > > 16/06/13 21:26:02 WARN MemoryStore: Failed to reserve initial memory > > threshold of 1024.0 KB for computing block broadcast_69652 in memory. > > 16/06/13 21:26:02 WARN MemoryStore: Not enough space to cache > > broadcast_69652 in memory! (computed 496.0 B so far) > > 16/06/13 21:26:02 INFO MemoryStore: Memory use = 556.1 KB (blocks) + 2.6 > GB > > (scratch space shared across 0 tasks(s)) = 2.6 GB. Storage limit = 2.6 > GB. > > 16/06/13 21:26:02 WARN MemoryStore: Persisting block broadcast_69652 to > disk > > instead. > > 16/06/13 21:26:02 INFO BlockManager: Found block rdd_100761_1 locally > > 16/06/13 21:26:02 INFO Executor: Finished task 0.0 in stage 71577.0 (TID > > 452316). 2043 bytes result sent to driver > > > > > > Thanks, > > > > L > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
