Hi, > > What do you see under Executors and Details for Stage (for the > affected stages)? Anything weird memory-related? > Under executor Tab, logs throw these warning -
16/06/16 20:45:40 INFO TorrentBroadcast: Reading broadcast variable 422145 took 1 ms 16/06/16 20:45:40 WARN MemoryStore: Failed to reserve initial memory threshold of 1024.0 KB for computing block broadcast_422145 in memory. 16/06/16 20:45:40 WARN MemoryStore: Not enough space to cache broadcast_422145 in memory! (computed 496.0 B so far) 16/06/16 20:45:40 INFO MemoryStore: Memory use = 147.9 KB (blocks) + 2.2 GB (scratch space shared across 0 tasks(s)) = 2.2 GB. Storage limit = 2.2 GB. 16/06/16 20:45:40 WARN MemoryStore: Persisting block broadcast_422145 to disk instead. 16/06/16 20:45:40 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 70278, fetching them 16/06/16 20:45:40 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = AkkaRpcEndpointRef(Actor[akka.tcp:// sparkDriver@17.40.240.71:46187/user/MapOutputTracker#-1794035569]) I dont see any memory related errors on 'stages' Tab. > > How does your "I am reading data from Kafka into Spark and writing it > into Cassandra after processing it." pipeline look like? > > This part has no issues. Reading from Kafka is always up to date. There are no offset lags. Writting to Cassandra is also fine with less than 1ms to write data. > Pozdrawiam, > Jacek Laskowski > ---- > https://medium.com/@jaceklaskowski/ > Mastering Apache Spark http://bit.ly/mastering-apache-spark > Follow me at https://twitter.com/jaceklaskowski > > > On Mon, Jun 13, 2016 at 11:56 PM, Cassa L <lcas...@gmail.com> wrote: > > Hi, > > > > I'm using spark 1.5.1 version. I am reading data from Kafka into Spark > and > > writing it into Cassandra after processing it. Spark job starts fine and > > runs all good for some time until I start getting below errors. Once > these > > errors come, job start to lag behind and I see that job has scheduling > and > > processing delays in streaming UI. > > > > Worker memory is 6GB, executor-memory is 5GB, I also tried to tweak > > memoryFraction parameters. Nothing works. > > > > > > 16/06/13 21:26:02 INFO MemoryStore: ensureFreeSpace(4044) called with > > curMem=565394, maxMem=2778495713 > > 16/06/13 21:26:02 INFO MemoryStore: Block broadcast_69652_piece0 stored > as > > bytes in memory (estimated size 3.9 KB, free 2.6 GB) > > 16/06/13 21:26:02 INFO TorrentBroadcast: Reading broadcast variable 69652 > > took 2 ms > > 16/06/13 21:26:02 WARN MemoryStore: Failed to reserve initial memory > > threshold of 1024.0 KB for computing block broadcast_69652 in memory. > > 16/06/13 21:26:02 WARN MemoryStore: Not enough space to cache > > broadcast_69652 in memory! (computed 496.0 B so far) > > 16/06/13 21:26:02 INFO MemoryStore: Memory use = 556.1 KB (blocks) + 2.6 > GB > > (scratch space shared across 0 tasks(s)) = 2.6 GB. Storage limit = 2.6 > GB. > > 16/06/13 21:26:02 WARN MemoryStore: Persisting block broadcast_69652 to > disk > > instead. > > 16/06/13 21:26:02 INFO BlockManager: Found block rdd_100761_1 locally > > 16/06/13 21:26:02 INFO Executor: Finished task 0.0 in stage 71577.0 (TID > > 452316). 2043 bytes result sent to driver > > > > > > Thanks, > > > > L >
