Hi Prateek, How many cores (threads) do you assign to spark in local mode? It is very likely the local spark does not have enough resource to proceed. You can check http://yourip:4040 to check the details.
Thanks! Terry On Fri, Oct 9, 2015 at 10:34 PM, Prateek . <prat...@aricent.com> wrote: > Hi All, > > > > In my application I have a serializable class which is taking > InputDStream from Kafka. The inputDStream contains JSON which is stored in > serializable case class. Transformations are applied and saveToCassandra() > is executed. > > I was getting task not serializable exception , so I made the class > serializable. > > > > Now ,the application is working fine in standalone mode, but not able to > receive data in local mode with the below mentioned log. > > > > What is internally happening?, if anyone have some insights Please share! > > > > Thank You in advance > > Regards, > > Prateek > > > > > > *From:* Prateek . > *Sent:* Friday, October 09, 2015 6:55 PM > *To:* user@spark.apache.org > *Subject:* Streaming Application Unable to get Stream from Kafka > > > > Hi, > > > > I have Spark Streaming application running with the following log on > console, I don’t get any exception but I am not able to receive the data > from Kafka Stream. > > > > Can anyone please provide any insight what is happening with Spark > Streaming. Is Receiver is not able to read the stream? How shall I debug > it? > > > > JobScheduler: Added jobs for time 1444396043000 ms > > 15/10/09 18:37:23 INFO MemoryStore: ensureFreeSpace(922) called with > curMem=1009312, maxMem=278302556 > > 15/10/09 18:37:23 INFO MemoryStore: Block input-0-1444396043000 stored as > bytes in memory (estimated size 922.0 B, free 264.4 MB) > > 15/10/09 18:37:23 INFO BlockManagerInfo: Added input-0-1444396043000 in > memory on webanalytics03:51843 (size: 922.0 B, free: 264.4 MB) > > 15/10/09 18:37:23 INFO BlockManagerMaster: Updated info of block > input-0-1444396043000 > > 15/10/09 18:37:23 WARN BlockManager: Block input-0-1444396043000 already > exists on this machine; not re-adding it > > 15/10/09 18:37:23 INFO BlockGenerator: Pushed block input-0-1444396043000 > > 15/10/09 18:37:23 INFO MemoryStore: ensureFreeSpace(926) called with > curMem=1010234, maxMem=278302556 > > 15/10/09 18:37:23 INFO MemoryStore: Block input-0-1444396043200 stored as > bytes in memory (estimated size 926.0 B, free 264.4 MB) > > 15/10/09 18:37:23 INFO BlockManagerInfo: Added input-0-1444396043200 in > memory on webanalytics03:51843 (size: 926.0 B, free: 264.4 MB) > > 15/10/09 18:37:23 INFO BlockManagerMaster: Updated info of block > input-0-1444396043200 > > 15/10/09 18:37:23 WARN BlockManager: Block input-0-1444396043200 already > exists on this machine; not re-adding it > > 15/10/09 18:37:23 INFO BlockGenerator: Pushed block input-0-1444396043200 > > 15/10/09 18:37:23 INFO MemoryStore: ensureFreeSpace(923) called with > curMem=1011160, maxMem=278302556 > > 15/10/09 18:37:23 INFO MemoryStore: Block input-0-1444396043400 stored as > bytes in memory (estimated size 923.0 B, free 264.4 MB) > > 15/10/09 18:37:23 INFO BlockManagerInfo: Added input-0-1444396043400 in > memory on webanalytics03:51843 (size: 923.0 B, free: 264.4 MB) > > 15/10/09 18:37:23 INFO BlockManagerMaster: Updated info of block > input-0-1444396043400 > > 15/10/09 18:37:23 WARN BlockManager: Block input-0-1444396043400 already > exists on this machine; not re-adding it > > 15/10/09 18:37:23 INFO BlockGenerator: Pushed block input-0-1444396043400 > > 15/10/09 18:37:23 INFO MemoryStore: ensureFreeSpace(917) called with > curMem=1012083, maxMem=278302556 > > 15/10/09 18:37:23 INFO MemoryStore: Block input-0-1444396043600 stored as > bytes in memory (estimated size 917.0 B, free 264.4 MB) > > 15/10/09 18:37:23 INFO BlockManagerInfo: Added input-0-1444396043600 in > memory on webanalytics03:51843 (size: 917.0 B, free: 264.4 MB) > > 15/10/09 18:37:23 INFO BlockManagerMaster: Updated info of block > input-0-1444396043600 > > 15/10/09 18:37:23 WARN BlockManager: Block input-0-1444396043600 already > exists on this machine; not re-adding it > > 15/10/09 18:37:23 INFO BlockGenerator: Pushed block input-0-1444396043600 > > 15/10/09 18:37:24 INFO ReceiverTracker: Stream 0 received 5 blocks > > 15/10/09 18:37:24 INFO MemoryStore: ensureFreeSpace(922) called with > curMem=1013000, maxMem=278302556 > > 15/10/09 18:37:24 INFO MemoryStore: Block input-0-1444396043800 stored as > bytes in memory (estimated size 922.0 B, free 264.4 MB) > > 15/10/09 18:37:24 INFO BlockManagerInfo: Added input-0-1444396043800 in > memory on webanalytics03:51843 (size: 922.0 B, free: 264.4 MB) > > 15/10/09 18:37:24 INFO BlockManagerMaster: Updated info of block > input-0-1444396043800 > > 15/10/09 18:37:24 WARN BlockManager: Block input-0-1444396043800 already > exists on this machine; not re-adding it > > 15/10/09 18:37:24 INFO BlockGenerator: Pushed block input-0-1444396043800 > > > > Thanks in advance > > Prateek > > "DISCLAIMER: This message is proprietary to Aricent and is intended solely > for the use of the individual to whom it is addressed. It may contain > privileged or confidential information and should not be circulated or used > for any purpose other than for what it is intended. If you have received > this message in error, please notify the originator immediately. If you are > not the intended recipient, you are notified that you are strictly > prohibited from using, copying, altering, or disclosing the contents of > this message. Aricent accepts no responsibility for loss or damage arising > from the use of the information transmitted by this email including damage > from virus." >
