Re: Spark streaming spilling all the data to disk even if memory available

Use StorageLevel MEMORY_ONLY. Also have a look at the createDirectStream API. Most likely in your case your batch duration must be less than your processing time and the addition of delay probably blows up the memory. On Mar 31, 2016 6:13 PM, "Mayur Mohite" wrote: > We

Re: Spark streaming spilling all the data to disk even if memory available

We are using KafkaUtils.createStream API to read data from kafka topics and we are using StorageLevel.MEMORY_AND_DISK_SER option while configuring kafka streams. On Wed, Mar 30, 2016 at 12:58 PM, Akhil Das wrote: > Can you elaborate more on from where you are

Re: Spark streaming spilling all the data to disk even if memory available

Can you elaborate more on from where you are streaming the data and what type of consumer you are using etc? Thanks Best Regards On Tue, Mar 29, 2016 at 6:10 PM, Mayur Mohite wrote: > Hi, > > We are running spark streaming app on a single machine and we have >