We are using KafkaUtils.createStream API to read data from kafka topics and we are using StorageLevel.MEMORY_AND_DISK_SER option while configuring kafka streams.
On Wed, Mar 30, 2016 at 12:58 PM, Akhil Das <ak...@sigmoidanalytics.com> wrote: > Can you elaborate more on from where you are streaming the data and what > type of consumer you are using etc? > > Thanks > Best Regards > > On Tue, Mar 29, 2016 at 6:10 PM, Mayur Mohite <mayur.moh...@applift.com> > wrote: > >> Hi, >> >> We are running spark streaming app on a single machine and we have >> configured spark executor memory to 30G. >> We noticed that after running the app for 12 hours, spark streaming >> started spilling ALL the data to disk even though we have configured >> sufficient memory for spark to use for storage. >> >> -Mayur >> >> Learn more about our inaugural *FirstScreen Conference >> <http://www.firstscreenconf.com/>*! >> *Where the worlds of mobile advertising and technology meet!* >> >> June 15, 2016 @ Urania Berlin >> > > -- *Mayur Mohite* Senior Software Engineer Phone: +91 9035867742 Skype: mayur.mohite_applift *AppLift India* 107/3, 80 Feet Main Road, Koramangala 4th Block, Bangalore - 560034 www.AppLift.com <http://www.applift.com/> -- Learn more about our inaugural *FirstScreen Conference <http://www.firstscreenconf.com/>*! *Where the worlds of mobile advertising and technology meet!* June 15, 2016 @ Urania Berlin
