Which version of spark are you running? If you are running the latest one, then could try running not a window but a simple event count on every 2 second batch, and see if you are still running out of memory?
TD On Thu, Sep 11, 2014 at 10:34 AM, Aniket Bhatnagar < aniket.bhatna...@gmail.com> wrote: > I did change it to be 1 gb. It still ran out of memory but a little later. > > The streaming job isnt handling a lot of data. In every 2 seconds, it > doesn't get more than 50 records. Each record size is not more than 500 > bytes. > On Sep 11, 2014 10:54 PM, "Bharat Venkat" <bvenkat.sp...@gmail.com> > wrote: > >> You could set "spark.executor.memory" to something bigger than the >> default (512mb) >> >> >> On Thu, Sep 11, 2014 at 8:31 AM, Aniket Bhatnagar < >> aniket.bhatna...@gmail.com> wrote: >> >>> I am running a simple Spark Streaming program that pulls in data from >>> Kinesis at a batch interval of 10 seconds, windows it for 10 seconds, maps >>> data and persists to a store. >>> >>> The program is running in local mode right now and runs out of memory >>> after a while. I am yet to investigate heap dumps but I think Spark isn't >>> releasing memory after processing is complete. I have even tried changing >>> storage level to disk only. >>> >>> Help! >>> >>> Thanks, >>> Aniket >>> >> >>
