Re: [SparkStreaming]StackOverflow on restart

2014-08-04 Thread Tathagata Das
What is your checkpoint interval of the updateStateByKey's DStream? Did you modify it? Also, do you have a simple program and the step-by-step process by which I can reproduce the issue? If not, can you give me the full DEBUG level logs of the program before and after restart? TD On Mon, Aug 4, 2

[SparkStreaming]StackOverflow on restart

2014-08-04 Thread Yana Kadiyska
Hi Spark users, I'm trying to get a pretty simple streaming program going. My context is created via StreamingContext.getOrCreate(checkpointDir,createFn) creating a context works fine but when trying to start from a checkpoint I get a stack overflow. Any pointers what could be going wrong? My ba