Re: Why Spark Stream job stops producing outputs after a while?

Any suggestions? Is there anyway that I could debug this issue? Cheers, Uthay On 11 October 2015 at 18:39, Uthayan Suthakar wrote: > Hello all, > > I have a Spark Streaming job that run and produce results successfully. > However, after a few days the job stop

Re: Why Spark Stream job stops producing outputs after a while?

Are you sure that there are not log4j errors in the driver logs? What if you try enabling debug level? And what does the streaming UI say? On Mon, Oct 12, 2015 at 12:50 PM, Uthayan Suthakar < uthayan.sutha...@gmail.com> wrote: > Any suggestions? Is there anyway that I could debug this issue? >

Re: Why Spark Stream job stops producing outputs after a while?

Hi Tathagata, Yes, I'm pretty sure there are no errors in driver logs and workers logs. The streaming UI appears to be showing that job is running fine. I can see the tasks are being completed. I also, see the receiver is picking up new messages (in UI). I'm running this same job twice that read

Why Spark Stream job stops producing outputs after a while?

Hello all, I have a Spark Streaming job that run and produce results successfully. However, after a few days the job stop producing any output. I can see the job is still running ( polling data from Flume, completing jobs and it's subtasks) however, it is failing to produce any output. I have to