RE: Spark/Kafka Streaming Job Gets Stuck

m). We're in the process of re-evaluating. > -- > Nick > > -Original Message- > From: Adrian Tanase [mailto:atan...@adobe.com] > Sent: Wednesday, October 28, 2015 4:53 PM > To: Afshartous, Nick <nafshart...@turbine.com> > Cc: user@spark.apache.org > Subject: Re: Sp

Re: Spark/Kafka Streaming Job Gets Stuck

Original Message- >From: Adrian Tanase [mailto:atan...@adobe.com] >Sent: Wednesday, October 28, 2015 4:53 PM >To: Afshartous, Nick <nafshart...@turbine.com> >Cc: user@spark.apache.org >Subject: Re: Spark/Kafka Streaming Job Gets Stuck > >Does it work as expected with smalle

Re: Spark/Kafka Streaming Job Gets Stuck

Sent: Wednesday, October 28, 2015 4:53 PM >> To: Afshartous, Nick <nafshart...@turbine.com> >> Cc: user@spark.apache.org >> Subject: Re: Spark/Kafka Streaming Job Gets Stuck >> >> Does it work as expected with smaller batch or smaller load? Could it be >> tha

Re: Spark/Kafka Streaming Job Gets Stuck

Other than @Adrian suggestions, check if the processing delay is more than the batch processing time. On Thu, Oct 29, 2015 at 2:23 AM, Adrian Tanase wrote: > Does it work as expected with smaller batch or smaller load? Could it be > that it's accumulating too many events over

Spark/Kafka Streaming Job Gets Stuck

Hi, we are load testing our Spark 1.3 streaming (reading from Kafka) job and seeing a problem. This is running in AWS/Yarn and the streaming batch interval is set to 3 minutes and this is a ten node cluster. Testing at 30,000 events per second we are seeing the streaming job get stuck

Re: Spark/Kafka Streaming Job Gets Stuck

Does it work as expected with smaller batch or smaller load? Could it be that it's accumulating too many events over 3 minutes? You could also try increasing the parallelism via repartition to ensure smaller tasks that can safely fit in working memory. Sent from my iPhone > On 28 Oct 2015, at