Re: Spark Kafka stream processing time increasing gradually

2016-06-22 Thread Roshan Singh
ese keys will be perpetually > held in memory even though there is no need for them to be there. This > growth in the size of the generated window rdd is what's giving rise to the > deteriorating processing time in your case. > > A filter function that's equivalent of &qu

Re: Spark Kafka stream processing time increasing gradually

2016-06-16 Thread Roshan Singh
that you are > using. For fixing this issue, you have to use different flavor of that > API, specifically the 2 versions that allow you to give a 'Filter function' > to them. Putting in the filter functions helped stabilize our application > too. > > HTH > NB > > > On Sun,