Can having a smaller block interval only resolve this? Regards, Praveen On 18 Feb 2016 21:13, "Cody Koeninger" <c...@koeninger.org> wrote:
> Backpressure won't help you with the first batch, you'd need > spark.streaming.kafka.maxRatePerPartition > for that > > On Thu, Feb 18, 2016 at 9:40 AM, praveen S <mylogi...@gmail.com> wrote: > >> Have a look at >> >> spark.streaming.backpressure.enabled >> Property >> >> Regards, >> Praveen >> On 18 Feb 2016 00:13, "Abhishek Anand" <abhis.anan...@gmail.com> wrote: >> >>> I have a spark streaming application running in production. I am trying >>> to find a solution for a particular use case when my application has a >>> downtime of say 5 hours and is restarted. Now, when I start my streaming >>> application after 5 hours there would be considerable amount of data then >>> in the Kafka and my cluster would be unable to repartition and process that. >>> >>> Is there any workaround so that when my streaming application starts it >>> starts taking data for 1-2 hours, process it , then take the data for next >>> 1 hour process it. Now when its done processing of previous 5 hours data >>> which missed, normal streaming should start with the given slide interval. >>> >>> Please suggest any ideas and feasibility of this. >>> >>> >>> Thanks !! >>> Abhi >>> >> >
