Backpressure won't help you with the first batch, you'd need spark.streaming.kafka.maxRatePerPartition for that
On Thu, Feb 18, 2016 at 9:40 AM, praveen S <mylogi...@gmail.com> wrote: > Have a look at > > spark.streaming.backpressure.enabled > Property > > Regards, > Praveen > On 18 Feb 2016 00:13, "Abhishek Anand" <abhis.anan...@gmail.com> wrote: > >> I have a spark streaming application running in production. I am trying >> to find a solution for a particular use case when my application has a >> downtime of say 5 hours and is restarted. Now, when I start my streaming >> application after 5 hours there would be considerable amount of data then >> in the Kafka and my cluster would be unable to repartition and process that. >> >> Is there any workaround so that when my streaming application starts it >> starts taking data for 1-2 hours, process it , then take the data for next >> 1 hour process it. Now when its done processing of previous 5 hours data >> which missed, normal streaming should start with the given slide interval. >> >> Please suggest any ideas and feasibility of this. >> >> >> Thanks !! >> Abhi >> >
