Hi Jungtaek, Thanks for the input. I did tried and it worked. I got confused earlier after reading some blogs.
Regards Amit On Friday, August 28, 2020, Jungtaek Lim <kabhwan.opensou...@gmail.com> wrote: > Hi Amit, > > if I remember correctly, you don't need to restart the query to reflect > the newly added topic and partition, if your subscription covers the topic > (like subscribe pattern). Please try it out. > > Hope this helps. > > Thanks, > Jungtaek Lim (HeartSaVioR) > > On Fri, Aug 28, 2020 at 1:56 PM Amit Joshi <mailtojoshia...@gmail.com> > wrote: > >> Any pointers will be appreciated. >> >> On Thursday, August 27, 2020, Amit Joshi <mailtojoshia...@gmail.com> >> wrote: >> >>> Hi All, >>> >>> I am trying to understand the effect of adding topics and partitions to >>> a topic in kafka, which is being consumed by spark structured streaming >>> applications. >>> >>> Do we have to restart the spark structured streaming application to read >>> from the newly added topic? >>> Do we have to restart the spark structured streaming application to read >>> from the newly added partition to a topic? >>> >>> Kafka consumers have a meta data refresh property that works without >>> restarting. >>> >>> Thanks advance. >>> >>> Regards >>> Amit Joshi >>> >>