Hi Amit, if I remember correctly, you don't need to restart the query to reflect the newly added topic and partition, if your subscription covers the topic (like subscribe pattern). Please try it out.
Hope this helps. Thanks, Jungtaek Lim (HeartSaVioR) On Fri, Aug 28, 2020 at 1:56 PM Amit Joshi <mailtojoshia...@gmail.com> wrote: > Any pointers will be appreciated. > > On Thursday, August 27, 2020, Amit Joshi <mailtojoshia...@gmail.com> > wrote: > >> Hi All, >> >> I am trying to understand the effect of adding topics and partitions to a >> topic in kafka, which is being consumed by spark structured streaming >> applications. >> >> Do we have to restart the spark structured streaming application to read >> from the newly added topic? >> Do we have to restart the spark structured streaming application to read >> from the newly added partition to a topic? >> >> Kafka consumers have a meta data refresh property that works without >> restarting. >> >> Thanks advance. >> >> Regards >> Amit Joshi >> >
