Ok, thanks for the clarification. I will try to migrate my project to structured streaming .
Regards, Renu On Fri, Mar 12, 2021 at 7:38 PM Gabor Somogyi <gabor.g.somo...@gmail.com> wrote: > Mainly bugfixes and no breaking AFAIK. > > As a side note there were intentions to close DStreams and discontinue > as-is. > It's not yet happened but it's on the road so I strongly recommend to > migrate to Structured Streaming... > We simply can't support 2 streaming engines for huge amount of time. > > G > > > On Fri, Mar 12, 2021 at 3:02 PM Renu Yadav <yren...@gmail.com> wrote: > >> Hi Gabor, >> >> It seems like it is better to upgrade my spark version . >> >> Are there major changes in terms of streaming from spark 2.2 to spark 2.4? >> >> PS: I am using KafkaUtils api to create steam >> >> Thanks & Regards, >> Renu yadav >> >> On Fri, Mar 12, 2021 at 7:25 PM Renu Yadav <yren...@gmail.com> wrote: >> >>> Thanks Gabor, >>> This is very useful. >>> >>> Regards, >>> Renu Yadav >>> >>> On Fri, Mar 12, 2021 at 5:36 PM Gabor Somogyi <gabor.g.somo...@gmail.com> >>> wrote: >>> >>>> Kafka client upgrade is not a trivial change which may or may not work >>>> since new versions can contain incompatible API and/or behavior changes. >>>> I've collected how Spark evolved in terms of Kafka client and there >>>> I've gathered the breaking changes to make our life easier. >>>> Have a look and based on that you can make your choice: >>>> https://gist.github.com/gaborgsomogyi/3476c32d69ff2087ed5d7d031653c7a9 >>>> >>>> As a general suggestion it would be best to upgrade Spark as-is because >>>> we've added many fixes which one can face... >>>> >>>> Hope this helps! >>>> >>>> G >>>> >>>> >>>> On Fri, Mar 12, 2021 at 9:45 AM Renu Yadav <yren...@gmail.com> wrote: >>>> >>>>> Hi Team, >>>>> I am using spark -2.2 and spark_streamin_kafka 2.2 , which is >>>>> pointing to kafka-client 0.10 . How can I upgrade a kafka client to kafka >>>>> 2.2.0 ? >>>>> >>>>> Thanks & Regards, >>>>> Renu Yadav >>>>> >>>>
