We were worried about regression when adding Kafka source v2 because it had lots of changes. Hence we copy-pasted codes to keep the Kafka source v1 untouched and provided a config to fallback to v1.
On Mon, Aug 26, 2019 at 7:05 AM Jungtaek Lim <kabh...@gmail.com> wrote: > Thanks! The patch is here: https://github.com/apache/spark/pull/25583 > > On Mon, Aug 26, 2019 at 11:02 PM Gabor Somogyi <gabor.g.somo...@gmail.com> > wrote: > >> Just checked this and it's a copy-paste :) It works properly when >> KafkaSourceInitialOffsetWriter used. Pull me in if review needed. >> >> BR, >> G >> >> >> On Mon, Aug 26, 2019 at 3:57 PM Jungtaek Lim <kabh...@gmail.com> wrote: >> >>> Nice finding! I don't see any reason to not use >>> KafkaSourceInitialOffsetWriter from KafkaSource, as they're identical. I >>> guess it was copied and pasted sometime before and not addressed yet. >>> As you haven't submit a patch, I'll submit a patch shortly, with >>> mentioning credit. I'd close mine and wait for your patch if you plan to do >>> it. Please let me know. >>> >>> Thanks, >>> Jungtaek Lim (HeartSaVioR) >>> >>> >>> On Mon, Aug 26, 2019 at 8:03 PM Jacek Laskowski <ja...@japila.pl> wrote: >>> >>>> Hi, >>>> >>>> Just found out that KafkaSource [1] does not >>>> use KafkaSourceInitialOffsetWriter (of KafkaMicroBatchStream) [2] for >>>> initial offsets. >>>> >>>> Any reason for that? Should I report an issue? Just checking out as I'm >>>> with 2.4.3 exclusively and have no idea what's coming for 3.0. >>>> >>>> [1] >>>> https://github.com/apache/spark/blob/master/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala#L102 >>>> >>>> [2] >>>> https://github.com/apache/spark/blob/master/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala#L281 >>>> >>>> Pozdrawiam, >>>> Jacek Laskowski >>>> ---- >>>> https://about.me/JacekLaskowski >>>> The Internals of Spark SQL https://bit.ly/spark-sql-internals >>>> The Internals of Spark Structured Streaming >>>> https://bit.ly/spark-structured-streaming >>>> The Internals of Apache Kafka https://bit.ly/apache-kafka-internals >>>> Follow me at https://twitter.com/jaceklaskowski >>>> >>>> >>> >>> -- >>> Name : Jungtaek Lim >>> Blog : http://medium.com/@heartsavior >>> Twitter : http://twitter.com/heartsavior >>> LinkedIn : http://www.linkedin.com/in/heartsavior >>> >> > > -- > Name : Jungtaek Lim > Blog : http://medium.com/@heartsavior > Twitter : http://twitter.com/heartsavior > LinkedIn : http://www.linkedin.com/in/heartsavior > -- Best Regards, Ryan
