Nice finding! I don't see any reason to not use KafkaSourceInitialOffsetWriter from KafkaSource, as they're identical. I guess it was copied and pasted sometime before and not addressed yet. As you haven't submit a patch, I'll submit a patch shortly, with mentioning credit. I'd close mine and wait for your patch if you plan to do it. Please let me know.
Thanks, Jungtaek Lim (HeartSaVioR) On Mon, Aug 26, 2019 at 8:03 PM Jacek Laskowski <[email protected]> wrote: > Hi, > > Just found out that KafkaSource [1] does not > use KafkaSourceInitialOffsetWriter (of KafkaMicroBatchStream) [2] for > initial offsets. > > Any reason for that? Should I report an issue? Just checking out as I'm > with 2.4.3 exclusively and have no idea what's coming for 3.0. > > [1] > https://github.com/apache/spark/blob/master/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala#L102 > > [2] > https://github.com/apache/spark/blob/master/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala#L281 > > Pozdrawiam, > Jacek Laskowski > ---- > https://about.me/JacekLaskowski > The Internals of Spark SQL https://bit.ly/spark-sql-internals > The Internals of Spark Structured Streaming > https://bit.ly/spark-structured-streaming > The Internals of Apache Kafka https://bit.ly/apache-kafka-internals > Follow me at https://twitter.com/jaceklaskowski > > -- Name : Jungtaek Lim Blog : http://medium.com/@heartsavior Twitter : http://twitter.com/heartsavior LinkedIn : http://www.linkedin.com/in/heartsavior
