Hi Devs,
Thanks all for a very prompt response! That was insanely quick. Merci
beaucoup! :)
Pozdrawiam,
Jacek Laskowski
https://about.me/JacekLaskowski
The Internals of Spark SQL https://bit.ly/spark-sql-internals
The Internals of Spark Structured Streaming
We were worried about regression when adding Kafka source v2 because it had
lots of changes. Hence we copy-pasted codes to keep the Kafka source v1
untouched and provided a config to fallback to v1.
On Mon, Aug 26, 2019 at 7:05 AM Jungtaek Lim wrote:
> Thanks! The patch is here:
OK, starting with this tomorrow...
On Mon, 26 Aug 2019, 16:05 Jungtaek Lim, wrote:
> Thanks! The patch is here: https://github.com/apache/spark/pull/25583
>
> On Mon, Aug 26, 2019 at 11:02 PM Gabor Somogyi
> wrote:
>
>> Just checked this and it's a copy-paste :) It works properly when
>>
Thanks! The patch is here: https://github.com/apache/spark/pull/25583
On Mon, Aug 26, 2019 at 11:02 PM Gabor Somogyi
wrote:
> Just checked this and it's a copy-paste :) It works properly when
> KafkaSourceInitialOffsetWriter used. Pull me in if review needed.
>
> BR,
> G
>
>
> On Mon, Aug 26,
Just checked this and it's a copy-paste :) It works properly when
KafkaSourceInitialOffsetWriter used. Pull me in if review needed.
BR,
G
On Mon, Aug 26, 2019 at 3:57 PM Jungtaek Lim wrote:
> Nice finding! I don't see any reason to not use
> KafkaSourceInitialOffsetWriter from KafkaSource, as
Nice finding! I don't see any reason to not use
KafkaSourceInitialOffsetWriter from KafkaSource, as they're identical. I
guess it was copied and pasted sometime before and not addressed yet.
As you haven't submit a patch, I'll submit a patch shortly, with mentioning
credit. I'd close mine and wait
Hi,
Just found out that KafkaSource [1] does not
use KafkaSourceInitialOffsetWriter (of KafkaMicroBatchStream) [2] for
initial offsets.
Any reason for that? Should I report an issue? Just checking out as I'm
with 2.4.3 exclusively and have no idea what's coming for 3.0.
[1]
