Re: [DISCUSS] Flip the default value of Kafka offset fetching config (spark.sql.streaming.kafka.useDeprecatedOffsetFetching)

2022-10-18 Thread Jungtaek Lim
No further voice so far. I'm going to submit a PR. Thanks again for the feedback! On Mon, Oct 17, 2022 at 9:30 AM Jungtaek Lim wrote: > Thanks Gabor and Dongjoon for supporting this! > > Bump to reach more eyes. If there is no further voice on this in a couple > of days, I'll consider it as a

Re: [DISCUSS] Flip the default value of Kafka offset fetching config (spark.sql.streaming.kafka.useDeprecatedOffsetFetching)

2022-10-16 Thread Jungtaek Lim
Thanks Gabor and Dongjoon for supporting this! Bump to reach more eyes. If there is no further voice on this in a couple of days, I'll consider it as a lazy consensus and submit a PR to this. On Sat, Oct 15, 2022 at 3:32 AM Dongjoon Hyun wrote: > +1 > > I agree with Jungtaek and Gabor about

Re: [DISCUSS] Flip the default value of Kafka offset fetching config (spark.sql.streaming.kafka.useDeprecatedOffsetFetching)

2022-10-14 Thread Dongjoon Hyun
+1 I agree with Jungtaek and Gabor about switching the default value of configurations with the migration guide. Dongjoon On Thu, Oct 13, 2022 at 12:46 AM Gabor Somogyi wrote: > Hi Jungtaek, > > Good to hear that the new approach is working fine. +1 from my side. > > BR, > G > > > On Thu, Oct

Re: [DISCUSS] Flip the default value of Kafka offset fetching config (spark.sql.streaming.kafka.useDeprecatedOffsetFetching)

2022-10-13 Thread Gabor Somogyi
Hi Jungtaek, Good to hear that the new approach is working fine. +1 from my side. BR, G On Thu, Oct 13, 2022 at 4:12 AM Jungtaek Lim wrote: > Hi all, > > I would like to propose flipping the default value of Kafka offset > fetching config. The context is following: > > Before Spark 3.1,

[DISCUSS] Flip the default value of Kafka offset fetching config (spark.sql.streaming.kafka.useDeprecatedOffsetFetching)

2022-10-12 Thread Jungtaek Lim
Hi all, I would like to propose flipping the default value of Kafka offset fetching config. The context is following: Before Spark 3.1, there was only one approach on fetching offset, using consumer.poll(0). This has been pointed out as a root cause for hang since there is no timeout for