Re: [VOTE} KIP-733: change Kafka Streams default replication factor config

[Sophie Blee-Goldman]

That's a good summary of the situation in my mind. Definitely +1 to
highlighting each of these
specifically in the KIP

On Fri, Apr 23, 2021 at 6:19 PM Ismael Juma  wrote:

> Thanks for the comments. Is the following fair?
>
> 1. It's good that upgrades are not affected under normal circumstances and
> it mitigates the compatibility impact greatly.
> 2. We're saying that the previous behavior of defaulting to 1 is dangerous
> and we don't want to fallback to that.
> 3. The main motivation for the previous default was to make the quickstart
> easy and we can achieve that with the new default.
> 4. This change raises the minimum Kafka version required by Kafka Streams
> with the default configuration. We should highlight that and update the
> docs accordingly.
> 5. Kafka 2.3.x clusters are not that uncommon, but we're ok with users
> having to set this config explicitly when deploying new applications in
> such clusters.
>
> If so, I'm fine with the KIP, but I think we should update it to cover the
> points above (some may already be covered).
>
> Ismael
>
> On Tue, Apr 20, 2021 at 12:59 PM Sophie Blee-Goldman
>  wrote:
>
> > FWIW if someone is deploying a new Streams application to production and
> > they don't
> > set this config, either because they are not reading the docs or they
> just
> > forgot, then imo
> > it's actually better for them to hit this exception and fail-fast if the
> > brokers don't support it.
> >
> > Currently, the default value is 1, which means if they forget to set this
> > config then they're
> > running a fragile application in production with potential for data loss.
> > Setting it to -1 will
> > mean in most cases the appropriate value for that environment is chosen.
> > And in the subset
> > of applications where the user has forgotten to set this config and are
> > running against older
> > brokers, better to fail-fast and alert the user to set this config then
> to
> > silently downgrade and
> > pick a value that might be completely wrong for that environment.
> >
> > I don't necessarily agree that 2.3 is so old that we can assume most
> > clusters will understand
> > this value, my experience has very much been otherwise. However I do
> think
> > it's likely to be
> > quite rare for this to impact any users on upgrade. For that to happen,
> > they would need to be
> > on older brokers and running an application with the default replication
> > value in their production
> > environment, AND they would need to have deleted their internal topics
> for
> > some reason prior
> > to the upgrade. Even if all of those things are true, then (1) if the
> > internal topics were deleted
> > then they have essentially reset their app, which means this case is
> really
> > more like starting up
> > a new app than upgrading a live one, and (2) they probably do not mean to
> > run with this setting
> > in production in the first place, and they have an opportunity to fix it
> > when they recreate the topics
> >
> > Just my 2 cents
> >
> >
> > On Mon, Apr 19, 2021 at 8:54 PM Matthias J. Sax 
> wrote:
> >
> > > Thanks Ismael,
> > >
> > > just to clarify, any app that is running and is regularly upgraded is
> > > not affected: As pointed out in the KIP, if there is a running app that
> > > does not overwrite the default, and the app is upgraded, it won't be
> > > affected because the repartition/changelog topics already exist.
> > >
> > > Second, as the default value is currently 1, for production application
> > > I would assume that most people change the default anyway and thus
> won't
> > > be affected either. Thus, if one deploys a new production app, or
> resets
> > > a production app with reasonable config overwrites, they won't be
> > affected.
> > >
> > > Hence, the impact on production deployment should basically be
> > > non-existent, and for those rare cases were user would be affected it
> > > should be just a small hick-up because they would either deploy a new
> > > application (maybe annoying but no harm done; and actually, they should
> > > detect in staging) or would have reset an existing app for data
> > > reprocessing, ie, they did some manual cleanup and might hit this
> corner
> > > case only when re-deploying an previously stopped app, ie, an app that
> > > is currently offline anyway. Again, no harm done, just some delay to
> > > redeploy the app.
> > >
> > > Also, we point the change out in the upgrade notes. Obviously, not
> > > everybody might read them, but in the end there are multiple guards and
> > > if a user really has issues, it sounds like a very rare corner case.
> > >
> > > Last, while it might be possible to do so, the question is really,
> which
> > > value should we pick?
> > >
> > > Thus, overall I personally don't see any need to cover this case
> > > automatically. It might not be too hard to write code and handle this
> > > case, but I would rather keep it simple as it should really not impact
> > > users in any critical way.
> > >
> > >
> > > 

Re: [VOTE} KIP-733: change Kafka Streams default replication factor config

[Ismael Juma]

Thanks for the comments. Is the following fair?

1. It's good that upgrades are not affected under normal circumstances and
it mitigates the compatibility impact greatly.
2. We're saying that the previous behavior of defaulting to 1 is dangerous
and we don't want to fallback to that.
3. The main motivation for the previous default was to make the quickstart
easy and we can achieve that with the new default.
4. This change raises the minimum Kafka version required by Kafka Streams
with the default configuration. We should highlight that and update the
docs accordingly.
5. Kafka 2.3.x clusters are not that uncommon, but we're ok with users
having to set this config explicitly when deploying new applications in
such clusters.

If so, I'm fine with the KIP, but I think we should update it to cover the
points above (some may already be covered).

Ismael

On Tue, Apr 20, 2021 at 12:59 PM Sophie Blee-Goldman
 wrote:

> FWIW if someone is deploying a new Streams application to production and
> they don't
> set this config, either because they are not reading the docs or they just
> forgot, then imo
> it's actually better for them to hit this exception and fail-fast if the
> brokers don't support it.
>
> Currently, the default value is 1, which means if they forget to set this
> config then they're
> running a fragile application in production with potential for data loss.
> Setting it to -1 will
> mean in most cases the appropriate value for that environment is chosen.
> And in the subset
> of applications where the user has forgotten to set this config and are
> running against older
> brokers, better to fail-fast and alert the user to set this config then to
> silently downgrade and
> pick a value that might be completely wrong for that environment.
>
> I don't necessarily agree that 2.3 is so old that we can assume most
> clusters will understand
> this value, my experience has very much been otherwise. However I do think
> it's likely to be
> quite rare for this to impact any users on upgrade. For that to happen,
> they would need to be
> on older brokers and running an application with the default replication
> value in their production
> environment, AND they would need to have deleted their internal topics for
> some reason prior
> to the upgrade. Even if all of those things are true, then (1) if the
> internal topics were deleted
> then they have essentially reset their app, which means this case is really
> more like starting up
> a new app than upgrading a live one, and (2) they probably do not mean to
> run with this setting
> in production in the first place, and they have an opportunity to fix it
> when they recreate the topics
>
> Just my 2 cents
>
>
> On Mon, Apr 19, 2021 at 8:54 PM Matthias J. Sax  wrote:
>
> > Thanks Ismael,
> >
> > just to clarify, any app that is running and is regularly upgraded is
> > not affected: As pointed out in the KIP, if there is a running app that
> > does not overwrite the default, and the app is upgraded, it won't be
> > affected because the repartition/changelog topics already exist.
> >
> > Second, as the default value is currently 1, for production application
> > I would assume that most people change the default anyway and thus won't
> > be affected either. Thus, if one deploys a new production app, or resets
> > a production app with reasonable config overwrites, they won't be
> affected.
> >
> > Hence, the impact on production deployment should basically be
> > non-existent, and for those rare cases were user would be affected it
> > should be just a small hick-up because they would either deploy a new
> > application (maybe annoying but no harm done; and actually, they should
> > detect in staging) or would have reset an existing app for data
> > reprocessing, ie, they did some manual cleanup and might hit this corner
> > case only when re-deploying an previously stopped app, ie, an app that
> > is currently offline anyway. Again, no harm done, just some delay to
> > redeploy the app.
> >
> > Also, we point the change out in the upgrade notes. Obviously, not
> > everybody might read them, but in the end there are multiple guards and
> > if a user really has issues, it sounds like a very rare corner case.
> >
> > Last, while it might be possible to do so, the question is really, which
> > value should we pick?
> >
> > Thus, overall I personally don't see any need to cover this case
> > automatically. It might not be too hard to write code and handle this
> > case, but I would rather keep it simple as it should really not impact
> > users in any critical way.
> >
> >
> > Thoughts?
> >
> >
> > -Matthias
> >
> > On 4/18/21 8:00 PM, Ismael Juma wrote:
> > > Is the following accurate? Do we have data that not many users would be
> > > affected?
> > >
> > > "We also believe that 2.3.0 broker a sufficiently old such that not too
> > > many user may be affected"
> > >
> > > I wonder if we should fallback to an actual value if the brokers are
> > 2.3.x
> > > or older.

Re: [DISCUSS] KIP-735: Increase default consumer session timeout

[Ismael Juma]

Thanks for the KIP, +1.

Ismael

On Fri, Apr 23, 2021 at 9:29 AM Jason Gustafson 
wrote:

> Hi All,
>
> I've created a KIP to change the default consumer session timeout as well
> as some related behavior:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-735%3A+Increase+default+consumer+session+timeout
> .
> Have a look and let me know what you think.
>
> -Jason
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #66

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-725: Streamlining configurations for TimeWindowedDeserializer.

[Sophie Blee-Goldman]

Thanks Sagar, these changes make sense to me. I don't think we need to call
for a
new vote unless there are any concerns raised, but I feel this is probably
not
too controversial.

Thanks for the update

On Fri, Apr 23, 2021 at 3:17 AM Sagar  wrote:

> Hi All,
>
> After working on the changes proposed in the KIP, it was pointed out by
> Sophie that the KIP needs to be upgraded to include Serialisers as well. In
> line with this, I. have updated the KIP with the following changes:
>
> 1) Renamed the newly proposed config window.inner.class.deserialiser to
> windowed.inner.class.serde. 2 changes here are => changed window to
> windowed and replaced deserialiser with serde. The second change will
> ensure the config works for both serialisers and deserialisers.
> 2).Added  better formatting in the page to make it more readable.
>
> i am not sure if we will need a new vote for these so please let me know!
>
> Thanks!
> Sagar.
>
> On Sun, Apr 4, 2021 at 7:55 AM Sophie Blee-Goldman
>  wrote:
>
> > I think you can start the vote, we can always return to the discussion if
> > someone raises a
> > concern during voting.
> >
> > Anyways I think/hope this won't be too controversial since we went
> through
> > and agreed on
> > a similar interface not long ago with KIP-659
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-659%3A+Improve+TimeWindowedDeserializer+and+TimeWindowedSerde+to+handle+window+size
> > >
> >
> > On Sat, Apr 3, 2021 at 4:26 AM Sagar  wrote:
> >
> > > Thanks Leah/Sophie.
> > >
> > > I have updated the KIP with the new config.
> > >
> > > Do you think we can proceed with the voting process for the KIP or
> should
> > > we wait a bit longer?
> > >
> > > Thanks!
> > > Sagar.
> > >
> > > On Fri, Apr 2, 2021 at 11:59 PM Sophie Blee-Goldman
> > >  wrote:
> > >
> > > > Yeah sure, window.inner.class.deserializer sounds good to me
> > > >
> > > > On Fri, Apr 2, 2021 at 11:28 AM Leah Thomas
> >  > > >
> > > > wrote:
> > > >
> > > > > Hey Sagar,
> > > > >
> > > > > Thanks for picking this up! The proposal looks good to me after
> > Sophie
> > > > and
> > > > > John's changes.
> > > > >
> > > > > Cheers,
> > > > > Leah
> > > > >
> > > > > On Fri, Apr 2, 2021 at 6:21 AM Sagar 
> > > wrote:
> > > > >
> > > > > > Thanks John!
> > > > > >
> > > > > > Yeah I think window.inner.class.deserializer sounds good. Your
> > > thoughts
> > > > > > @Sophie?
> > > > > >
> > > > > > Thanks!
> > > > > > Sagar.
> > > > > >
> > > > > >
> > > > > > On Fri, Apr 2, 2021 at 5:23 AM John Roesler  >
> > > > wrote:
> > > > > >
> > > > > > > Hi Sagar,
> > > > > > >
> > > > > > > I think this is a good proposal.
> > > > > > >
> > > > > > > The only change I might recommend is to change the config to
> more
> > > > > closely
> > > > > > > align with the other one, for example:
> > > > > “window.inner.class.deserializer”
> > > > > > >
> > > > > > > But it’s very minor; I leave it to your judgement.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John
> > > > > > >
> > > > > > > On Fri, Mar 26, 2021, at 03:36, Sagar wrote:
> > > > > > > > Hi Sophie,
> > > > > > > >
> > > > > > > > Thanks for the feedback! I have updated the KIP inline with
> > > > whatever
> > > > > > you
> > > > > > > > suggested.
> > > > > > > >
> > > > > > > > Regarding point 5, I have added the note as it makes sense to
> > not
> > > > set
> > > > > > the
> > > > > > > > config via the KafkaStreams app.
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > > Sagar.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Mar 24, 2021 at 7:52 AM Sophie Blee-Goldman
> > > > > > > >  wrote:
> > > > > > > >
> > > > > > > > > Hey Sagar,
> > > > > > > > >
> > > > > > > > > Thanks for the KIP! The overall proposal looks good to me,
> > but
> > > I
> > > > > had
> > > > > > > some
> > > > > > > > > miscellaneous notes:
> > > > > > > > >
> > > > > > > > > 1) Some general KIP-writing advice:
> > > > > > > > > - You don't need to list any implementation details,
> only
> > > > > public
> > > > > > > > > interfaces that are being added, modified, or
> > > > > > > > >   deprecated. It's better to describe your changes in
> > > words,
> > > > or
> > > > > > > > > occasionally psuedo-code for more complicated
> > > > > > > > >   changes or algorithms. The KIP is a public contract,
> so
> > > you
> > > > > > > generally
> > > > > > > > > want to agree upon the high-level
> > > > > > > > >   proposal and avoid getting locked in to specific code
> > > which
> > > > > you
> > > > > > > may
> > > > > > > > > end up wanting to change once you
> > > > > > > > >   start on the PR.
> > > > > > > > >   In this KIP, you can remove the code block under
> > > > > > > > > *TimeWindowedDeserializer
> > > > > > > > > *and just describe the desired
> > > > > > > > >   semantics: eg that you should only use the
> constructor
> > > > within
> > > > > > > > > Streams, the configs within the console consumer,
> > > > > > > > >   or either for a plain 

Re: [DISCUSS] KIP-735: Increase default consumer session timeout

[Bill Bejeck]

Hi Jason,

Thanks for the well-written KIP.
>From the reasons outlined in the KIP,  I think this a great change.

-Bill

On Fri, Apr 23, 2021 at 2:52 PM Sophie Blee-Goldman
 wrote:

> Hey Jason,
>
> Thanks for the KIP, I think increasing the default session timeout would be
> great. The choice of
> 45s sounds reasonable to me.
>
> -Sophie
>
> On Fri, Apr 23, 2021 at 9:29 AM Jason Gustafson  >
> wrote:
>
> > Hi All,
> >
> > I've created a KIP to change the default consumer session timeout as well
> > as some related behavior:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-735%3A+Increase+default+consumer+session+timeout
> > .
> > Have a look and let me know what you think.
> >
> > -Jason
> >
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #65

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-735: Increase default consumer session timeout

[Sophie Blee-Goldman]

Hey Jason,

Thanks for the KIP, I think increasing the default session timeout would be
great. The choice of
45s sounds reasonable to me.

-Sophie

On Fri, Apr 23, 2021 at 9:29 AM Jason Gustafson 
wrote:

> Hi All,
>
> I've created a KIP to change the default consumer session timeout as well
> as some related behavior:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-735%3A+Increase+default+consumer+session+timeout
> .
> Have a look and let me know what you think.
>
> -Jason
>

RE: Re: [DISCUSS] KIP-679 Producer will enable the strongest delivery guarantee by default

[Soumyajit Sahu]

I would like to raise an issue with making enable.idempotence=true by
default.

This has the ability to crash a broker with OutOfMemory exception whenever
there are too many producerIds.

This can take many current deployments by surprise.

I would suggest to hold on to this change until we have figured a way to
check the above behavior.

I have a left a comment too in the JIRA tracking this task:
https://issues.apache.org/jira/plugins/servlet/mobile#issue/KAFKA-10619

Regards,
Soumyajit

On 2020/11/04 19:49:46 Cheng Tan wrote:
> I made the following changes since I sent out the last discussion message:
>
> 1. Rename org.apache.kafka.server.authorizer.Authorizer#authorizeAny to
org.apache.kafka.server.authorizer.Authorizer#authorizeByResourceType
> 2. Optimized the interface default of
org.apache.kafka.server.authorizer.Authorizer#authorizeAny, now it’s not
coupled with org.apache.kafka.server.authorizer.Authorizer#authorize any
more and having a better performance.
>
> Please feel free to comment and leave any thoughts. Any feedback will be
appreciated. Thanks.
>
> Best, - Cheng
>
> > On Oct 19, 2020, at 9:15 PM, Cheng Tan  wrote:
> >
> > Hi all,
> >
> > I’m proposing a new KIP for enabling the strongest delivery guarantee
by default. Today Kafka support EOS and N-1 concurrent failure tolerance
but the default settings haven’t bring them out of the box. The proposal is
discussing the best approach to change the producer defaults to `ack=all`
and `enable.idempotence=true`. Please join the discussion here:
> >
> >
https://cwiki.apache.org/confluence/display/KAFKA/KIP-679%3A+Producer+will+enable+the+strongest+delivery+guarantee+by+default
<
https://cwiki.apache.org/confluence/display/KAFKA/KIP-679:+Producer+will+enable+the+strongest+delivery+guarantee+by+default
>
> >
> > Thanks
> >
> > - Cheng Tan
>
>

[jira] [Created] (KAFKA-12712) KRaft: Missing controller.quorom.voters config not properly handled

[Magnus Edenhill (Jira)]

Magnus Edenhill created KAFKA-12712:
---

 Summary: KRaft: Missing controller.quorom.voters config not 
properly handled
 Key: KAFKA-12712
 URL: https://issues.apache.org/jira/browse/KAFKA-12712
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 2.8.0
Reporter: Magnus Edenhill


When trying out KRaft in 2.8 I mispelled controller.quorum.voters as 
controller.quorum.voters, but the broker did not fail to start, nor did it 
print any warning.

 

Instead it raised this error:

 
{code:java}
[2021-04-23 18:25:13,484] INFO Starting controller 
(kafka.server.ControllerServer)[2021-04-23 18:25:13,484] INFO Starting 
controller (kafka.server.ControllerServer)[2021-04-23 18:25:13,485] ERROR 
[kafka-raft-io-thread]: Error due to 
(kafka.raft.KafkaRaftManager$RaftIoThread)java.lang.IllegalArgumentException: 
bound must be positive at java.util.Random.nextInt(Random.java:388) at 
org.apache.kafka.raft.RequestManager.findReadyVoter(RequestManager.java:57) at 
org.apache.kafka.raft.KafkaRaftClient.maybeSendAnyVoterFetch(KafkaRaftClient.java:1778)
 at 
org.apache.kafka.raft.KafkaRaftClient.pollUnattachedAsObserver(KafkaRaftClient.java:2080)
 at 
org.apache.kafka.raft.KafkaRaftClient.pollUnattached(KafkaRaftClient.java:2061) 
at 
org.apache.kafka.raft.KafkaRaftClient.pollCurrentState(KafkaRaftClient.java:2096)
 at org.apache.kafka.raft.KafkaRaftClient.poll(KafkaRaftClient.java:2181) at 
kafka.raft.KafkaRaftManager$RaftIoThread.doWork(RaftManager.scala:53) at 
kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
{code}
which I guess eventually (1 minute later) lead to this error which terminated 
the broker:
{code:java}
[2021-04-23 18:26:14,435] ERROR [BrokerLifecycleManager id=2] Shutting down 
because we were unable to register with the controller quorum. 
(kafka.server.BrokerLifecycleManager)[2021-04-23 18:26:14,435] ERROR 
[BrokerLifecycleManager id=2] Shutting down because we were unable to register 
with the controller quorum. (kafka.server.BrokerLifecycleManager)[2021-04-23 
18:26:14,436] INFO [BrokerLifecycleManager id=2] registrationTimeout: shutting 
down event queue. (org.apache.kafka.queue.KafkaEventQueue)[2021-04-23 
18:26:14,437] INFO [BrokerLifecycleManager id=2] Transitioning from STARTING to 
SHUTTING_DOWN. (kafka.server.BrokerLifecycleManager)[2021-04-23 18:26:14,437] 
INFO [broker-2-to-controller-send-thread]: Shutting down 
(kafka.server.BrokerToControllerRequestThread)[2021-04-23 18:26:14,438] INFO 
[broker-2-to-controller-send-thread]: Stopped 
(kafka.server.BrokerToControllerRequestThread)[2021-04-23 18:26:14,438] INFO 
[broker-2-to-controller-send-thread]: Shutdown completed 
(kafka.server.BrokerToControllerRequestThread)[2021-04-23 18:26:14,441] ERROR 
[BrokerServer id=2] Fatal error during broker startup. Prepare to shutdown 
(kafka.server.BrokerServer)java.util.concurrent.CancellationException at 
java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276) at 
kafka.server.BrokerLifecycleManager$ShutdownEvent.run(BrokerLifecycleManager.scala:474)
 at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:174)
 at java.lang.Thread.run(Thread.java:748)
{code}
But since the client listeners were made available prior to shutting down, the 
broker was deemed up and operational by the (naiive) monitoring tool.

So..:

 - Broker should fail on startup on invalid/unknown config properties. I 
understand this is tehcnically tricky, so at least a warning log should be 
printed.

 - Perhaps not create client listeners before control plane is somewhat happy.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[DISCUSS] KIP-735: Increase default consumer session timeout

[Jason Gustafson]

Hi All,

I've created a KIP to change the default consumer session timeout as well
as some related behavior:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-735%3A+Increase+default+consumer+session+timeout.
Have a look and let me know what you think.

-Jason

[jira] [Created] (KAFKA-12711) Add a back off option to Replace thread

[Walker Carlson (Jira)]

Walker Carlson created KAFKA-12711:
--

 Summary: Add a back off option to Replace thread
 Key: KAFKA-12711
 URL: https://issues.apache.org/jira/browse/KAFKA-12711
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Walker Carlson


There should be a native option to set a back off when replacing a thread.

 

Either there should be a config and a user chosen strategy or a value you can 
set in the handler that causes a delay in creating the new thread.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

CFP for ApacheCon 2021 closes in ONE WEEK

[Rich Bowen]

[You are receiving this because you're subscribed to one or more dev@
mailing lists for an Apache project, or the ApacheCon Announce list.]

Time is running out to submit your talk for ApacheCon 2021.

The Call for Presentations for ApacheCon @Home 2021, focused on Europe
and North America time zones, closes May 3rd, and is at
https://www.apachecon.com/acah2021/cfp.html

The CFP for ApacheCon Asia, focused on Asia/Pacific time zones, is at
https://apachecon.com/acasia2021/cfp.html and also closes on May 3rd.

ApacheCon is our main event, featuring content from any and all of our
projects, and is your best opportunity to get your project in front of
the largest audience of enthusiasts.

Please don't wait for the last minute. Get your talks in today!

--
Rich Bowen, VP Conferences
The Apache Software Foundation
https://apachecon.com/
@apachecon

[GitHub] [kafka-site] cadonna merged pull request #353: Add Bruno Cadonna to committers

[GitBox]

cadonna merged pull request #353:
URL: https://github.com/apache/kafka-site/pull/353


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[VOTE] KIP-734: Improve AdminClient.listOffsets to return timestamp and offset for the record with the largest timestamp

[Thomas Scott]

Hey all,

  I'm starting the voting on KIP-734.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-734%3A+Improve+AdminClient.listOffsets+to+return+timestamp+and+offset+for+the+record+with+the+largest+timestamp

Thanks

  Tom

Re: [DISCUSS] KIP-725: Streamlining configurations for TimeWindowedDeserializer.

[Sagar]

Hi All,

After working on the changes proposed in the KIP, it was pointed out by
Sophie that the KIP needs to be upgraded to include Serialisers as well. In
line with this, I. have updated the KIP with the following changes:

1) Renamed the newly proposed config window.inner.class.deserialiser to
windowed.inner.class.serde. 2 changes here are => changed window to
windowed and replaced deserialiser with serde. The second change will
ensure the config works for both serialisers and deserialisers.
2).Added  better formatting in the page to make it more readable.

i am not sure if we will need a new vote for these so please let me know!

Thanks!
Sagar.

On Sun, Apr 4, 2021 at 7:55 AM Sophie Blee-Goldman
 wrote:

> I think you can start the vote, we can always return to the discussion if
> someone raises a
> concern during voting.
>
> Anyways I think/hope this won't be too controversial since we went through
> and agreed on
> a similar interface not long ago with KIP-659
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-659%3A+Improve+TimeWindowedDeserializer+and+TimeWindowedSerde+to+handle+window+size
> >
>
> On Sat, Apr 3, 2021 at 4:26 AM Sagar  wrote:
>
> > Thanks Leah/Sophie.
> >
> > I have updated the KIP with the new config.
> >
> > Do you think we can proceed with the voting process for the KIP or should
> > we wait a bit longer?
> >
> > Thanks!
> > Sagar.
> >
> > On Fri, Apr 2, 2021 at 11:59 PM Sophie Blee-Goldman
> >  wrote:
> >
> > > Yeah sure, window.inner.class.deserializer sounds good to me
> > >
> > > On Fri, Apr 2, 2021 at 11:28 AM Leah Thomas
>  > >
> > > wrote:
> > >
> > > > Hey Sagar,
> > > >
> > > > Thanks for picking this up! The proposal looks good to me after
> Sophie
> > > and
> > > > John's changes.
> > > >
> > > > Cheers,
> > > > Leah
> > > >
> > > > On Fri, Apr 2, 2021 at 6:21 AM Sagar 
> > wrote:
> > > >
> > > > > Thanks John!
> > > > >
> > > > > Yeah I think window.inner.class.deserializer sounds good. Your
> > thoughts
> > > > > @Sophie?
> > > > >
> > > > > Thanks!
> > > > > Sagar.
> > > > >
> > > > >
> > > > > On Fri, Apr 2, 2021 at 5:23 AM John Roesler 
> > > wrote:
> > > > >
> > > > > > Hi Sagar,
> > > > > >
> > > > > > I think this is a good proposal.
> > > > > >
> > > > > > The only change I might recommend is to change the config to more
> > > > closely
> > > > > > align with the other one, for example:
> > > > “window.inner.class.deserializer”
> > > > > >
> > > > > > But it’s very minor; I leave it to your judgement.
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > > On Fri, Mar 26, 2021, at 03:36, Sagar wrote:
> > > > > > > Hi Sophie,
> > > > > > >
> > > > > > > Thanks for the feedback! I have updated the KIP inline with
> > > whatever
> > > > > you
> > > > > > > suggested.
> > > > > > >
> > > > > > > Regarding point 5, I have added the note as it makes sense to
> not
> > > set
> > > > > the
> > > > > > > config via the KafkaStreams app.
> > > > > > >
> > > > > > > Thanks!
> > > > > > > Sagar.
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Mar 24, 2021 at 7:52 AM Sophie Blee-Goldman
> > > > > > >  wrote:
> > > > > > >
> > > > > > > > Hey Sagar,
> > > > > > > >
> > > > > > > > Thanks for the KIP! The overall proposal looks good to me,
> but
> > I
> > > > had
> > > > > > some
> > > > > > > > miscellaneous notes:
> > > > > > > >
> > > > > > > > 1) Some general KIP-writing advice:
> > > > > > > > - You don't need to list any implementation details, only
> > > > public
> > > > > > > > interfaces that are being added, modified, or
> > > > > > > >   deprecated. It's better to describe your changes in
> > words,
> > > or
> > > > > > > > occasionally psuedo-code for more complicated
> > > > > > > >   changes or algorithms. The KIP is a public contract, so
> > you
> > > > > > generally
> > > > > > > > want to agree upon the high-level
> > > > > > > >   proposal and avoid getting locked in to specific code
> > which
> > > > you
> > > > > > may
> > > > > > > > end up wanting to change once you
> > > > > > > >   start on the PR.
> > > > > > > >   In this KIP, you can remove the code block under
> > > > > > > > *TimeWindowedDeserializer
> > > > > > > > *and just describe the desired
> > > > > > > >   semantics: eg that you should only use the constructor
> > > within
> > > > > > > > Streams, the configs within the console consumer,
> > > > > > > >   or either for a plain consumer client provided they
> > match.
> > > > > > > >   The code block under *StreamsConfig *however is a good
> > > > example
> > > > > of
> > > > > > > > what should be in a KIP. Only one nit:
> > > > > > > >   in the doc string, avoid saying "windowed key" and just
> > say
> > > > > > "windowed
> > > > > > > > record" or something like that.
> > > > > > > > - The *Motivation* section should be focused on any
> > > background
> > > > or
> > > > > > > > additional context that's necessary to
> > > > > > > >   understand the KIP, as 

[DISCUSS] KIP-334 Include partitions in exceptions during consumer record deserialization/validation (Round 2)

[Sarwar Bhuiyan]

Hi all,

I'm reviving this discussion after perusing the vote thread created by
Stan. I believe the KIP is already revised to include Jason and Colin's
suggestion of using a RecordDeserializationException. It signals to the
user that something needs to be done (whether skipping the record as shown
in the example and/or do something else). This is backward compatible with
what we have today and is clean.

Mathias suggested that we split the exception types into a
RecordDeserializationException and a CorruptRecordException with the
intention of auto-skipping corrupt records. I would prefer to use another
KIP for that type of functionality.

KIP:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=87297793

JIRAs:
https://issues.apache.org/jira/browse/KAFKA-5682

Last Vote thread comment from Stan:
https://www.mail-archive.com/dev@kafka.apache.org/msg92392.html

On another note, I need edit access to the KIP page. Could somebody give me
access should I need to make changes? My username is sarwarb.

Thanks.

Sarwar

[GitHub] [kafka-site] cadonna opened a new pull request #353: Add Bruno Cadonna to committers

[GitBox]

cadonna opened a new pull request #353:
URL: https://github.com/apache/kafka-site/pull/353


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [VOTE] KIP-334 Include partitions in exceptions raised during consumer record deserialization/validation

[Sarwar Bhuiyan]

Thanks Colin, I'll open a new discussion thread. It does look like the KIP
has been updated with your and Jason's suggestions but not Mathias' which
is a different direction (splitting into separate exceptions for "corrupt"
records and auto-skipping those).



On Thu, Apr 22, 2021 at 10:30 PM Colin McCabe  wrote:

> Hi Sarwar,
>
> It seems like we didn't reach a consensus last time about this KIP.  So I
> would suggest revising the KIP, and then creating a new DISCUSS thread when
> it's ready for review.
>
> regards,
> Colin
>
>
> On Wed, Apr 14, 2021, at 01:26, Sarwar Bhuiyan wrote:
> > Forgive me while I find my feet, but since this reached a VOTE stage, is
> it
> > really required to go through a DISCUSS thread again or can we continue
> to
> > address the issues on this thread since the issues were raised on this
> > thread? If I do have to create a new DISCUSS thread, do I just copy the
> > messages from the old thread to it to address them?
> >
> > Thank you.
> >
> > Sarwar
> >
> > On Wed, Apr 14, 2021 at 9:18 AM Stanislav Kozlovski <
> stanis...@confluent.io>
> > wrote:
> >
> > > Hey all,
> > >
> > > To revive this old KIP, Sarwar Bhuiyan has volunteered to take over
> > > ownership.
> > > He will continue to drive this KIP to approval and completion - I
> > > understand that he will re-start the discussion with a new [DISCUSS] or
> > > [VOTE] thread.
> > >
> > > Thank you Sarwar!
> > >
> > > Best,
> > > Stanislav
> > >
> > > On Fri, Jan 10, 2020 at 5:55 PM Gwen Shapira 
> wrote:
> > >
> > >> Sorry for the super late reply, but since the vote thread showed up,
> > >> I've read the KIP and have a concern.
> > >> The concern was raised by Matthias Sax earlier, but I didn't see it
> > >> addressed.
> > >>
> > >> Basically the current iteration of the KIP unifies deserialization
> > >> errors with corruption. As was pointed out, these are not the same
> > >> thing. Corruption means that the message itself (including envelope,
> > >> not just the payload) is broken. De-serialization errors mean that
> > >> either key or value serializers have a problem. It can even be a
> > >> temporary problem of connecting to schema registry, I believe. Corrupt
> > >> messages can only be skipped. De-serialization errors can (and
> > >> arguably should) be retried with a different serializer. Something
> > >> like Connect will need to skip corrupt messages, but messages with
> > >> SerDe issues should probably go into a dead-letter queue.
> > >>
> > >> Anyway, IMO we need exceptions that will let us tell the difference.
> > >>
> > >> Gwen
> > >>
> > >> On Fri, Oct 11, 2019 at 10:05 AM Stanislav Kozlovski
> > >>  wrote:
> > >> >
> > >> > Thanks Jason. I've edited the KIP with the latest proposal.
> > >> >
> > >> > On Thu, Oct 10, 2019 at 2:00 AM Jason Gustafson  >
> > >> wrote:
> > >> >
> > >> > > Hi Stanislav,
> > >> > >
> > >> > > Sorry for the late comment. I'm happy with the current proposal.
> Just
> > >> one
> > >> > > small request is to include an example which shows how a user
> could
> > >> use
> > >> > > this to skip over a record.
> > >> > >
> > >> > > I'd suggest pushing this to a vote to see if anyone else has
> feedback.
> > >> > >
> > >> > > Thanks,
> > >> > > Jason
> > >> > >
> > >> > > On Sat, Jul 13, 2019 at 2:27 PM Stanislav Kozlovski <
> > >> > > stanis...@confluent.io>
> > >> > > wrote:
> > >> > >
> > >> > > > Hello,
> > >> > > >
> > >> > > > Could we restart the discussion here again?
> > >> > > >
> > >> > > > My last message was sent on the 3rd of June but I haven't
> received
> > >> > > replies
> > >> > > > since then.
> > >> > > >
> > >> > > > I'd like to get this KIP to a finished state, regardless of
> whether
> > >> that
> > >> > > is
> > >> > > > merged-in or discarded. It has been almost one year since the
> > >> publication
> > >> > > > of the KIP.
> > >> > > >
> > >> > > > Thanks,
> > >> > > > Stanislav
> > >> > > >
> > >> > > > On Mon, Jun 3, 2019 at 11:19 AM Stanislav Kozlovski <
> > >> > > > stanis...@confluent.io>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Do people agree with the approach I outlined in my last reply?
> > >> > > > >
> > >> > > > > On Mon, May 6, 2019 at 2:12 PM Stanislav Kozlovski <
> > >> > > > stanis...@confluent.io>
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > >> Hey there Kamal,
> > >> > > > >>
> > >> > > > >> I'm sincerely sorry for missing your earlier message. As I
> open
> > >> this
> > >> > > > >> thread up, I see I have an unsent draft message about
> resuming
> > >> > > > discussion
> > >> > > > >> from some time ago.
> > >> > > > >>
> > >> > > > >> In retrospect, I think I may have been too pedantic with the
> > >> exception
> > >> > > > >> naming and hierarchy.
> > >> > > > >> I now believe a single exception type of
> > >> > > > `RecordDeserializationException`
> > >> > > > >> is enough. Let's go with that.
> > >> > > > >>
> > >> > > > >> On Mon, May 6, 2019 at 6:40 AM Kamal Chandraprakash <
> > >> > > > >> kamal.chandraprak...@gmail.com>