Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2774

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

[Justine Olshan]

Hi Jun,

20. I can update the KIP.

21. This is used to complete some of the work with KIP-360. (We use
previous producer ID there, but never persisted it which was in the KIP
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89068820)
The KIP also mentions including previous epoch but we explained in this KIP
how we can figure this out.

Justine



On Mon, Apr 1, 2024 at 3:56 PM Jun Rao  wrote:

> Hi, Justine,
>
> Thanks for the updated KIP. A couple of more comments.
>
> 20. Could we show the output of version-mapping?
>
> 21. "Transaction version 1 will include the flexible fields in the
> transaction state log, and transaction version 2 will include the changes
> to the transactional protocol as described by KIP-890 (epoch bumps and
> implicit add partitions.)"
>   So TV 1 enables the writing of new tagged fields like PrevProducerId? But
> those fields are only usable after the epoch bump, right? What
> functionality does TV 1 achieve?
>
> Jun
>
>
> On Mon, Apr 1, 2024 at 2:06 PM Justine Olshan  >
> wrote:
>
> > I have also updated the KIP to mention the feature tool's --metadata flag
> > will be deprecated.
> > It will still work for users as they learn the new flag, but a warning
> > indicating the alternatives will be shown.
> >
> > Justine
> >
> > On Thu, Mar 28, 2024 at 11:03 AM Justine Olshan 
> > wrote:
> >
> > > Hi Jun,
> > >
> > > For both transaction state and group coordinator state, there are only
> > > version 0 records.
> > > KIP-915 introduced flexible versions, but it was never put to use. MV
> has
> > > never gated these. This KIP will do that. I can include this context in
> > the
> > > KIP.
> > >
> > > I'm happy to modify his 1 and 2 to 0 and 1.
> > >
> > > Justine
> > >
> > > On Thu, Mar 28, 2024 at 10:57 AM Jun Rao 
> > wrote:
> > >
> > >> Hi, David,
> > >>
> > >> Thanks for the reply.
> > >>
> > >> Historically, the format of all records were controlled by MV. Now,
> > >> records
> > >> in _offset_commit will be controlled by `group.coordinator.version`,
> is
> > >> that right? It would be useful to document that.
> > >>
> > >> Also, we should align on the version numbering. "kafka-feature
> disable"
> > >> says "Disable one or more feature flags. This is the same as
> downgrading
> > >> the version to zero". So, in the `group.coordinator.version' case, we
> > >> probably should use version 0 for the old consumer protocol.
> > >>
> > >> Jun
> > >>
> > >> On Thu, Mar 28, 2024 at 2:13 AM Andrew Schofield <
> > >> andrew_schofield_j...@outlook.com> wrote:
> > >>
> > >> > Hi David,
> > >> > I agree that we should use the same mechanism to gate KIP-932 once
> > that
> > >> > feature reaches production readiness. The precise details of the
> > values
> > >> > will
> > >> > depend upon the current state of all these flags when that release
> > >> comes.
> > >> >
> > >> > Thanks,
> > >> > Andrew
> > >> >
> > >> > > On 28 Mar 2024, at 07:11, David Jacot  >
> > >> > wrote:
> > >> > >
> > >> > > Hi, Jun, Justine,
> > >> > >
> > >> > > Regarding `group.coordinator.version`, the idea is to use it to
> gate
> > >> > > records and APIs of the group coordinator. The first use case will
> > be
> > >> > > KIP-848. We will use version 2 of the flag to gate all the new
> > records
> > >> > and
> > >> > > the new ConsumerGroupHeartbeat/Describe APIs present in AK 3.8. So
> > >> > version
> > >> > > 1 will be the only the old protocol and version 2 will be the
> > >> currently
> > >> > > implemented new protocol. I don't think that we have any
> dependency
> > on
> > >> > the
> > >> > > metadata version at the moment. The changes are orthogonal. I
> think
> > >> that
> > >> > we
> > >> > > could mention KIP-848 as the first usage of this flag in the KIP.
> I
> > >> will
> > >> > > also update KIP-848 to include it when this KIP is accepted.
> Another
> > >> use
> > >> > > case is the Queues KIP. I think that we should also use this new
> > flag
> > >> to
> > >> > > gate it.
> > >> > >
> > >> > > Best,
> > >> > > David
> > >> > >
> > >> > > On Thu, Mar 28, 2024 at 1:14 AM Jun Rao  >
> > >> > wrote:
> > >> > >
> > >> > >> Hi, Justine,
> > >> > >>
> > >> > >> Thanks for the reply.
> > >> > >>
> > >> > >> So, "dependencies" and "version-mapping" will be added to both
> > >> > >> kafka-feature and kafka-storage? Could we document that in the
> tool
> > >> > format
> > >> > >> section?
> > >> > >>
> > >> > >> Jun
> > >> > >>
> > >> > >> On Wed, Mar 27, 2024 at 4:01 PM Justine Olshan
> > >> > >> 
> > >> > >> wrote:
> > >> > >>
> > >> > >>> Ok. I can remove the info from the describe output.
> > >> > >>>
> > >> > >>> Dependencies is needed for the storage tool because we want to
> > make
> > >> > sure
> > >> > >>> the desired versions we are setting will be valid. Version
> mapping
> > >> > should
> > >> > >>> be for both tools since we have --release-version for both
> tools.
> > >> > >>>
> > >> > >>> I was considering changing the IV strings, but I wasn't sure if
> > >> there
> > >> > >> 

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

[Jun Rao]

Hi, Justine,

Thanks for the updated KIP. A couple of more comments.

20. Could we show the output of version-mapping?

21. "Transaction version 1 will include the flexible fields in the
transaction state log, and transaction version 2 will include the changes
to the transactional protocol as described by KIP-890 (epoch bumps and
implicit add partitions.)"
  So TV 1 enables the writing of new tagged fields like PrevProducerId? But
those fields are only usable after the epoch bump, right? What
functionality does TV 1 achieve?

Jun


On Mon, Apr 1, 2024 at 2:06 PM Justine Olshan 
wrote:

> I have also updated the KIP to mention the feature tool's --metadata flag
> will be deprecated.
> It will still work for users as they learn the new flag, but a warning
> indicating the alternatives will be shown.
>
> Justine
>
> On Thu, Mar 28, 2024 at 11:03 AM Justine Olshan 
> wrote:
>
> > Hi Jun,
> >
> > For both transaction state and group coordinator state, there are only
> > version 0 records.
> > KIP-915 introduced flexible versions, but it was never put to use. MV has
> > never gated these. This KIP will do that. I can include this context in
> the
> > KIP.
> >
> > I'm happy to modify his 1 and 2 to 0 and 1.
> >
> > Justine
> >
> > On Thu, Mar 28, 2024 at 10:57 AM Jun Rao 
> wrote:
> >
> >> Hi, David,
> >>
> >> Thanks for the reply.
> >>
> >> Historically, the format of all records were controlled by MV. Now,
> >> records
> >> in _offset_commit will be controlled by `group.coordinator.version`, is
> >> that right? It would be useful to document that.
> >>
> >> Also, we should align on the version numbering. "kafka-feature disable"
> >> says "Disable one or more feature flags. This is the same as downgrading
> >> the version to zero". So, in the `group.coordinator.version' case, we
> >> probably should use version 0 for the old consumer protocol.
> >>
> >> Jun
> >>
> >> On Thu, Mar 28, 2024 at 2:13 AM Andrew Schofield <
> >> andrew_schofield_j...@outlook.com> wrote:
> >>
> >> > Hi David,
> >> > I agree that we should use the same mechanism to gate KIP-932 once
> that
> >> > feature reaches production readiness. The precise details of the
> values
> >> > will
> >> > depend upon the current state of all these flags when that release
> >> comes.
> >> >
> >> > Thanks,
> >> > Andrew
> >> >
> >> > > On 28 Mar 2024, at 07:11, David Jacot 
> >> > wrote:
> >> > >
> >> > > Hi, Jun, Justine,
> >> > >
> >> > > Regarding `group.coordinator.version`, the idea is to use it to gate
> >> > > records and APIs of the group coordinator. The first use case will
> be
> >> > > KIP-848. We will use version 2 of the flag to gate all the new
> records
> >> > and
> >> > > the new ConsumerGroupHeartbeat/Describe APIs present in AK 3.8. So
> >> > version
> >> > > 1 will be the only the old protocol and version 2 will be the
> >> currently
> >> > > implemented new protocol. I don't think that we have any dependency
> on
> >> > the
> >> > > metadata version at the moment. The changes are orthogonal. I think
> >> that
> >> > we
> >> > > could mention KIP-848 as the first usage of this flag in the KIP. I
> >> will
> >> > > also update KIP-848 to include it when this KIP is accepted. Another
> >> use
> >> > > case is the Queues KIP. I think that we should also use this new
> flag
> >> to
> >> > > gate it.
> >> > >
> >> > > Best,
> >> > > David
> >> > >
> >> > > On Thu, Mar 28, 2024 at 1:14 AM Jun Rao 
> >> > wrote:
> >> > >
> >> > >> Hi, Justine,
> >> > >>
> >> > >> Thanks for the reply.
> >> > >>
> >> > >> So, "dependencies" and "version-mapping" will be added to both
> >> > >> kafka-feature and kafka-storage? Could we document that in the tool
> >> > format
> >> > >> section?
> >> > >>
> >> > >> Jun
> >> > >>
> >> > >> On Wed, Mar 27, 2024 at 4:01 PM Justine Olshan
> >> > >> 
> >> > >> wrote:
> >> > >>
> >> > >>> Ok. I can remove the info from the describe output.
> >> > >>>
> >> > >>> Dependencies is needed for the storage tool because we want to
> make
> >> > sure
> >> > >>> the desired versions we are setting will be valid. Version mapping
> >> > should
> >> > >>> be for both tools since we have --release-version for both tools.
> >> > >>>
> >> > >>> I was considering changing the IV strings, but I wasn't sure if
> >> there
> >> > >> would
> >> > >>> be some disagreement with the decision. Not sure if that breaks
> >> > >>> compatibility etc. Happy to hear everyone's thoughts.
> >> > >>>
> >> > >>> Justine
> >> > >>>
> >> > >>> On Wed, Mar 27, 2024 at 3:36 PM Jun Rao  >
> >> > >> wrote:
> >> > >>>
> >> >  Hi, Justine,
> >> > 
> >> >  Thanks for the reply.
> >> > 
> >> >  Having "kafka-feature dependencies" seems enough to me. We don't
> >> need
> >> > >> to
> >> >  include the dependencies in the output of "kafka-feature
> describe".
> >> > 
> >> >  We only support "dependencies" in kafka-feature, not
> >> kafka-storage. We
> >> >  probably should do the same for "version-mapping".
> >> > 
> >> >  

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

[Justine Olshan]

I have also updated the KIP to mention the feature tool's --metadata flag
will be deprecated.
It will still work for users as they learn the new flag, but a warning
indicating the alternatives will be shown.

Justine

On Thu, Mar 28, 2024 at 11:03 AM Justine Olshan 
wrote:

> Hi Jun,
>
> For both transaction state and group coordinator state, there are only
> version 0 records.
> KIP-915 introduced flexible versions, but it was never put to use. MV has
> never gated these. This KIP will do that. I can include this context in the
> KIP.
>
> I'm happy to modify his 1 and 2 to 0 and 1.
>
> Justine
>
> On Thu, Mar 28, 2024 at 10:57 AM Jun Rao  wrote:
>
>> Hi, David,
>>
>> Thanks for the reply.
>>
>> Historically, the format of all records were controlled by MV. Now,
>> records
>> in _offset_commit will be controlled by `group.coordinator.version`, is
>> that right? It would be useful to document that.
>>
>> Also, we should align on the version numbering. "kafka-feature disable"
>> says "Disable one or more feature flags. This is the same as downgrading
>> the version to zero". So, in the `group.coordinator.version' case, we
>> probably should use version 0 for the old consumer protocol.
>>
>> Jun
>>
>> On Thu, Mar 28, 2024 at 2:13 AM Andrew Schofield <
>> andrew_schofield_j...@outlook.com> wrote:
>>
>> > Hi David,
>> > I agree that we should use the same mechanism to gate KIP-932 once that
>> > feature reaches production readiness. The precise details of the values
>> > will
>> > depend upon the current state of all these flags when that release
>> comes.
>> >
>> > Thanks,
>> > Andrew
>> >
>> > > On 28 Mar 2024, at 07:11, David Jacot 
>> > wrote:
>> > >
>> > > Hi, Jun, Justine,
>> > >
>> > > Regarding `group.coordinator.version`, the idea is to use it to gate
>> > > records and APIs of the group coordinator. The first use case will be
>> > > KIP-848. We will use version 2 of the flag to gate all the new records
>> > and
>> > > the new ConsumerGroupHeartbeat/Describe APIs present in AK 3.8. So
>> > version
>> > > 1 will be the only the old protocol and version 2 will be the
>> currently
>> > > implemented new protocol. I don't think that we have any dependency on
>> > the
>> > > metadata version at the moment. The changes are orthogonal. I think
>> that
>> > we
>> > > could mention KIP-848 as the first usage of this flag in the KIP. I
>> will
>> > > also update KIP-848 to include it when this KIP is accepted. Another
>> use
>> > > case is the Queues KIP. I think that we should also use this new flag
>> to
>> > > gate it.
>> > >
>> > > Best,
>> > > David
>> > >
>> > > On Thu, Mar 28, 2024 at 1:14 AM Jun Rao 
>> > wrote:
>> > >
>> > >> Hi, Justine,
>> > >>
>> > >> Thanks for the reply.
>> > >>
>> > >> So, "dependencies" and "version-mapping" will be added to both
>> > >> kafka-feature and kafka-storage? Could we document that in the tool
>> > format
>> > >> section?
>> > >>
>> > >> Jun
>> > >>
>> > >> On Wed, Mar 27, 2024 at 4:01 PM Justine Olshan
>> > >> 
>> > >> wrote:
>> > >>
>> > >>> Ok. I can remove the info from the describe output.
>> > >>>
>> > >>> Dependencies is needed for the storage tool because we want to make
>> > sure
>> > >>> the desired versions we are setting will be valid. Version mapping
>> > should
>> > >>> be for both tools since we have --release-version for both tools.
>> > >>>
>> > >>> I was considering changing the IV strings, but I wasn't sure if
>> there
>> > >> would
>> > >>> be some disagreement with the decision. Not sure if that breaks
>> > >>> compatibility etc. Happy to hear everyone's thoughts.
>> > >>>
>> > >>> Justine
>> > >>>
>> > >>> On Wed, Mar 27, 2024 at 3:36 PM Jun Rao 
>> > >> wrote:
>> > >>>
>> >  Hi, Justine,
>> > 
>> >  Thanks for the reply.
>> > 
>> >  Having "kafka-feature dependencies" seems enough to me. We don't
>> need
>> > >> to
>> >  include the dependencies in the output of "kafka-feature describe".
>> > 
>> >  We only support "dependencies" in kafka-feature, not
>> kafka-storage. We
>> >  probably should do the same for "version-mapping".
>> > 
>> >  bin/kafka-features.sh downgrade --feature metadata.version=16
>> >  --transaction.protocol.version=2
>> >  We need to add the --feature flag for the second feature, right?
>> > 
>> >  In "kafka-features.sh describe", we only show the IV string for
>> >  metadata.version. Should we also show the level number?
>> > 
>> >  Thanks,
>> > 
>> >  Jun
>> > 
>> >  On Wed, Mar 27, 2024 at 1:52 PM Justine Olshan
>> >  
>> >  wrote:
>> > 
>> > > I had already included this example
>> > > bin/kafka-features.sh downgrade --feature metadata.version=16
>> > > --transaction.protocol.version=2 // Throws error if metadata
>> version
>> > >>> is <
>> > > 16, and this would be an upgrade
>> > > But I have updated the KIP to explicitly say the text you
>> mentioned.
>> > >
>> > > Justine
>> > >
>> > 

Re: [DISCUSS] KIP-1024: Make the restore behavior of GlobalKTables with custom processors configureable

[Bruno Cadonna]

Hi Walker and Matthias,

(2)
That is exactly my point about having a compile time error versus a 
runtime error. The added flexibility as proposed by Matthias sounds good 
to me.


Regarding the Named parameter, I was not aware that the processor that 
writes records to the global state store is named according to the name 
passed in by Consumed. I thought Consumed strictly specifies the names 
of source processors. So I am fine with not having an overload with a 
Named parameter.


Best,
Bruno

On 3/31/24 11:30 AM, Matthias J. Sax wrote:

Two more follow up thoughts:

(1) I am still not a big fan of the boolean parameter we introduce. Did 
you consider to use different method names, like 
`addReadOnlyGlobalStore()` (for the optimized method, that would not 
reprocess data on restore), and maybe add `addModifiableGlobalStore()` 
(not a good name, but we cannot re-use existing `addGlobalStore()` -- 
maybe somebody else has a good idea about a better `addXxxGlobalStore` 
that would describe it well).


(2) I was thinking about Bruno's comment to limit the scope the store 
builder for the optimized case. I think we should actually do something 
about it, because in the end, the runtime (ie, the `Processor` we hard 
wire) would need to pick a store it supports and cast to the 
corresponding store? If the cast fails, we hit a runtime exception, but 
by putting the store we cast to into the signature we can actually 
convert it into a compile time error what seems better. -- If we want, 
we could make it somewhat flexible and support both `KeyValueStore` and 
`TimestampedKeyValueStore` -- ie, the signature would be `KeyValueStore` 
but we explicitly check if the builder gives us a 
`TimestampedKeyValueStore` instance and use it properly.


If putting the signature does not work for some reason, we should at 
least clearly call it out in the JavaDocs what store type is expected.




-Matthias



On 3/28/24 5:05 PM, Walker Carlson wrote:

Hey all,

Thanks for the feedback Bruno, Almog and Matthias!

Almog: I like the idea, but I agree with Matthais. I actually looked at
that ticket a bit when doing this and found that while similar they are
actually pretty unrelated codewise. I would love to see it get taken care
of.

Bruno and Matthias: The Named parameter doesn't really make sense to 
me to

put it here. The store in the Store builder is already named through what
Matthais described and the processor doesn't actually have a name. That
would be the processor node that gets named via the Named parameter  (in
the DSL) and the internal streams builder uses the consumed object to 
make

a source name. I think we should just keep the Consumed object and used
that for the processor node name.

As for the limitation of the store builder interface I don't think it is
necessary. It could be something we add later if we really want to.

Anyways I think we are getting close enough to consensus that I'm 
going to

open a vote and hopefully we can get it voted on soon!

best,
Walker

On Thu, Mar 28, 2024 at 5:55 AM Matthias J. Sax  wrote:


Hey,

looking into the API, I am wondering why we would need to add an
overload talking a `Named` parameter?

StreamsBuilder.addGlobalStore() (and .addGlobalTable()) already takes a
`Consumed` parameter that allows to set a name.



2.
I do not understand what you mean with "maximum flexibility". The

built-in processor needs to assume a given state store interface. That
means, users have to provide a state store that offers that 
interface. If

they do not they will get a runtime exception. If we require a store
builder for a given interface, we can catch the mistake at compile time.
Let me know whether I misunderstood something.

Yes, we could catch it at runtime. But I guess what I was trying to say
is different: I was trying to say, we should not limit the API to always
require a specific store, such that global stores can only be of a
certain type. Global Stores should be allowed to be of any type. Hence,
if we add a built-in processor, it can only be one option, and we always
need to support custom processor, and might also want to try to allow
the restore optimization for custom processor (and thus other store
types), not just for our built-in processor (and our built-in stores).
Coupling the optimization to built-in stores would prevent us to apply
the optimization to custom stores.



@Almog: interesting idea. I tend to think that both issues are
orthogonal. If users pick to apply the optimization "added" by this KIP,
the bug you mentioned would still apply to global stores, and thus this
KIP is not addressing the issue you mentioned.

Personally, I also think that we don't need a KIP to fix the ticket you
mentioned? In the end, we need to skip records during restore, and it
seems it does not make sense to make this configurable?



-Matthias


On 3/26/24 4:24 PM, Almog Gavra wrote:

Thanks for the thoughts Bruno!


Do you mean a API to configure restoration instead of boolean 

[jira] [Resolved] (KAFKA-16435) Add test for KAFKA-16428

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16435.

Fix Version/s: 3.8.0
   Resolution: Fixed

> Add test for KAFKA-16428
> 
>
> Key: KAFKA-16435
> URL: https://issues.apache.org/jira/browse/KAFKA-16435
> Project: Kafka
>  Issue Type: Bug
>Reporter: Colin McCabe
>Assignee: Kuan Po Tseng
>Priority: Major
> Fix For: 3.8.0
>
>
> Add a test for KAFKA-16428: Fix bug where config change notification znode 
> may not get created during migration #15608



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1032: Upgrade to Jakarta and JavaEE 9 in Kafka 4.0

[Christopher Shannon]

Greg,

1. Ok sounds good we can target JDK 17 in this KIP if we decide to do that.
2. For the EE version, I don't think it really matters since we won't be
using any new features that I am aware of. It's just something I noticed
that we will need to pick because Jetty 12 supports multiple versions so it
would affect which spec jars we use.  In the past Jetty versions have been
tied to a specific Servlet spec but the new Jetty 12 they have abstracted
things away and they support multiple versions simultaneously. There's
different versions for all the specs but the primary one to note for us
would be that JavaEE 9 uses the Servlet 5.0 spec and JavaEE 10 uses the
Servlet 6.0 spec. JavaEE 11 is under development and will use the Servlet
6.1 spec. So we may not really need to call out the EE version at all if it
doesn't matter and we are not using specific features but I wanted to bring
it up since multiple versions are listed as being compatible with Jetty 12
so we need to pick one. On the main page they list the different servlet
specs they support: https://eclipse.dev/jetty/
3. Right, I didn't mean we should include it in the KIP, I was more asking
I guess how to go about things. It looks like we could use a lot of it and
adapt the work already done. While it's under the Apache 2.0 license and we
could use it, someone else wrote it so it would still be good to properly
credit that person as you mentioned. If I work on it I would probably start
over with a new branch and just use the old PR as a guide and then maybe
figure out a way to credit the original author. There's always that
co-author tag that could be used I think.

Chris


On Mon, Apr 1, 2024 at 12:11 PM Greg Harris 
wrote:

> Hey Chris,
>
> Thanks for your questions!
>
> 1. KIPs are generally immutable once they've been voted on. In this
> case, I think it's better that this KIP include the bump to Java 17
> just for Connect and MirrorMaker 2, and should include that in the KIP
> title.
> 2. I'm not familiar with the difference, can you provide some more
> context that would help us make a decision? AFAIU we haven't specified
> an EE version in the past, and we don't do any sort of automated
> testing for compatibility. I think it would be good to call out which
> components have JavaEE-sensitive dependencies (just connect-runtime?).
> We do not want to accidentally give users the idea that the clients
> depend on the JavaEE version, as that could be very confusing.
> 3. That's an implementation detail left up to anyone that effects this
> KIP on the repo, and doesn't need to be mentioned in the KIP itself. I
> have seen people adopt changes from un-merged PRs after the original
> contributor has lost interest, while still crediting the original
> contributor for their portion of the changes. If you're doing this,
> then it's ultimately up to your judgement.
>
> Thanks,
> Greg
>
> On Mon, Apr 1, 2024 at 6:30 AM Christopher Shannon
>  wrote:
> >
> > Hi Greg,
> >
> > Thanks for the detailed analysis on the connect framework. It sounds like
> > hopefully we can go ahead and require JDK 17+ and bump that dependency
> for
> > the ConnectRestExtensionContext.
> >
> > I agree we can leave it open and hear what others think as well about the
> > requirement, if everyone ends up agreeing, I can update the KIP to
> reflect
> > requiring JDK 17 and going with Jetty 12.
> >
> > I had a couple of questions
> > 1) If we go with JDK 17 as a requirement for the Connect framework,
> should
> > we modify and make that part of KIP-1013 or keep it in this one?
> > 2) Should we go with JavaEE 9 or JavaEE 10? I'm not sure how much it
> > matters in this case.
> > 3) Can we just re-open https://github.com/apache/kafka/pull/10176 as a
> > starting point or maybe we can create a new PR and use it as a basis?
> It's
> > a bit old so I suspect there would be a ton of conflicts so it might be
> > best to start over and use it as a guide. I can work on a new PR once we
> > are all on the same page. I don't think it would take too long to put
> > together since most of the work is just dependency updates and package
> > renaming.
> >
> > Chris
> >
> >
> > On Fri, Mar 29, 2024 at 8:59 PM Greg Harris  >
> > wrote:
> >
> > > Hey all,
> > >
> > > I looked into how Debezium handled the javax->jakarta changeover for
> > > Quarkus, and found this release note:
> > >
> > >
> https://debezium.io/blog/2023/04/20/debezium-2-2-final-released/#new-quarkus-3
> > > It appears that Debezium 2.1 required Quarkus < 3.0, and Debezium 2.2
> > > required Quarkus >= 3.0. The upgrade for Kafka could be very similar
> > > and not incur a major version release.
> > >
> > >  We can leave some time to hear from anyone else that is impacted by
> > > this change, but from the open source projects present on github, this
> > > LGTM.
> > >
> > > Thanks,
> > > Greg
> > >
> > > On Fri, Mar 29, 2024 at 5:27 PM Greg Harris 
> wrote:
> > > >
> > > > Hi Chris,
> > > >
> > > > Thank you so much for opening this 

[VOTE] KIP-1020 Move `window.size.ms` and `windowed.inner.serde.class` from `StreamsConfig` to TimeWindowedDe/Serializer class

[Lucia Cerchie]

Hello everyone,

I'd like to call a vote on KIP-1020
.
It has been under discussion since Feb 15, and has received edits to the
KIP and approval by discussion participants.

Best,
Lucia Cerchie

[jira] [Created] (KAFKA-16455) Check partition exists before send reassignments to server in ReassignPartitionsCommand

[Kuan Po Tseng (Jira)]

Kuan Po Tseng created KAFKA-16455:
-

 Summary: Check partition exists before send reassignments to 
server in ReassignPartitionsCommand
 Key: KAFKA-16455
 URL: https://issues.apache.org/jira/browse/KAFKA-16455
 Project: Kafka
  Issue Type: Improvement
  Components: tools
Reporter: Kuan Po Tseng
Assignee: Kuan Po Tseng


Currently, when executing {{kafka-reassign-partitions.sh}} with the 
{{--execute}} option, if a partition number specified in the JSON file does not 
exist, this check occurs only when submitting the reassignments to 
{{alterPartitionReassignments}} on the server-side.

We can perform this check in advance before submitting the reassignments to the 
server side.

For example, suppose we have three brokers with IDs 1001, 1002, and 1003, and a 
topic named {{first_topic}} with only three partitions. And execute 
{code:bash}
bin/kafka-reassign-partitions.sh 
  --bootstrap-server 192.168.0.128:9092 
  --reassignment-json-file reassignment.json 
  --execute
{code}
Where reassignment.json contains
{code:json}
{
  "version": 1,
  "partitions": [
{
  "topic": "first_topic",
  "partition": 20,
  "replicas": [1002, 1001, 1003],
  "log_dirs": ["any", "any", "any"]
}
  ]
}
{code}
The console outputs
{code:java}
Current partition replica assignment

{"version":1,"partitions":[]}

Save this to use as the --reassignment-json-file option during rollback
Error reassigning partition(s):
first_topic-20: The partition does not exist.
{code}
Apart from the output {{\{"version":1,"partitions":[]\}}} which doesn't provide 
much help, the error {{first_topic-20: The partition does not exist.}} is 
reported back to the tool from the server-side, as mentioned earlier. This 
check could be moved earlier before sending reassignments to server side



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1032: Upgrade to Jakarta and JavaEE 9 in Kafka 4.0

[Greg Harris]

Hey Chris,

Thanks for your questions!

1. KIPs are generally immutable once they've been voted on. In this
case, I think it's better that this KIP include the bump to Java 17
just for Connect and MirrorMaker 2, and should include that in the KIP
title.
2. I'm not familiar with the difference, can you provide some more
context that would help us make a decision? AFAIU we haven't specified
an EE version in the past, and we don't do any sort of automated
testing for compatibility. I think it would be good to call out which
components have JavaEE-sensitive dependencies (just connect-runtime?).
We do not want to accidentally give users the idea that the clients
depend on the JavaEE version, as that could be very confusing.
3. That's an implementation detail left up to anyone that effects this
KIP on the repo, and doesn't need to be mentioned in the KIP itself. I
have seen people adopt changes from un-merged PRs after the original
contributor has lost interest, while still crediting the original
contributor for their portion of the changes. If you're doing this,
then it's ultimately up to your judgement.

Thanks,
Greg

On Mon, Apr 1, 2024 at 6:30 AM Christopher Shannon
 wrote:
>
> Hi Greg,
>
> Thanks for the detailed analysis on the connect framework. It sounds like
> hopefully we can go ahead and require JDK 17+ and bump that dependency for
> the ConnectRestExtensionContext.
>
> I agree we can leave it open and hear what others think as well about the
> requirement, if everyone ends up agreeing, I can update the KIP to reflect
> requiring JDK 17 and going with Jetty 12.
>
> I had a couple of questions
> 1) If we go with JDK 17 as a requirement for the Connect framework, should
> we modify and make that part of KIP-1013 or keep it in this one?
> 2) Should we go with JavaEE 9 or JavaEE 10? I'm not sure how much it
> matters in this case.
> 3) Can we just re-open https://github.com/apache/kafka/pull/10176 as a
> starting point or maybe we can create a new PR and use it as a basis? It's
> a bit old so I suspect there would be a ton of conflicts so it might be
> best to start over and use it as a guide. I can work on a new PR once we
> are all on the same page. I don't think it would take too long to put
> together since most of the work is just dependency updates and package
> renaming.
>
> Chris
>
>
> On Fri, Mar 29, 2024 at 8:59 PM Greg Harris 
> wrote:
>
> > Hey all,
> >
> > I looked into how Debezium handled the javax->jakarta changeover for
> > Quarkus, and found this release note:
> >
> > https://debezium.io/blog/2023/04/20/debezium-2-2-final-released/#new-quarkus-3
> > It appears that Debezium 2.1 required Quarkus < 3.0, and Debezium 2.2
> > required Quarkus >= 3.0. The upgrade for Kafka could be very similar
> > and not incur a major version release.
> >
> >  We can leave some time to hear from anyone else that is impacted by
> > this change, but from the open source projects present on github, this
> > LGTM.
> >
> > Thanks,
> > Greg
> >
> > On Fri, Mar 29, 2024 at 5:27 PM Greg Harris  wrote:
> > >
> > > Hi Chris,
> > >
> > > Thank you so much for opening this KIP, and making sure Kafka keeps up
> > > with the rest of the Java ecosystem!
> > >
> > > I took a look around at some Open Source connector implementations,
> > > and checked their Java version support:
> > > * The Aiven connect plugins (http, bigquery, jdbc, elasticsearch,
> > > opensearch, commons, s3, transforms, gcs), 6/9 are tested with JDK 17
> > > in CI, 2/9 JDK 11, and 1/9 JDK 8. I'll look into improving the testing
> > > matrix, but I don't expect substantial problems with requiring JDK 17.
> > > * The Debezium Project lists Java 11+ compatibility:
> > > https://debezium.io/releases/ and appears to use Java 22 (ga) and 23
> > > (ea) in their CI:
> > >
> > https://github.com/debezium/debezium/blob/9cdaa38453c9f065c6075d31636592a5b147518f/.github/workflows/jdk-outreach-workflow.yml#L20
> > >
> > > I think the bigger problem really is the ConnectRestExtension, since
> > > we've baked the rs-api type into the signature of
> > > ConnectRestExtensionContext.
> > > * Aiven doesn't have any ConnectRestExtensions, so this isn't a concern
> > for us.
> > > * The Debezium Project has at least 6 ConnectRestExtension
> > > implementations:
> > >
> > https://github.com/search?q=repo%3Adebezium%2Fdebezium+ConnectRestExtension+language%3AJava=code=Java
> > > . Some of these are baked into artifacts that I know for a fact are
> > > used in normal connect deployments.
> > > * I found a healthcheck extension that looks unmaintained:
> > >
> > https://github.com/LoObp4ck/kafka-connect-healthchecks/blob/2d9dbfee900d9f85e6acd9a09bd04969afa46261/src/main/java/com/loobpack/data/kafka/connect/healthcheck/extension/HealthCheckConnectRestExtension.java#L9
> > >
> > > I figure that adopting this KIP would mean that the Debezium project
> > > would be forced to bump their major version 3.0 to be compatible with
> > > Connect 4.0, or otherwise change their packaging, 

Re: [DISCUSS] KIP-1032: Upgrade to Jakarta and JavaEE 9 in Kafka 4.0

[Christopher Shannon]

Hi Greg,

Thanks for the detailed analysis on the connect framework. It sounds like
hopefully we can go ahead and require JDK 17+ and bump that dependency for
the ConnectRestExtensionContext.

I agree we can leave it open and hear what others think as well about the
requirement, if everyone ends up agreeing, I can update the KIP to reflect
requiring JDK 17 and going with Jetty 12.

I had a couple of questions
1) If we go with JDK 17 as a requirement for the Connect framework, should
we modify and make that part of KIP-1013 or keep it in this one?
2) Should we go with JavaEE 9 or JavaEE 10? I'm not sure how much it
matters in this case.
3) Can we just re-open https://github.com/apache/kafka/pull/10176 as a
starting point or maybe we can create a new PR and use it as a basis? It's
a bit old so I suspect there would be a ton of conflicts so it might be
best to start over and use it as a guide. I can work on a new PR once we
are all on the same page. I don't think it would take too long to put
together since most of the work is just dependency updates and package
renaming.

Chris


On Fri, Mar 29, 2024 at 8:59 PM Greg Harris 
wrote:

> Hey all,
>
> I looked into how Debezium handled the javax->jakarta changeover for
> Quarkus, and found this release note:
>
> https://debezium.io/blog/2023/04/20/debezium-2-2-final-released/#new-quarkus-3
> It appears that Debezium 2.1 required Quarkus < 3.0, and Debezium 2.2
> required Quarkus >= 3.0. The upgrade for Kafka could be very similar
> and not incur a major version release.
>
>  We can leave some time to hear from anyone else that is impacted by
> this change, but from the open source projects present on github, this
> LGTM.
>
> Thanks,
> Greg
>
> On Fri, Mar 29, 2024 at 5:27 PM Greg Harris  wrote:
> >
> > Hi Chris,
> >
> > Thank you so much for opening this KIP, and making sure Kafka keeps up
> > with the rest of the Java ecosystem!
> >
> > I took a look around at some Open Source connector implementations,
> > and checked their Java version support:
> > * The Aiven connect plugins (http, bigquery, jdbc, elasticsearch,
> > opensearch, commons, s3, transforms, gcs), 6/9 are tested with JDK 17
> > in CI, 2/9 JDK 11, and 1/9 JDK 8. I'll look into improving the testing
> > matrix, but I don't expect substantial problems with requiring JDK 17.
> > * The Debezium Project lists Java 11+ compatibility:
> > https://debezium.io/releases/ and appears to use Java 22 (ga) and 23
> > (ea) in their CI:
> >
> https://github.com/debezium/debezium/blob/9cdaa38453c9f065c6075d31636592a5b147518f/.github/workflows/jdk-outreach-workflow.yml#L20
> >
> > I think the bigger problem really is the ConnectRestExtension, since
> > we've baked the rs-api type into the signature of
> > ConnectRestExtensionContext.
> > * Aiven doesn't have any ConnectRestExtensions, so this isn't a concern
> for us.
> > * The Debezium Project has at least 6 ConnectRestExtension
> > implementations:
> >
> https://github.com/search?q=repo%3Adebezium%2Fdebezium+ConnectRestExtension+language%3AJava=code=Java
> > . Some of these are baked into artifacts that I know for a fact are
> > used in normal connect deployments.
> > * I found a healthcheck extension that looks unmaintained:
> >
> https://github.com/LoObp4ck/kafka-connect-healthchecks/blob/2d9dbfee900d9f85e6acd9a09bd04969afa46261/src/main/java/com/loobpack/data/kafka/connect/healthcheck/extension/HealthCheckConnectRestExtension.java#L9
> >
> > I figure that adopting this KIP would mean that the Debezium project
> > would be forced to bump their major version 3.0 to be compatible with
> > Connect 4.0, or otherwise change their packaging, so I'd like to hear
> > from the Debezium folks what they think of this proposal.
> >
> > Thanks,
> > Greg
> >
> > On Wed, Mar 27, 2024 at 4:43 PM Christopher Shannon
> >  wrote:
> > >
> > > Hi,
> > >
> > > I'm proposing a KIP for Kafka 4.0 to upgrade to to Jakarta and JavaEE 9
> > > APIs. This will also upgrade dependencies like Jetty and move away from
> > > the depcrated javax namespace to be in line with other libraries and
> > > frameworks. There was some initial
> > > 
> > > discussion and below is the KIP.
> > >
> > > Please take a look and let me know what you think:
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1032%3A+Upgrade+to+Jakarta+and+JavaEE+9+in+Kafka+4.0
> > >
> > > Thanks,
> > > Chris
>

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2772

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 466466 lines...]
[2024-04-01T11:36:09.105Z] > Task :storage:storage-api:compileTestJava
[2024-04-01T11:36:09.105Z] > Task :storage:storage-api:testClasses
[2024-04-01T11:36:10.212Z] > Task :server:compileTestJava
[2024-04-01T11:36:10.212Z] > Task :server:testClasses
[2024-04-01T11:36:11.321Z] > Task :server-common:compileTestJava
[2024-04-01T11:36:11.321Z] > Task :server-common:testClasses
[2024-04-01T11:36:13.809Z] > Task :raft:compileTestJava
[2024-04-01T11:36:13.809Z] > Task :raft:testClasses
[2024-04-01T11:36:15.900Z] 
[2024-04-01T11:36:15.900Z] > Task :clients:javadoc
[2024-04-01T11:36:15.900Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/clients/src/main/java/org/apache/kafka/clients/admin/ScramMechanism.java:32:
 warning - Tag @see: missing final '>': "https://cwiki.apache.org/confluence/display/KAFKA/KIP-554%3A+Add+Broker-side+SCRAM+Config+API;>KIP-554:
 Add Broker-side SCRAM Config API
[2024-04-01T11:36:15.900Z] 
[2024-04-01T11:36:15.900Z]  This code is duplicated in 
org.apache.kafka.common.security.scram.internals.ScramMechanism.
[2024-04-01T11:36:15.900Z]  The type field in both files must match and must 
not change. The type field
[2024-04-01T11:36:15.900Z]  is used both for passing ScramCredentialUpsertion 
and for the internal
[2024-04-01T11:36:15.900Z]  UserScramCredentialRecord. Do not change the type 
field."
[2024-04-01T11:36:15.900Z] 
[2024-04-01T11:36:15.900Z] > Task :core:compileScala
[2024-04-01T11:36:17.063Z] > Task :group-coordinator:compileTestJava
[2024-04-01T11:36:17.063Z] > Task :group-coordinator:testClasses
[2024-04-01T11:36:17.063Z] > Task 
:streams:generateMetadataFileForMavenJavaPublication
[2024-04-01T11:36:17.063Z] 
[2024-04-01T11:36:17.063Z] > Task :clients:javadoc
[2024-04-01T11:36:17.063Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/package-info.java:21:
 warning - Tag @link: reference not found: 
org.apache.kafka.common.security.oauthbearer
[2024-04-01T11:36:17.063Z] 2 warnings
[2024-04-01T11:36:18.169Z] 
[2024-04-01T11:36:18.169Z] > Task :clients:javadocJar
[2024-04-01T11:36:19.274Z] > Task :clients:srcJar
[2024-04-01T11:36:19.274Z] > Task :metadata:compileTestJava
[2024-04-01T11:36:19.274Z] > Task :metadata:testClasses
[2024-04-01T11:36:19.274Z] > Task :clients:testJar
[2024-04-01T11:36:19.274Z] > Task :clients:testSrcJar
[2024-04-01T11:36:20.381Z] > Task 
:clients:publishMavenJavaPublicationToMavenLocal
[2024-04-01T11:36:20.381Z] > Task :clients:publishToMavenLocal
[2024-04-01T11:36:20.381Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2024-04-01T11:36:20.381Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2024-04-01T11:36:20.381Z] > Task :connect:api:testClasses UP-TO-DATE
[2024-04-01T11:36:20.381Z] > Task :connect:api:testJar
[2024-04-01T11:36:20.381Z] > Task :connect:api:testSrcJar
[2024-04-01T11:36:20.381Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2024-04-01T11:36:20.381Z] > Task :connect:api:publishToMavenLocal
[2024-04-01T11:36:25.846Z] > Task :streams:javadoc
[2024-04-01T11:36:25.846Z] > Task :streams:javadocJar
[2024-04-01T11:36:25.846Z] > Task :streams:srcJar
[2024-04-01T11:36:26.954Z] > Task :streams:processTestResources UP-TO-DATE
[2024-04-01T11:37:11.642Z] > Task :core:classes
[2024-04-01T11:37:11.642Z] > Task :core:compileTestJava NO-SOURCE
[2024-04-01T11:37:40.404Z] > Task :core:compileTestScala
[2024-04-01T11:38:32.300Z] > Task :core:testClasses
[2024-04-01T11:39:00.934Z] > Task :streams:compileTestJava
[2024-04-01T11:40:38.889Z] > Task :streams:testClasses
[2024-04-01T11:40:38.889Z] > Task :streams:testJar
[2024-04-01T11:40:38.889Z] > Task :streams:testSrcJar
[2024-04-01T11:40:38.889Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2024-04-01T11:40:38.889Z] > Task :streams:publishToMavenLocal
[2024-04-01T11:40:38.889Z] 
[2024-04-01T11:40:38.889Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 9.0.
[2024-04-01T11:40:38.889Z] 
[2024-04-01T11:40:38.889Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2024-04-01T11:40:38.889Z] 
[2024-04-01T11:40:38.889Z] For more on this, please refer to 
https://docs.gradle.org/8.6/userguide/command_line_interface.html#sec:command_line_warnings
 in the Gradle documentation.
[2024-04-01T11:40:38.889Z] 
[2024-04-01T11:40:38.889Z] BUILD SUCCESSFUL in 5m 9s
[2024-04-01T11:40:38.889Z] 96 actionable tasks: 41 executed, 55 up-to-date
[2024-04-01T11:40:38.889Z] 
[2024-04-01T11:40:38.889Z] Publishing build scan...
[2024-04-01T11:40:38.889Z] https://ge.apache.org/s/4q44pcy4lz4yo
[2024-04-01T11:40:38.889Z] 
[Pipeline] sh
[2024-04-01T11:40:41.729Z] + + grep ^version= gradle.properties

[jira] [Created] (KAFKA-16454) Snapshot the state of remote log metadata for all the partitions

[Kamal Chandraprakash (Jira)]

Kamal Chandraprakash created KAFKA-16454:


 Summary: Snapshot the state of remote log metadata for all the 
partitions
 Key: KAFKA-16454
 URL: https://issues.apache.org/jira/browse/KAFKA-16454
 Project: Kafka
  Issue Type: Task
Reporter: Kamal Chandraprakash


When restarting the broker, we are reading from the beginning of the  
{{__remote_log_metadata}} topic to reconstruct the state of remote log 
segments, instead we can snapshot the state of remote log segments under each 
partition directory. 

Previous work to snapshot the state of the remote log metadata are removed as 
the solution is not complete. 

https://github.com/apache/kafka/pull/15636



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16453) Improve documentation and error message for not supporting zookeeper mode

[Vedarth Sharma (Jira)]

Vedarth Sharma created KAFKA-16453:
--

 Summary: Improve documentation and error message for not 
supporting zookeeper mode
 Key: KAFKA-16453
 URL: https://issues.apache.org/jira/browse/KAFKA-16453
 Project: Kafka
  Issue Type: Sub-task
Reporter: Vedarth Sharma
Assignee: Vedarth Sharma


Docker image doesn't support zookeeper mode. This needs to be stated explicitly 
in docs and communicated clearly in the error message.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2771

[Apache Jenkins Server]

See