Request to contribute to Kafka

[Doe John]

my Jira ID: JohnDoe

on email zh2725284...@gmail.com

Thanks!

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1824

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-14084) Support SCRAM when using KRaft mode

[Proven Provenzano (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Proven Provenzano resolved KAFKA-14084.
---
Resolution: Fixed

SCRAM support in KRaft is implemented.

Test of SCRAM authorization have been updated to work in KRaft mode.

Migration support of SCRAM credentials from ZK to SCRAM have been implemented 
and tested.

> Support SCRAM when using KRaft mode
> ---
>
> Key: KAFKA-14084
> URL: https://issues.apache.org/jira/browse/KAFKA-14084
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Colin McCabe
>Assignee: Proven Provenzano
>Priority: Major
>  Labels: kip-500
> Fix For: 3.5.0
>
>
> Support SCRAM when using KRaft mode, as specified in KIP-631



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14970) Dual write mode testing for SCRAM and Quota

[Proven Provenzano (Jira)]

Proven Provenzano created KAFKA-14970:
-

 Summary: Dual write mode testing for SCRAM and Quota
 Key: KAFKA-14970
 URL: https://issues.apache.org/jira/browse/KAFKA-14970
 Project: Kafka
  Issue Type: Test
  Components: kraft
Reporter: Proven Provenzano
Assignee: Proven Provenzano


SCRAM and Quota are stored together in ZK and we need better testing to 
validate the dual write mode support for them.

I will add some additional tests for this.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Kafka client needs KAFKA-10337 to cover async commit use case

[Philip Nee]

Hey Eric,

Maybe its more straightforward to open a new PR.

Thanks!
P
On Fri, May 5, 2023 at 9:36 AM Erik van Oosten  wrote:

> If I were to rebase the old pull request and re-open KAFKA-10337, would
> it be considered for merging?
>
> Kind regards,
>  Erik.
>
>
> Op 03-05-2023 om 09:21 schreef Erik van Oosten:
> > Hi Philip,
> >
> >> Firstly, could you explain the situation
> >> in that you would prefer to invoke commitAsync over commitSync in the
> >> rebalance listener?
> >
> > Of course!
> >
> > Short answer: we prefer commitAsync because we want to handle multiple
> > partitions concurrently using the ZIO runtime.
> >
> > Long answer: this is in the context of zio-kafka. In zio-kafka the
> > user writes code for a stream that processes data and does commits. In
> > the library we intercept those commits and pass them to the
> > KafkaConsumer. We also keep track of the offsets of handed out
> > records. Together this information allows us to track when a stream is
> > ready processing a partition and that it is safe to start the rebalance.
> >
> > All of this happens concurrently and asynchronously using the ZIO
> > runtime. When calling commit inside the onPartitionRevoked callback
> > the library does a thread-id check; we can only call the KafkaConsumer
> > from the same thread that invoked us. This is unfortunate because it
> > forces us to spin up a specialized single-threaded ZIO runtime inside
> > the callback method. Though this runtime can run blocking methods like
> > commitSync, it will need careful programming since all other tasks
> > need to wait.
> >
> > (BTW, it would be great if there is an option to disable the thread-id
> > check. It has more use cases, see for example KAFKA-7143.)
> >
> >> is it your concern that we
> >> currently don't have a way to invoke the callback, and the user won't
> >> be to
> >> correctly handle these failed/successful async commits?
> >
> > Yes, that is correct.
> >
> >> Sorry - I dug a bit into the old PR. Seems like the issue is there's
> >> broken
> >> contract that if the commitSync won't wait for the previous async
> >> commits
> >> to complete if it tries to commit an empty offset map.
> >
> > Indeed! I am assuming the same is true for commitAsync. The important
> > thing is that we need something to get those callbacks. I would prefer
> > commitAsync but if only commitSync gets fixed we can use that as well.
> > If there is another method completely for this task, that would be
> > good as well.
> >
> > Kind regards,
> > Erik.
> >
> >
> > Philip Nee schreef op 2023-05-02 21:49:
> >> Hey Erik,
> >>
> >> Just a couple of questions to you: Firstly, could you explain the
> >> situation
> >> in that you would prefer to invoke commitAsync over commitSync in the
> >> rebalance listener?  Typically we would use the synchronized method to
> >> ensure the commits are completed before moving on with the rebalancing,
> >> which leads to my second comment/question.  is it your concern that we
> >> currently don't have a way to invoke the callback, and the user won't
> >> be to
> >> correctly handle these failed/successful async commits?
> >>
> >> Thanks,
> >> P
> >>
> >> On Tue, May 2, 2023 at 12:22 PM Erik van Oosten
> >>  wrote:
> >>
> >>> Dear developers of the Kafka java client,
> >>>
> >>> It seems I have found a feature gap in the Kafka java client.
> >>> KAFKA-10337 and its associated pull request on Github (from 2020!)
> >>> would
> >>> solve this, but it was closed without merging. We would love to see it
> >>> being reconsidered for merging. This mail has the arguments for
> >>> doing so.
> >>>
> >>> The javadoc of `ConsumerRebalanceListener` method `onPartitionsRevoked`
> >>> recommends you commit all offsets within the method, thereby holding up
> >>> the rebalance until those commits are done. The (perceived) feature gap
> >>> is when the user is trying to do async commits from the rebalance
> >>> listener; there is nothing available to trigger the callbacks of
> >>> completed commits. Without these callback, there is no way to know when
> >>> it is safe to return from onPartitionsRevoked. (We cannot call `poll`
> >>> because the rebalance listener is already called from inside a poll.)
> >>>
> >>> Calling `commitAsync` with an empty offsets parameter seems a perfect
> >>> candidate for triggering callbacks of earlier commits. Unfortunately,
> >>> commitAsync doesn't behave that way. This is fixed by mentioned pull
> >>> request.
> >>>
> >>> The pull request conversation has a comment saying that calling
> >>> `commit`
> >>> with an empty offsets parameter is not something that should happen. I
> >>> found this a strange thing to say. First of all, the method does have
> >>> special handling for this situation, negating the comment outright. In
> >>> addition this special handling violates the contract of the method (as
> >>> specified in the javadoc section about ordering). Therefore, this pull
> >>> request has 2 advantages:
> >>>
> >>> 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1823

[Apache Jenkins Server]

See 

Re: Kafka client needs KAFKA-10337 to cover async commit use case

[Erik van Oosten]

If I were to rebase the old pull request and re-open KAFKA-10337, would 
it be considered for merging?


Kind regards,
    Erik.


Op 03-05-2023 om 09:21 schreef Erik van Oosten:

Hi Philip,


Firstly, could you explain the situation
in that you would prefer to invoke commitAsync over commitSync in the
rebalance listener?


Of course!

Short answer: we prefer commitAsync because we want to handle multiple 
partitions concurrently using the ZIO runtime.


Long answer: this is in the context of zio-kafka. In zio-kafka the 
user writes code for a stream that processes data and does commits. In 
the library we intercept those commits and pass them to the 
KafkaConsumer. We also keep track of the offsets of handed out 
records. Together this information allows us to track when a stream is 
ready processing a partition and that it is safe to start the rebalance.


All of this happens concurrently and asynchronously using the ZIO 
runtime. When calling commit inside the onPartitionRevoked callback 
the library does a thread-id check; we can only call the KafkaConsumer 
from the same thread that invoked us. This is unfortunate because it 
forces us to spin up a specialized single-threaded ZIO runtime inside 
the callback method. Though this runtime can run blocking methods like 
commitSync, it will need careful programming since all other tasks 
need to wait.


(BTW, it would be great if there is an option to disable the thread-id 
check. It has more use cases, see for example KAFKA-7143.)



is it your concern that we
currently don't have a way to invoke the callback, and the user won't 
be to

correctly handle these failed/successful async commits?


Yes, that is correct.

Sorry - I dug a bit into the old PR. Seems like the issue is there's 
broken
contract that if the commitSync won't wait for the previous async 
commits

to complete if it tries to commit an empty offset map.


Indeed! I am assuming the same is true for commitAsync. The important 
thing is that we need something to get those callbacks. I would prefer 
commitAsync but if only commitSync gets fixed we can use that as well. 
If there is another method completely for this task, that would be 
good as well.


Kind regards,
    Erik.


Philip Nee schreef op 2023-05-02 21:49:

Hey Erik,

Just a couple of questions to you: Firstly, could you explain the 
situation

in that you would prefer to invoke commitAsync over commitSync in the
rebalance listener?  Typically we would use the synchronized method to
ensure the commits are completed before moving on with the rebalancing,
which leads to my second comment/question.  is it your concern that we
currently don't have a way to invoke the callback, and the user won't 
be to

correctly handle these failed/successful async commits?

Thanks,
P

On Tue, May 2, 2023 at 12:22 PM Erik van Oosten
 wrote:


Dear developers of the Kafka java client,

It seems I have found a feature gap in the Kafka java client.
KAFKA-10337 and its associated pull request on Github (from 2020!) 
would

solve this, but it was closed without merging. We would love to see it
being reconsidered for merging. This mail has the arguments for 
doing so.


The javadoc of `ConsumerRebalanceListener` method `onPartitionsRevoked`
recommends you commit all offsets within the method, thereby holding up
the rebalance until those commits are done. The (perceived) feature gap
is when the user is trying to do async commits from the rebalance
listener; there is nothing available to trigger the callbacks of
completed commits. Without these callback, there is no way to know when
it is safe to return from onPartitionsRevoked. (We cannot call `poll`
because the rebalance listener is already called from inside a poll.)

Calling `commitAsync` with an empty offsets parameter seems a perfect
candidate for triggering callbacks of earlier commits. Unfortunately,
commitAsync doesn't behave that way. This is fixed by mentioned pull
request.

The pull request conversation has a comment saying that calling 
`commit`

with an empty offsets parameter is not something that should happen. I
found this a strange thing to say. First of all, the method does have
special handling for this situation, negating the comment outright. In
addition this special handling violates the contract of the method (as
specified in the javadoc section about ordering). Therefore, this pull
request has 2 advantages:

 1. KafkaConsumer.commitAsync will be more in line with its javadoc,
 2. the feature gap is gone.

Of course, it might be that I missed something and that there are other
ways to trigger the commit callbacks. I would be very happy to hear
about that because it means I do not have to wait for a release cycle.

If you agree these arguments are sound, I would be happy to make the
pull request mergable again.

Curious to your thoughts and kind regards,
 Erik.


--
Erik van Oosten
e.vanoos...@grons.nl
https://day-to-day-stuff.blogspot.com
Committer of 

Re: [DISCUSS] KIP-917: Additional custom metadata for remote log segment

[Ivan Yurchenko]

Hi Alexandre,

> combining custom
> metadata with rlmMetadata increases coupling between Kafka and the
> plugin.

This is true. However, (if I understand your concern correctly,)
rlmMetadata in the current form may be independent from RSM plugins, but
data they point to are accessible only via the particular plugin (the one
that wrote the data or a compatible one). It seems, this coupling already
exists, but it is implicit. To make my point more concrete, imagine an S3
RSM which maps RemoteLogSegmentMetadata objects to S3 object keys. This
mapping logic is a part of the RSM plugin and without it the metadata is
useless. I think it will not get worse if (to follow the example) the
plugin makes the said S3 object keys explicit by adding them to the
metadata. From the high level point of view, moving the custom metadata to
a separate topic doesn't change the picture: it's still the plugin that
binds the standard and custom metadata together.


> For instance, the custom metadata may need to be modified
> outside of Kafka, but the rlmMetadata would still be cached on brokers
> independently of any update of custom metadata. Since both types of
> metadata are authored by different systems, and are cached in
> different layers, this may become a problem, or make plugin migration
> more difficult. What do you think?

This is indeed a problem. I think a solution to this would be to clearly
state that metadata being modified outside of Kafka is out of scope and
instruct the plugin authors that custom metadata could be provided only
reactively from the copyLogSegmentData method and must remain immutable
after that. Does it make sense?


> Yes, you are right that the suggested alternative is to let the plugin
> store its own metadata separately with a solution chosen by the admin
> or plugin provider. For instance, it could be using a dedicated topic
> if chosen to, or relying on an external key-value store.

I see. Yes, this option always exists and doesn't even require a KIP. The
biggest drawback I see is that a plugin will need to reimplement the
consumer/producer + caching mechanics that will exist on the broker side
for the standard remote metadata. I'd like to avoid this and this KIP is
the best solution I see.

Best,
Ivan



On Tue, 18 Apr 2023 at 13:02, Alexandre Dupriez 
wrote:

> Hi Ivan,
>
> Thanks for the follow-up.
>
> Yes, you are right that the suggested alternative is to let the plugin
> store its own metadata separately with a solution chosen by the admin
> or plugin provider. For instance, it could be using a dedicated topic
> if chosen to, or relying on an external key-value store.
>
> I agree with you on the existing risks associated with running
> third-party code inside Apache Kafka. That said, combining custom
> metadata with rlmMetadata increases coupling between Kafka and the
> plugin. For instance, the custom metadata may need to be modified
> outside of Kafka, but the rlmMetadata would still be cached on brokers
> independently of any update of custom metadata. Since both types of
> metadata are authored by different systems, and are cached in
> different layers, this may become a problem, or make plugin migration
> more difficult. What do you think?
>
> I have a vague memory of this being discussed back when the tiered
> storage KIP was started. Maybe Satish has more background on this.
>
> Thanks,
> Alexandre
>
> Le lun. 17 avr. 2023 à 16:50, Ivan Yurchenko
>  a écrit :
> >
> > Hi Alexandre,
> >
> > Thank you for your feedback!
> >
> > > One question I would have is, what is the benefit of adding these
> > > custom metadata in the rlmMetadata rather than letting the plugin
> > > manage access and persistence to them?
> >
> > Could you please elaborate? Do I understand correctly that the idea is
> that
> > the plugin will have its own storage for those custom metadata, for
> example
> > a special topic?
> >
> > > It would be possible for a user
> > > to use custom metadata large enough to adversely impact access to and
> > > caching of the rlmMetadata by Kafka.
> >
> > Since the custom metadata is 100% under control of the RSM plugin, the
> risk
> > is as big as the risk of running a third-party code (i.e. the RSM
> plugin).
> > The cluster admin must make the decision if they trust it.
> > To mitigate this risk and put it under control, the RSM plugin
> > implementations could document what custom metadata they use and estimate
> > their size.
> >
> > Best,
> > Ivan
> >
> >
> > On Mon, 17 Apr 2023 at 18:14, Alexandre Dupriez <
> alexandre.dupr...@gmail.com>
> > wrote:
> >
> > > Hi Ivan,
> > >
> > > Thank you for the KIP.
> > >
> > > I think the KIP clearly explains the need for out-of-band metadata
> > > authored and used by an implementation of the remote storage manager.
> > > One question I would have is, what is the benefit of adding these
> > > custom metadata in the rlmMetadata rather than letting the plugin
> > > manage access and persistence to them?
> > >
> > > Maybe one 

[GitHub] [kafka-site] namantam1 closed pull request #501: Update broken link of one of the Kafka Streams use cases.

[via GitHub]

namantam1 closed pull request #501: Update broken link of one of the Kafka 
Streams use cases.
URL: https://github.com/apache/kafka-site/pull/501


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] namantam1 commented on pull request #501: Update broken link of one of the Kafka Streams use cases.

[via GitHub]

namantam1 commented on PR #501:
URL: https://github.com/apache/kafka-site/pull/501#issuecomment-1536432784

   > Hey @namantam1 Thank you for the change. And welcome to the Kafka 
community (I see that this is your first change with this repo!).
   > 
   > We keep the source of truth (for most) of the doc at 
https://github.com/apache/kafka/tree/trunk/docs and during the release, we copy 
the docs from source of truth to this repository. Hence, your change should 
ideally be targeted at 
https://github.com/apache/kafka/blob/trunk/docs/streams/index.html. You can 
find more information on how to contribute to the documentation at 
https://cwiki.apache.org/confluence/display/KAFKA/Contributing+Website+Documentation+Changes
   > 
   > Could you please open a PR with that repository and close this one?
   > 
   > (as an aside, it helps the reviewers if you add a motivation and 
description of your change in the summary section above).
   
   Sure, closing this PR and thanks for the information. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Resolved] (KAFKA-14859) Support SCRAM ZK to KRaft Migration

[Proven Provenzano (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Proven Provenzano resolved KAFKA-14859.
---
Resolution: Fixed

Merged into trunk and 3.5

> Support SCRAM ZK to KRaft Migration
> ---
>
> Key: KAFKA-14859
> URL: https://issues.apache.org/jira/browse/KAFKA-14859
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Proven Provenzano
>Assignee: Proven Provenzano
>Priority: Major
>
> I want to allow existing ZK installations to be able to migrate to KRaft and 
> support their existing SCRAM credentials.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1822

[Apache Jenkins Server]

See 

Re: [DISCUSS] Apache Kafka 3.5.0 release

[Mickael Maison]

Hi David,

Thanks for the update!
You still own 4 other tickets targeting 3.5: KAFKA-14840, KAFKA-14805,
KAFKA-14918, KAFKA-14692. Should I move all of them to the next
release?
Also KAFKA-14698 and KAFKA-14084 are somewhat related to the
migration. Should I move them too?

Thanks,
Mickael

On Fri, May 5, 2023 at 4:27 PM David Arthur
 wrote:
>
> Hey Mickael, my two ZK migration fixes are in 3.5 now.
>
> Cheers,
> David
>
> On Fri, May 5, 2023 at 9:37 AM Mickael Maison 
> wrote:
>
> > Hi Divij,
> >
> > Some dependencies (ZooKeeper, Snappy, Swagger, zstd, etc) have been
> > updated since 3.4.
> > Regarding your PR, I would have been in favor of bringing this to 3.5
> > a couple of weeks ago, but we're now a week past code freeze for 3.5.
> > Apart if this fixes CVEs, or significant bugs, I think we should only
> > merge it in trunk.
> >
> > Thanks,
> > Mickael
> >
> > On Fri, May 5, 2023 at 1:49 PM Divij Vaidya 
> > wrote:
> > >
> > > Hey Mickael
> > >
> > > Should we consider performing an update of the minor versions of the
> > > dependencies in 3.5 (per https://github.com/apache/kafka/pull/13673)?
> > >
> > > --
> > > Divij Vaidya
> > >
> > >
> > >
> > > On Tue, May 2, 2023 at 5:48 PM Mickael Maison 
> > > wrote:
> > >
> > > > Hi Luke,
> > > >
> > > > Yes I think it makes sense to backport both to 3.5.
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > > On Tue, May 2, 2023 at 11:38 AM Luke Chen  wrote:
> > > > >
> > > > > Hi Mickael,
> > > > >
> > > > > There are 1 bug and 1 improvement that I'd like to backport to 3.5.
> > > > > 1. A small improvement for ZK migration based on KAFKA-14840
> > (mentioned
> > > > > above in David's mail). PR is already merged to trunk.
> > > > > https://issues.apache.org/jira/browse/KAFKA-14909
> > > > >
> > > > > 2. A bug will cause the KRaft controller node to shut down
> > unexpectedly.
> > > > PR
> > > > > is ready for review.
> > > > > https://issues.apache.org/jira/browse/KAFKA-14946
> > > > > https://github.com/apache/kafka/pull/13653
> > > > >
> > > > > Thanks.
> > > > > Luke
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Apr 28, 2023 at 4:18 PM Mickael Maison <
> > mickael.mai...@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi David,
> > > > > >
> > > > > > Yes you can backport these to 3.5. Let me know when you are done.
> > > > > >
> > > > > > Thanks,
> > > > > > Mickael
> > > > > >
> > > > > > On Thu, Apr 27, 2023 at 9:02 PM David Arthur
> > > > > >  wrote:
> > > > > > >
> > > > > > > Hey Mickael,
> > > > > > >
> > > > > > > I have one major ZK migration improvement (KAFKA-14805) that
> > landed
> > > > in
> > > > > > > trunk this week that I'd like to merge to 3.5 (once we fix some
> > test
> > > > > > > failures it introduced). After that, I have another PR for
> > > > KAFKA-14840
> > > > > > > which is essentially a huge bug in the ZK migration logic that
> > needs
> > > > to
> > > > > > > land in 3.5.
> > > > > > >
> > > > > > > https://issues.apache.org/jira/browse/KAFKA-14805 (done)
> > > > > > > https://issues.apache.org/jira/browse/KAFKA-14840 (nearly done)
> > > > > > >
> > > > > > > I just wanted to check with you before cherry-picking these to
> > 3.5
> > > > > > >
> > > > > > > David
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Apr 24, 2023 at 1:18 PM Mickael Maison <
> > > > mickael.mai...@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Justine,
> > > > > > > >
> > > > > > > > That makes sense. Feel free to revert that commit in 3.5.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Mickael
> > > > > > > >
> > > > > > > > On Mon, Apr 24, 2023 at 7:16 PM Mickael Maison <
> > > > > > mickael.mai...@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi Josep,
> > > > > > > > >
> > > > > > > > > Thanks for letting me know!
> > > > > > > > >
> > > > > > > > > On Mon, Apr 24, 2023 at 6:58 PM Justine Olshan
> > > > > > > >  wrote:
> > > > > > > > > >
> > > > > > > > > > Hey Mickael,
> > > > > > > > > >
> > > > > > > > > > I've just opened a blocker to revert KAFKA-14561 in 3.5.
> > There
> > > > are
> > > > > > a
> > > > > > > > few
> > > > > > > > > > blocker bugs that I don't think I can fix before the code
> > > > freeze,
> > > > > > so I
> > > > > > > > > > think for the quality of the release, we should just
> > revert the
> > > > > > commit.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Justine
> > > > > > > > > >
> > > > > > > > > > On Fri, Apr 21, 2023 at 1:23 PM Josep Prat
> > > > > >  > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Mickael,
> > > > > > > > > > >
> > > > > > > > > > > Greg Harris managed to fix a flaky test in
> > > > > > > > > > > https://github.com/apache/kafka/pull/13575, I
> > cherry-picked
> > > > it
> > > > > > to
> > > > > > > > the 3.5
> > > > > > > > > > > (and 3.4) branch. I updated the Jira to reflect that is
> > now
> > > > > > fixed on
> > > > > > > > 3.5.0
> > > > > > > > > > 

Re: [DISCUSS] Apache Kafka 3.5.0 release

[David Arthur]

Hey Mickael, my two ZK migration fixes are in 3.5 now.

Cheers,
David

On Fri, May 5, 2023 at 9:37 AM Mickael Maison 
wrote:

> Hi Divij,
>
> Some dependencies (ZooKeeper, Snappy, Swagger, zstd, etc) have been
> updated since 3.4.
> Regarding your PR, I would have been in favor of bringing this to 3.5
> a couple of weeks ago, but we're now a week past code freeze for 3.5.
> Apart if this fixes CVEs, or significant bugs, I think we should only
> merge it in trunk.
>
> Thanks,
> Mickael
>
> On Fri, May 5, 2023 at 1:49 PM Divij Vaidya 
> wrote:
> >
> > Hey Mickael
> >
> > Should we consider performing an update of the minor versions of the
> > dependencies in 3.5 (per https://github.com/apache/kafka/pull/13673)?
> >
> > --
> > Divij Vaidya
> >
> >
> >
> > On Tue, May 2, 2023 at 5:48 PM Mickael Maison 
> > wrote:
> >
> > > Hi Luke,
> > >
> > > Yes I think it makes sense to backport both to 3.5.
> > >
> > > Thanks,
> > > Mickael
> > >
> > > On Tue, May 2, 2023 at 11:38 AM Luke Chen  wrote:
> > > >
> > > > Hi Mickael,
> > > >
> > > > There are 1 bug and 1 improvement that I'd like to backport to 3.5.
> > > > 1. A small improvement for ZK migration based on KAFKA-14840
> (mentioned
> > > > above in David's mail). PR is already merged to trunk.
> > > > https://issues.apache.org/jira/browse/KAFKA-14909
> > > >
> > > > 2. A bug will cause the KRaft controller node to shut down
> unexpectedly.
> > > PR
> > > > is ready for review.
> > > > https://issues.apache.org/jira/browse/KAFKA-14946
> > > > https://github.com/apache/kafka/pull/13653
> > > >
> > > > Thanks.
> > > > Luke
> > > >
> > > >
> > > >
> > > > On Fri, Apr 28, 2023 at 4:18 PM Mickael Maison <
> mickael.mai...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi David,
> > > > >
> > > > > Yes you can backport these to 3.5. Let me know when you are done.
> > > > >
> > > > > Thanks,
> > > > > Mickael
> > > > >
> > > > > On Thu, Apr 27, 2023 at 9:02 PM David Arthur
> > > > >  wrote:
> > > > > >
> > > > > > Hey Mickael,
> > > > > >
> > > > > > I have one major ZK migration improvement (KAFKA-14805) that
> landed
> > > in
> > > > > > trunk this week that I'd like to merge to 3.5 (once we fix some
> test
> > > > > > failures it introduced). After that, I have another PR for
> > > KAFKA-14840
> > > > > > which is essentially a huge bug in the ZK migration logic that
> needs
> > > to
> > > > > > land in 3.5.
> > > > > >
> > > > > > https://issues.apache.org/jira/browse/KAFKA-14805 (done)
> > > > > > https://issues.apache.org/jira/browse/KAFKA-14840 (nearly done)
> > > > > >
> > > > > > I just wanted to check with you before cherry-picking these to
> 3.5
> > > > > >
> > > > > > David
> > > > > >
> > > > > >
> > > > > > On Mon, Apr 24, 2023 at 1:18 PM Mickael Maison <
> > > mickael.mai...@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Justine,
> > > > > > >
> > > > > > > That makes sense. Feel free to revert that commit in 3.5.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Mickael
> > > > > > >
> > > > > > > On Mon, Apr 24, 2023 at 7:16 PM Mickael Maison <
> > > > > mickael.mai...@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi Josep,
> > > > > > > >
> > > > > > > > Thanks for letting me know!
> > > > > > > >
> > > > > > > > On Mon, Apr 24, 2023 at 6:58 PM Justine Olshan
> > > > > > >  wrote:
> > > > > > > > >
> > > > > > > > > Hey Mickael,
> > > > > > > > >
> > > > > > > > > I've just opened a blocker to revert KAFKA-14561 in 3.5.
> There
> > > are
> > > > > a
> > > > > > > few
> > > > > > > > > blocker bugs that I don't think I can fix before the code
> > > freeze,
> > > > > so I
> > > > > > > > > think for the quality of the release, we should just
> revert the
> > > > > commit.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Justine
> > > > > > > > >
> > > > > > > > > On Fri, Apr 21, 2023 at 1:23 PM Josep Prat
> > > > >  > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Mickael,
> > > > > > > > > >
> > > > > > > > > > Greg Harris managed to fix a flaky test in
> > > > > > > > > > https://github.com/apache/kafka/pull/13575, I
> cherry-picked
> > > it
> > > > > to
> > > > > > > the 3.5
> > > > > > > > > > (and 3.4) branch. I updated the Jira to reflect that is
> now
> > > > > fixed on
> > > > > > > 3.5.0
> > > > > > > > > > as well as 3.6.0.
> > > > > > > > > > Let me know if I forgot anything.
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > >
> > > > > > > > > > On Fri, Apr 21, 2023 at 3:44 PM Mickael Maison <
> > > > > > > mickael.mai...@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi,
> > > > > > > > > > >
> > > > > > > > > > > Just a quick reminder that code freeze is next week.
> > > > > > > > > > > We still have 27 JIRAs targeting 3.5 [0] including
> quite a
> > > few
> > > > > bugs
> > > > > > > > > > > and flaky test issues opened recently. If you have
> time,
> > > take
> > > > > one
> > > > > > > of
> > > > > > > > 

Re: [DISCUSS] Apache Kafka 3.5.0 release

[Mickael Maison]

Hi Divij,

Some dependencies (ZooKeeper, Snappy, Swagger, zstd, etc) have been
updated since 3.4.
Regarding your PR, I would have been in favor of bringing this to 3.5
a couple of weeks ago, but we're now a week past code freeze for 3.5.
Apart if this fixes CVEs, or significant bugs, I think we should only
merge it in trunk.

Thanks,
Mickael

On Fri, May 5, 2023 at 1:49 PM Divij Vaidya  wrote:
>
> Hey Mickael
>
> Should we consider performing an update of the minor versions of the
> dependencies in 3.5 (per https://github.com/apache/kafka/pull/13673)?
>
> --
> Divij Vaidya
>
>
>
> On Tue, May 2, 2023 at 5:48 PM Mickael Maison 
> wrote:
>
> > Hi Luke,
> >
> > Yes I think it makes sense to backport both to 3.5.
> >
> > Thanks,
> > Mickael
> >
> > On Tue, May 2, 2023 at 11:38 AM Luke Chen  wrote:
> > >
> > > Hi Mickael,
> > >
> > > There are 1 bug and 1 improvement that I'd like to backport to 3.5.
> > > 1. A small improvement for ZK migration based on KAFKA-14840 (mentioned
> > > above in David's mail). PR is already merged to trunk.
> > > https://issues.apache.org/jira/browse/KAFKA-14909
> > >
> > > 2. A bug will cause the KRaft controller node to shut down unexpectedly.
> > PR
> > > is ready for review.
> > > https://issues.apache.org/jira/browse/KAFKA-14946
> > > https://github.com/apache/kafka/pull/13653
> > >
> > > Thanks.
> > > Luke
> > >
> > >
> > >
> > > On Fri, Apr 28, 2023 at 4:18 PM Mickael Maison  > >
> > > wrote:
> > >
> > > > Hi David,
> > > >
> > > > Yes you can backport these to 3.5. Let me know when you are done.
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > > On Thu, Apr 27, 2023 at 9:02 PM David Arthur
> > > >  wrote:
> > > > >
> > > > > Hey Mickael,
> > > > >
> > > > > I have one major ZK migration improvement (KAFKA-14805) that landed
> > in
> > > > > trunk this week that I'd like to merge to 3.5 (once we fix some test
> > > > > failures it introduced). After that, I have another PR for
> > KAFKA-14840
> > > > > which is essentially a huge bug in the ZK migration logic that needs
> > to
> > > > > land in 3.5.
> > > > >
> > > > > https://issues.apache.org/jira/browse/KAFKA-14805 (done)
> > > > > https://issues.apache.org/jira/browse/KAFKA-14840 (nearly done)
> > > > >
> > > > > I just wanted to check with you before cherry-picking these to 3.5
> > > > >
> > > > > David
> > > > >
> > > > >
> > > > > On Mon, Apr 24, 2023 at 1:18 PM Mickael Maison <
> > mickael.mai...@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Justine,
> > > > > >
> > > > > > That makes sense. Feel free to revert that commit in 3.5.
> > > > > >
> > > > > > Thanks,
> > > > > > Mickael
> > > > > >
> > > > > > On Mon, Apr 24, 2023 at 7:16 PM Mickael Maison <
> > > > mickael.mai...@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi Josep,
> > > > > > >
> > > > > > > Thanks for letting me know!
> > > > > > >
> > > > > > > On Mon, Apr 24, 2023 at 6:58 PM Justine Olshan
> > > > > >  wrote:
> > > > > > > >
> > > > > > > > Hey Mickael,
> > > > > > > >
> > > > > > > > I've just opened a blocker to revert KAFKA-14561 in 3.5. There
> > are
> > > > a
> > > > > > few
> > > > > > > > blocker bugs that I don't think I can fix before the code
> > freeze,
> > > > so I
> > > > > > > > think for the quality of the release, we should just revert the
> > > > commit.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Justine
> > > > > > > >
> > > > > > > > On Fri, Apr 21, 2023 at 1:23 PM Josep Prat
> > > >  > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Mickael,
> > > > > > > > >
> > > > > > > > > Greg Harris managed to fix a flaky test in
> > > > > > > > > https://github.com/apache/kafka/pull/13575, I cherry-picked
> > it
> > > > to
> > > > > > the 3.5
> > > > > > > > > (and 3.4) branch. I updated the Jira to reflect that is now
> > > > fixed on
> > > > > > 3.5.0
> > > > > > > > > as well as 3.6.0.
> > > > > > > > > Let me know if I forgot anything.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > >
> > > > > > > > > On Fri, Apr 21, 2023 at 3:44 PM Mickael Maison <
> > > > > > mickael.mai...@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > Just a quick reminder that code freeze is next week.
> > > > > > > > > > We still have 27 JIRAs targeting 3.5 [0] including quite a
> > few
> > > > bugs
> > > > > > > > > > and flaky test issues opened recently. If you have time,
> > take
> > > > one
> > > > > > of
> > > > > > > > > > these items or help with the reviews.
> > > > > > > > > >
> > > > > > > > > > I'll send another update next once we've entered code
> > freeze.
> > > > > > > > > >
> > > > > > > > > > 0:
> > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > >
> > https://issues.apache.org/jira/browse/KAFKA-13421?jql=project%20%3D%20KAFKA%20AND%20fixVersion%20%3D%203.5.0%20AND%20status%20not%20in%20(resolved%2C%20closed)%20ORDER%20BY%20priority%20DESC%2C%20status%20DESC%2C%20updated%20DESC
> > > > > > > 

[jira] [Created] (KAFKA-14969) Upgrade Mockito to 4.11.0

[Divij Vaidya (Jira)]

Divij Vaidya created KAFKA-14969:


 Summary: Upgrade Mockito to 4.11.0
 Key: KAFKA-14969
 URL: https://issues.apache.org/jira/browse/KAFKA-14969
 Project: Kafka
  Issue Type: Improvement
  Components: unit tests
Reporter: Divij Vaidya
 Fix For: 3.6.0


Upgrading to Mockito 4.11.0 leads to following errors. Need to fix them before 
upgrade. Note that we cannot upgrade to Mockito 5 because it doesn't support 
JDK 8.


{code:java}
[2023-05-05T12:21:58.628Z] > Task :core:compileTestScala
[2023-05-05T12:21:58.628Z] [Error] 
/home/jenkins/jenkins-agent/712657a4/workspace/Kafka_kafka-pr_PR-13673/core/src/test/scala/unit/kafka/cluster/PartitionTest.scala:2654:31:
 ambiguous reference to overloaded definition,
[2023-05-05T12:21:58.628Z] both method spy in class Mockito of type [T](x$1: 
T*)T
[2023-05-05T12:21:58.628Z] and  method spy in class Mockito of type [T](x$1: T)T
[2023-05-05T12:21:58.628Z] match argument types 
(kafka.server.metadata.MockConfigRepository)
[2023-05-05T12:21:58.628Z] [Error] 
/home/jenkins/jenkins-agent/712657a4/workspace/Kafka_kafka-pr_PR-13673/core/src/test/scala/unit/kafka/cluster/PartitionTest.scala:2658:25:
 ambiguous reference to overloaded definition,
[2023-05-05T12:21:58.628Z] both method spy in class Mockito of type [T](x$1: 
T*)T
[2023-05-05T12:21:58.628Z] and  method spy in class Mockito of type [T](x$1: T)T
[2023-05-05T12:21:58.628Z] match argument types (kafka.log.LogManager)
[2023-05-05T12:21:58.628Z] [Error] 
/home/jenkins/jenkins-agent/712657a4/workspace/Kafka_kafka-pr_PR-13673/core/src/test/scala/unit/kafka/cluster/PartitionTest.scala:2688:31:
 ambiguous reference to overloaded definition,
[2023-05-05T12:21:58.628Z] both method spy in class Mockito of type [T](x$1: 
T*)T
[2023-05-05T12:21:58.628Z] and  method spy in class Mockito of type [T](x$1: T)T
[2023-05-05T12:21:58.628Z] match argument types 
(kafka.server.metadata.MockConfigRepository)
[2023-05-05T12:21:58.628Z] [Error] 
/home/jenkins/jenkins-agent/712657a4/workspace/Kafka_kafka-pr_PR-13673/core/src/test/scala/unit/kafka/cluster/PartitionTest.scala:2692:25:
 ambiguous reference to overloaded definition,
[2023-05-05T12:21:58.628Z] both method spy in class Mockito of type [T](x$1: 
T*)T
[2023-05-05T12:21:58.628Z] and  method spy in class Mockito of type [T](x$1: T)T
[2023-05-05T12:21:58.628Z] match argument types (kafka.log.LogManager)
[2023-05-05T12:21:58.628Z] [Error] 
/home/jenkins/jenkins-agent/712657a4/workspace/Kafka_kafka-pr_PR-13673/core/src/test/scala/unit/kafka/cluster/PartitionTest.scala:2728:31:
 ambiguous reference to overloaded definition,
[2023-05-05T12:21:58.628Z] both method spy in class Mockito of type [T](x$1: 
T*)T
[2023-05-05T12:21:58.628Z] and  method spy in class Mockito of type [T](x$1: T)T
[2023-05-05T12:21:58.628Z] match argument types 
(kafka.server.metadata.MockConfigRepository)
[2023-05-05T12:21:58.628Z] [Error] 
/home/jenkins/jenkins-agent/712657a4/workspace/Kafka_kafka-pr_PR-13673/core/src/test/scala/unit/kafka/cluster/PartitionTest.scala:2734:25:
 ambiguous reference to overloaded definition,
[2023-05-05T12:21:58.628Z] both method spy in class Mockito of type [T](x$1: 
T*)T
[2023-05-05T12:21:58.628Z] and  method spy in class Mockito of type [T](x$1: T)T
[2023-05-05T12:21:58.628Z] match argument types (kafka.log.LogManager)
[2023-05-05T12:22:06.267Z] [Error] 
/home/jenkins/jenkins-agent/712657a4/workspace/Kafka_kafka-pr_PR-13673/core/src/test/scala/unit/kafka/controller/ControllerIntegrationTest.scala:807:21:
 ambiguous reference to overloaded definition,
[2023-05-05T12:22:06.267Z] both method spy in class Mockito of type [T](x$1: 
T*)T
[2023-05-05T12:22:06.267Z] and  method spy in class Mockito of type [T](x$1: T)T
[2023-05-05T12:22:06.267Z] match argument types 
(controller.eventManager.ControllerEventThread)
[2023-05-05T12:22:06.267Z] [Error] 
/home/jenkins/jenkins-agent/712657a4/workspace/Kafka_kafka-pr_PR-13673/core/src/test/scala/unit/kafka/controller/ControllerIntegrationTest.scala:838:21:
 ambiguous reference to overloaded definition,
[2023-05-05T12:22:06.267Z] both method spy in class Mockito of type [T](x$1: 
T*)T
[2023-05-05T12:22:06.267Z] and  method spy in class Mockito of type [T](x$1: T)T
[2023-05-05T12:22:06.267Z] match argument types 
(controller.eventManager.ControllerEventThread){code}
{code:java}
[2023-05-05T12:22:40.865Z] > Task :core:compileTestScala
[2023-05-05T12:22:40.865Z] [Error] 
/home/jenkins/jenkins-agent/712657a4/workspace/Kafka_kafka-pr_PR-13673/core/src/test/scala/unit/kafka/log/LogManagerTest.scala:171:18:
 ambiguous reference to overloaded definition,
[2023-05-05T12:22:40.865Z] both method spy in class Mockito of type [T](x$1: 
T*)T
[2023-05-05T12:22:40.865Z] and  method spy in class Mockito of type [T](x$1: T)T
[2023-05-05T12:22:40.865Z] match argument types (kafka.log.LogManager) and 
expected result type 

Re: Request to contribute to Kafka

[Mickael Maison]

Hi,

I've granted you permissions on the wiki and Jira. Thanks for your
interest in Kafka!

Thanks,
Mickael

On Thu, May 4, 2023 at 3:15 PM Lianet M.  wrote:
>
> Wiki ID: lianetmr
>
> Jira ID: lianetm
>
> Both on email liane...@gmail.com
>
>
> Thanks!
>
> Lianet Magrans

[jira] [Created] (KAFKA-14968) Consumer Notification Callback

[Hendrik Jan van Randen (Jira)]

Hendrik Jan van Randen created KAFKA-14968:
--

 Summary: Consumer Notification Callback
 Key: KAFKA-14968
 URL: https://issues.apache.org/jira/browse/KAFKA-14968
 Project: Kafka
  Issue Type: New Feature
  Components: consumer
Reporter: Hendrik Jan van Randen


We want to use Kafka for all the communication between our services. 

We have many high frequency topics (many messages per second), for which the 
Kafka mechanism of polling by consumers is efficient and suitable.

However, we also have low frequency topics (no or few messages per day). If 
such a low frequency message is produced, all consumers should consume it 
immediately. But it would be inefficient if all those consumers are polling 
continuously with a high frequency (e.g. each second), while no or few such 
messages are produced each day. Most efficient would be if for such a low 
frequency topic the consumers can register a callback (e.g. a REST endpoint) 
with Kafka, and Kafka calls that callback when a message is produced. Either 
supplying the message in the call (pro: consumer can handle it immediately), or 
just to notify the consumer that it should do a one time poll (pro: same 
message transfer mechanism as everywhere).

I just talked this over with Ryan Corrigan of Confluent.

In the coming weeks I will supply more information about the urgency of this 
issue.

I will be present at the Kafka Summit in London on May 16 & 17, 2023. Who of 
the Confluent technical staff, or Kafka committers, is present at the Kafka 
Summit to investigate possible solution directions for this?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Question: CI Pipeline in Kafka GitHub Pull Requests & Github Actions Usage

[aaron ai]

Dear David:

I greatly appreciate your welcome and the explanation that you offered
regarding the Jenkins system!

I will try to familiarize myself with the standards and contribution
process of the Kafka community, and look forward to making some
contributions in the coming days.

Best regards,
Aaron

On Thu, May 4, 2023 at 11:47 PM David Arthur
 wrote:

> Hello, Aaron and welcome to the project!
>
> If you look towards the bottom of a Pull Request there will be a section
> reporting the status of the Checks for the latest commit.
> There is a "Details" link that takes you to the Jenkins job for that PR.
>
> The default Jenkins view is the new UI (called Blue Ocean I think?). The
> "Classic" Jenkins view is also available by clicking the button near the
> top:
>
> [image: image.png]
>
> From the Classic view it's pretty straightforward to download the console
> log for the job to see what went wrong.
>
> Our build steps are defined in the Jenkinsfile
>  checked into the
> repository. We do have a compile + static analysis step that happens before
> running tests.
>
> -David
>
> On Thu, May 4, 2023 at 5:34 AM aaron ai  wrote:
>
>> Hey folks,
>>
>> I hope this email finds you well. My name is Aaron Ai, and I'm a beginner
>> interested in the Kafka community. I am eager to contribute to Kafka's
>> development and be part of this amazing project.
>>
>> However, I have noticed some confusion surrounding the CI pipeline in
>> Kafka
>> GitHub pull requests. It appears that the builds often fail, and the
>> results are not very clear to analyse. How can I determine if a pull
>> request build is okay for now, given the current CI pipeline situation?
>> Furthermore, is it a good idea to use Github Action to help with some
>> static checks? If so, I would be more than happy to contribute. By the
>> way,
>> is there any plan to migrate to Github Actions in the near future?
>>
>> I eagerly await your response and appreciate any feedback you might have
>> on
>> this matter. Thank you for your time and consideration.
>>
>> Best regards,
>> Aaron
>>
>
>
> --
> -David
>

[jira] [Resolved] (KAFKA-14926) Remove metrics on Log Cleaner shutdown

[David Jacot (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-14926.
-
Resolution: Fixed

> Remove metrics on Log Cleaner shutdown
> --
>
> Key: KAFKA-14926
> URL: https://issues.apache.org/jira/browse/KAFKA-14926
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Reporter: Divij Vaidya
>Assignee: Divij Vaidya
>Priority: Minor
> Fix For: 3.6.0
>
>
> We register metrics with the KafkaMetricsGroup in LogCleaner.scala but we 
> don't remove them on shutdown.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Apache Kafka 3.5.0 release

[Divij Vaidya]

Hey Mickael

Should we consider performing an update of the minor versions of the
dependencies in 3.5 (per https://github.com/apache/kafka/pull/13673)?

--
Divij Vaidya



On Tue, May 2, 2023 at 5:48 PM Mickael Maison 
wrote:

> Hi Luke,
>
> Yes I think it makes sense to backport both to 3.5.
>
> Thanks,
> Mickael
>
> On Tue, May 2, 2023 at 11:38 AM Luke Chen  wrote:
> >
> > Hi Mickael,
> >
> > There are 1 bug and 1 improvement that I'd like to backport to 3.5.
> > 1. A small improvement for ZK migration based on KAFKA-14840 (mentioned
> > above in David's mail). PR is already merged to trunk.
> > https://issues.apache.org/jira/browse/KAFKA-14909
> >
> > 2. A bug will cause the KRaft controller node to shut down unexpectedly.
> PR
> > is ready for review.
> > https://issues.apache.org/jira/browse/KAFKA-14946
> > https://github.com/apache/kafka/pull/13653
> >
> > Thanks.
> > Luke
> >
> >
> >
> > On Fri, Apr 28, 2023 at 4:18 PM Mickael Maison  >
> > wrote:
> >
> > > Hi David,
> > >
> > > Yes you can backport these to 3.5. Let me know when you are done.
> > >
> > > Thanks,
> > > Mickael
> > >
> > > On Thu, Apr 27, 2023 at 9:02 PM David Arthur
> > >  wrote:
> > > >
> > > > Hey Mickael,
> > > >
> > > > I have one major ZK migration improvement (KAFKA-14805) that landed
> in
> > > > trunk this week that I'd like to merge to 3.5 (once we fix some test
> > > > failures it introduced). After that, I have another PR for
> KAFKA-14840
> > > > which is essentially a huge bug in the ZK migration logic that needs
> to
> > > > land in 3.5.
> > > >
> > > > https://issues.apache.org/jira/browse/KAFKA-14805 (done)
> > > > https://issues.apache.org/jira/browse/KAFKA-14840 (nearly done)
> > > >
> > > > I just wanted to check with you before cherry-picking these to 3.5
> > > >
> > > > David
> > > >
> > > >
> > > > On Mon, Apr 24, 2023 at 1:18 PM Mickael Maison <
> mickael.mai...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi Justine,
> > > > >
> > > > > That makes sense. Feel free to revert that commit in 3.5.
> > > > >
> > > > > Thanks,
> > > > > Mickael
> > > > >
> > > > > On Mon, Apr 24, 2023 at 7:16 PM Mickael Maison <
> > > mickael.mai...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > Hi Josep,
> > > > > >
> > > > > > Thanks for letting me know!
> > > > > >
> > > > > > On Mon, Apr 24, 2023 at 6:58 PM Justine Olshan
> > > > >  wrote:
> > > > > > >
> > > > > > > Hey Mickael,
> > > > > > >
> > > > > > > I've just opened a blocker to revert KAFKA-14561 in 3.5. There
> are
> > > a
> > > > > few
> > > > > > > blocker bugs that I don't think I can fix before the code
> freeze,
> > > so I
> > > > > > > think for the quality of the release, we should just revert the
> > > commit.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Justine
> > > > > > >
> > > > > > > On Fri, Apr 21, 2023 at 1:23 PM Josep Prat
> > >  > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Mickael,
> > > > > > > >
> > > > > > > > Greg Harris managed to fix a flaky test in
> > > > > > > > https://github.com/apache/kafka/pull/13575, I cherry-picked
> it
> > > to
> > > > > the 3.5
> > > > > > > > (and 3.4) branch. I updated the Jira to reflect that is now
> > > fixed on
> > > > > 3.5.0
> > > > > > > > as well as 3.6.0.
> > > > > > > > Let me know if I forgot anything.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > >
> > > > > > > > On Fri, Apr 21, 2023 at 3:44 PM Mickael Maison <
> > > > > mickael.mai...@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > Just a quick reminder that code freeze is next week.
> > > > > > > > > We still have 27 JIRAs targeting 3.5 [0] including quite a
> few
> > > bugs
> > > > > > > > > and flaky test issues opened recently. If you have time,
> take
> > > one
> > > > > of
> > > > > > > > > these items or help with the reviews.
> > > > > > > > >
> > > > > > > > > I'll send another update next once we've entered code
> freeze.
> > > > > > > > >
> > > > > > > > > 0:
> > > > > > > > >
> > > > > > > >
> > > > >
> > >
> https://issues.apache.org/jira/browse/KAFKA-13421?jql=project%20%3D%20KAFKA%20AND%20fixVersion%20%3D%203.5.0%20AND%20status%20not%20in%20(resolved%2C%20closed)%20ORDER%20BY%20priority%20DESC%2C%20status%20DESC%2C%20updated%20DESC
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Mickael
> > > > > > > > >
> > > > > > > > > On Thu, Apr 20, 2023 at 9:14 PM Mickael Maison <
> > > > > mickael.mai...@gmail.com
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi Ron,
> > > > > > > > > >
> > > > > > > > > > Yes feel free to merge that fix. Thanks for letting me
> know!
> > > > > > > > > >
> > > > > > > > > > Mickael
> > > > > > > > > >
> > > > > > > > > > On Thu, Apr 20, 2023 at 8:15 PM Ron Dagostino <
> > > rndg...@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi Mickael.  I would like to merge
> > > > > > > > > > > 

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1821

[Apache Jenkins Server]

See 

[GitHub] [kafka-site] showuon merged pull request #365: Update ops.html

[via GitHub]

showuon merged PR #365:
URL: https://github.com/apache/kafka-site/pull/365


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] divijvaidya commented on pull request #501: Update broken link of one of the Kafka Streams use cases.

[via GitHub]

divijvaidya commented on PR #501:
URL: https://github.com/apache/kafka-site/pull/501#issuecomment-1535927663

   Hey @namantam1 
   Thank you for the change. And welcome to the Kafka community (I see that 
this is your first change with this repo!).
   
   We keep the source of truth (for most) of the doc at 
https://github.com/apache/kafka/tree/trunk/docs and during the release, we copy 
the docs from source of truth to this repository. Hence, your change should 
ideally be targeted at 
https://github.com/apache/kafka/blob/trunk/docs/streams/index.html. You can 
find more information on how to contribute to the documentation at 
https://cwiki.apache.org/confluence/display/KAFKA/Contributing+Website+Documentation+Changes
 
   
   Could you please open a PR with that repository and close this one?
   
   (as an aside, it helps the reviewers if you add a motivation and description 
of your change in the summary section above).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] divijvaidya commented on pull request #365: Update ops.html

[via GitHub]

divijvaidya commented on PR #365:
URL: https://github.com/apache/kafka-site/pull/365#issuecomment-1535919387

   Hey @jaapterwoerds 
   Thank you for the contributions and welcome to the Kafka community (I 
noticed that this is your first contribution!).
   
   As you might already be aware, this bug has already been fixed in all 
versions released since late 2021, as part of 
https://github.com/apache/kafka/pull/10433. The change you are submitting here 
is for documentation of version 2.8.
   As part of our [end of life 
policy](https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan#TimeBasedReleasePlan-WhatIsOurEOLPolicy?),
 that is an unsupported version and we usually don't backport changes 
(documentation or otherwise) to the unsupported versions. 
   
   Having said that, since this is your first commit to our community, I will 
request the reviewers to merge this change in. 
   
   (As an aside, it helps the reviewers if you add a motivation and the 
description of the change in the summary section)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Resolved] (KAFKA-14963) Incorrect partition count metrics for kraft controllers

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-14963.
---
Resolution: Fixed

> Incorrect partition count metrics for kraft controllers
> ---
>
> Key: KAFKA-14963
> URL: https://issues.apache.org/jira/browse/KAFKA-14963
> Project: Kafka
>  Issue Type: Bug
>  Components: controller, kraft
>Affects Versions: 3.4.0
>Reporter: José Armando García Sancio
>Assignee: José Armando García Sancio
>Priority: Major
> Fix For: 3.4.1
>
>
> It is possible for the KRaft controller to report more partitions than are 
> available in the cluster. This is because the following test fail against 
> 3.4.0:
> {code:java}
>        @Test
>       public void testPartitionCountDecreased() {
>           ControllerMetrics metrics = new MockControllerMetrics();
>           ControllerMetricsManager manager = new 
> ControllerMetricsManager(metrics);          Uuid createTopicId = 
> Uuid.randomUuid();
>           Uuid createPartitionTopicId = new Uuid(
>               createTopicId.getMostSignificantBits(),
>               createTopicId.getLeastSignificantBits()
>           );
>           Uuid removeTopicId = new 
> Uuid(createTopicId.getMostSignificantBits(), 
> createTopicId.getLeastSignificantBits());
>           manager.replay(topicRecord("test", createTopicId));
>           manager.replay(partitionRecord(createPartitionTopicId, 0, 0, 
> Arrays.asList(0, 1, 2)));
>           manager.replay(partitionRecord(createPartitionTopicId, 1, 0, 
> Arrays.asList(0, 1, 2)));
>           manager.replay(removeTopicRecord(removeTopicId));
>           assertEquals(0, metrics.globalPartitionCount());
>       }
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)