[jira] [Created] (KAFKA-16758) Extend Consumer#close with option to leave the group or not

[A. Sophie Blee-Goldman (Jira)]

A. Sophie Blee-Goldman created KAFKA-16758:
--

 Summary: Extend Consumer#close with option to leave the group or 
not
 Key: KAFKA-16758
 URL: https://issues.apache.org/jira/browse/KAFKA-16758
 Project: Kafka
  Issue Type: New Feature
  Components: consumer
Reporter: A. Sophie Blee-Goldman


See comments on https://issues.apache.org/jira/browse/KAFKA-16514 for the full 
context.

Essentially we would get rid of the "internal.leave.group.on.close" config that 
is used as a backdoor by Kafka Streams right now to prevent closed consumers 
from leaving the group, thus reducing unnecessary task movements after a simple 
bounce. 

This would be replaced by an actual public API that would allow the caller to 
opt in or out to the LeaveGroup when close is called. This would be similar to 
the KafkaStreams#close(CloseOptions) API, and in fact would be how that API 
will be implemented (since it only works for static groups at the moment as 
noted in KAFKA-16514 )

This has several benefits over the current situation:
 # It allows plain consumer apps to opt-out of leaving the group when closed, 
which is currently not possible through any public API (only an internal 
backdoor config)
 # It enables the caller to dynamically select the appropriate action depending 
on why the client is being closed – for example, you would not want the 
consumer to leave the group during a simple restart, but would want it to leave 
the group when shutting down the app or if scaling down the node. This is not 
possible today, even with the internal config, since configs are immutable
 # It can be leveraged to "fix" the KafkaStreams#close(closeOptions) API so 
that the user's choice to leave the group during close will be respected for 
non-static members



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-1033: Add Kafka Streams exception handler for exceptions occurring during processing

[Matthias J. Sax]

+1 (binding)

On 5/13/24 5:54 PM, Sophie Blee-Goldman wrote:

Thanks for the KIP guys!

+1 (binding)

On Mon, May 13, 2024 at 6:02 AM Bill Bejeck  wrote:


Thanks for the KIP, this will be a great addition!

+1(binding)

-Bill

On Fri, May 3, 2024 at 4:48 AM Bruno Cadonna  wrote:


Hi Damien, Sébastien, and Loïc,

Thanks for the KIP!

+1 (binding)

Best,
Bruno


On 4/26/24 4:00 PM, Damien Gasparina wrote:

Hi all,

We would like to start a vote for KIP-1033: Add Kafka Streams
exception handler for exceptions occurring during processing

The KIP is available on




https://cwiki.apache.org/confluence/display/KAFKA/KIP-1033%3A+Add+Kafka+Streams+exception+handler+for+exceptions+occurring+during+processing


If you have any suggestions or feedback, feel free to participate to
the discussion thread:
https://lists.apache.org/thread/1nhhsrogmmv15o7mk9nj4kvkb5k2bx9s

Best regards,
Damien Sebastien and Loic

Re: [KAFKA-16361] Rack aware sticky assignor minQuota violations

[Xiangyuan LI]

Please check the pr  , the
logic in AbstractStickyAssignor has many mistakes, though it's been a long
time since pr submitted, I couldn't remember the whole detail now.

Julien Opoix  于2024年5月14日周二 03:55写道：

> Hi there,
>
> We're facing a critical issue with Java Kafka client version 3.5+ and
> sticky assignors (details here:
> https://issues.apache.org/jira/browse/KAFKA-16361).
> Could we request prioritization for this issue? We're willing to assist,
> if necessary, though guidance would be appreciated given the complexity of
> diving into the code.
>
> Best regards.
>

Re: [KAFKA-16361] Rack aware sticky assignor minQuota violations

[Sophie Blee-Goldman]

Hey,

Took a look at the ticket and left a comment. Seems like narrowing down
which versions this affects is a good first step, after that I'm happy to
help you understand the code and put together a fix if you're willing. I'd
recommend starting out by going through the "constrainedAssign" method to
familiarize yourself with what it's trying to do, and what condition is
being violated.

 We can continue the discussion on the ticket though

Cheers,
Sophie

On Mon, May 13, 2024 at 7:52 AM Julien Opoix  wrote:

> Hi there,
>
> We're facing a critical issue with Java Kafka client version 3.5+ and
> sticky assignors (details here:
> https://issues.apache.org/jira/browse/KAFKA-16361).
> Could we request prioritization for this issue? We're willing to assist,
> if necessary, though guidance would be appreciated given the complexity of
> diving into the code.
>
> Best regards.
>
>

Re: [VOTE] KIP-1033: Add Kafka Streams exception handler for exceptions occurring during processing

[Sophie Blee-Goldman]

Thanks for the KIP guys!

+1 (binding)

On Mon, May 13, 2024 at 6:02 AM Bill Bejeck  wrote:

> Thanks for the KIP, this will be a great addition!
>
> +1(binding)
>
> -Bill
>
> On Fri, May 3, 2024 at 4:48 AM Bruno Cadonna  wrote:
>
> > Hi Damien, Sébastien, and Loïc,
> >
> > Thanks for the KIP!
> >
> > +1 (binding)
> >
> > Best,
> > Bruno
> >
> >
> > On 4/26/24 4:00 PM, Damien Gasparina wrote:
> > > Hi all,
> > >
> > > We would like to start a vote for KIP-1033: Add Kafka Streams
> > > exception handler for exceptions occurring during processing
> > >
> > > The KIP is available on
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1033%3A+Add+Kafka+Streams+exception+handler+for+exceptions+occurring+during+processing
> > >
> > > If you have any suggestions or feedback, feel free to participate to
> > > the discussion thread:
> > > https://lists.apache.org/thread/1nhhsrogmmv15o7mk9nj4kvkb5k2bx9s
> > >
> > > Best regards,
> > > Damien Sebastien and Loic
> >
>

[jira] [Created] (KAFKA-16757) Fix broker re-registration issues around MV 3.7-IV2

[Colin McCabe (Jira)]

Colin McCabe created KAFKA-16757:


 Summary: Fix broker re-registration issues around MV 3.7-IV2
 Key: KAFKA-16757
 URL: https://issues.apache.org/jira/browse/KAFKA-16757
 Project: Kafka
  Issue Type: Bug
Reporter: Colin McCabe
Assignee: Colin McCabe


When upgrading from a MetadataVersion older than 3.7-IV2, we need to resend the 
broker registration, so that the controller can record the storage directories. 
The current code for doing this has several problems, however. One is that it 
tends to trigger even in cases where we don't actually need it. Another is that 
when re-registering the broker, the broker is marked as fenced.

This PR moves the handling of the re-registration case out of 
BrokerMetadataPublisher and into BrokerRegistrationTracker. The re-registration 
code there will only trigger in the case where the broker sees an existing 
registration for itself with no directories set. This is much more targetted 
than the original code.

Additionally, in ClusterControlManager, when re-registering the same broker, we 
now preserve its fencing and shutdown state, rather than clearing those. (There 
isn't any good reason re-registering the same broker should clear these 
things... this was purely an oversight.) Note that we can tell the broker is 
"the same" because it has the same IncarnationId.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1038: Add Custom Error Handler to Producer

[Alieh Saeedi]

Hi all,


Thanks for all the valid points you listed.


KIP updates and addressing concerns:


1) The KIP now suggests two Response types: `RetryableResponse` and
`NonRetryableResponse`


2) `custom.exception.handler` is changed to `custom.exception.handler.class`


3) The KIP clarifies that `In the case of an implemented handler for the
specified exception, the handler takes precedence.`


4)  There is now a `default` implementation for both handle() methods.


5)  @Chris: for `UnknownTopicOrPartition`, the default is already retrying
for 60s. (In fact, the default value of `max.block.ms`). If the handler
instructs to FAIL or SWALLOW, there will be no retry, and if the handler
instructs to RETRY, that will be the default behavior, which follows the
values in already existing config parameters such as `max.block.ms`. Does
that make sense?


Hope the changes and explanations are convincing :)


Cheers,

Alieh

On Mon, May 13, 2024 at 6:40 PM Justine Olshan 
wrote:

> Oh I see. The type isn't the error type but a newly defined type for the
> response. Makes sense and works for me.
>
> Justine
>
> On Mon, May 13, 2024 at 9:13 AM Chris Egerton 
> wrote:
>
> > If we have dedicated methods for each kind of exception
> > (handleRecordTooLarge, handleUnknownTopicOrPartition, etc.), doesn't that
> > provide sufficient constraint? I'm not suggesting we eliminate these
> > methods, just that we change their return types to something more
> flexible.
> >
> > On Mon, May 13, 2024, 12:07 Justine Olshan  >
> > wrote:
> >
> > > I'm not sure I agree with the Retriable and NonRetriableResponse
> comment.
> > > This doesn't limit the blast radius or enforce certain errors are used.
> > > I think we might disagree on how controlled these interfaces can be...
> > >
> > > Justine
> > >
> > > On Mon, May 13, 2024 at 8:40 AM Chris Egerton  >
> > > wrote:
> > >
> > > > Hi Alieh,
> > > >
> > > > Thanks for the updates! I just have a few more thoughts:
> > > >
> > > > - I don't think a boolean property is sufficient to dictate retries
> for
> > > > unknown topic partitions, though. These errors can occur if a topic
> has
> > > > just been created, which can occur if, for example, automatic topic
> > > > creation is enabled for a multi-task connector. This is why I
> proposed
> > a
> > > > timeout instead of a boolean (and see my previous email for why
> > reducing
> > > > max.block.ms for a producer is not a viable alternative). If it
> helps,
> > > one
> > > > way to reproduce this yourself is to add the line
> > > > `fooProps.put(TASKS_MAX_CONFIG, "10");` to the integration test here:
> > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/blob/5439914c32fa00d634efa7219699f1bc21add839/connect/runtime/src/test/java/org/apache/kafka/connect/integration/SourceConnectorsIntegrationTest.java#L134
> > > > and then check the logs afterward for messages like "Error while
> > fetching
> > > > metadata with correlation id  :
> > > {foo-topic=UNKNOWN_TOPIC_OR_PARTITION}".
> > > >
> > > > - I also don't think we need custom XxxResponse enums for every
> > possible
> > > > method; it seems like this will lead to a lot of duplication and
> > > cognitive
> > > > overhead if we want to expand the error handler in the future.
> > Something
> > > > more flexible like RetriableResponse and NonRetriableResponse could
> > > > suffice.
> > > >
> > > > - Finally, the KIP still doesn't state how the handler will or won't
> > take
> > > > precedence over existing retry properties. If I set `retries` or `
> > > > delivery.timeout.ms` or `max.block.ms` to low values, will that
> cause
> > > > retries to cease even if my custom handler would otherwise keep
> > returning
> > > > RETRY for an error?
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Mon, May 13, 2024 at 11:02 AM Andrew Schofield <
> > > > andrew_schofi...@live.com>
> > > > wrote:
> > > >
> > > > > Hi Alieh,
> > > > > Just a few more comments on the KIP. It is looking much less risky
> > now
> > > > the
> > > > > scope
> > > > > is tighter.
> > > > >
> > > > > [AJS1] It would be nice to have default implementations of the
> handle
> > > > > methods
> > > > > so an implementor would not need to implement both themselves.
> > > > >
> > > > > [AJS2] Producer configurations which are class names usually end in
> > > > > “.class”.
> > > > > I suggest “custom.exception.handler.class”.
> > > > >
> > > > > [AJS3] If I implemented a handler, and I set a non-default value
> for
> > > one
> > > > > of the
> > > > > new configuations, what happens? I would expect that the handler
> > takes
> > > > > precedence. I wasn’t quite clear what “the control will follow the
> > > > handler
> > > > > instructions” meant.
> > > > >
> > > > > [AJS4] Because you now have an enum for the
> > > > > RecordTooLargeExceptionResponse,
> > > > > I don’t think you need to state in the comment for
> > > > > ProducerExceptionHandler that
> > > > > RETRY will be interpreted as FAIL.
> > > > >
> > > > > Thanks,

Re: DISCUSS KIP-984 Add pluggable compression interface to Kafka

[Greg Harris]

Hi Assane,

Thank you for the further information about your motivation and
intended use-cases, that adds a lot of context.

> Our motivation here is to accelerate compression with the use of hardware 
> accelerators.

This is a very broad statement, so let me break it down into cases,
and what I would recommend in each:

Case A: Open source accelerators for supported compression codecs (e.g. zstd)
1. Try to add your accelerator to an existing upstream implementation
(e.g. zstd-jni), so that whenever that library is used, people benefit
from your accelerator.
2. Fork an existing implementation, and propose that the Kafka project
use your fork.

Case B: Closed-source accelerators for supported compression codecs (e.g. zstd)
1. Fork an existing implementation, and structure your fork such that
it can be swapped out at runtime by operators that want a particular
accelerator.
2. Kafka can add a Java pluggable interface to the broker and clients
to pick among the accelerated and non-accelerated plugins, falling
back to non-accelerated "reference implementations" as necessary. This
wouldn't require protocol changes.

Case C: Accelerators for unsupported open source compression codecs
(e.g. brotli)
1. I think that these should be proposed as official codecs for Kafka
to support, and then the acceleration can be implemented as in Case A
or B.

Case D: Accelerators for unsupported closed-source compression codecs
These are codecs that would require a fully pluggable implementation,
and reserved bits in the binary protocol. They are also the codecs
which are most damaging to the ecosystem. If you have a specific
proprietary codec in mind please say so, otherwise I want to invoke
the YAGNI principle here.

Thanks,
Greg





On Mon, May 13, 2024 at 11:22 AM Diop, Assane  wrote:
>
> Hi Greg,
>
> Thank you for your thoughtful response. Resending this email to continue 
> engagement on the KIP discussion.
>
> Our motivation here is to accelerate compression with the use of hardware 
> accelerators.
>
> If the community prefers, we would be happy to contribute code to support 
> compression accelerators, but we believe that introducing a pluggable 
> compression framework   is more scalable than enabling new compression 
> algorithms in an ad hoc manner.
>
> A pluggable compression interface would enable hardware accelerators without 
> requiring vendor-specific code in Kafka code base.
>
> We aim to ensure robustness by supporting all possible language-clients. In 
> this latest iteration, this design provides a path to support other languages 
> where each client has its own topic holding the plugin information for that 
> language.
>
> The pluggable interface does not replace the built-in functionally, rather, 
> it is an optional compression path seamlessly added for Kafka users who would 
> like to use custom compression algorithms or simply accelerate current 
> algorithms. In this latter case, a vendor providing acceleration for 
> compression will need to support their plugins.
>
> As far as your concerns, I appreciate you taking the time to respond. Let me 
> address them the best I can:
> 1) When an operator adds a plugin to a cluster, they must ensure that the 
> compression algorithms for all the supported language-clients of that plugin 
> are compatible . For the plugin to be installed, the language must support 
> dynamic loading or linking of libraries and these mechanisms exist in at 
> least Java, C, Go and Python. Clients written in a language that does not 
> support dynamic loading or linking can still use built-in codecs and coexist 
> in a cluster where plugins were registered. This coexistence highlights that 
> the use of plugins is an optional feature.
>
> 2) Plugins source should come from a reputable developer. This is true of any 
> dependencies. Should an operator register a plugin, the plugin should have a 
> path for support including deprecation of such plugin. If the community finds 
> it useful, there could be an official Kafka repository and we are open to 
> discussing ways to provide governance of the plugin ecosystem.
>
> 3) We do not see this as a fork of the binary protocol, but rather an 
> evolution of the protocol to provide additional flexibility for compression.
> Once a plugin is registered, it is compatible with all the “flavors” of 
> the plugins which here means different minor versions of a codec. Compression 
> algorithms typically follow semantic versioning where v1.x is compatible with 
> v1.y and where v2.x is not necessarily compatible with v1.x.
> If a plugin version breaks compatibility with an older version, then it 
> should be treated as a new plugin with a new plugin alias.
> In parallel to the plugin topic holding plugin information during 
> registration, additional topics holding the plugin binaries can be published 
> by the plugin admin tool during installation to ensure compatibility. We view 
> this as improving performance at the cost of extra 

[KAFKA-16361] Rack aware sticky assignor minQuota violations

[Julien Opoix]

Hi there,

We're facing a critical issue with Java Kafka client version 3.5+ and sticky 
assignors (details here: https://issues.apache.org/jira/browse/KAFKA-16361).
Could we request prioritization for this issue? We're willing to assist, if 
necessary, though guidance would be appreciated given the complexity of diving 
into the code.

Best regards.

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2898

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-1028: Docker Official Image for Apache Kafka

[Vedarth Sharma]

Hey Chris,

Once we provide the definitions to docker, they should take care of
everything from there. They mentioned here

that
the image will be rebuilt when the base image is updated. Hence active
rebuilds won't require any changes from our side.
If we are packaging something which may contain a CVE, like some jar, then
the onus will be on us to patch it, but it will be upto us whether we
consider the threat severe enough to fix and when we want to provide the
fixed version. Having Docker Official Image will not impact the frequency
of our releases. It will be the Apache Kafka community's call on when a
release goes and Docker Official Image will be released accordingly as per
the KIP. source


As mentioned in Docker's documentation as well "In essence we strive to
heed upstream's recommendations on how they intend for their software to be
consumed." source

Docker Official Image will rely on upstream's recommendation for
functionality. But I do agree that since Docker's stance on this might
change in future it makes sense to put a safeguard that will not allow any
functionality changes get incorporated as part of the vetting process. I
have updated the KIP to reflect the same.

KIP-975 has a well defined public interface based on how configs can be
supplied and how it can be used. I am not sure if we put that label on it
during discussions. I am happy to have a separate email thread on it to
iron things out.

I hope this addresses all of your concerns!

Thanks and regards,
Vedarth

On Mon, May 13, 2024 at 10:55 PM Chris Egerton 
wrote:

> Thanks both for your responses! Friendly reminder: again, better to provide
> a quote instead of just a link :)
>
> I've seen a bit about image rebuilding to handle CVEs but I'm a little
> unclear on how this would work in practice, and I couldn't find any
> concrete details in any of the links. Does Docker do this automatically for
> DOIs? Or will the onus be on us to put out patched images? Would this lead
> to us putting out images more quickly than we put out standard releases? As
> a plus, it does look like DOIs get the benefit of Docker Scout [1] for
> free, which is nice, but it's still unclear who'd be doing the rest of the
> work on that front.
>
> As far as this point from Vedarth goes:
>
> > By incorporating the source code of the Docker Official Image into our
> > AK ecosystem, we gain control over its functionality, ensuring alignment
> > with the OSS Docker image. This ensures a seamless experience for users
> who
> > may need to transition between these images.
>
> This captures my concern with the KIP pretty well. If there's any
> significant divergence in behavior (not just build methodology) between the
> apache/kafka image and what Docker requires for a Kafka DOI, how are we
> going to vet these changes moving forward? Under the "Post Release Process
> - if Dockerhub folks suggest changes to the Dockerfiles:" header, this KIP
> proposes that we port all suggested changes for the DOI to
> the docker/jvm/Dockerfile image, but this seems a bit too permissive. As an
> alternative, we could state that all build-related changes can be done with
> a PR on the apache/kafka GitHub repo (which will require approval from a
> single committer), but any functional changes will require a KIP.
>
> Finally, during KIP-975 was there discussion on what we would count as the
> public interface for the apache/kafka image? If not, it'd be nice to get
> that ironed out since it may make future discussions around our Docker
> images quicker, but I don't think this is necessary for KIP-1028.
>
> [1] - https://www.docker.com/products/docker-scout/
>
> On Mon, May 13, 2024 at 4:37 AM Prabha Manepalli
>  wrote:
>
> > Hi Chris,
> >
> > Sharing the requested links explaining why Docker Official images are
> > considered more secure -
> >
> >
> https://www.docker.com/blog/enhancing-security-and-transparency-with-docker-official-images/
> > and
> >
> >
> https://github.com/docker-library/faq#why-does-my-security-scanner-show-that-an-image-has-cves
> >
> > I hope these links help you understand why we need Docker Official images
> > for organisations with stringent security compliance requirements for
> their
> > Kafka workloads.
> >
> > Thank you.
> >
> >
> >
> > On Sun, May 12, 2024 at 3:33 PM Vedarth Sharma  >
> > wrote:
> >
> > > Hey Chris!
> > >
> > > Functionality wise, we don't intend to have any differences between OSS
> > > Docker Image and Docker Official Image.
> > > The Docker Official Image will be the recommended one.
> > > Since the Docker Official Image might be delayed due to review done by
> > > Docker, images on apache/kafka (OSS Docker Image) can be used by users.
> > >
> > > 1) I read 

[jira] [Resolved] (KAFKA-15578) Run System Tests for Old protocol in the New Coordinator

[David Jacot (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-15578.
-
Resolution: Fixed

> Run System Tests for Old protocol in the New Coordinator
> 
>
> Key: KAFKA-15578
> URL: https://issues.apache.org/jira/browse/KAFKA-15578
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Ritika Reddy
>Assignee: Ritika Reddy
>Priority: Major
>  Labels: kip-848-preview
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> Change existing system tests related to the consumer group protocol and group 
> coordinator to test the old protocol running with the new coordinator.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-1036: Extend RecordDeserializationException exception

[Kirk True]

+1 (non-binding)

Thanks Fred!

> On May 13, 2024, at 5:46 AM, Bill Bejeck  wrote:
> 
> Thanks for the KIP!
> 
> +1 (binding)
> 
> -Bill
> 
> 
> On Tue, May 7, 2024 at 6:16 PM Sophie Blee-Goldman 
> wrote:
> 
>> +1 (binding)
>> 
>> thanks for the KIP!
>> 
>> On Fri, May 3, 2024 at 9:13 AM Matthias J. Sax  wrote:
>> 
>>> +1 (binding)
>>> 
>>> On 5/3/24 8:52 AM, Federico Valeri wrote:
 Hi Fred, this is a useful addition.
 
 +1 non binding
 
 Thanks
 
 On Fri, May 3, 2024 at 4:11 PM Andrew Schofield
  wrote:
> 
> Hi Fred,
> Thanks for the KIP. It’s turned out nice and elegant I think.
>>> Definitely a worthwhile improvement.
> 
> +1 (non-binding)
> 
> Thanks,
> Andrew
> 
>> On 30 Apr 2024, at 14:02, Frédérik Rouleau
>>>  wrote:
>> 
>> Hi all,
>> 
>> As there is no more activity for a while on the discuss thread, I
>>> think we
>> can start a vote.
>> The KIP is available on
>> 
>>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1036%3A+Extend+RecordDeserializationException+exception
>> 
>> 
>> If you have some feedback or suggestions, please participate to the
>> discussion thread:
>> https://lists.apache.org/thread/or85okygtfywvnsfd37kwykkq5jq7fy5
>> 
>> Best regards,
>> Fred
> 
>>> 
>> 

[jira] [Resolved] (KAFKA-16117) Add Integration test for checking if the correct assignor is chosen

[David Jacot (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-16117.
-
Fix Version/s: 3.8.0
   Resolution: Fixed

> Add Integration test for checking if the correct assignor is chosen
> ---
>
> Key: KAFKA-16117
> URL: https://issues.apache.org/jira/browse/KAFKA-16117
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Ritika Reddy
>Priority: Minor
> Fix For: 3.8.0
>
>
> h4.  We are trying to test this section of the KIP-848
> h4. Assignor Selection
> The group coordinator has to determine which assignment strategy must be used 
> for the group. The group's members may not have exactly the same assignors at 
> any given point in time - e.g. they may migrate from an assignor to another 
> one for instance. The group coordinator will chose the assignor as follow:
>  * A client side assignor is used if possible. This means that a client side 
> assignor must be supported by all the members. If multiple are, it will 
> respect the precedence defined by the members when they advertise their 
> supported client side assignors.
>  * A server side assignor is used otherwise. If multiple server side 
> assignors are specified in the group, the group coordinator uses the most 
> common one. If a member does not provide an assignor, the group coordinator 
> will default to the first one in {{{}group.consumer.assignors{}}}.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16735) Deprecate offsets.commit.required.acks in 3.8

[David Jacot (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-16735.
-
Resolution: Fixed

> Deprecate offsets.commit.required.acks in 3.8
> -
>
> Key: KAFKA-16735
> URL: https://issues.apache.org/jira/browse/KAFKA-16735
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: David Jacot
>Assignee: David Jacot
>Priority: Blocker
> Fix For: 3.8.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

RE: DISCUSS KIP-984 Add pluggable compression interface to Kafka

[Diop, Assane]

Hi Greg, 

Thank you for your thoughtful response. Resending this email to continue 
engagement on the KIP discussion. 

Our motivation here is to accelerate compression with the use of hardware 
accelerators. 

If the community prefers, we would be happy to contribute code to support 
compression accelerators, but we believe that introducing a pluggable 
compression framework   is more scalable than enabling new compression 
algorithms in an ad hoc manner.

A pluggable compression interface would enable hardware accelerators without 
requiring vendor-specific code in Kafka code base.

We aim to ensure robustness by supporting all possible language-clients. In 
this latest iteration, this design provides a path to support other languages 
where each client has its own topic holding the plugin information for that 
language.

The pluggable interface does not replace the built-in functionally, rather, it 
is an optional compression path seamlessly added for Kafka users who would like 
to use custom compression algorithms or simply accelerate current algorithms. 
In this latter case, a vendor providing acceleration for compression will need 
to support their plugins.

As far as your concerns, I appreciate you taking the time to respond. Let me 
address them the best I can: 
1) When an operator adds a plugin to a cluster, they must ensure that the 
compression algorithms for all the supported language-clients of that plugin 
are compatible . For the plugin to be installed, the language must support 
dynamic loading or linking of libraries and these mechanisms exist in at least 
Java, C, Go and Python. Clients written in a language that does not support 
dynamic loading or linking can still use built-in codecs and coexist in a 
cluster where plugins were registered. This coexistence highlights that the use 
of plugins is an optional feature. 

2) Plugins source should come from a reputable developer. This is true of any 
dependencies. Should an operator register a plugin, the plugin should have a 
path for support including deprecation of such plugin. If the community finds 
it useful, there could be an official Kafka repository and we are open to 
discussing ways to provide governance of the plugin ecosystem. 

3) We do not see this as a fork of the binary protocol, but rather an evolution 
of the protocol to provide additional flexibility for compression. 
Once a plugin is registered, it is compatible with all the “flavors” of the 
plugins which here means different minor versions of a codec. Compression 
algorithms typically follow semantic versioning where v1.x is compatible with 
v1.y and where v2.x is not necessarily compatible with v1.x. 
If a plugin version breaks compatibility with an older version, then it should 
be treated as a new plugin with a new plugin alias. 
In parallel to the plugin topic holding plugin information during registration, 
additional topics holding the plugin binaries can be published by the plugin 
admin tool during installation to ensure compatibility. We view this as 
improving performance at the cost of extra operator work.

4) We only require the operator to register and then install the plugins. 
During the registration  process,  the plugin admin tool  takes in a plugin 
information (plugin alias and classname/library) and then  internally assigns 
the pluginID.  The operator is only responsible for providing the plugin alias 
and the className/library. The plugin admin tool is a new Java class in Tools 
that interacts with the operator to setup the plugins in the cluster. At this 
stage of the KIP, we have assumed a manual installation of the plugin. 
Installing here means the deployment of the plugin binary making it ready to be 
dynamically loaded/linked when needed. 
We are looking at an option for dynamic installation of the plugin which would 
require the operator to install the binary using the plugin admin tool. Using 
the same concept as plugin registration, the operator can install the plugin 
binary by publishing it to a topic using the plugin admin tool. Clients that 
register a plugin by consuming the plugin list would also consume the necessary 
binaries from the cluster. 

5) When a plugin is used, the set of built-in codecs is augmented by the set of 
plugins described in the plugin topic. The additional set of codecs is 
cluster-dependent, so, while a given batch of records stays within a cluster, 
they remain self-contained. If these batches are produced into another cluster, 
then the operator needs to either recompress data using builtins/available 
plugins or install plugins in the dependent cluster. In this scenario a 
consumer would decompress the data, and, if the mirrored data needs the same 
compression plugin, then the operator is required to register and install the 
plugins in the secondary cluster.  
Our assertion is that the additional work required by an operator could be 
justified by improved performance.

6) There is a finite number 

Re: [DISCUSS] KIP-891: Running multiple versions of Connector plugins

[Snehashis]

Hi Greg,

That is much appreciated. No complaints on the additional scope, I will
make some time out to work on this once we have approval.

Thanks
Snehashis

On Fri, May 10, 2024 at 9:28 PM Greg Harris 
wrote:

> Hey Snehashis,
>
> I'm glad to hear you're still interested in this KIP!
> I'm happy to let you drive this, and I apologize for increasing the
> scope of work so drastically. To make up for that, I'll volunteer to
> be the primary PR reviewer to help get this done quickly once the KIP
> is approved.
>
> Thanks,
> Greg
>
>
> On Fri, May 10, 2024 at 3:51 AM Snehashis 
> wrote:
> >
> > Hi Greg,
> >
> > Thanks for the follow up to my original KIP, I am in favour of the
> > additions made to expand its scope, the addition of range versions
> > specifically make a lot of sense.
> >
> > Apologies if I have not publicly worked on this KIP for a long time. The
> > original work was done when the move to service loading was in discussion
> > and I wanted to loop back to this only after that work was completed.
> Post
> > its conclusion, I have not been able to take this up due to other
> > priorities. If it's okay with you, I would still like to get this
> > implemented myself, including the additional scope.
> >
> > Thanks and regards
> > Snehashis
> >
> > On Fri, May 10, 2024 at 12:45 AM Greg Harris
> 
> > wrote:
> >
> > > Hi all,
> > >
> > > I'd like to reboot the discussion on KIP-891:
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+Connector+plugins
> > >
> > > I've made some changes, most notably:
> > >
> > > 1. Specifying versions for all plugins in Connector configs
> > > (converters, header converters, transforms, and predicates) not just
> > > connectors & tasks
> > > 2. Specifying a range of versions instead of an exact match
> > > 3. New metrics to observe what versions are in-use
> > >
> > > Thanks to Snehashis for the original KIP idea!
> > >
> > > Thanks,
> > > Greg
> > >
> > > On Tue, Jan 2, 2024 at 11:49 AM Greg Harris 
> wrote:
> > > >
> > > > Hi Snehashis,
> > > >
> > > > Thank you for the KIP! This is something I've wanted for a long time.
> > > >
> > > > I know the discussion has gone cold, are you still interested in
> > > > pursuing this feature? I'll make time to review the KIP if you are
> > > > still accepting comments.
> > > >
> > > > Thanks,
> > > > Greg
> > > >
> > > > On Tue, Nov 22, 2022 at 12:29 PM Snehashis  >
> > > wrote:
> > > > >
> > > > > Thanks for the points Sagar.
> > > > >
> > > > > > 1) Should we update the GET /connectors endpoint to include the
> > > version of
> > > > > > the plugin that is running? It could be useful to figure out the
> > > version
> > > > > of
> > > > > > the plugin or I am assuming it gets returned by the expand=info
> call?
> > > > >
> > > > > I think this is good to have and possible future enhancement. The
> > > version
> > > > > info will be present in the config of the connector if the user has
> > > > > specified the version. Otherwise it is the latest version which the
> > > user
> > > > > can find out from the connector-plugin endpoint. The information
> can be
> > > > > introduced to the response of the GET /connectors endpoint itself,
> > > however
> > > > > the most ideal way of doing this would be to get the currently
> running
> > > > > instance of the connector and get the version directly from there.
> > > This is
> > > > > slightly tricky as the connector could be running in a different
> node.
> > > > > One way to do this would be to persist the version information in
> the
> > > > > status backing store during instantiation of the connector. It
> requires
> > > > > some more thought and since the version is part of the configs if
> > > provided
> > > > > and evident otherwise, I have not included it in this KIP.
> > > > >
> > > > > > 2) I am not aware of this and hence asking, can 2 connectors with
> > > > > different
> > > > > > versions have the same name? Does the plugin isolation allow
> this?
> > > This
> > > > > > could have a bearing when using the lifecycle endpoints for
> > > connectors
> > > > > like
> > > > > > DELETE etc.
> > > > >
> > > > > All connectors in a cluster need to have uniquire connector names
> > > > > regardless of what version of the plugin the connector is running
> > > > > underneath. This is something enforced by the connect runtime
> itself.
> > > All
> > > > > connect CRUD operations are keyed on the connector name so there
> will
> > > not
> > > > > be an issue.
> > > > >
> > > > > Regards
> > > > > Snehashis
> > > > >
> > > > > On Tue, Nov 22, 2022 at 3:16 PM Sagar 
> > > wrote:
> > > > >
> > > > > > Hey Snehashsih,
> > > > > >
> > > > > > Thanks for the KIP. It looks like a very useful feature. Couple
> of
> > > > > > small-ish points, let me know what you think:
> > > > > >
> > > > > > 1) Should we update the GET /connectors endpoint to include the
> > > version of
> > > > > > the plugin that is running? It could be useful to figure 

[jira] [Created] (KAFKA-16755) Implement lock timeout functionality in SharePartition

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-16755:
-

 Summary: Implement lock timeout functionality in SharePartition
 Key: KAFKA-16755
 URL: https://issues.apache.org/jira/browse/KAFKA-16755
 Project: Kafka
  Issue Type: Sub-task
Reporter: Apoorv Mittal
Assignee: Abhinav Dixit






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16756) Implement max delivery count functionality in SharePartition

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-16756:
-

 Summary: Implement max delivery count functionality in 
SharePartition
 Key: KAFKA-16756
 URL: https://issues.apache.org/jira/browse/KAFKA-16756
 Project: Kafka
  Issue Type: Sub-task
Reporter: Apoorv Mittal
Assignee: Chirag Wadhwa






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16754) Implement release acquired records functionality in SharePartition

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-16754:
-

 Summary: Implement release acquired records functionality in 
SharePartition
 Key: KAFKA-16754
 URL: https://issues.apache.org/jira/browse/KAFKA-16754
 Project: Kafka
  Issue Type: Sub-task
Reporter: Apoorv Mittal
Assignee: Abhinav Dixit






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16753) Implement acknowledge functionality in SharePartition

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-16753:
-

 Summary: Implement acknowledge functionality in SharePartition
 Key: KAFKA-16753
 URL: https://issues.apache.org/jira/browse/KAFKA-16753
 Project: Kafka
  Issue Type: Sub-task
Reporter: Apoorv Mittal
Assignee: Apoorv Mittal






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16752) Implement acquire functionality in SharePartition

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-16752:
-

 Summary: Implement acquire functionality in SharePartition
 Key: KAFKA-16752
 URL: https://issues.apache.org/jira/browse/KAFKA-16752
 Project: Kafka
  Issue Type: Sub-task
Reporter: Apoorv Mittal
Assignee: Apoorv Mittal






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16749) Implement share fetch messages

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-16749:
-

 Summary: Implement share fetch messages
 Key: KAFKA-16749
 URL: https://issues.apache.org/jira/browse/KAFKA-16749
 Project: Kafka
  Issue Type: Sub-task
Reporter: Apoorv Mittal
Assignee: Apoorv Mittal






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16751) Implement release acquired records in SharePartitionManager

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-16751:
-

 Summary: Implement release acquired records in 
SharePartitionManager
 Key: KAFKA-16751
 URL: https://issues.apache.org/jira/browse/KAFKA-16751
 Project: Kafka
  Issue Type: Sub-task
Reporter: Apoorv Mittal
Assignee: Abhinav Dixit






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16750) Implement acknowledge API in SharePartitionManager

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-16750:
-

 Summary: Implement acknowledge API in SharePartitionManager
 Key: KAFKA-16750
 URL: https://issues.apache.org/jira/browse/KAFKA-16750
 Project: Kafka
  Issue Type: Sub-task
Reporter: Apoorv Mittal
Assignee: Chirag Wadhwa






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16747) Implement share sessions and context

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-16747:
-

 Summary: Implement share sessions and context
 Key: KAFKA-16747
 URL: https://issues.apache.org/jira/browse/KAFKA-16747
 Project: Kafka
  Issue Type: Sub-task
Reporter: Apoorv Mittal
Assignee: Abhinav Dixit






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16748) Implement share response handling in SharePartitionManager

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-16748:
-

 Summary: Implement share response handling in SharePartitionManager
 Key: KAFKA-16748
 URL: https://issues.apache.org/jira/browse/KAFKA-16748
 Project: Kafka
  Issue Type: Sub-task
Reporter: Apoorv Mittal
Assignee: Abhinav Dixit






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16746) Add support for share acknowledgement request in KafkaApis

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-16746:
-

 Summary: Add support for share acknowledgement request in KafkaApis
 Key: KAFKA-16746
 URL: https://issues.apache.org/jira/browse/KAFKA-16746
 Project: Kafka
  Issue Type: Sub-task
Reporter: Apoorv Mittal
Assignee: Chirag Wadhwa






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16745) Add support for share fetch request in KafkaApis

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-16745:
-

 Summary: Add support for share fetch request in KafkaApis
 Key: KAFKA-16745
 URL: https://issues.apache.org/jira/browse/KAFKA-16745
 Project: Kafka
  Issue Type: Sub-task
Reporter: Apoorv Mittal
Assignee: Chirag Wadhwa






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16744) Add support for share group describe in KafkaApis

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-16744:
-

 Summary: Add support for share group describe in KafkaApis
 Key: KAFKA-16744
 URL: https://issues.apache.org/jira/browse/KAFKA-16744
 Project: Kafka
  Issue Type: Sub-task
Reporter: Apoorv Mittal
Assignee: Apoorv Mittal






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16743) Add support for share group heartbeat in KafkaApis

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-16743:
-

 Summary: Add support for share group heartbeat in KafkaApis
 Key: KAFKA-16743
 URL: https://issues.apache.org/jira/browse/KAFKA-16743
 Project: Kafka
  Issue Type: Sub-task
Reporter: Apoorv Mittal






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16741) Add SharGroupHeartbeat API support in GroupCoordinator

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-16741:
-

 Summary: Add SharGroupHeartbeat API support in GroupCoordinator
 Key: KAFKA-16741
 URL: https://issues.apache.org/jira/browse/KAFKA-16741
 Project: Kafka
  Issue Type: Sub-task
Reporter: Apoorv Mittal
Assignee: Apoorv Mittal






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16742) Add ShareGroupDescribe API support in GroupCoordinator

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-16742:
-

 Summary: Add ShareGroupDescribe API support in GroupCoordinator
 Key: KAFKA-16742
 URL: https://issues.apache.org/jira/browse/KAFKA-16742
 Project: Kafka
  Issue Type: Sub-task
Reporter: Apoorv Mittal
Assignee: Apoorv Mittal






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16740) Define skeleton for SharePartitionManager and SharePartition

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-16740:
-

 Summary: Define skeleton for SharePartitionManager and 
SharePartition
 Key: KAFKA-16740
 URL: https://issues.apache.org/jira/browse/KAFKA-16740
 Project: Kafka
  Issue Type: Sub-task
Reporter: Apoorv Mittal
Assignee: Apoorv Mittal


Add high level design for broker side implementation for fetching and 
acknowledging messages.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16739) Exclude protected variables from aggregated JavaDocs

[Greg Harris (Jira)]

Greg Harris created KAFKA-16739:
---

 Summary: Exclude protected variables from aggregated JavaDocs
 Key: KAFKA-16739
 URL: https://issues.apache.org/jira/browse/KAFKA-16739
 Project: Kafka
  Issue Type: Bug
  Components: documentation
Affects Versions: 3.7.0, 3.6.0, 3.5.0
Reporter: Greg Harris
Assignee: Greg Harris


Protected variables were excluded from the jars generated by `javadoc` in 
KAFKA-14839 but not from `aggregatedJavadoc`, which is used when generating the 
Apache Kafka web documentation. This means that there are two different 
versions of the 3.5-3.7 docs published: one without the protected classes and 
methods, and one with.

We should align these two javadoc configurations so that protected classes 
methods and variables are hidden everywhere.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1028: Docker Official Image for Apache Kafka

[Chris Egerton]

Thanks both for your responses! Friendly reminder: again, better to provide
a quote instead of just a link :)

I've seen a bit about image rebuilding to handle CVEs but I'm a little
unclear on how this would work in practice, and I couldn't find any
concrete details in any of the links. Does Docker do this automatically for
DOIs? Or will the onus be on us to put out patched images? Would this lead
to us putting out images more quickly than we put out standard releases? As
a plus, it does look like DOIs get the benefit of Docker Scout [1] for
free, which is nice, but it's still unclear who'd be doing the rest of the
work on that front.

As far as this point from Vedarth goes:

> By incorporating the source code of the Docker Official Image into our
> AK ecosystem, we gain control over its functionality, ensuring alignment
> with the OSS Docker image. This ensures a seamless experience for users
who
> may need to transition between these images.

This captures my concern with the KIP pretty well. If there's any
significant divergence in behavior (not just build methodology) between the
apache/kafka image and what Docker requires for a Kafka DOI, how are we
going to vet these changes moving forward? Under the "Post Release Process
- if Dockerhub folks suggest changes to the Dockerfiles:" header, this KIP
proposes that we port all suggested changes for the DOI to
the docker/jvm/Dockerfile image, but this seems a bit too permissive. As an
alternative, we could state that all build-related changes can be done with
a PR on the apache/kafka GitHub repo (which will require approval from a
single committer), but any functional changes will require a KIP.

Finally, during KIP-975 was there discussion on what we would count as the
public interface for the apache/kafka image? If not, it'd be nice to get
that ironed out since it may make future discussions around our Docker
images quicker, but I don't think this is necessary for KIP-1028.

[1] - https://www.docker.com/products/docker-scout/

On Mon, May 13, 2024 at 4:37 AM Prabha Manepalli
 wrote:

> Hi Chris,
>
> Sharing the requested links explaining why Docker Official images are
> considered more secure -
>
> https://www.docker.com/blog/enhancing-security-and-transparency-with-docker-official-images/
> and
>
> https://github.com/docker-library/faq#why-does-my-security-scanner-show-that-an-image-has-cves
>
> I hope these links help you understand why we need Docker Official images
> for organisations with stringent security compliance requirements for their
> Kafka workloads.
>
> Thank you.
>
>
>
> On Sun, May 12, 2024 at 3:33 PM Vedarth Sharma 
> wrote:
>
> > Hey Chris!
> >
> > Functionality wise, we don't intend to have any differences between OSS
> > Docker Image and Docker Official Image.
> > The Docker Official Image will be the recommended one.
> > Since the Docker Official Image might be delayed due to review done by
> > Docker, images on apache/kafka (OSS Docker Image) can be used by users.
> >
> > 1) I read https://docs.docker.com/trusted-content/official-images/ and
> > > didn't find anything on that page or immediately around it that
> explains
> > > what compliance requirements might be satisfied by a DOI that couldn't
> be
> > > satisfied by the existing apache/kafka image. Can you elaborate? Feel
> > free
> > > to provide another link, but please also quote the relevant sections
> from
> > > it (as StackOverflow likes to say, links can grow stale over time).
> >
> >
> >- If a user's selection is confined solely to Docker Official Images,
> >this Docker Image will ensure their access to Kafka.
> >- Details on specific advantages of Docker Official Images can be
> found
> >at
> >
> >
> https://github.com/docker-library/official-images?tab=readme-ov-file#what-are-official-images
> >.
> >- The Docker Official Image will be actively rebuilt for updates and
> >security fixes.
> >- It's true we can provide exactly the same benefits in the OSS Docker
> >Image as well. But it won't have the Docker Official Image badge and
> it
> >will add additional overhead for Apache Kafka community.
> >- The fact that it will have Docker Official Image badge will set it
> >apart from the OSS Docker Image.
> >- Also the ability to do just docker pull kafka to get the kafka
> docker
> >image will only be possible with Docker Official Image.
> >
> >
> > 2) It would be great to see a brief summary of the differences in these
> > > images included in the KIP, in order to try to gauge how this would
> look
> > to
> > > our users.
> >
> >
> >- Functionally, both Docker images will remain identical.
> >- The variance lies primarily in the methodologies of building and
> >validation, as outlined in the updated KIP.
> >
> >
> > 3) What I suggested last time was not a separate apache/apache-docker
> > > repository, but a repository controlled entirely by Docker. The DOI
> docs
> > > [1] state that "While it's 

Re: [DISCUSS] KIP-1038: Add Custom Error Handler to Producer

[Justine Olshan]

Oh I see. The type isn't the error type but a newly defined type for the
response. Makes sense and works for me.

Justine

On Mon, May 13, 2024 at 9:13 AM Chris Egerton 
wrote:

> If we have dedicated methods for each kind of exception
> (handleRecordTooLarge, handleUnknownTopicOrPartition, etc.), doesn't that
> provide sufficient constraint? I'm not suggesting we eliminate these
> methods, just that we change their return types to something more flexible.
>
> On Mon, May 13, 2024, 12:07 Justine Olshan 
> wrote:
>
> > I'm not sure I agree with the Retriable and NonRetriableResponse comment.
> > This doesn't limit the blast radius or enforce certain errors are used.
> > I think we might disagree on how controlled these interfaces can be...
> >
> > Justine
> >
> > On Mon, May 13, 2024 at 8:40 AM Chris Egerton 
> > wrote:
> >
> > > Hi Alieh,
> > >
> > > Thanks for the updates! I just have a few more thoughts:
> > >
> > > - I don't think a boolean property is sufficient to dictate retries for
> > > unknown topic partitions, though. These errors can occur if a topic has
> > > just been created, which can occur if, for example, automatic topic
> > > creation is enabled for a multi-task connector. This is why I proposed
> a
> > > timeout instead of a boolean (and see my previous email for why
> reducing
> > > max.block.ms for a producer is not a viable alternative). If it helps,
> > one
> > > way to reproduce this yourself is to add the line
> > > `fooProps.put(TASKS_MAX_CONFIG, "10");` to the integration test here:
> > >
> > >
> >
> https://github.com/apache/kafka/blob/5439914c32fa00d634efa7219699f1bc21add839/connect/runtime/src/test/java/org/apache/kafka/connect/integration/SourceConnectorsIntegrationTest.java#L134
> > > and then check the logs afterward for messages like "Error while
> fetching
> > > metadata with correlation id  :
> > {foo-topic=UNKNOWN_TOPIC_OR_PARTITION}".
> > >
> > > - I also don't think we need custom XxxResponse enums for every
> possible
> > > method; it seems like this will lead to a lot of duplication and
> > cognitive
> > > overhead if we want to expand the error handler in the future.
> Something
> > > more flexible like RetriableResponse and NonRetriableResponse could
> > > suffice.
> > >
> > > - Finally, the KIP still doesn't state how the handler will or won't
> take
> > > precedence over existing retry properties. If I set `retries` or `
> > > delivery.timeout.ms` or `max.block.ms` to low values, will that cause
> > > retries to cease even if my custom handler would otherwise keep
> returning
> > > RETRY for an error?
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Mon, May 13, 2024 at 11:02 AM Andrew Schofield <
> > > andrew_schofi...@live.com>
> > > wrote:
> > >
> > > > Hi Alieh,
> > > > Just a few more comments on the KIP. It is looking much less risky
> now
> > > the
> > > > scope
> > > > is tighter.
> > > >
> > > > [AJS1] It would be nice to have default implementations of the handle
> > > > methods
> > > > so an implementor would not need to implement both themselves.
> > > >
> > > > [AJS2] Producer configurations which are class names usually end in
> > > > “.class”.
> > > > I suggest “custom.exception.handler.class”.
> > > >
> > > > [AJS3] If I implemented a handler, and I set a non-default value for
> > one
> > > > of the
> > > > new configuations, what happens? I would expect that the handler
> takes
> > > > precedence. I wasn’t quite clear what “the control will follow the
> > > handler
> > > > instructions” meant.
> > > >
> > > > [AJS4] Because you now have an enum for the
> > > > RecordTooLargeExceptionResponse,
> > > > I don’t think you need to state in the comment for
> > > > ProducerExceptionHandler that
> > > > RETRY will be interpreted as FAIL.
> > > >
> > > > Thanks,
> > > > Andrew
> > > >
> > > > > On 13 May 2024, at 14:53, Alieh Saeedi
>  > >
> > > > wrote:
> > > > >
> > > > > Hi all,
> > > > >
> > > > >
> > > > > Thanks for the very interesting discussion during my PTO.
> > > > >
> > > > >
> > > > > KIP updates and addressing concerns:
> > > > >
> > > > >
> > > > > 1) Two handle() methods are defined in ProducerExceptionHandler for
> > the
> > > > two
> > > > > exceptions with different input parameters so that we have
> > > > > handle(RecordTooLargeException e, ProducerRecord record) and
> > > > > handle(UnknownTopicOrPartitionException e, ProducerRecord record)
> > > > >
> > > > >
> > > > > 2) The ProducerExceptionHandler extends `Closable` as well.
> > > > >
> > > > >
> > > > > 3) The KIP suggests having two more configuration parameters with
> > > boolean
> > > > > values:
> > > > >
> > > > > - `drop.invalid.large.records` with a default value of `false` for
> > > > > swallowing too large records.
> > > > >
> > > > > - `retry.unknown.topic.partition` with a default value of `true`
> that
> > > > > performs RETRY for `max.block.ms` ms, encountering the
> > > > > UnknownTopicOrPartitionException.
> > > > >
> > > > >
> > > > > Hope the main concerns 

Re: [DISCUSS] KIP-1038: Add Custom Error Handler to Producer

[Chris Egerton]

If we have dedicated methods for each kind of exception
(handleRecordTooLarge, handleUnknownTopicOrPartition, etc.), doesn't that
provide sufficient constraint? I'm not suggesting we eliminate these
methods, just that we change their return types to something more flexible.

On Mon, May 13, 2024, 12:07 Justine Olshan 
wrote:

> I'm not sure I agree with the Retriable and NonRetriableResponse comment.
> This doesn't limit the blast radius or enforce certain errors are used.
> I think we might disagree on how controlled these interfaces can be...
>
> Justine
>
> On Mon, May 13, 2024 at 8:40 AM Chris Egerton 
> wrote:
>
> > Hi Alieh,
> >
> > Thanks for the updates! I just have a few more thoughts:
> >
> > - I don't think a boolean property is sufficient to dictate retries for
> > unknown topic partitions, though. These errors can occur if a topic has
> > just been created, which can occur if, for example, automatic topic
> > creation is enabled for a multi-task connector. This is why I proposed a
> > timeout instead of a boolean (and see my previous email for why reducing
> > max.block.ms for a producer is not a viable alternative). If it helps,
> one
> > way to reproduce this yourself is to add the line
> > `fooProps.put(TASKS_MAX_CONFIG, "10");` to the integration test here:
> >
> >
> https://github.com/apache/kafka/blob/5439914c32fa00d634efa7219699f1bc21add839/connect/runtime/src/test/java/org/apache/kafka/connect/integration/SourceConnectorsIntegrationTest.java#L134
> > and then check the logs afterward for messages like "Error while fetching
> > metadata with correlation id  :
> {foo-topic=UNKNOWN_TOPIC_OR_PARTITION}".
> >
> > - I also don't think we need custom XxxResponse enums for every possible
> > method; it seems like this will lead to a lot of duplication and
> cognitive
> > overhead if we want to expand the error handler in the future. Something
> > more flexible like RetriableResponse and NonRetriableResponse could
> > suffice.
> >
> > - Finally, the KIP still doesn't state how the handler will or won't take
> > precedence over existing retry properties. If I set `retries` or `
> > delivery.timeout.ms` or `max.block.ms` to low values, will that cause
> > retries to cease even if my custom handler would otherwise keep returning
> > RETRY for an error?
> >
> > Cheers,
> >
> > Chris
> >
> > On Mon, May 13, 2024 at 11:02 AM Andrew Schofield <
> > andrew_schofi...@live.com>
> > wrote:
> >
> > > Hi Alieh,
> > > Just a few more comments on the KIP. It is looking much less risky now
> > the
> > > scope
> > > is tighter.
> > >
> > > [AJS1] It would be nice to have default implementations of the handle
> > > methods
> > > so an implementor would not need to implement both themselves.
> > >
> > > [AJS2] Producer configurations which are class names usually end in
> > > “.class”.
> > > I suggest “custom.exception.handler.class”.
> > >
> > > [AJS3] If I implemented a handler, and I set a non-default value for
> one
> > > of the
> > > new configuations, what happens? I would expect that the handler takes
> > > precedence. I wasn’t quite clear what “the control will follow the
> > handler
> > > instructions” meant.
> > >
> > > [AJS4] Because you now have an enum for the
> > > RecordTooLargeExceptionResponse,
> > > I don’t think you need to state in the comment for
> > > ProducerExceptionHandler that
> > > RETRY will be interpreted as FAIL.
> > >
> > > Thanks,
> > > Andrew
> > >
> > > > On 13 May 2024, at 14:53, Alieh Saeedi  >
> > > wrote:
> > > >
> > > > Hi all,
> > > >
> > > >
> > > > Thanks for the very interesting discussion during my PTO.
> > > >
> > > >
> > > > KIP updates and addressing concerns:
> > > >
> > > >
> > > > 1) Two handle() methods are defined in ProducerExceptionHandler for
> the
> > > two
> > > > exceptions with different input parameters so that we have
> > > > handle(RecordTooLargeException e, ProducerRecord record) and
> > > > handle(UnknownTopicOrPartitionException e, ProducerRecord record)
> > > >
> > > >
> > > > 2) The ProducerExceptionHandler extends `Closable` as well.
> > > >
> > > >
> > > > 3) The KIP suggests having two more configuration parameters with
> > boolean
> > > > values:
> > > >
> > > > - `drop.invalid.large.records` with a default value of `false` for
> > > > swallowing too large records.
> > > >
> > > > - `retry.unknown.topic.partition` with a default value of `true` that
> > > > performs RETRY for `max.block.ms` ms, encountering the
> > > > UnknownTopicOrPartitionException.
> > > >
> > > >
> > > > Hope the main concerns are addressed so that we can go forward with
> > > voting.
> > > >
> > > >
> > > > Cheers,
> > > >
> > > > Alieh
> > > >
> > > > On Thu, May 9, 2024 at 11:25 PM Artem Livshits
> > > >  wrote:
> > > >
> > > >> Hi Mathias,
> > > >>
> > > >>> [AL1] While I see the point, I would think having a different
> > callback
> > > >> for every exception might not really be elegant?
> > > >>
> > > >> I'm not sure how to assess the level of elegance of the 

Re: [DISCUSS] KIP-1038: Add Custom Error Handler to Producer

[Justine Olshan]

I'm not sure I agree with the Retriable and NonRetriableResponse comment.
This doesn't limit the blast radius or enforce certain errors are used.
I think we might disagree on how controlled these interfaces can be...

Justine

On Mon, May 13, 2024 at 8:40 AM Chris Egerton 
wrote:

> Hi Alieh,
>
> Thanks for the updates! I just have a few more thoughts:
>
> - I don't think a boolean property is sufficient to dictate retries for
> unknown topic partitions, though. These errors can occur if a topic has
> just been created, which can occur if, for example, automatic topic
> creation is enabled for a multi-task connector. This is why I proposed a
> timeout instead of a boolean (and see my previous email for why reducing
> max.block.ms for a producer is not a viable alternative). If it helps, one
> way to reproduce this yourself is to add the line
> `fooProps.put(TASKS_MAX_CONFIG, "10");` to the integration test here:
>
> https://github.com/apache/kafka/blob/5439914c32fa00d634efa7219699f1bc21add839/connect/runtime/src/test/java/org/apache/kafka/connect/integration/SourceConnectorsIntegrationTest.java#L134
> and then check the logs afterward for messages like "Error while fetching
> metadata with correlation id  : {foo-topic=UNKNOWN_TOPIC_OR_PARTITION}".
>
> - I also don't think we need custom XxxResponse enums for every possible
> method; it seems like this will lead to a lot of duplication and cognitive
> overhead if we want to expand the error handler in the future. Something
> more flexible like RetriableResponse and NonRetriableResponse could
> suffice.
>
> - Finally, the KIP still doesn't state how the handler will or won't take
> precedence over existing retry properties. If I set `retries` or `
> delivery.timeout.ms` or `max.block.ms` to low values, will that cause
> retries to cease even if my custom handler would otherwise keep returning
> RETRY for an error?
>
> Cheers,
>
> Chris
>
> On Mon, May 13, 2024 at 11:02 AM Andrew Schofield <
> andrew_schofi...@live.com>
> wrote:
>
> > Hi Alieh,
> > Just a few more comments on the KIP. It is looking much less risky now
> the
> > scope
> > is tighter.
> >
> > [AJS1] It would be nice to have default implementations of the handle
> > methods
> > so an implementor would not need to implement both themselves.
> >
> > [AJS2] Producer configurations which are class names usually end in
> > “.class”.
> > I suggest “custom.exception.handler.class”.
> >
> > [AJS3] If I implemented a handler, and I set a non-default value for one
> > of the
> > new configuations, what happens? I would expect that the handler takes
> > precedence. I wasn’t quite clear what “the control will follow the
> handler
> > instructions” meant.
> >
> > [AJS4] Because you now have an enum for the
> > RecordTooLargeExceptionResponse,
> > I don’t think you need to state in the comment for
> > ProducerExceptionHandler that
> > RETRY will be interpreted as FAIL.
> >
> > Thanks,
> > Andrew
> >
> > > On 13 May 2024, at 14:53, Alieh Saeedi 
> > wrote:
> > >
> > > Hi all,
> > >
> > >
> > > Thanks for the very interesting discussion during my PTO.
> > >
> > >
> > > KIP updates and addressing concerns:
> > >
> > >
> > > 1) Two handle() methods are defined in ProducerExceptionHandler for the
> > two
> > > exceptions with different input parameters so that we have
> > > handle(RecordTooLargeException e, ProducerRecord record) and
> > > handle(UnknownTopicOrPartitionException e, ProducerRecord record)
> > >
> > >
> > > 2) The ProducerExceptionHandler extends `Closable` as well.
> > >
> > >
> > > 3) The KIP suggests having two more configuration parameters with
> boolean
> > > values:
> > >
> > > - `drop.invalid.large.records` with a default value of `false` for
> > > swallowing too large records.
> > >
> > > - `retry.unknown.topic.partition` with a default value of `true` that
> > > performs RETRY for `max.block.ms` ms, encountering the
> > > UnknownTopicOrPartitionException.
> > >
> > >
> > > Hope the main concerns are addressed so that we can go forward with
> > voting.
> > >
> > >
> > > Cheers,
> > >
> > > Alieh
> > >
> > > On Thu, May 9, 2024 at 11:25 PM Artem Livshits
> > >  wrote:
> > >
> > >> Hi Mathias,
> > >>
> > >>> [AL1] While I see the point, I would think having a different
> callback
> > >> for every exception might not really be elegant?
> > >>
> > >> I'm not sure how to assess the level of elegance of the proposal, but
> I
> > can
> > >> comment on the technical characteristics:
> > >>
> > >> 1. Having specific interfaces that codify the logic that is currently
> > >> prescribed in the comments reduce the chance of making a mistake.
> > >> Commments may get ignored, misuderstood or etc. but if the contract is
> > >> codified, the compilier will help to enforce the contract.
> > >> 2. Given that the logic is trickier than it seems (the
> record-too-large
> > is
> > >> an example that can easily confuse someone who's not intimately
> familiar
> > >> with the nuances of the batching logic), having 

Re: [DISCUSS] KIP-950: Tiered Storage Disablement

[Christo Lolov]

Heya!

re Kamal - Okay, I believe I understand what you mean and I agree. I have
made the following change

```

During tiered storage disablement, when RemoteLogManager#stopPartition() is
called:

   - Tasks scheduled for the topic-partitions in the
   RemoteStorageCopierThreadPool will be canceled.
   - If the disablement policy is retain, scheduled tasks for the
   topic-partitions in the RemoteDataExpirationThreadPool will remain
   unchanged.
   - If the disablement policy is delete, we will first advance the log
   start offset and we will let tasks scheduled for the topic-partitions in
   the RemoteDataExpirationThreadPool to successfully delete all remote
   segments before the log start offset and then unregister themselves.

```

re Luke - I checked once again. As far as I understand when a broker goes
down all replicas it hosts go to OfflineReplica state in the state machine
the controller maintains. The moment the broker comes back up again the
state machine resends StopReplica based on
```

* OfflineReplica -> ReplicaDeletionStarted
* --send StopReplicaRequest to the replica (with deletion)

```
from ReplicaStateMachine.scala. So I was wrong and you are right, we do not
appear to be sending constant requests today. I believe it is safe for us
to follow a similar approach i.e. if a replica comes online again we resend
the StopReplica.

If you don't notice any more problems I will aim to start a VOTE tomorrow
so we can get at least part of this KIP in 3.8.

Best,
Christo

On Fri, 10 May 2024 at 11:11, Luke Chen  wrote:

> Hi Christo,
>
> > 1. I am not certain I follow the question. From DISABLED you can only go
> to
> ENABLED regardless of whether your cluster is backed by Zookeeper or KRaft.
> Am I misunderstanding your point?
>
> Yes, you're right.
>
> > 4. I was thinking that if there is a mismatch we will just fail accepting
> the request for disablement. This should be the same in both Zookeeper and
> KRaft. Or am I misunderstanding your question?
>
> OK, sounds good.
>
> > 6. I think my current train of thought is that there will be unlimited
> retries until all brokers respond in a similar way to how deletion of a
> topic works today in ZK. In the meantime the state will continue to be
> DISABLING. Do you have a better suggestion?
>
> I don't think infinite retries is a good idea since if a broker is down
> forever, this request will never complete.
> You mentioned the existing topic deletion is using the similar pattern, how
> does it handle this issue?
>
> Thanks.
> Luke
>
> On Thu, May 9, 2024 at 9:21 PM Christo Lolov 
> wrote:
>
> > Heya!
> >
> > re: Luke
> >
> > 1. I am not certain I follow the question. From DISABLED you can only go
> to
> > ENABLED regardless of whether your cluster is backed by Zookeeper or
> KRaft.
> > Am I misunderstanding your point?
> >
> > 2. Apologies, this was a leftover from previous versions. I have updated
> > the Zookeeper section. The steps ought to be: controller receives change,
> > commits necessary data to Zookeeper, enqueues disablement and starts
> > sending StopReplicas request to brokers; brokers receive StopReplicas and
> > propagate them all the way to RemoteLogManager#stopPartitions which takes
> > care of the rest.
> >
> > 3. Correct, it should say DISABLED - this should now be corrected.
> >
> > 4. I was thinking that if there is a mismatch we will just fail accepting
> > the request for disablement. This should be the same in both Zookeeper
> and
> > KRaft. Or am I misunderstanding your question?
> >
> > 5. Yeah. I am now doing a second pass on all diagrams and will update
> them
> > by the end of the day!
> >
> > 6. I think my current train of thought is that there will be unlimited
> > retries until all brokers respond in a similar way to how deletion of a
> > topic works today in ZK. In the meantime the state will continue to be
> > DISABLING. Do you have a better suggestion?
> >
> > re: Kamal
> >
> > Yep, I will update all diagrams
> >
> > I am not certain I follow the reasoning for making retain and delete the
> > same. Deletion when the policy is retain happens asynchronously due to
> > expiration. I think that deletion when the policy is delete ought to (at
> > least for the initial implementation) happen synchronously. Should people
> > run into timeout problems we can always then have a follow-up KIP where
> we
> > make it asynchronous.
> >
> > Best,
> > Christo
> >
> > On Tue, 7 May 2024 at 10:04, Kamal Chandraprakash <
> > kamal.chandraprak...@gmail.com> wrote:
> >
> > > Hi Christo,
> > >
> > > Thanks for the update!
> > >
> > > For both the policies "retain" and "delete", can we maintain the same
> > > approach to delete the segments async?
> > >
> > > > If the disablement policy is set to delete, the Log start offset
> (LSO)
> > is
> > > updated to match the Local Log Start Offset and the remote log is
> deleted
> > > by calling the RemoteStorageManager#deleteLogSegmentData().
> > >
> > > In the KIP, it's mentioned that when 

Re: [DISCUSS] KIP-1038: Add Custom Error Handler to Producer

[Chris Egerton]

Hi Alieh,

Thanks for the updates! I just have a few more thoughts:

- I don't think a boolean property is sufficient to dictate retries for
unknown topic partitions, though. These errors can occur if a topic has
just been created, which can occur if, for example, automatic topic
creation is enabled for a multi-task connector. This is why I proposed a
timeout instead of a boolean (and see my previous email for why reducing
max.block.ms for a producer is not a viable alternative). If it helps, one
way to reproduce this yourself is to add the line
`fooProps.put(TASKS_MAX_CONFIG, "10");` to the integration test here:
https://github.com/apache/kafka/blob/5439914c32fa00d634efa7219699f1bc21add839/connect/runtime/src/test/java/org/apache/kafka/connect/integration/SourceConnectorsIntegrationTest.java#L134
and then check the logs afterward for messages like "Error while fetching
metadata with correlation id  : {foo-topic=UNKNOWN_TOPIC_OR_PARTITION}".

- I also don't think we need custom XxxResponse enums for every possible
method; it seems like this will lead to a lot of duplication and cognitive
overhead if we want to expand the error handler in the future. Something
more flexible like RetriableResponse and NonRetriableResponse could suffice.

- Finally, the KIP still doesn't state how the handler will or won't take
precedence over existing retry properties. If I set `retries` or `
delivery.timeout.ms` or `max.block.ms` to low values, will that cause
retries to cease even if my custom handler would otherwise keep returning
RETRY for an error?

Cheers,

Chris

On Mon, May 13, 2024 at 11:02 AM Andrew Schofield 
wrote:

> Hi Alieh,
> Just a few more comments on the KIP. It is looking much less risky now the
> scope
> is tighter.
>
> [AJS1] It would be nice to have default implementations of the handle
> methods
> so an implementor would not need to implement both themselves.
>
> [AJS2] Producer configurations which are class names usually end in
> “.class”.
> I suggest “custom.exception.handler.class”.
>
> [AJS3] If I implemented a handler, and I set a non-default value for one
> of the
> new configuations, what happens? I would expect that the handler takes
> precedence. I wasn’t quite clear what “the control will follow the handler
> instructions” meant.
>
> [AJS4] Because you now have an enum for the
> RecordTooLargeExceptionResponse,
> I don’t think you need to state in the comment for
> ProducerExceptionHandler that
> RETRY will be interpreted as FAIL.
>
> Thanks,
> Andrew
>
> > On 13 May 2024, at 14:53, Alieh Saeedi 
> wrote:
> >
> > Hi all,
> >
> >
> > Thanks for the very interesting discussion during my PTO.
> >
> >
> > KIP updates and addressing concerns:
> >
> >
> > 1) Two handle() methods are defined in ProducerExceptionHandler for the
> two
> > exceptions with different input parameters so that we have
> > handle(RecordTooLargeException e, ProducerRecord record) and
> > handle(UnknownTopicOrPartitionException e, ProducerRecord record)
> >
> >
> > 2) The ProducerExceptionHandler extends `Closable` as well.
> >
> >
> > 3) The KIP suggests having two more configuration parameters with boolean
> > values:
> >
> > - `drop.invalid.large.records` with a default value of `false` for
> > swallowing too large records.
> >
> > - `retry.unknown.topic.partition` with a default value of `true` that
> > performs RETRY for `max.block.ms` ms, encountering the
> > UnknownTopicOrPartitionException.
> >
> >
> > Hope the main concerns are addressed so that we can go forward with
> voting.
> >
> >
> > Cheers,
> >
> > Alieh
> >
> > On Thu, May 9, 2024 at 11:25 PM Artem Livshits
> >  wrote:
> >
> >> Hi Mathias,
> >>
> >>> [AL1] While I see the point, I would think having a different callback
> >> for every exception might not really be elegant?
> >>
> >> I'm not sure how to assess the level of elegance of the proposal, but I
> can
> >> comment on the technical characteristics:
> >>
> >> 1. Having specific interfaces that codify the logic that is currently
> >> prescribed in the comments reduce the chance of making a mistake.
> >> Commments may get ignored, misuderstood or etc. but if the contract is
> >> codified, the compilier will help to enforce the contract.
> >> 2. Given that the logic is trickier than it seems (the record-too-large
> is
> >> an example that can easily confuse someone who's not intimately familiar
> >> with the nuances of the batching logic), having a little more hoops to
> jump
> >> would give a greater chance that whoever tries to add a new cases pauses
> >> and thinks a bit more.
> >> 3. As Justine pointed out, having different method will be a forcing
> >> function to go through a KIP rather than smuggle new cases through
> >> implementation.
> >> 4. Sort of a consequence of the previous 3 -- all those things reduce
> the
> >> chance of someone writing the code that works with 2 errors and then
> when
> >> more errors are added in the future will suddenly incorrectly ignore new
> >> errors (the example 

[jira] [Created] (KAFKA-16738) Returns BaseRecords instead of MemoryRecords

[Zhenyu Luo (Jira)]

Zhenyu Luo created KAFKA-16738:
--

 Summary: Returns BaseRecords instead of MemoryRecords
 Key: KAFKA-16738
 URL: https://issues.apache.org/jira/browse/KAFKA-16738
 Project: Kafka
  Issue Type: Improvement
  Components: protocol
Reporter: Zhenyu Luo


We can write a record which is a subtype of 
[BaseRecords|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/record/BaseRecords.java],
 but we can not read a record which is a subtype of 
[BaseRecords|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/record/BaseRecords.java].
 If we change the return type of 
[Readable#readRecords|https://github.com/apache/kafka/blob/5439914c32fa00d634efa7219699f1bc21add839/clients/src/main/java/org/apache/kafka/common/protocol/Readable.java#L56]
 from MemoryRecords to BaseRecords, we can override the implementation of 
readRecords easily.

We known that the MemoryRecords is based on JDK's ByteBuffer. We are developing 
a netty project([kroxylicious|https://github.com/kroxylicious/kroxylicious/]) 
and we want to create a subtype of BaseRecords like MemoryRecords based on 
netty's ByteBuf.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1038: Add Custom Error Handler to Producer

[Andrew Schofield]

Hi Alieh,
Just a few more comments on the KIP. It is looking much less risky now the scope
is tighter.

[AJS1] It would be nice to have default implementations of the handle methods
so an implementor would not need to implement both themselves.

[AJS2] Producer configurations which are class names usually end in “.class”.
I suggest “custom.exception.handler.class”.

[AJS3] If I implemented a handler, and I set a non-default value for one of the
new configuations, what happens? I would expect that the handler takes
precedence. I wasn’t quite clear what “the control will follow the handler
instructions” meant.

[AJS4] Because you now have an enum for the RecordTooLargeExceptionResponse,
I don’t think you need to state in the comment for ProducerExceptionHandler that
RETRY will be interpreted as FAIL.

Thanks,
Andrew

> On 13 May 2024, at 14:53, Alieh Saeedi  wrote:
>
> Hi all,
>
>
> Thanks for the very interesting discussion during my PTO.
>
>
> KIP updates and addressing concerns:
>
>
> 1) Two handle() methods are defined in ProducerExceptionHandler for the two
> exceptions with different input parameters so that we have
> handle(RecordTooLargeException e, ProducerRecord record) and
> handle(UnknownTopicOrPartitionException e, ProducerRecord record)
>
>
> 2) The ProducerExceptionHandler extends `Closable` as well.
>
>
> 3) The KIP suggests having two more configuration parameters with boolean
> values:
>
> - `drop.invalid.large.records` with a default value of `false` for
> swallowing too large records.
>
> - `retry.unknown.topic.partition` with a default value of `true` that
> performs RETRY for `max.block.ms` ms, encountering the
> UnknownTopicOrPartitionException.
>
>
> Hope the main concerns are addressed so that we can go forward with voting.
>
>
> Cheers,
>
> Alieh
>
> On Thu, May 9, 2024 at 11:25 PM Artem Livshits
>  wrote:
>
>> Hi Mathias,
>>
>>> [AL1] While I see the point, I would think having a different callback
>> for every exception might not really be elegant?
>>
>> I'm not sure how to assess the level of elegance of the proposal, but I can
>> comment on the technical characteristics:
>>
>> 1. Having specific interfaces that codify the logic that is currently
>> prescribed in the comments reduce the chance of making a mistake.
>> Commments may get ignored, misuderstood or etc. but if the contract is
>> codified, the compilier will help to enforce the contract.
>> 2. Given that the logic is trickier than it seems (the record-too-large is
>> an example that can easily confuse someone who's not intimately familiar
>> with the nuances of the batching logic), having a little more hoops to jump
>> would give a greater chance that whoever tries to add a new cases pauses
>> and thinks a bit more.
>> 3. As Justine pointed out, having different method will be a forcing
>> function to go through a KIP rather than smuggle new cases through
>> implementation.
>> 4. Sort of a consequence of the previous 3 -- all those things reduce the
>> chance of someone writing the code that works with 2 errors and then when
>> more errors are added in the future will suddenly incorrectly ignore new
>> errors (the example I gave in the previous email).
>>
>>> [AL2 cont.] Similar to AL1, I see such a handler to some extend as
>> business logic. If a user puts a bad filter condition in their KS app, and
>> drops messages
>>
>> I agree that there is always a chance to get a bug and lose messages, but
>> there are generally separation of concerns that has different risk profile:
>> the filtering logic may be more rigorously tested and rarely changed (say
>> an application developer does it), but setting the topics to produce may be
>> done via configuration (e.g. a user of the application does it) and it's
>> generally an expectation that users would get an error when configuration
>> is incorrect.
>>
>> What could be worse is that UnknownTopicOrPartitionException can be an
>> intermittent error, i.e. with a generally correct configuration, there
>> could be metadata propagation problem on the cluster and then a random set
>> of records could get lost.
>>
>>> [AL3] Maybe I misunderstand what you are saying, but to me, checking the
>> size of the record upfront is exactly what the KIP proposes? No?
>>
>> It achieves the same result but solves it differently, my proposal:
>>
>> 1. Application checks the validity of a record (maybe via a new
>> validateRecord method) before producing it, and can just exclude it or
>> return an error to the user.
>> 2. Application produces the record -- at this point there are no records
>> that could return record too large, they were either skipped at step 1 or
>> we didn't get here because step 1 failed.
>>
>> Vs. KIP's proposal
>>
>> 1. Application produces the record.
>> 2. Application gets a callback.
>> 3. Application returns the action on how to proceed.
>>
>> The advantage of the former is the clarity of semantics -- the record is
>> invalid (property of the record, not a function of 

[KAFKA-16361] Rack aware sticky assignor minQuota violations

[Julien Opoix]

Hi there,

We're facing a critical issue with Java Kafka client version 3.5+ and sticky 
assignors (details here: https://issues.apache.org/jira/browse/KAFKA-16361).
Could we request prioritization for this issue? We're willing to assist, if 
necessary, though guidance would be appreciated given the complexity of diving 
into the code.

Best regards.

Re: KIP-1040: Improve handling of nullable values in InsertField/ExtractField transformations

[Mario Fiore Vitale]

Hi all,

Any other suggestions/objections/dubs?

On Fri, May 10, 2024 at 5:10 PM Chris Egerton 
wrote:

>  Done
>
> On Fri, May 10, 2024, 10:55 Mario Fiore Vitale  wrote:
>
> > Thanks a lot! I have just a minor comment, should we also update the
> title
> > to be more generic since now it's also related to other SMTs?
> >
> > On Fri, May 10, 2024 at 4:44 PM Chris Egerton 
> > wrote:
> >
> > > I've finished updating the KIP; @Mario, please let me know what you
> > think.
> > >
> > > On Fri, May 10, 2024 at 10:26 AM Chris Egerton 
> wrote:
> > >
> > > > I can do it :)
> > > >
> > > > On Fri, May 10, 2024 at 10:02 AM Mario Fiore Vitale <
> > mvit...@redhat.com>
> > > > wrote:
> > > >
> > > >> Yes, I agree. Unfortunately due to the issue of the creation of a
> new
> > > >> account for the WIKI, I asked Mickael Maison to create the KIP for
> me.
> > > >>
> > > >> I'll ask him to update the KIP. Do you have other alternatives?
> > > >>
> > > >> Thanks,
> > > >> Mario.
> > > >>
> > > >> On Fri, May 10, 2024 at 3:40 PM Chris Egerton
>  > >
> > > >> wrote:
> > > >>
> > > >> > Yes, I think we should just do one KIP for all the SMTs. You don't
> > > have
> > > >> to
> > > >> > implement everything all at once or by yourself, but I don't see
> why
> > > we
> > > >> > should require one or more follow-up KIPs to apply the exact same
> > > >> changes
> > > >> > to the SMTs we missed the first time.
> > > >> >
> > > >> > On Fri, May 10, 2024 at 5:20 AM Mario Fiore Vitale <
> > > mvit...@redhat.com>
> > > >> > wrote:
> > > >> >
> > > >> > > Hi Chris,
> > > >> > >
> > > >> > > Thanks for the survey. Do you think I need to update the KIP to
> > put
> > > >> all
> > > >> > of
> > > >> > > these?
> > > >> > >
> > > >> > > On Thu, May 9, 2024 at 4:14 PM Chris Egerton
> > >  > > >> >
> > > >> > > wrote:
> > > >> > >
> > > >> > > > After doing a brief survey of the SMTs that ship with Connect,
> > it
> > > >> seems
> > > >> > > > like these would also benefit:
> > > >> > > >
> > > >> > > > - HeaderFrom, which populates record headers with subfields of
> > > >> > > keys/values
> > > >> > > > [1]
> > > >> > > > - Cast, which performs type transformation on subfields of
> > > >> keys/values
> > > >> > > [2]
> > > >> > > > - SetSchemaMetadata, which (when the record key/value is a
> > struct)
> > > >> > copies
> > > >> > > > fields from the input struct to the output struct (which uses
> a
> > > >> > different
> > > >> > > > schema) [3]
> > > >> > > > - TimestampConverter, which does similar input/output field
> > > copying
> > > >> to
> > > >> > > > SetSchemaMetadata [4]
> > > >> > > > - ReplaceField, which does similar input/output field copying
> to
> > > >> > > > SetSchemaMetadata and TimestampConverter
> > > >> > > >
> > > >> > > > [1] -
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/apache/kafka/blob/f4fdaa702a2e718bdb44b9c5fec254f32a33f0d8/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/HeaderFrom.java#L143
> > > >> > > > [2] -
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/apache/kafka/blob/f4fdaa702a2e718bdb44b9c5fec254f32a33f0d8/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/Cast.java#L178
> > > >> > > > [3] -
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/apache/kafka/blob/f4fdaa702a2e718bdb44b9c5fec254f32a33f0d8/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/SetSchemaMetadata.java#L174
> > > >> > > > [4] -
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/apache/kafka/blob/2a5efe4a334611fc7c15f76b322482bad0942db0/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/TimestampConverter.java#L420
> > > >> > > > [5] -
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/apache/kafka/blob/f4fdaa702a2e718bdb44b9c5fec254f32a33f0d8/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/ReplaceField.java#L183
> > > >> > > >
> > > >> > > > On Thu, May 9, 2024 at 3:28 AM Mario Fiore Vitale <
> > > >> mvit...@redhat.com>
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > Hi Chris,
> > > >> > > > >
> > > >> > > > > > Wouldn't ValueToKey [1] be applicable as well, for
> example?
> > > >> > > > > Yes, also that one can be affected.
> > > >> > > > >
> > > >> > > > > On Wed, May 8, 2024 at 5:59 PM Chris Egerton
> > > >>  > > >> > >
> > > >> > > > > wrote:
> > > >> > > > >
> > > >> > > > > > Wait, just one more thing--are there any other SMTs that
> > could
> > > >> > > benefit
> > > >> > > > > from
> > > >> > > > > > this? Wouldn't ValueToKey [1] be applicable as well, for
> > > >> example?
> > > >> > > > > >
> > > >> > > > > > [1] -
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> 

Re: [DISCUSS] KIP-1038: Add Custom Error Handler to Producer

[Alieh Saeedi]

Hi all,


Thanks for the very interesting discussion during my PTO.


KIP updates and addressing concerns:


1) Two handle() methods are defined in ProducerExceptionHandler for the two
exceptions with different input parameters so that we have
handle(RecordTooLargeException e, ProducerRecord record) and
handle(UnknownTopicOrPartitionException e, ProducerRecord record)


2) The ProducerExceptionHandler extends `Closable` as well.


3) The KIP suggests having two more configuration parameters with boolean
values:

- `drop.invalid.large.records` with a default value of `false` for
swallowing too large records.

- `retry.unknown.topic.partition` with a default value of `true` that
performs RETRY for `max.block.ms` ms, encountering the
UnknownTopicOrPartitionException.


Hope the main concerns are addressed so that we can go forward with voting.


Cheers,

Alieh

On Thu, May 9, 2024 at 11:25 PM Artem Livshits
 wrote:

> Hi Mathias,
>
> > [AL1] While I see the point, I would think having a different callback
> for every exception might not really be elegant?
>
> I'm not sure how to assess the level of elegance of the proposal, but I can
> comment on the technical characteristics:
>
> 1. Having specific interfaces that codify the logic that is currently
> prescribed in the comments reduce the chance of making a mistake.
> Commments may get ignored, misuderstood or etc. but if the contract is
> codified, the compilier will help to enforce the contract.
> 2. Given that the logic is trickier than it seems (the record-too-large is
> an example that can easily confuse someone who's not intimately familiar
> with the nuances of the batching logic), having a little more hoops to jump
> would give a greater chance that whoever tries to add a new cases pauses
> and thinks a bit more.
> 3. As Justine pointed out, having different method will be a forcing
> function to go through a KIP rather than smuggle new cases through
> implementation.
> 4. Sort of a consequence of the previous 3 -- all those things reduce the
> chance of someone writing the code that works with 2 errors and then when
> more errors are added in the future will suddenly incorrectly ignore new
> errors (the example I gave in the previous email).
>
> > [AL2 cont.] Similar to AL1, I see such a handler to some extend as
> business logic. If a user puts a bad filter condition in their KS app, and
> drops messages
>
> I agree that there is always a chance to get a bug and lose messages, but
> there are generally separation of concerns that has different risk profile:
> the filtering logic may be more rigorously tested and rarely changed (say
> an application developer does it), but setting the topics to produce may be
> done via configuration (e.g. a user of the application does it) and it's
> generally an expectation that users would get an error when configuration
> is incorrect.
>
> What could be worse is that UnknownTopicOrPartitionException can be an
> intermittent error, i.e. with a generally correct configuration, there
> could be metadata propagation problem on the cluster and then a random set
> of records could get lost.
>
> > [AL3] Maybe I misunderstand what you are saying, but to me, checking the
> size of the record upfront is exactly what the KIP proposes? No?
>
> It achieves the same result but solves it differently, my proposal:
>
> 1. Application checks the validity of a record (maybe via a new
> validateRecord method) before producing it, and can just exclude it or
> return an error to the user.
> 2. Application produces the record -- at this point there are no records
> that could return record too large, they were either skipped at step 1 or
> we didn't get here because step 1 failed.
>
> Vs. KIP's proposal
>
> 1. Application produces the record.
> 2. Application gets a callback.
> 3. Application returns the action on how to proceed.
>
> The advantage of the former is the clarity of semantics -- the record is
> invalid (property of the record, not a function of server state or server
> configuration) and we can clearly know that it is the record that is bad
> and can never succeed.
>
> The KIP-proposed way actually has a very tricky point: it actually handles
> a subset of record-too-large exceptions.  The broker can return
> record-too-large and reject the whole batch (but we don't want to ignore
> those because then we can skip random records that just happened to be in
> the same batch), in some sense we use the same error for 2 different
> conditions and understanding that requires pretty deep understanding of
> Kafka internals.
>
> -Artem
>
>
> On Wed, May 8, 2024 at 9:47 AM Justine Olshan  >
> wrote:
>
> > My concern with respect to it being fragile: the code that ensures the
> > error type is internal to the producer. Someone may see it and say, I
> want
> > to add such and such error. This looks like internal code, so I don't
> need
> > a KIP, and then they can change it to whatever they want thinking it is
> > within the typical 

[jira] [Created] (KAFKA-16737) Clean up KafkaConsumerTest TODOs enabling tests for new consumer

[Lianet Magrans (Jira)]

Lianet Magrans created KAFKA-16737:
--

 Summary: Clean up KafkaConsumerTest TODOs enabling tests for new 
consumer
 Key: KAFKA-16737
 URL: https://issues.apache.org/jira/browse/KAFKA-16737
 Project: Kafka
  Issue Type: Task
  Components: consumer
Reporter: Lianet Magrans


KafkaConsumerTest.java contains lots of TODOs (50+) related to tests that are 
only enabled for the CLASSIC protocol and should be reviewed and enabled for 
the new CONSUMER group protocol when applicable. Some tests also have TODOs to 
enable them for the new consumer when certain features/bugs are addressed. 

The new protocol and consumer implementation have evolved a lot since those 
TODOs where added, so we should review them all, enable tests for the new 
protocol when applicable and removing the TODOs from the code. Note that there 
is another AsyncKafkaConsumerTest.java, testing logic specific to the internals 
of the new consumer, but still many tests in the KafkaConsumerTest apply to 
both the new and legacy consumer, and we should enable them for both. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-1033: Add Kafka Streams exception handler for exceptions occurring during processing

[Bill Bejeck]

Thanks for the KIP, this will be a great addition!

+1(binding)

-Bill

On Fri, May 3, 2024 at 4:48 AM Bruno Cadonna  wrote:

> Hi Damien, Sébastien, and Loïc,
>
> Thanks for the KIP!
>
> +1 (binding)
>
> Best,
> Bruno
>
>
> On 4/26/24 4:00 PM, Damien Gasparina wrote:
> > Hi all,
> >
> > We would like to start a vote for KIP-1033: Add Kafka Streams
> > exception handler for exceptions occurring during processing
> >
> > The KIP is available on
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1033%3A+Add+Kafka+Streams+exception+handler+for+exceptions+occurring+during+processing
> >
> > If you have any suggestions or feedback, feel free to participate to
> > the discussion thread:
> > https://lists.apache.org/thread/1nhhsrogmmv15o7mk9nj4kvkb5k2bx9s
> >
> > Best regards,
> > Damien Sebastien and Loic
>

Re: [VOTE] KIP-1036: Extend RecordDeserializationException exception

[Bill Bejeck]

Thanks for the KIP!

+1 (binding)

-Bill


On Tue, May 7, 2024 at 6:16 PM Sophie Blee-Goldman 
wrote:

> +1 (binding)
>
> thanks for the KIP!
>
> On Fri, May 3, 2024 at 9:13 AM Matthias J. Sax  wrote:
>
> > +1 (binding)
> >
> > On 5/3/24 8:52 AM, Federico Valeri wrote:
> > > Hi Fred, this is a useful addition.
> > >
> > > +1 non binding
> > >
> > > Thanks
> > >
> > > On Fri, May 3, 2024 at 4:11 PM Andrew Schofield
> > >  wrote:
> > >>
> > >> Hi Fred,
> > >> Thanks for the KIP. It’s turned out nice and elegant I think.
> > Definitely a worthwhile improvement.
> > >>
> > >> +1 (non-binding)
> > >>
> > >> Thanks,
> > >> Andrew
> > >>
> > >>> On 30 Apr 2024, at 14:02, Frédérik Rouleau
> >  wrote:
> > >>>
> > >>> Hi all,
> > >>>
> > >>> As there is no more activity for a while on the discuss thread, I
> > think we
> > >>> can start a vote.
> > >>> The KIP is available on
> > >>>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1036%3A+Extend+RecordDeserializationException+exception
> > >>>
> > >>>
> > >>> If you have some feedback or suggestions, please participate to the
> > >>> discussion thread:
> > >>> https://lists.apache.org/thread/or85okygtfywvnsfd37kwykkq5jq7fy5
> > >>>
> > >>> Best regards,
> > >>> Fred
> > >>
> >
>

[jira] [Created] (KAFKA-16736) Remove offsets.commit.required.acks in 4.0

[David Jacot (Jira)]

David Jacot created KAFKA-16736:
---

 Summary: Remove offsets.commit.required.acks in 4.0
 Key: KAFKA-16736
 URL: https://issues.apache.org/jira/browse/KAFKA-16736
 Project: Kafka
  Issue Type: Sub-task
Affects Versions: 4.0.0
Reporter: David Jacot






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16735) Deprecate offsets.commit.required.acks in 3.8

[David Jacot (Jira)]

David Jacot created KAFKA-16735:
---

 Summary: Deprecate offsets.commit.required.acks in 3.8
 Key: KAFKA-16735
 URL: https://issues.apache.org/jira/browse/KAFKA-16735
 Project: Kafka
  Issue Type: Sub-task
Reporter: David Jacot
Assignee: David Jacot






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16734) Add support for formatting records written to share-group state topic

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-16734:


 Summary: Add support for formatting records written to share-group 
state topic
 Key: KAFKA-16734
 URL: https://issues.apache.org/jira/browse/KAFKA-16734
 Project: Kafka
  Issue Type: Sub-task
Reporter: Andrew Schofield
Assignee: Andrew Schofield






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16730) Initial code for share-group consumer

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-16730:


 Summary: Initial code for share-group consumer
 Key: KAFKA-16730
 URL: https://issues.apache.org/jira/browse/KAFKA-16730
 Project: Kafka
  Issue Type: Sub-task
Reporter: Andrew Schofield
Assignee: Andrew Schofield






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16733) Add support for formatting new records written to offsets topic

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-16733:


 Summary: Add support for formatting new records written to offsets 
topic
 Key: KAFKA-16733
 URL: https://issues.apache.org/jira/browse/KAFKA-16733
 Project: Kafka
  Issue Type: Sub-task
Reporter: Andrew Schofield
Assignee: Andrew Schofield






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16732) Support for share-coordinator-metrics in the broker

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-16732:


 Summary: Support for share-coordinator-metrics in the broker
 Key: KAFKA-16732
 URL: https://issues.apache.org/jira/browse/KAFKA-16732
 Project: Kafka
  Issue Type: Sub-task
Reporter: Andrew Schofield
Assignee: Andrew Schofield






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16731) Support for share-group-metrics in the broker

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-16731:


 Summary: Support for share-group-metrics in the broker
 Key: KAFKA-16731
 URL: https://issues.apache.org/jira/browse/KAFKA-16731
 Project: Kafka
  Issue Type: Sub-task
Reporter: Andrew Schofield
Assignee: Andrew Schofield






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16728) Add dynamic group configuration for heartbeat interval and session timeout

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-16728:


 Summary: Add dynamic group configuration for heartbeat interval 
and session timeout
 Key: KAFKA-16728
 URL: https://issues.apache.org/jira/browse/KAFKA-16728
 Project: Kafka
  Issue Type: Sub-task
Reporter: Andrew Schofield
Assignee: Andrew Schofield






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16729) Support for read-committed isolation level

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-16729:


 Summary: Support for read-committed isolation level
 Key: KAFKA-16729
 URL: https://issues.apache.org/jira/browse/KAFKA-16729
 Project: Kafka
  Issue Type: Sub-task
Reporter: Andrew Schofield
Assignee: Andrew Schofield






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-1041: Drop `offsets.commit.required.acks` config in 4.0 (deprecate in 3.8)

[David Jacot]

+1 (binding) from me too.

The KIP passes with binding votes from Justine, Manikumar and me; and
non-binding votes from Andrew and Federico.

Thanks,
David

On Mon, May 13, 2024 at 1:52 PM Manikumar  wrote:

> +1 (binding).
>
> Thanks for the KIP.
>
> Manikumar
>
> On Wed, May 8, 2024 at 9:55 PM Justine Olshan
>  wrote:
> >
> > +1 (binding)
> >
> > Thanks,
> > Justine
> >
> > On Wed, May 8, 2024 at 8:36 AM Federico Valeri 
> wrote:
> >
> > > +1 non binding
> > >
> > > Thanks
> > >
> > > On Wed, May 8, 2024 at 5:27 PM Andrew Schofield
> > >  wrote:
> > > >
> > > > Hi,
> > > > Thanks for the KIP.
> > > >
> > > > +1 (non-binding)
> > > >
> > > > Thanks,
> > > > Andrew
> > > >
> > > > > On 8 May 2024, at 15:48, David Jacot 
> > > wrote:
> > > > >
> > > > > Hi folks,
> > > > >
> > > > > I'd like to start a voting thread for KIP-1041: Drop
> > > > > `offsets.commit.required.acks` config in 4.0 (deprecate in 3.8).
> > > > >
> > > > > KIP: https://cwiki.apache.org/confluence/x/9YobEg
> > > > >
> > > > > Best,
> > > > > David
> > > >
> > >
>

[jira] [Created] (KAFKA-16726) Add dynamic group configuration for offset reset

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-16726:


 Summary: Add dynamic group configuration for offset reset
 Key: KAFKA-16726
 URL: https://issues.apache.org/jira/browse/KAFKA-16726
 Project: Kafka
  Issue Type: Sub-task
Reporter: Andrew Schofield
Assignee: Andrew Schofield






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16725) Add broker configurations

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-16725:


 Summary: Add broker configurations
 Key: KAFKA-16725
 URL: https://issues.apache.org/jira/browse/KAFKA-16725
 Project: Kafka
  Issue Type: Sub-task
Reporter: Andrew Schofield
Assignee: Andrew Schofield






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16727) Add dynamic group configuration for record lock duration

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-16727:


 Summary: Add dynamic group configuration for record lock duration
 Key: KAFKA-16727
 URL: https://issues.apache.org/jira/browse/KAFKA-16727
 Project: Kafka
  Issue Type: Sub-task
Reporter: Andrew Schofield
Assignee: Andrew Schofield






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16724) Add new options for kafka-producer-perf-test.sh

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-16724:


 Summary: Add new options for kafka-producer-perf-test.sh
 Key: KAFKA-16724
 URL: https://issues.apache.org/jira/browse/KAFKA-16724
 Project: Kafka
  Issue Type: Sub-task
Reporter: Andrew Schofield
Assignee: Andrew Schofield






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16722) Add ConsumerGroupPartitionAssignor and ShareGroupPartitionAssignor

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-16722:


 Summary: Add ConsumerGroupPartitionAssignor and 
ShareGroupPartitionAssignor
 Key: KAFKA-16722
 URL: https://issues.apache.org/jira/browse/KAFKA-16722
 Project: Kafka
  Issue Type: Sub-task
Reporter: Andrew Schofield
Assignee: Andrew Schofield






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16723) Add kafka-console-share-consumer.sh

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-16723:


 Summary: Add kafka-console-share-consumer.sh
 Key: KAFKA-16723
 URL: https://issues.apache.org/jira/browse/KAFKA-16723
 Project: Kafka
  Issue Type: Sub-task
Reporter: Andrew Schofield
Assignee: Andrew Schofield






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16721) Add exceptions for the new error codes

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-16721:


 Summary: Add exceptions for the new error codes
 Key: KAFKA-16721
 URL: https://issues.apache.org/jira/browse/KAFKA-16721
 Project: Kafka
  Issue Type: Sub-task
Reporter: Andrew Schofield
Assignee: Andrew Schofield






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16718) Add AdminClient.deleteShareGroupOffsets

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-16718:


 Summary: Add AdminClient.deleteShareGroupOffsets
 Key: KAFKA-16718
 URL: https://issues.apache.org/jira/browse/KAFKA-16718
 Project: Kafka
  Issue Type: Sub-task
Reporter: Andrew Schofield
Assignee: Andrew Schofield






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16719) Add AdminClient.deleteShareGroups

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-16719:


 Summary: Add AdminClient.deleteShareGroups
 Key: KAFKA-16719
 URL: https://issues.apache.org/jira/browse/KAFKA-16719
 Project: Kafka
  Issue Type: Sub-task
Reporter: Andrew Schofield
Assignee: Andrew Schofield






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16717) Add AdminClient.alterShareGroupOffsets

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-16717:


 Summary: Add AdminClient.alterShareGroupOffsets
 Key: KAFKA-16717
 URL: https://issues.apache.org/jira/browse/KAFKA-16717
 Project: Kafka
  Issue Type: Sub-task
Reporter: Andrew Schofield
Assignee: Andrew Schofield






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16720) Add AdminClient.listShareGroupOffsets

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-16720:


 Summary: Add AdminClient.listShareGroupOffsets
 Key: KAFKA-16720
 URL: https://issues.apache.org/jira/browse/KAFKA-16720
 Project: Kafka
  Issue Type: Sub-task
Reporter: Andrew Schofield
Assignee: Andrew Schofield






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16716) Add AdminClient.describeShareGroups and AdminClient.listShareGroups

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-16716:


 Summary: Add AdminClient.describeShareGroups and 
AdminClient.listShareGroups
 Key: KAFKA-16716
 URL: https://issues.apache.org/jira/browse/KAFKA-16716
 Project: Kafka
  Issue Type: Sub-task
Reporter: Andrew Schofield
Assignee: Andrew Schofield






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16715) Create KafkaShareConsumer interface

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-16715:


 Summary: Create KafkaShareConsumer interface
 Key: KAFKA-16715
 URL: https://issues.apache.org/jira/browse/KAFKA-16715
 Project: Kafka
  Issue Type: Sub-task
Reporter: Andrew Schofield
Assignee: Andrew Schofield






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16714) kafka-share-groups.sh supporting list and describe

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-16714:


 Summary: kafka-share-groups.sh supporting list and describe
 Key: KAFKA-16714
 URL: https://issues.apache.org/jira/browse/KAFKA-16714
 Project: Kafka
  Issue Type: Sub-task
Reporter: Andrew Schofield
Assignee: Andrew Schofield






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16713) Add new RPC definitions

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-16713:


 Summary: Add new RPC definitions
 Key: KAFKA-16713
 URL: https://issues.apache.org/jira/browse/KAFKA-16713
 Project: Kafka
  Issue Type: Sub-task
Reporter: Andrew Schofield
Assignee: Andrew Schofield






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-1041: Drop `offsets.commit.required.acks` config in 4.0 (deprecate in 3.8)

[Manikumar]

+1 (binding).

Thanks for the KIP.

Manikumar

On Wed, May 8, 2024 at 9:55 PM Justine Olshan
 wrote:
>
> +1 (binding)
>
> Thanks,
> Justine
>
> On Wed, May 8, 2024 at 8:36 AM Federico Valeri  wrote:
>
> > +1 non binding
> >
> > Thanks
> >
> > On Wed, May 8, 2024 at 5:27 PM Andrew Schofield
> >  wrote:
> > >
> > > Hi,
> > > Thanks for the KIP.
> > >
> > > +1 (non-binding)
> > >
> > > Thanks,
> > > Andrew
> > >
> > > > On 8 May 2024, at 15:48, David Jacot 
> > wrote:
> > > >
> > > > Hi folks,
> > > >
> > > > I'd like to start a voting thread for KIP-1041: Drop
> > > > `offsets.commit.required.acks` config in 4.0 (deprecate in 3.8).
> > > >
> > > > KIP: https://cwiki.apache.org/confluence/x/9YobEg
> > > >
> > > > Best,
> > > > David
> > >
> >

[jira] [Created] (KAFKA-16712) Race while setting RemoteLogMetadataTopicPartitioner in TopicBasedRemoteLogMetadataManagerMultipleSubscriptionsTest

[Gaurav Narula (Jira)]

Gaurav Narula created KAFKA-16712:
-

 Summary: Race while setting RemoteLogMetadataTopicPartitioner in 
TopicBasedRemoteLogMetadataManagerMultipleSubscriptionsTest
 Key: KAFKA-16712
 URL: https://issues.apache.org/jira/browse/KAFKA-16712
 Project: Kafka
  Issue Type: Test
Reporter: Gaurav Narula


>From the discussion in 
>[PR-15885|https://github.com/apache/kafka/pull/15885#discussion_r1598102664], 
>{{TopicBasedRemoteLogMetadataManagerMultipleSubscriptionsTest}} is prone to a 
>race while setting {{RemoteLogMetadataTopicPartitioner}} instance at 
>https://github.com/apache/kafka/blob/8a9dd2beda90f04180e71b406657d9da388d359e/storage/src/test/java/org/apache/kafka/server/log/remote/metadata/storage/TopicBasedRemoteLogMetadataManagerMultipleSubscriptionsTest.java#L125



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16711) highestOffsetInRemoteStorage is not updated after logDir altering within broker

[Luke Chen (Jira)]

Luke Chen created KAFKA-16711:
-

 Summary: highestOffsetInRemoteStorage is not updated after logDir 
altering within broker
 Key: KAFKA-16711
 URL: https://issues.apache.org/jira/browse/KAFKA-16711
 Project: Kafka
  Issue Type: Sub-task
Affects Versions: 3.7.0
Reporter: Luke Chen
Assignee: Luke Chen


We use topicIdPartition as the key for each RLM task. It will cause 
highestOffsetInRemoteStorage in log not updated after logDir alter completion. 
The reproducing flow is like this:

 
 # tp-0 locating in dirA is the leader of the partition
 # tp-0 is altering logDir to dirB
 # tp-0 is copying segments to remote storage (note: the log in dirA)
 # The logDir altering for tp-0 is completed
 # remoteLogManager#onLeadershipChange is invoked, copiedOffsetOption is reset 
to Optional.empty()
 # The copying segments to remote storage in step 3 for tp-0 is completed, 
updating copiedOffsetOption to new offset, as well as the 
log#highestOffsetInRemoteStorage. (note: the log in dirA)
 # In the next run of RLMTask, the log will be the one in target dir (dirB), 
and the log#highestOffsetInRemoteStorage (dirB) will be the default value (-1), 
which will block the log cleanup operation.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16587) Store subscription model for consumer group in group state

[David Jacot (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-16587.
-
Fix Version/s: 3.8.0
 Reviewer: David Jacot
   Resolution: Fixed

> Store subscription model for consumer group in group state
> --
>
> Key: KAFKA-16587
> URL: https://issues.apache.org/jira/browse/KAFKA-16587
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Ritika Reddy
>Assignee: Ritika Reddy
>Priority: Major
> Fix For: 3.8.0
>
>
> Currently we iterate through all the subscribed topics for each member in the 
> consumer group to determine whether all the members are subscribed to the 
> same set of topics aka it has a homogeneous subscription model.
> Instead of iterating and comparing the topicIds on every rebalance, we want 
> to maintain this information in the group state



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1028: Docker Official Image for Apache Kafka

[Prabha Manepalli]

Hi Chris,

Sharing the requested links explaining why Docker Official images are
considered more secure -
https://www.docker.com/blog/enhancing-security-and-transparency-with-docker-official-images/
and
https://github.com/docker-library/faq#why-does-my-security-scanner-show-that-an-image-has-cves

I hope these links help you understand why we need Docker Official images
for organisations with stringent security compliance requirements for their
Kafka workloads.

Thank you.



On Sun, May 12, 2024 at 3:33 PM Vedarth Sharma 
wrote:

> Hey Chris!
>
> Functionality wise, we don't intend to have any differences between OSS
> Docker Image and Docker Official Image.
> The Docker Official Image will be the recommended one.
> Since the Docker Official Image might be delayed due to review done by
> Docker, images on apache/kafka (OSS Docker Image) can be used by users.
>
> 1) I read https://docs.docker.com/trusted-content/official-images/ and
> > didn't find anything on that page or immediately around it that explains
> > what compliance requirements might be satisfied by a DOI that couldn't be
> > satisfied by the existing apache/kafka image. Can you elaborate? Feel
> free
> > to provide another link, but please also quote the relevant sections from
> > it (as StackOverflow likes to say, links can grow stale over time).
>
>
>- If a user's selection is confined solely to Docker Official Images,
>this Docker Image will ensure their access to Kafka.
>- Details on specific advantages of Docker Official Images can be found
>at
>
> https://github.com/docker-library/official-images?tab=readme-ov-file#what-are-official-images
>.
>- The Docker Official Image will be actively rebuilt for updates and
>security fixes.
>- It's true we can provide exactly the same benefits in the OSS Docker
>Image as well. But it won't have the Docker Official Image badge and it
>will add additional overhead for Apache Kafka community.
>- The fact that it will have Docker Official Image badge will set it
>apart from the OSS Docker Image.
>- Also the ability to do just docker pull kafka to get the kafka docker
>image will only be possible with Docker Official Image.
>
>
> 2) It would be great to see a brief summary of the differences in these
> > images included in the KIP, in order to try to gauge how this would look
> to
> > our users.
>
>
>- Functionally, both Docker images will remain identical.
>- The variance lies primarily in the methodologies of building and
>validation, as outlined in the updated KIP.
>
>
> 3) What I suggested last time was not a separate apache/apache-docker
> > repository, but a repository controlled entirely by Docker. The DOI docs
> > [1] state that "While it's preferable to have upstream software authors
> > maintaining their Docker Official Images, this isn't a strict
> requirement",
> > which I take to mean that it's not required for an Apache Kafka DOI to
> live
> > under the apache organization on GitHub. It also seems like there's
> > precedent for this: images for MySQL [2] and PHP [3] already exist under
> > the control of Docker. The reason I think this is worth considering is
> that
> > Docker can arbitrarily change the eligibility requirements for their
> > official images at any time, and it doesn't seem like there's a clear
> > process in the KIP for judging how we should respond to these changes (in
> > fact, it seems like the idea in the KIP is that we should make any change
> > required with no further vetting beyond possibly a pull request on
> > apache/kafka, which would require approval by a committer). By hosting
> the
> > DOI definitions ourselves (either in apache/kafka, or in a theoretical
> > apache/docker-kafka repository), we take responsibility for the image,
> even
> > if the owner on Docker Hub is Docker, not Apache. If the code lives
> > elsewhere, then (as long as basic trademark and possibly security
> > guidelines are respected) Apache doesn't have to concern itself at all
> with
> > the image and the maintainers are free to make whatever changes they want
> > to it in order to meet Docker's requirements.
>
>
>- By incorporating the source code of the Docker Official Image into our
>AK ecosystem, we gain control over its functionality, ensuring alignment
>with the OSS Docker image. This ensures a seamless experience for users
> who
>may need to transition between these images.
>- Maintaining both images within the same community facilitates ease of
>management and fosters a singular source of truth.
>- While Apache may not retain ownership of the hosted Docker Official
>Image, we are, in essence, providing Docker with a foundation that
> aligns
>with their established guidelines as well as remains consistent with OSS
>Docker Image apache/kafka.
>- Any future alterations to the functionality can be seamlessly
>propagated across both the OSS and Official Docker Images.
>

Re: [DISCUSS] KIP-1033: Add Kafka Streams exception handler for exceptions occuring during processing

[Damien Gasparina]

Thanks for the feedback, I think we should keep two separate callbacks
for serialization and error handlers. It makes sense for type safety
(ProducerRecord vs POJO) and also for backward
compatibility. On top of that, all metadata provided in the #handle
method would need to be held in memory until the producer invokes its
callback, in the future, having two callbacks might avoid confusion as
some metadata might be provided to #handle and not
#handleSerializationException. I do think that one method would be
cleaner but for backward compatibility, type safety and memory
reasons, we should keep two separate callbacks.

As you suggested Sophie, I updated the KIP to introduce a
SerializationExceptionOrigin enum and added the "origin" parameter to
the #handleSerializationException method.

On Sat, 11 May 2024 at 07:30, Sophie Blee-Goldman  wrote:
>
> Whoops, just noticed there is already a voting thread for this. Hard to
> keep track with all the KIPs going on right now!
>
> In that case I'll just wait for the SerializationExceptionOrigin thing to
> be added and then I'll vote. Should definitely be able to make 3.8 in this
> case :D
>
> On Fri, May 10, 2024 at 10:10 PM Sophie Blee-Goldman 
> wrote:
>
> > Sounds like we're more or less in agreement here. I think the KIP just
> > needs one small update still, which is the SerializationExceptionOrigin
> > enum.
> >
> > As discussed there are a few options for this and we're all happy to defer
> > to the preference of the KIP authors, but if we keep the KIP as-is with the
> > two separate methods in the ProductionExceptionHandler, then imo it makes
> > the most sense to add the SerializationExceptionOrigin enum to the
> > ProductionExceptionHandler interface and then add an "origin" parameter to
> > the new  #handleSerializationException method. However you decide to do it,
> > I'm personally happy to vote on this KIP once the KIP is updated.
> >
> >  Just FYI the 3.8 KIP freeze is this upcoming Wednesday, so if you guys
> > would like to target 3.8 for this feature, just make sure to update the KIP
> > and kick off a [VOTE] thread by EOD Monday so that you can close the vote
> > by EOD Wednesday (since it has to be open for 72 hours).
> >
> > Thanks again for this sorely needed feature!
> >
> > On Fri, May 10, 2024 at 10:06 AM Bill Bejeck  wrote:
> >
> >> Great KIP discussion so far by everyone.
> >> At this point, I'm in agreement with the direction and current state of
> >> the
> >> KIP.
> >>
> >> As for having two separate callbacks for the ProductionExceptionHandler,
> >> I'm somewhat split in that I agree with points raised by Sophie and
> >> Matthias with my final
> >> position being to maintain both callbacks.  IMHO, while there are several
> >> things that could go wrong with producing a message, it seems that
> >> serialization exceptions would be the most common, although I don't have
> >> any data to back that opinion up.  But having said that, should the KIP
> >> authors decide otherwise, I would be fine with that approach as well.
> >>
> >> I'm at the point where I'm comfortable voting for this KIP.
> >>
> >> Thanks,
> >> Bill
> >>
> >> On Thu, May 9, 2024 at 4:28 PM Sophie Blee-Goldman  >> >
> >> wrote:
> >>
> >> > The type safety issue is definitely not solved by having two separate
> >> > callbacks. I just think it gets a bit worse by mashing them into one
> >> > method. At least in the plain #handle method you can be sure that the
> >> type
> >> > is ProducerRecord and in #handleSerialization the type
> >> is
> >> > some POJO.
> >> >
> >> > And in theory you can just embed the mapping of sink topics to
> >> type/Serde
> >> > based on your topology. Or let's say your output record keys & values
> >> are
> >> > all Strings, and you want to print the String representation in your
> >> > handler, rather than the bytes.
> >> > Having a separate callback means knowing you can simply print the
> >> > ProducerRecord's key/value in the #handleSerialization method, and will
> >> > have to use a StringDeserializer to convert the key/value to its String
> >> > form to print it in the #handle method.
> >> >
> >> > Again, I just feel this will be more straightforward and easy for users
> >> to
> >> > use correctly, but am satisfied either way. I'll shut up now and wait
> >> for
> >> > the KIP authors to make a call on this one way or another, and then I'm
> >> > happy to cast my vote
> >> >
> >> > On Thu, May 9, 2024 at 10:15 AM Matthias J. Sax 
> >> wrote:
> >> >
> >> > > Thanks Sophie! Makes it much clearer where you are coming from.
> >> > >
> >> > > About the Type unsafety: isn't this also an issue for the
> >> > > `handleSerialziationException` case, because the handler is used for
> >> all
> >> > > sink topics, and thus key/value types are not really know w/o taking
> >> the
> >> > > sink topic into account? -- So I am not sure if having two handler
> >> > > methods really helps much with regard to type safety?
> >> > >
> >> > > Just want to make this 

[jira] [Created] (KAFKA-16710) Continuously `makeFollower` may cause the replica fetcher thread to encounter an offset mismatch exception when `processPartitionData`

[hudeqi (Jira)]

hudeqi created KAFKA-16710:
--

 Summary: Continuously `makeFollower` may cause the replica fetcher 
thread to encounter an offset mismatch exception when `processPartitionData`
 Key: KAFKA-16710
 URL: https://issues.apache.org/jira/browse/KAFKA-16710
 Project: Kafka
  Issue Type: Bug
  Components: core, replication
Affects Versions: 2.8.1
Reporter: hudeqi
Assignee: hudeqi






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16663) CoordinatorRuntime write timer tasks should be cancelled once HWM advances

[David Jacot (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-16663.
-
Fix Version/s: 3.8.0
 Reviewer: David Jacot
   Resolution: Fixed

> CoordinatorRuntime write timer tasks should be cancelled once HWM advances
> --
>
> Key: KAFKA-16663
> URL: https://issues.apache.org/jira/browse/KAFKA-16663
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jeff Kim
>Assignee: Jeff Kim
>Priority: Major
> Fix For: 3.8.0
>
>
> Otherwise, we pile up the number of timer tasks which are no-ops if 
> replication was successful. They stay in memory for 15 seconds and as the 
> rate of write increases, this may heavily impact memory usage.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16677) Replace ClusterType#ALL and ClusterType#DEFAULT by Array

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16677.

Fix Version/s: 3.8.0
   Resolution: Fixed

> Replace ClusterType#ALL and ClusterType#DEFAULT by Array
> 
>
> Key: KAFKA-16677
> URL: https://issues.apache.org/jira/browse/KAFKA-16677
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Chia-Ping Tsai
>Assignee: PoAn Yang
>Priority: Minor
> Fix For: 3.8.0
>
>
> Both ClusterType#ALL and ClusterType#DEFAULT are a kind of "tag" instead of 
> true "type". It seems to me they can be removed by using Array. For example:
> ClusterType#ALL -> {Type.ZK, Type.KRAFT, Type.CO_KRAFT}
> ClusterType#DEFAULT -> {}
> There are two benefits
> 1. That is more readable for "ALL type". 
> 2. We don't throw the awkward "exception" when seeing "DEFAULT".



--
This message was sent by Atlassian Jira
(v8.20.10#820010)