Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2673

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2672

[Apache Jenkins Server]

See 

[DISCUSS] Uninitialized metadata cache during broker startup

[Chris Wildman]

Hi All,

I recently discovered the race condition where Kafka clients may request
metadata from brokers who have not yet received a snapshot of the
cluster metadata. At scale with ZK managed clusters we see this happen
frequently when brokers are restarted, possibly due to the admin client's
preference for the least loaded node. This results in unexpected behavior
for some metadata requests on the Admin api. For example describeTopics
will fail with an UnknownTopicOrPartition exception for topics which do
exist in the cluster.

I also learned that consumers and producers ignore uninitialized metadata
by detecting if the set of brokers in a metadata response is empty:
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java#L1187-L1191

The only Admin API that has this protection seems to be listConsumerGroups
which throws a StaleMetadataException for the empty brokers condition:
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java#L3369-L3371

I'm interested in making it clearer to users of the Admin api when they
should retry a metadata request because it was handled by a broker with
uninitialized metadata. I think the proper fix here would be to have
brokers respond with a UninitializedMetadataException when handling
metadata requests if they haven't yet received a metadata snapshot. That is
a big change that would need to be handled appropriately in all clients. A
more immediate fix would be to change the KafkaAdminClient to *always*
detect the empty brokers condition when getting a MetadataResponse and
throw a StaleMetadataException or some other RetriableException.

Some questions I have:

1. Does the likelihood of a broker responding with stale metadata decrease
significantly or entirely when using KRAFT? I can understand not fixing
this if that is the case. I tried but could not reproduce this behavior
using kafka integration tests for both ZK and KRAFT.

2. Do we want to go for the proper fix or the more immediate one?

3. Would the immediate fix mentioned above, patching the KafkaAdminClient,
require a KIP or should I just PR that?

4. Is StaleMetadataException the exception we want to use for the
unitialized metadata case? From the docs for both StaleMetadataException
and InvalidMetadataException it seems more geared toward old data, not
uninitialized data.

Thanks for your time and I hope this is an appropriate discussion!

Chris Wildman

Re: [DISCUSS] KIP-853: KRaft Controller Membership Changes

[Jun Rao]

Hi, Jose,

Thanks for the reply.

15. VotersRecord: Ok. I got the need for including the listener name in the
endpoint. Currently, controller.quorum.voters can only specify one endpoint
per voter. So, we can only support one listener. It's fine if we want to
extend that in this KIP.
15.1 "In this configuration, the local replica needs to use
CONTROLLER_PLAINTEXT because that is what is supported by all of the
voters."
   Hmm, my understanding is that what listener to use can be made pairwise
between the sender and the receiver. There is no need to pick the same
listener across all voters, right?
15.2  When bootstrapping with just one controller, we could register a
single voter with multiple listeners from controller.listener.names and
listeners. However, when bootstrapping with multiple controllers,
kafka-storage only has the option to specify one listener
(--controller-quorum-voters -@:) per
voter. Should we make them consistent?

18.3 : "It will not use the metadata layer (QuorumController) to update and
persist the finalized kraft.version."
So, we depend on KRaftVersionRecord to propagate finalized kraft.version to
brokers/controllers? It would be useful to document that.

27. "My plan is to rely on KIP-996: Pre-vote:".
Hmm, Diego's paper says "Unfortunately, the Pre-Vote phase does not solve
the problem of disruptive servers". Is that a real issue?

30. So, VotersRecords and LeaderChangeRecord are controlled by
both kraft.version and metadata.version? How do we resolve conflicts? For
example, if a user downgrades kraft.version, do we automatically downgrade
metadata.version to the matching level?

31. kafka-features now has --release-software. Could we
change kafka-storage to be consistent? Currently, kafka-storage takes
--release-version.

32. Now that we have more than one feature, should we extend kafka-storage
to also support --feature FEATURE like kafka-features?

Jun





On Fri, Feb 23, 2024 at 12:00 PM José Armando García Sancio
 wrote:

> Jun, I updated one of the rejected ideas to better explain why KRaft
> can't rely on information stored by the metadata layer.
>
> Thanks,
> --
> -José
>

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

[José Armando García Sancio]

Hi Calvin

On Fri, Feb 23, 2024 at 9:23 AM Calvin Liu  wrote:
> As we agreed to implement the pagination for the new API
> DescribeTopicPartitions, the client side must also add a proper interface
> to handle the pagination.
> The current KafkaAdminClient.describeTopics returns
> the DescribeTopicsResult which is the future for querying all the topics.
> It is awkward to fit the pagination into it because

I suggest taking a look at Java's Flow API:
https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/util/concurrent/Flow.html
It was design for this specific use case and many libraries integrate with it.

If the Kafka client cannot be upgraded to support the Java 9 which
introduced that API, you can copy the same interface and semantics.
This would allow users to easily integrate with reactive libraries
since they all integrate with Java Flow.

Thanks,
-- 
-José

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2671

[Apache Jenkins Server]

See 

Re: [PR] 3.7: Add blog post for Kafka 3.7 [kafka-site]

[via GitHub]

pprovenzano commented on code in PR #578:
URL: https://github.com/apache/kafka-site/pull/578#discussion_r1501132621


##
blog.html:
##
@@ -22,6 +22,125 @@
 
 
 Blog
+
+
+
+Apache 
Kafka 3.7.0 Release Announcement
+
+February 2024 - Stanislav Kozlovski (https://twitter.com/BdKozlovski";>@BdKozlovski)
+We are proud to announce the release of Apache Kafka 3.7.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.7.0/RELEASE_NOTES.html";>release 
notes.
+See the https://kafka.apache.org/36/documentation.html#upgrade_3_7_0";>Upgrading 
to 3.7.0 from any version 0.8.x through 3.6.x section in the documentation 
for the list of notable changes and detailed upgrade steps.
+
+In the last release, 3.6,
+https://kafka.apache.org/documentation/#kraft_zk_migration";>the ability 
to migrate Kafka clusters from a ZooKeeper metadata system
+to a KRaft metadata system was ready for usage in 
production environments with one caveat -- JBOD was not yet available for KRaft 
clusters.
+In this release, we are shipping an early access release 
of JBOD in KRaft. (See https://cwiki.apache.org/confluence/display/KAFKA/KIP-858%3A+Handle+JBOD+broker+disk+failure+in+KRaft";>KIP-858
 for details)
+
+
+Additionally, client APIs released prior to Apache Kafka 
2.1 are now marked deprecated in 3.7 and will be removed in Apache Kafka 4.0. 
See https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0";>KIP-896
 for details and RPC versions that are now deprecated.
+
+
+Java 11 support for the Kafka broker is also marked 
deprecated in 3.7, and is planned to be removed in Kafka 4.0. See https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789510";>KIP-1013
 for more details
+
+
+Note: ZooKeeper is marked as deprecated since the 3.5.0 
release. ZooKeeper is planned to be removed in Apache Kafka 4.0. For more 
information, please see the documentation for https://kafka.apache.org/documentation/#zk_depr";>ZooKeeper 
Deprecation.
+
+
+Kafka Broker, Controller, Producer, Consumer and Admin 
Client
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-858%3A+Handle+JBOD+broker+disk+failure+in+KRaft";>(Early
 Access) KIP-858 Handle JBOD broker disk failure in KRaft:

Review Comment:
   LGTM



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] KIP-853: KRaft Controller Membership Changes

[José Armando García Sancio]

Jun, I updated one of the rejected ideas to better explain why KRaft
can't rely on information stored by the metadata layer.

Thanks,
-- 
-José

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

[Andrew Schofield]

This is an interesting problem. While it would be possible to use a Consumer,
it can’t handle exceptions really. A java.util.Stream has a similar problem.

I wonder whether an interface which looks like 
java.util.concurrent.Flow.Subscriber
would be good. Something like:

public interface AdminResultsSubscriber {
  void onComplete();
  void onError(Exception e):
  void onNext(T result);
}

And then:
Admin.describeTopics(TopicCollection, DescribeTopicOptions, 
AdminResultsSubscriber)

Thanks,
Andrew


> On 23 Feb 2024, at 17:56, David Arthur  
> wrote:
> 
> Thanks for raising this here, Calvin. Since this is the first "streaming
> results" type API in KafkaAdminClient (as far as I know), we're treading
> new ground here.
> 
> As you mentioned, we can either accept a consumer or return some iterable
> result. Returning a java.util.Stream is also an option, and a bit more
> modern/convenient than java.util.Iterator. Personally, I like the consumer
> approach, but I'm interested in hearing other's opinions.
> 
> This actually brings up another question: Do we think it's safe to assume
> that one topic's description can fit into memory? The RPC supports paging
> across partitions within a single topic, so maybe the admin API should as
> well?
> 
> -David
> 
> On Fri, Feb 23, 2024 at 12:22 PM Calvin Liu  wrote:
> 
>> Hey,
>> As we agreed to implement the pagination for the new API
>> DescribeTopicPartitions, the client side must also add a proper interface
>> to handle the pagination.
>> The current KafkaAdminClient.describeTopics returns
>> the DescribeTopicsResult which is the future for querying all the topics.
>> It is awkward to fit the pagination into it because
>> 
>>   1. Each future corresponds to a topic. We also want to have the
>>   pagination on huge topics for their partitions.
>>   2. To avoid OOM, we should only fetch the new topics when we need them
>>   and release the used topics. Especially the main use case of looping the
>>   topic list is when the client prints all the topics.
>> 
>> So, to better serve the pagination, @David Arthur
>>  suggested to add a new interface in the Admin
>> client between the following 2.
>> 
>> describeTopics(TopicCollection topics, DescribeTopicsOptions options, 
>> Consumer);
>> 
>> Iterator describeTopics(TopicCollection topics, 
>> DescribeTopicsOptions options);
>> 
>> David and I would prefer the first Consumer version which works better as a 
>> stream purposes.
>> 
>> 
>> On Wed, Oct 11, 2023 at 4:28 PM Calvin Liu  wrote:
>> 
>>> Hi David,
>>> Thanks for the comment.
>>> Yes, we can separate the ELR enablement from the metadata version. It is
>>> also helpful to avoid blocking the following MV releases if the user is not
>>> ready for ELR.
>>> One thing to correct is that, the Unclean recovery is controlled
>>> by unclean.recovery.manager.enabled, a separate config
>>> from unclean.recovery.strategy. It determines whether unclean recovery will
>>> be used in an unclean leader election.
>>> Thanks
>>> 
>>> On Wed, Oct 11, 2023 at 4:11 PM David Arthur  wrote:
>>> 
 One thing we should consider is a static config to totally enable/disable
 the ELR feature. If I understand the KIP correctly, we can effectively
 disable the unclean recovery by setting the recovery strategy config to
 "none".
 
 This would make development and rollout of this feature a bit smoother.
 Consider the case that we find bugs in ELR after a cluster has updated to
 its MetadataVersion. It's simpler to disable the feature through config
 rather than going through a MetadataVersion downgrade (once that's
 supported).
 
 Does that make sense?
 
 -David
 
 On Wed, Oct 11, 2023 at 1:40 PM Calvin Liu 
 wrote:
 
> Hi Jun
> -Good catch, yes, we don't need the -1 in the DescribeTopicRequest.
> -No new value is added. The LeaderRecoveryState will still be set to 1
 if
> we have an unclean leader election. The unclean leader election
 includes
> the old random way and the unclean recovery. During the unclean
 recovery,
> the LeaderRecoveryState will not change until the controller decides to
> update the records with the new leader.
> Thanks
> 
> On Wed, Oct 11, 2023 at 9:02 AM Jun Rao 
 wrote:
> 
>> Hi, Calvin,
>> 
>> Another thing. Currently, when there is an unclean leader election,
 we
> set
>> the LeaderRecoveryState in PartitionRecord and PartitionChangeRecord
 to
> 1.
>> With the KIP, will there be new values for LeaderRecoveryState? If
 not,
>> when will LeaderRecoveryState be set to 1?
>> 
>> Thanks,
>> 
>> Jun
>> 
>> On Tue, Oct 10, 2023 at 4:24 PM Jun Rao  wrote:
>> 
>>> Hi, Calvin,
>>> 
>>> One more comment.
>>> 
>>> "The first partition to fetch details for. -1 means to fetch all
>>> partitions." It seems that FirstPartitionId of 0 naturally means
> fetc

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

[David Arthur]

Thanks for raising this here, Calvin. Since this is the first "streaming
results" type API in KafkaAdminClient (as far as I know), we're treading
new ground here.

As you mentioned, we can either accept a consumer or return some iterable
result. Returning a java.util.Stream is also an option, and a bit more
modern/convenient than java.util.Iterator. Personally, I like the consumer
approach, but I'm interested in hearing other's opinions.

This actually brings up another question: Do we think it's safe to assume
that one topic's description can fit into memory? The RPC supports paging
across partitions within a single topic, so maybe the admin API should as
well?

-David

On Fri, Feb 23, 2024 at 12:22 PM Calvin Liu  wrote:

> Hey,
> As we agreed to implement the pagination for the new API
> DescribeTopicPartitions, the client side must also add a proper interface
> to handle the pagination.
> The current KafkaAdminClient.describeTopics returns
> the DescribeTopicsResult which is the future for querying all the topics.
> It is awkward to fit the pagination into it because
>
>1. Each future corresponds to a topic. We also want to have the
>pagination on huge topics for their partitions.
>2. To avoid OOM, we should only fetch the new topics when we need them
>and release the used topics. Especially the main use case of looping the
>topic list is when the client prints all the topics.
>
> So, to better serve the pagination, @David Arthur
>  suggested to add a new interface in the Admin
> client between the following 2.
>
> describeTopics(TopicCollection topics, DescribeTopicsOptions options, 
> Consumer);
>
> Iterator describeTopics(TopicCollection topics, 
> DescribeTopicsOptions options);
>
> David and I would prefer the first Consumer version which works better as a 
> stream purposes.
>
>
> On Wed, Oct 11, 2023 at 4:28 PM Calvin Liu  wrote:
>
>> Hi David,
>> Thanks for the comment.
>> Yes, we can separate the ELR enablement from the metadata version. It is
>> also helpful to avoid blocking the following MV releases if the user is not
>> ready for ELR.
>> One thing to correct is that, the Unclean recovery is controlled
>> by unclean.recovery.manager.enabled, a separate config
>> from unclean.recovery.strategy. It determines whether unclean recovery will
>> be used in an unclean leader election.
>> Thanks
>>
>> On Wed, Oct 11, 2023 at 4:11 PM David Arthur  wrote:
>>
>>> One thing we should consider is a static config to totally enable/disable
>>> the ELR feature. If I understand the KIP correctly, we can effectively
>>> disable the unclean recovery by setting the recovery strategy config to
>>> "none".
>>>
>>> This would make development and rollout of this feature a bit smoother.
>>> Consider the case that we find bugs in ELR after a cluster has updated to
>>> its MetadataVersion. It's simpler to disable the feature through config
>>> rather than going through a MetadataVersion downgrade (once that's
>>> supported).
>>>
>>> Does that make sense?
>>>
>>> -David
>>>
>>> On Wed, Oct 11, 2023 at 1:40 PM Calvin Liu 
>>> wrote:
>>>
>>> > Hi Jun
>>> > -Good catch, yes, we don't need the -1 in the DescribeTopicRequest.
>>> > -No new value is added. The LeaderRecoveryState will still be set to 1
>>> if
>>> > we have an unclean leader election. The unclean leader election
>>> includes
>>> > the old random way and the unclean recovery. During the unclean
>>> recovery,
>>> > the LeaderRecoveryState will not change until the controller decides to
>>> > update the records with the new leader.
>>> > Thanks
>>> >
>>> > On Wed, Oct 11, 2023 at 9:02 AM Jun Rao 
>>> wrote:
>>> >
>>> > > Hi, Calvin,
>>> > >
>>> > > Another thing. Currently, when there is an unclean leader election,
>>> we
>>> > set
>>> > > the LeaderRecoveryState in PartitionRecord and PartitionChangeRecord
>>> to
>>> > 1.
>>> > > With the KIP, will there be new values for LeaderRecoveryState? If
>>> not,
>>> > > when will LeaderRecoveryState be set to 1?
>>> > >
>>> > > Thanks,
>>> > >
>>> > > Jun
>>> > >
>>> > > On Tue, Oct 10, 2023 at 4:24 PM Jun Rao  wrote:
>>> > >
>>> > > > Hi, Calvin,
>>> > > >
>>> > > > One more comment.
>>> > > >
>>> > > > "The first partition to fetch details for. -1 means to fetch all
>>> > > > partitions." It seems that FirstPartitionId of 0 naturally means
>>> > fetching
>>> > > > all partitions?
>>> > > >
>>> > > > Thanks,
>>> > > >
>>> > > > Jun
>>> > > >
>>> > > > On Tue, Oct 10, 2023 at 12:40 PM Calvin Liu
>>> >> > >
>>> > > > wrote:
>>> > > >
>>> > > >> Hi Jun,
>>> > > >> Yeah, with the current Metadata request handling, we only return
>>> > errors
>>> > > on
>>> > > >> the Topic level, like topic not found. It seems that querying a
>>> > specific
>>> > > >> partition is not a valid use case. Will update.
>>> > > >> Thanks
>>> > > >>
>>> > > >> On Tue, Oct 10, 2023 at 11:55 AM Jun Rao >> >
>>> > > >> wrote:
>>> > > >>
>>> > > >> > Hi, Calvin,
>>> > > >> >
>>> > > >> > 60.  If the range query has errors for s

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

[Calvin Liu]

Hey,
As we agreed to implement the pagination for the new API
DescribeTopicPartitions, the client side must also add a proper interface
to handle the pagination.
The current KafkaAdminClient.describeTopics returns
the DescribeTopicsResult which is the future for querying all the topics.
It is awkward to fit the pagination into it because

   1. Each future corresponds to a topic. We also want to have the
   pagination on huge topics for their partitions.
   2. To avoid OOM, we should only fetch the new topics when we need them
   and release the used topics. Especially the main use case of looping the
   topic list is when the client prints all the topics.

So, to better serve the pagination, @David Arthur
 suggested to add a new interface in the Admin
client between the following 2.

describeTopics(TopicCollection topics, DescribeTopicsOptions options,
Consumer);

Iterator describeTopics(TopicCollection topics,
DescribeTopicsOptions options);

David and I would prefer the first Consumer version which works better
as a stream purposes.


On Wed, Oct 11, 2023 at 4:28 PM Calvin Liu  wrote:

> Hi David,
> Thanks for the comment.
> Yes, we can separate the ELR enablement from the metadata version. It is
> also helpful to avoid blocking the following MV releases if the user is not
> ready for ELR.
> One thing to correct is that, the Unclean recovery is controlled
> by unclean.recovery.manager.enabled, a separate config
> from unclean.recovery.strategy. It determines whether unclean recovery will
> be used in an unclean leader election.
> Thanks
>
> On Wed, Oct 11, 2023 at 4:11 PM David Arthur  wrote:
>
>> One thing we should consider is a static config to totally enable/disable
>> the ELR feature. If I understand the KIP correctly, we can effectively
>> disable the unclean recovery by setting the recovery strategy config to
>> "none".
>>
>> This would make development and rollout of this feature a bit smoother.
>> Consider the case that we find bugs in ELR after a cluster has updated to
>> its MetadataVersion. It's simpler to disable the feature through config
>> rather than going through a MetadataVersion downgrade (once that's
>> supported).
>>
>> Does that make sense?
>>
>> -David
>>
>> On Wed, Oct 11, 2023 at 1:40 PM Calvin Liu 
>> wrote:
>>
>> > Hi Jun
>> > -Good catch, yes, we don't need the -1 in the DescribeTopicRequest.
>> > -No new value is added. The LeaderRecoveryState will still be set to 1
>> if
>> > we have an unclean leader election. The unclean leader election includes
>> > the old random way and the unclean recovery. During the unclean
>> recovery,
>> > the LeaderRecoveryState will not change until the controller decides to
>> > update the records with the new leader.
>> > Thanks
>> >
>> > On Wed, Oct 11, 2023 at 9:02 AM Jun Rao 
>> wrote:
>> >
>> > > Hi, Calvin,
>> > >
>> > > Another thing. Currently, when there is an unclean leader election, we
>> > set
>> > > the LeaderRecoveryState in PartitionRecord and PartitionChangeRecord
>> to
>> > 1.
>> > > With the KIP, will there be new values for LeaderRecoveryState? If
>> not,
>> > > when will LeaderRecoveryState be set to 1?
>> > >
>> > > Thanks,
>> > >
>> > > Jun
>> > >
>> > > On Tue, Oct 10, 2023 at 4:24 PM Jun Rao  wrote:
>> > >
>> > > > Hi, Calvin,
>> > > >
>> > > > One more comment.
>> > > >
>> > > > "The first partition to fetch details for. -1 means to fetch all
>> > > > partitions." It seems that FirstPartitionId of 0 naturally means
>> > fetching
>> > > > all partitions?
>> > > >
>> > > > Thanks,
>> > > >
>> > > > Jun
>> > > >
>> > > > On Tue, Oct 10, 2023 at 12:40 PM Calvin Liu
>> > > >
>> > > > wrote:
>> > > >
>> > > >> Hi Jun,
>> > > >> Yeah, with the current Metadata request handling, we only return
>> > errors
>> > > on
>> > > >> the Topic level, like topic not found. It seems that querying a
>> > specific
>> > > >> partition is not a valid use case. Will update.
>> > > >> Thanks
>> > > >>
>> > > >> On Tue, Oct 10, 2023 at 11:55 AM Jun Rao > >
>> > > >> wrote:
>> > > >>
>> > > >> > Hi, Calvin,
>> > > >> >
>> > > >> > 60.  If the range query has errors for some of the partitions,
>> do we
>> > > >> expect
>> > > >> > different responses when querying particular partitions?
>> > > >> >
>> > > >> > Thanks,
>> > > >> >
>> > > >> > Jun
>> > > >> >
>> > > >> > On Tue, Oct 10, 2023 at 10:50 AM Calvin Liu
>> > > > > > >> >
>> > > >> > wrote:
>> > > >> >
>> > > >> > > Hi Jun
>> > > >> > > 60. Yes, it is a good question. I was thinking the API could be
>> > > >> flexible
>> > > >> > to
>> > > >> > > query the particular partitions if the range query has errors
>> for
>> > > >> some of
>> > > >> > > the partitions. Not sure whether it is a valid assumption,
>> what do
>> > > you
>> > > >> > > think?
>> > > >> > >
>> > > >> > > 61. Good point, I will update them to partition level with the
>> > same
>> > > >> > limit.
>> > > >> > >
>> > > >> > > 62. Sure, will do.
>> > > >> > >
>> > > >> > > Thanks
>> > > >> > >
>> > > >> > > On Tue, Oct 10, 2

Re: [DISCUSS] KIP-939: Support Participation in 2PC

[Jun Rao]

Hi, Artem,

Thanks for the reply.

For 3b, it would be useful to understand the reason why an admin doesn't
authorize 2PC for self-hosted Flink. Is the main reason that 2PC has
unbounded timeout that could lead to unbounded outstanding transactions? If
so, another way to address that is to allow the admin to set a timeout even
for the 2PC case. The timeout would be long enough for behavioring
applications to complete 2PC operations, but not too long for non-behaving
applications' transactions to hang.

Jun

On Wed, Feb 21, 2024 at 4:34 PM Artem Livshits
 wrote:

> Hi Jun,
>
> > 20A. One option is to make the API initTransactions(boolean enable2PC).
>
> We could do that.  I think there is a little bit of symmetry between the
> client and server that would get lost with this approach (server has
> enable2PC as config), but I don't really see a strong reason for enable2PC
> to be a config vs. an argument for initTransactions.  But let's see if we
> find 20B to be a strong consideration for keeping a separate flag for
> keepPreparedTxn.
>
> > 20B. But realistically, we want Flink (and other apps) to have a single
> implementation
>
> That's correct and here's what I think can happen if we don't allow
> independent keepPreparedTxn:
>
> 1. Pre-KIP-939 self-hosted Flink vs. any Kafka cluster -- reflection is
> used, which effectively implements keepPreparedTxn=true without our
> explicit support.
> 2. KIP-939 self-hosted Flink vs. pre-KIP-939 Kafka cluster -- we can
> either fall back to reflection or we just say we don't support this, have
> to upgrade Kafka cluster first.
> 3. KIP-939 self-hosted Flink vs. KIP-939 Kafka cluster, this becomes
> interesting depending on whether the Kafka cluster authorizes 2PC or not:
>  3a. Kafka cluster autorizes 2PC for self-hosted Flink -- everything uses
> KIP-939 and there is no problem
>  3b. Kafka cluster doesn't authorize 2PC for self-hosted Flink -- we can
> either fallback to reflection or use keepPreparedTxn=true even if 2PC is
> not enabled.
>
> It seems to be ok to not support case 2 (i.e. require Kafka upgrade first),
> it shouldn't be an issue for cloud offerings as cloud providers are likely
> to upgrade their Kafka to the latest versions.
>
> The case 3b seems to be important to support, though -- the latest version
> of everything should work at least as well (and preferably better) than
> previous ones.  It's possible to downgrade to case 1, but it's probably not
> sustainable as newer versions of Flink would also add other features that
> the customers may want to take advantage of.
>
> If we enabled keepPreparedTxn=true even without 2PC, then we could enable
> case 3b without the need to fall back to reflection, so we could get rid of
> reflection-based logic and just have a single implementation based on
> KIP-939.
>
> > 32. My suggestion is to change
>
> Let me think about it and I'll come back to this.
>
> -Artem
>
> On Wed, Feb 21, 2024 at 3:40 PM Jun Rao  wrote:
>
> > Hi, Artem,
> >
> > Thanks for the reply.
> >
> > 20A. One option is to make the API initTransactions(boolean enable2PC).
> > Then, it's clear from the code whether 2PC related logic should be added.
> >
> > 20B. But realistically, we want Flink (and other apps) to have a single
> > implementation of the 2PC logic, not two different implementations,
> right?
> >
> > 32. My suggestion is to
> > change
> >
> kafka.server:type=transaction-coordinator-metrics,name=active-transaction-open-time-max
> > to sth like
> > Metric NameType  Group
> > Tags   Description
> > active-transaction-open-time-max   Max   transaction-coordinator-metrics
> >  none  The max time a currently-open transaction has been open
> >
> > Jun
> >
> > On Wed, Feb 21, 2024 at 11:25 AM Artem Livshits
> >  wrote:
> >
> > > Hi Jun,
> > >
> > > > 20A.  This only takes care of the abort case. The application still
> > needs
> > > to be changed to handle the commit case properly
> > >
> > > My point here is that looking at the initTransactions() call it's not
> > clear
> > > what the semantics is.  Say I'm doing code review, I cannot say if the
> > code
> > > is correct or not -- if the config (that's something that's
> > > theoretically not known at the time of code review) is going to enable
> > 2PC,
> > > then the correct code should look one way, otherwise it would need to
> > look
> > > differently.  Also, say if code is written with InitTransaction()
> without
> > > explicit abort and then for whatever reason the code would get used
> with
> > > 2PC enabled (could be a library in a bigger product) it'll start
> breaking
> > > in a non-intuitive way.
> > >
> > > > 20B. Hmm, if the admin disables 2PC, there is likely a reason behind
> > that
> > >
> > > That's true, but reality may be more complicated.  Say a user wants to
> > run
> > > a self-managed Flink with Confluent cloud.  Confluent cloud adim may
> not
> > > be comfortable enabling 2PC to general user accounts that use services
> > not
> > > man

[jira] [Created] (KAFKA-16304) Disk I/O error does not offline log directory while handling Fetch requests

[Gaurav Narula (Jira)]

Gaurav Narula created KAFKA-16304:
-

 Summary: Disk I/O error does not offline log directory while 
handling Fetch requests
 Key: KAFKA-16304
 URL: https://issues.apache.org/jira/browse/KAFKA-16304
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.6.1, 3.7.0
Reporter: Gaurav Narula


The Fetch API handler delays reading of record batches from disk using 
{{FileRecords::writeTo}} until the bytes are ready to be sent to the wire.

Therefore, a disk I/O error that happens while reading the log segment in 
either {{SslTransportLayer::transferFrom}} or 
{{PlaintextTransportLayer::transferFrom}} does not cause the log directory to 
be offlined. Instead, we assume that such an {{IOException}} can only occur 
because of network related causes and close the socket instead 
[[0]|https://github.com/apache/kafka/blob/7ac50a86113c9a489667451eecf61b6ab80870db/clients/src/main/java/org/apache/kafka/common/network/Selector.java#L610]
 
[[1]|https://github.com/apache/kafka/blob/7ac50a86113c9a489667451eecf61b6ab80870db/clients/src/main/java/org/apache/kafka/common/network/Selector.java#L629]

We should disambiguate between IOExceptions due to network and disk operations 
and offline the log directory in case of the latter as we do for other disk 
related failures.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2670

[Apache Jenkins Server]

See 

Re: [VOTE] 3.7.0 RC4

[Stanislav Kozlovski]

Some quick updates:

There were some inconsistencies between the documentation in the
apache/kafka repo and the one in kafka-site. The process is such that the
apache/kafka docs are the source of truth, but we had a few divergences in
the other repo. I have worked on correcting those with:
- MINOR: Reconcile upgrade.html with kafka-site/36's version
 and cherry-picked it into the
3.6 and 3.7 branches too

Additionally, the 3.7 upgrade notes have been merged in apache/kafka - MINOR:
Add 3.7 upgrade notes .

With that, I have opened a PR to move them to the kafka-site repository -
https://github.com/apache/kafka-site/pull/587. That is awaiting review.

Similarly, the 3.7 blog post is ready for review again
 and awaiting a review on 37:
Update default docs to point to the 3.7.0 release docs


I also have a WIP for fixing the 3.6 docs in the kafka-site repo
. This isn't really related
to the release, but it's good to do.

On Wed, Feb 21, 2024 at 4:55 AM Luke Chen  wrote:

> Hi all,
>
> I found there is a bug (KAFKA-16283
> ) in the built-in
> `RoundRobinPartitioner`, and it will cause only half of the partitions
> receiving the records. (I'm surprised our tests didn't catch it!?)
> After further testing, I found this issue already existed in v3.0.0. (I
> didn't test 2.x versions)
> I think this should not be a blocker to v3.7.0 since it's been there for
> years.
> But I think we should, at least, document it to notify users about this
> known issue.
> I've created 2 PRs to document it:
> https://github.com/apache/kafka/pull/15400
> https://github.com/apache/kafka-site/pull/585
>
> Let me know what you think.
>
> Thanks.
> Luke
>
> On Wed, Feb 21, 2024 at 10:52 AM Proven Provenzano
>  wrote:
>
> > HI,
> >
> > I've downloaded, built from source and then validated JBOD with KRaft
> works
> > along with migrating a cluster with JBOD from ZK to KRaft works.
> >
> > +1 (nonbinding) from me.
> >
> > --Proven
> >
> > On Tue, Feb 20, 2024 at 2:13 PM Justine Olshan
> > 
> > wrote:
> >
> > > Hey folks,
> > >
> > > I've done the following to validate the release:
> > >
> > > -- validated the keys for all the artifacts
> > > -- built from source and started a ZK cluster -- ran a few workloads on
> > it.
> > > -- ran 2.12 Kraft cluster and ran a few workloads on it
> > >
> > > I see there is a lot of ongoing discussion about the upgrade notes. +1
> > > (binding) from me given Mickael is voting +1 as well.
> > >
> > > Justine
> > >
> > > On Tue, Feb 20, 2024 at 6:18 AM Divij Vaidya 
> > > wrote:
> > >
> > > > > I am a bit unclear on the precise process regarding what parts of
> > this
> > > > get merged at what time, and whether the release first needs to be
> done
> > > or
> > > > not.
> > > >
> > > > The order is as follows:
> > > >
> > > > 1. Release approved as part of this vote. After this we follow the
> > > > steps from here:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Process#ReleaseProcess-Afterthevotepasses
> > > >
> > > > 2. Upload artifacts to maven etc. These artifacts do not have RC
> suffix
> > > in
> > > > them. You need a PMC member to mark these artifacts as "production"
> in
> > > > apache svn.
> > > > 3. Update website changes (docs, blog etc.). This is where your PRs
> > > > on kafka-site repo get merged.
> > > > 4. Send a release announcement by email.
> > > >
> > > > --
> > > > Divij Vaidya
> > > >
> > > >
> > > >
> > > > On Tue, Feb 20, 2024 at 3:02 PM Stanislav Kozlovski
> > > >  wrote:
> > > >
> > > > > Thanks for testing the release! And thanks for the review on the
> > > > > documentation. Good catch on the license too.
> > > > >
> > > > > I have addressed the comments in the blog PR, and opened a few
> other
> > > PRs
> > > > to
> > > > > the website in relation to the release.
> > > > >
> > > > > - 37: Add download section for the latest 3.7 release
> > > > > 
> > > > > - 37: Update default docs to point to the 3.7.0 release docs
> > > > > 
> > > > > - 3.7: Add blog post for Kafka 3.7
> > > > > 
> > > > > - MINOR: Update stale upgrade_3_6_0 header links in documentation
> > > > > 
> > > > > - 37: Add upgrade notes for the 3.7.0 release
> > > > > 
> > > > >
> > > > > I am a bit unclear on the precise process regarding what parts of
> > this
> > > > get
> > > > > merged at what time, and whether the release first needs to be done
> > or
> > > > not.
> > > > >
> > > > > Best,
> > > > > Stanislav
> > > > >
> > > > > On Mon, Feb 19, 2024 at 8

Re: [PR] MINOR: Update stale `upgrade_3_6_0` header links in documentation [kafka-site]

[via GitHub]

stanislavkozlovski commented on PR #580:
URL: https://github.com/apache/kafka-site/pull/580#issuecomment-1961103486

   This will be done with https://github.com/apache/kafka-site/pull/586/files


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] MINOR: Update stale `upgrade_3_6_0` header links in documentation [kafka-site]

[via GitHub]

stanislavkozlovski closed pull request #580: MINOR: Update stale 
`upgrade_3_6_0` header links in documentation
URL: https://github.com/apache/kafka-site/pull/580


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] 3.7: Add blog post for Kafka 3.7 [kafka-site]

[via GitHub]

stanislavkozlovski commented on code in PR #578:
URL: https://github.com/apache/kafka-site/pull/578#discussion_r1500491362


##
blog.html:
##
@@ -22,6 +22,125 @@
 
 
 Blog
+
+
+
+Apache 
Kafka 3.7.0 Release Announcement
+
+February 2024 - Stanislav Kozlovski (https://twitter.com/BdKozlovski";>@BdKozlovski)
+We are proud to announce the release of Apache Kafka 3.7.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.7.0/RELEASE_NOTES.html";>release 
notes.
+See the https://kafka.apache.org/36/documentation.html#upgrade_3_7_0";>Upgrading 
to 3.7.0 from any version 0.8.x through 3.6.x section in the documentation 
for the list of notable changes and detailed upgrade steps.
+
+In the last release, 3.6,
+https://kafka.apache.org/documentation/#kraft_zk_migration";>the ability 
to migrate Kafka clusters from a ZooKeeper metadata system
+to a KRaft metadata system was ready for usage in 
production environments with one caveat -- JBOD was not yet available for KRaft 
clusters.
+In this release, we are shipping an early access release 
of JBOD in KRaft. (See https://cwiki.apache.org/confluence/display/KAFKA/KIP-858%3A+Handle+JBOD+broker+disk+failure+in+KRaft";>KIP-858
 for details)
+
+
+Additionally, client APIs released prior to Apache Kafka 
2.1 are now marked deprecated in 3.7 and will be removed in Apache Kafka 4.0. 
See https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0";>KIP-896
 for details and RPC versions that are now deprecated.
+
+
+Java 11 support for the Kafka broker is also marked 
deprecated in 3.7, and is planned to be removed in Kafka 4.0. See https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789510";>KIP-1013
 for more details
+
+
+Note: ZooKeeper is marked as deprecated since the 3.5.0 
release. ZooKeeper is planned to be removed in Apache Kafka 4.0. For more 
information, please see the documentation for https://kafka.apache.org/documentation/#zk_depr";>ZooKeeper 
Deprecation.
+
+
+Kafka Broker, Controller, Producer, Consumer and Admin 
Client
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-858%3A+Handle+JBOD+broker+disk+failure+in+KRaft";>(Early
 Access) KIP-858 Handle JBOD broker disk failure in KRaft:

Review Comment:
   added 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] 37: Add latest apache/kafka/3.7 site-docs [kafka-site]

[via GitHub]

divijvaidya commented on code in PR #587:
URL: https://github.com/apache/kafka-site/pull/587#discussion_r1500467325


##
37/generated/admin_client_config.html:
##
@@ -304,7 +304,7 @@ 

[PR] Fixed links to Streams Developer Guide subpages [kafka-site]

[via GitHub]

thuri opened a new pull request, #588:
URL: https://github.com/apache/kafka-site/pull/588

   As it seems the Developer Guide page has been changed for Version 1.0.x and 
sections for "Streams DSL", "Processor API" and "Interactive Queries" got new, 
separate pages.
   
   But some links have not been changed from links targeting anchors to links 
to the new subpages


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2669

[Apache Jenkins Server]

See 

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.7 #101

[Apache Jenkins Server]

See