Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2107

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-953: partition method to be overloaded to accept headers as well.

[Jack Tomy]

Thanks Sagar and Chris for your votes. I will add the details Chris has
asked for to the KIP.

Hey everyone,

Please consider this as a gentle reminder.


On Sat, Aug 12, 2023 at 10:48 AM Chris Egerton 
wrote:

> Hi Jack,
>
> +1 (binding)
>
> Some friendly, non-blocking suggestions:
>
> - IMO it's still worth specifying that the headers will be read-only; this
> clarifies the intended API contract both for reviewers of the KIP who
> haven't read the GitHub PR yet, and for developers who may leverage this
> new method
> - May be worth mentioning in the compatibility section that any
> partitioners that only implement the new interface will be incompatible
> with older Kafka clients versions (this is less likely to be a serious
> problem in the clients world, but it's a much hairier problem with Connect,
> where cross-compatibility between newer/older versions of connectors and
> the Kafka Connect runtime is a serious concern)
>
> Again, these are not blockers and I'm in favor of the KIP with or without
> them since I believe both can be addressed at least partially during PR
> review and don't have to be tackled at this stage.
>
> Cheers,
>
> Chris
>
> On Sat, Aug 12, 2023 at 12:43 AM Sagar  wrote:
>
> > Hey jack ,
> >
> > +1 (non binding)
> >
> > Sagar.
> >
> > On Sat, 12 Aug 2023 at 8:04 AM, Jack Tomy  wrote:
> >
> > > Hey everyone,
> > >
> > > Please consider this as a gentle reminder.
> > >
> > > On Mon, Aug 7, 2023 at 5:55 PM Jack Tomy 
> wrote:
> > >
> > > > Hey everyone.
> > > >
> > > > I would like to call for a vote on KIP-953: partition method to be
> > > > overloaded to accept headers as well.
> > > >
> > > > KIP :
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263424937
> > > > Discussion thread :
> > > > https://lists.apache.org/thread/0f20kvfqkmhdqrwcb8vqgqn80szcrcdd
> > > >
> > > > Thanks
> > > > --
> > > > Best Regards
> > > > *Jack*
> > > >
> > >
> > >
> > > --
> > > Best Regards
> > > *Jack*
> > >
> >
>


-- 
Best Regards
*Jack*

[jira] [Created] (KAFKA-15353) Empty ISR returned from controller after AlterPartition request

[Luke Chen (Jira)]

Luke Chen created KAFKA-15353:
-

 Summary: Empty ISR returned from controller after AlterPartition 
request
 Key: KAFKA-15353
 URL: https://issues.apache.org/jira/browse/KAFKA-15353
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 3.5.0
Reporter: Luke Chen


In 
[KIP-903|https://cwiki.apache.org/confluence/display/KAFKA/KIP-903%3A+Replicas+with+stale+broker+epoch+should+not+be+allowed+to+join+the+ISR],
 (more specifically this [PR|https://github.com/apache/kafka/pull/13408]), we 
bumped the AlterPartitionRequest version to 3 to use `NewIsrWithEpochs` field 
instead of `NewIsr` one. And when building the request, we'll manually 
convert/downgrade the request into the older version for backward compatibility 
[here|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/requests/AlterPartitionRequest.java#L85-L96],
 to extract ISR info from `NewIsrWithEpochs` and then fill in the `NewIsr` 
field, and then clear the `NewIsrWithEpochs` field. 

 

The problem is, when the AlterPartitionRequest sent out for the first time, 
there's some transient error (ex: NOT_CONTROLLER), we'll retry. On the retry, 
we'll build the AlterPartitionRequest again. But this time, the request data is 
the one that converted above. At this point, when we try to extract the ISR 
from `NewIsrWithEpochs`, we'll get empty. So, we'll send out an AlterPartition 
request with empty ISR, and impacting the kafka availability.

 

This will happen when users trying to upgrade from versions < 3.5.0 to 3.5.0 or 
later. During the rolling upgrade, there will be some nodes in v3.5.0, and some 
are not. So, for the node in v3.5.0 will try to build an old version of 
AlterPartitionRequest. And then, if it happen to have some transient error 
during the AlterPartitionRequest send, the ISR will be empty and no producers 
will be able to write data to the partitions.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Apache Kafka 3.6.0 release

[Satish Duggana]

Hi,
Please plan to merge PRs(including the major features) targeted for
3.6.0 by the end of Aug 20th UTC. Starting from August 21st, any pull
requests intended for the 3.6.0 release must include the changes
merged into the 3.6 branch as mentioned in the release plan.

Thanks,
Satish.

On Fri, 4 Aug 2023 at 18:39, Chris Egerton  wrote:
>
> Thanks for adding KIP-949, Satish!
>
> On Fri, Aug 4, 2023 at 7:06 AM Satish Duggana 
> wrote:
>
> > Hi,
> > Myself and Divij discussed and added the wiki for Kafka TieredStorage
> > Early Access Release[1]. If you have any comments or feedback, please
> > feel free to share them.
> >
> > 1.
> > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Tiered+Storage+Early+Access+Release+Notes
> >
> > Thanks,
> > Satish.
> >
> > On Fri, 4 Aug 2023 at 08:40, Satish Duggana 
> > wrote:
> > >
> > > Hi Chris,
> > > Thanks for the update. This looks to be a minor change and is also
> > > useful for backward compatibility. I added it to the release plan as
> > > an exceptional case.
> > >
> > > ~Satish.
> > >
> > > On Thu, 3 Aug 2023 at 21:34, Chris Egerton 
> > wrote:
> > > >
> > > > Hi Satish,
> > > >
> > > > Would it be possible to include KIP-949 (
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-949%3A+Add+flag+to+enable+the+usage+of+topic+separator+in+MM2+DefaultReplicationPolicy
> > )
> > > > in the 3.6.0 release? It passed voting yesterday, and is a very small,
> > > > low-risk change that we'd like to put out as soon as possible in order
> > to
> > > > patch an accidental break in backwards compatibility caused a few
> > versions
> > > > ago.
> > > >
> > > > Best,
> > > >
> > > > Chris
> > > >
> > > > On Fri, Jul 28, 2023 at 2:35 AM Satish Duggana <
> > satish.dugg...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > > Whoever has KIP entries in the 3.6.0 release plan. Please update it
> > > > > with the latest status by tomorrow(end of the day 29th Jul UTC ).
> > > > >
> > > > > Thanks
> > > > > Satish.
> > > > >
> > > > > On Fri, 28 Jul 2023 at 12:01, Satish Duggana <
> > satish.dugg...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > Thanks Ismael and Divij for the suggestions.
> > > > > >
> > > > > > One way was to follow the earlier guidelines that we set for any
> > early
> > > > > > access release. It looks Ismael already mentioned the example of
> > > > > > KRaft.
> > > > > >
> > > > > > KIP-405 mentions upgrade/downgrade and limitations sections. We can
> > > > > > clarify that in the release notes for users on how this feature
> > can be
> > > > > > used for early access.
> > > > > >
> > > > > > Divij, We do not want users to enable this feature on production
> > > > > > environments in early access release. Let us work together on the
> > > > > > followups Ismael suggested.
> > > > > >
> > > > > > ~Satish.
> > > > > >
> > > > > > On Fri, 28 Jul 2023 at 02:24, Divij Vaidya <
> > divijvaidy...@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > Those are great suggestions, thank you. We will continue this
> > > > > discussion
> > > > > > > forward in a separate KIP for release plan for Tiered Storage.
> > > > > > >
> > > > > > > On Thu 27. Jul 2023 at 21:46, Ismael Juma 
> > wrote:
> > > > > > >
> > > > > > > > Hi Divij,
> > > > > > > >
> > > > > > > > I think the points you bring up for discussion are all good.
> > My main
> > > > > > > > feedback is that they should be discussed in the context of
> > KIPs vs
> > > > > the
> > > > > > > > release template. That's why we have a backwards compatibility
> > > > > section for
> > > > > > > > every KIP, it's precisely to ensure we think carefully about
> > some of
> > > > > the
> > > > > > > > points you're bringing up. When it comes to defining the
> > meaning of
> > > > > early
> > > > > > > > access, we have two options:
> > > > > > > >
> > > > > > > > 1. Have a KIP specifically for tiered storage.
> > > > > > > > 2. Have a KIP to define general guidelines for what early
> > access
> > > > > means.
> > > > > > > >
> > > > > > > > Does this make sense?
> > > > > > > >
> > > > > > > > Ismael
> > > > > > > >
> > > > > > > > On Thu, Jul 27, 2023 at 6:38 PM Divij Vaidya <
> > > > > divijvaidy...@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Thank you for the response, Ismael.
> > > > > > > > >
> > > > > > > > > 1. Specifically in context of 3.6, I wanted this
> > compatibility
> > > > > > > > > guarantee point to encourage a discussion on
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-952%3A+Regenerate+segment-aligned+producer+snapshots+when+upgrading+to+a+Kafka+version+supporting+Tiered+Storage
> > > > > > > > > .
> > > > > > > > > Due to lack of producer snapshots in <2.8 versions, a
> > customer may
> > > > > not
> > > > > > > > > be able to upgrade to 3.6 and use TS on a topic which was
> > created
> > > > > when
> > > > > > > > > the cluster was on <2.8 version (see motivation for
> > d

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2108

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-15210) Mention vote should be open for at atleast 72 hours

[Divij Vaidya (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Divij Vaidya resolved KAFKA-15210.
--
Resolution: Fixed

> Mention vote should be open for at atleast 72 hours
> ---
>
> Key: KAFKA-15210
> URL: https://issues.apache.org/jira/browse/KAFKA-15210
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Divij Vaidya
>Assignee: Gantigmaa Selenge
>Priority: Minor
> Fix For: 3.6.0
>
>
> The voting deadline should be at least 3 days from the time VOTE email is 
> posted. Hence, the script should mention that the date should be at least 72 
> hours from now. The change needs to be done at the line below: 
> *** Please download, test and vote by  28, 9am PT>



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-953: partition method to be overloaded to accept headers as well.

[Hector Geraldino (BLOOMBERG/ 919 3RD A)]

+1 (non-binding) 

Thanks for your KIP!

From: dev@kafka.apache.org At: 08/16/23 04:48:13 UTC-4:00To:  
dev@kafka.apache.org
Subject: Re: [VOTE] KIP-953: partition method to be overloaded to accept 
headers as well.

Thanks Sagar and Chris for your votes. I will add the details Chris has
asked for to the KIP.

Hey everyone,

Please consider this as a gentle reminder.


On Sat, Aug 12, 2023 at 10:48 AM Chris Egerton 
wrote:

> Hi Jack,
>
> +1 (binding)
>
> Some friendly, non-blocking suggestions:
>
> - IMO it's still worth specifying that the headers will be read-only; this
> clarifies the intended API contract both for reviewers of the KIP who
> haven't read the GitHub PR yet, and for developers who may leverage this
> new method
> - May be worth mentioning in the compatibility section that any
> partitioners that only implement the new interface will be incompatible
> with older Kafka clients versions (this is less likely to be a serious
> problem in the clients world, but it's a much hairier problem with Connect,
> where cross-compatibility between newer/older versions of connectors and
> the Kafka Connect runtime is a serious concern)
>
> Again, these are not blockers and I'm in favor of the KIP with or without
> them since I believe both can be addressed at least partially during PR
> review and don't have to be tackled at this stage.
>
> Cheers,
>
> Chris
>
> On Sat, Aug 12, 2023 at 12:43 AM Sagar  wrote:
>
> > Hey jack ,
> >
> > +1 (non binding)
> >
> > Sagar.
> >
> > On Sat, 12 Aug 2023 at 8:04 AM, Jack Tomy  wrote:
> >
> > > Hey everyone,
> > >
> > > Please consider this as a gentle reminder.
> > >
> > > On Mon, Aug 7, 2023 at 5:55 PM Jack Tomy 
> wrote:
> > >
> > > > Hey everyone.
> > > >
> > > > I would like to call for a vote on KIP-953: partition method to be
> > > > overloaded to accept headers as well.
> > > >
> > > > KIP :
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263424937
> > > > Discussion thread :
> > > > https://lists.apache.org/thread/0f20kvfqkmhdqrwcb8vqgqn80szcrcdd
> > > >
> > > > Thanks
> > > > --
> > > > Best Regards
> > > > *Jack*
> > > >
> > >
> > >
> > > --
> > > Best Regards
> > > *Jack*
> > >
> >
>


-- 
Best Regards
*Jack*

[jira] [Created] (KAFKA-15354) Partition leader is not evenly distributed in kraft mode

[Deng Ziming (Jira)]

Deng Ziming created KAFKA-15354:
---

 Summary: Partition leader is not evenly distributed in kraft mode
 Key: KAFKA-15354
 URL: https://issues.apache.org/jira/browse/KAFKA-15354
 Project: Kafka
  Issue Type: Improvement
Reporter: Deng Ziming


In StripedReplicaPlacerTest, we can create a test below to reproduce this bug.
{code:java}
// code placeholder
@Test
public void testReplicaDistribution() {
MockRandom random = new MockRandom();
StripedReplicaPlacer placer = new StripedReplicaPlacer(random);
TopicAssignment assignment = place(placer, 0, 4, (short) 2, Arrays.asList(
new UsableBroker(0, Optional.of("0"), false),
new UsableBroker(1, Optional.of("0"), false),
new UsableBroker(2, Optional.of("1"), false),
new UsableBroker(3, Optional.of("1"), false)));
System.out.println(assignment);
} {code}
In StripedReplicaPlacer, we only ensure leader are distributed evenly across 
racks, but we didn't ensure leader are evenly distributed across nodes. in the 
test above, we have 4 node: 1 2 3 4, and create 4 partitions but the leaders 
are  1 2 1 2. while in zk mode, this is ensured, see 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-36+Rack+aware+replica+assignment



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2109

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-15355) Update metadata records

[Igor Soarez (Jira)]

Igor Soarez created KAFKA-15355:
---

 Summary: Update metadata records
 Key: KAFKA-15355
 URL: https://issues.apache.org/jira/browse/KAFKA-15355
 Project: Kafka
  Issue Type: Sub-task
Reporter: Igor Soarez


Includes:
 * BrokerRegistrationChangeRecord
 * PartitionChangeRecord
 * PartitionRecord
 * RegisterBrokerRecord



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15356) Generate and persist log directory UUIDs

[Igor Soarez (Jira)]

Igor Soarez created KAFKA-15356:
---

 Summary: Generate and persist log directory UUIDs
 Key: KAFKA-15356
 URL: https://issues.apache.org/jira/browse/KAFKA-15356
 Project: Kafka
  Issue Type: Sub-task
Reporter: Igor Soarez
Assignee: Igor Soarez


* The StorageTool format command should ensure each dir has a generated UUID
 * BrokerMetadataCheckpoint should parse directory.id from meta.properties, or 
generate and persist if missing

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15357) Propagates assignments and logdir failures to controller

[Igor Soarez (Jira)]

Igor Soarez created KAFKA-15357:
---

 Summary: Propagates assignments and logdir failures to controller
 Key: KAFKA-15357
 URL: https://issues.apache.org/jira/browse/KAFKA-15357
 Project: Kafka
  Issue Type: Sub-task
Reporter: Igor Soarez
Assignee: Igor Soarez


LogDirEventManager accumulates, batches, and sends assignments or failure 
events to the Controller, prioritizing assignments to ensure the Controller has 
the correct assignment view before processing log dir failures.

Assignments are sent via AssignReplicasToDirs, logdir failures are sent via 
BrokerHeartbeat.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15358) QueuedReplicaToDirAssignments metric

[Igor Soarez (Jira)]

Igor Soarez created KAFKA-15358:
---

 Summary: QueuedReplicaToDirAssignments metric
 Key: KAFKA-15358
 URL: https://issues.apache.org/jira/browse/KAFKA-15358
 Project: Kafka
  Issue Type: Sub-task
Reporter: Igor Soarez
Assignee: Igor Soarez






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15359) log.dir.failure.timeout.ms configuration

[Igor Soarez (Jira)]

Igor Soarez created KAFKA-15359:
---

 Summary: log.dir.failure.timeout.ms configuration
 Key: KAFKA-15359
 URL: https://issues.apache.org/jira/browse/KAFKA-15359
 Project: Kafka
  Issue Type: Sub-task
Reporter: Igor Soarez


If the Broker repeatedly cannot communicate fails to communicate a log 
directory failure after a configurable amount of time — 
{{log.dir.failure.timeout.ms}} — and it is the leader for any replicas in the 
failed log directory the broker will shutdown, as that is the only other way to 
guarantee that the controller will elect a new leader for those partitions.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15360) Include directory info in BrokerRegistration

[Igor Soarez (Jira)]

Igor Soarez created KAFKA-15360:
---

 Summary: Include directory info in BrokerRegistration
 Key: KAFKA-15360
 URL: https://issues.apache.org/jira/browse/KAFKA-15360
 Project: Kafka
  Issue Type: Sub-task
Reporter: Igor Soarez
Assignee: Igor Soarez


Brokers should correctly set the OnlineLogDirs and OfflineLogDirs fields in 
each BrokerRegistrationRequest.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15361) Process and persist dir info with broker registration

[Igor Soarez (Jira)]

Igor Soarez created KAFKA-15361:
---

 Summary: Process and persist dir info with broker registration
 Key: KAFKA-15361
 URL: https://issues.apache.org/jira/browse/KAFKA-15361
 Project: Kafka
  Issue Type: Sub-task
Reporter: Igor Soarez


Controllers should process and persist directory information from the broker 
registration request



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15362) Resolve offline replicas in metadata cache

[Igor Soarez (Jira)]

Igor Soarez created KAFKA-15362:
---

 Summary: Resolve offline replicas in metadata cache 
 Key: KAFKA-15362
 URL: https://issues.apache.org/jira/browse/KAFKA-15362
 Project: Kafka
  Issue Type: Sub-task
Reporter: Igor Soarez


Considering broker's offline log directories and replica to dir assignments



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15363) Broker log directory failure changes

[Igor Soarez (Jira)]

Igor Soarez created KAFKA-15363:
---

 Summary: Broker log directory failure changes
 Key: KAFKA-15363
 URL: https://issues.apache.org/jira/browse/KAFKA-15363
 Project: Kafka
  Issue Type: Sub-task
Reporter: Igor Soarez






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15364) Handle log directory failure in the Controller

[Igor Soarez (Jira)]

Igor Soarez created KAFKA-15364:
---

 Summary: Handle log directory failure in the Controller
 Key: KAFKA-15364
 URL: https://issues.apache.org/jira/browse/KAFKA-15364
 Project: Kafka
  Issue Type: Sub-task
Reporter: Igor Soarez






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15365) Replica management changes

[Igor Soarez (Jira)]

Igor Soarez created KAFKA-15365:
---

 Summary: Replica management changes
 Key: KAFKA-15365
 URL: https://issues.apache.org/jira/browse/KAFKA-15365
 Project: Kafka
  Issue Type: Sub-task
Reporter: Igor Soarez






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15366) Log directory failure integration test

[Igor Soarez (Jira)]

Igor Soarez created KAFKA-15366:
---

 Summary: Log directory failure integration test
 Key: KAFKA-15366
 URL: https://issues.apache.org/jira/browse/KAFKA-15366
 Project: Kafka
  Issue Type: Sub-task
Reporter: Igor Soarez






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15367) Test KRaft JBOD enabling migration

[Igor Soarez (Jira)]

Igor Soarez created KAFKA-15367:
---

 Summary: Test KRaft JBOD enabling migration
 Key: KAFKA-15367
 URL: https://issues.apache.org/jira/browse/KAFKA-15367
 Project: Kafka
  Issue Type: Sub-task
Reporter: Igor Soarez


A cluster running in KRaft without JBOD should be able to transition into JBOD 
mode without issues



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15368) Test ZK JBOD to KRaft migration

[Igor Soarez (Jira)]

Igor Soarez created KAFKA-15368:
---

 Summary: Test ZK JBOD to KRaft migration
 Key: KAFKA-15368
 URL: https://issues.apache.org/jira/browse/KAFKA-15368
 Project: Kafka
  Issue Type: Sub-task
Reporter: Igor Soarez


A ZK cluster running JBOD should be able to migrate to KRaft mode without issues



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[DISCUSS] KIP-968: Support single-key_multi-timestamp interactive queries (IQv2) for versioned state stores

[Alieh Saeedi]

Hi all,

I splitted KIP-960

into three separate KIPs. Therefore, please continue discussions
about single-key, multi-timestamp interactive queries here. You can see all
the addressed reviews on the following page. Thanks in advance.

KIP-968: Support single-key_multi-timestamp interactive queries (IQv2) for
versioned state stores


I look forward to your feedback!

Cheers,
Alieh

[DISCUSS] KIP-969: Support range interactive queries for versioned state stores

[Alieh Saeedi]

Hi all,

I splitted KIP-960

into
three separate KIPs. Therefore, please continue discussions about range
interactive queries here. You can see all the addressed reviews on the
following page. Thanks in advance.

KIP-969: Support range interactive queries (IQv2) for versioned state stores


I look forward to your feedback!

Cheers,
Alieh

[jira] [Resolved] (KAFKA-15228) Add sync-manifests subcommand to connect-plugin-path tool

[Greg Harris (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Greg Harris resolved KAFKA-15228.
-
Resolution: Fixed

> Add sync-manifests subcommand to connect-plugin-path tool
> -
>
> Key: KAFKA-15228
> URL: https://issues.apache.org/jira/browse/KAFKA-15228
> Project: Kafka
>  Issue Type: New Feature
>  Components: KafkaConnect, tools
>Reporter: Greg Harris
>Assignee: Greg Harris
>Priority: Major
> Fix For: 3.6.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15369) Allow AdminClient to Talk Directly with the KRaft Controller Quorum and add Controller Registration

[Colin McCabe (Jira)]

Colin McCabe created KAFKA-15369:


 Summary: Allow AdminClient to Talk Directly with the KRaft 
Controller Quorum and add Controller Registration
 Key: KAFKA-15369
 URL: https://issues.apache.org/jira/browse/KAFKA-15369
 Project: Kafka
  Issue Type: Improvement
Reporter: Colin McCabe






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15220) KRaftMetadataCache returns fenced brokers from getAliveBrokerNode

[Colin McCabe (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin McCabe resolved KAFKA-15220.
--
Fix Version/s: 3.6.0
   Resolution: Fixed

> KRaftMetadataCache returns fenced brokers from getAliveBrokerNode
> -
>
> Key: KAFKA-15220
> URL: https://issues.apache.org/jira/browse/KAFKA-15220
> Project: Kafka
>  Issue Type: Bug
>Reporter: David Mao
>Assignee: David Mao
>Priority: Major
> Fix For: 3.6.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2111

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-15370) Support Participation in 2PC (KIP-939)

[Artem Livshits (Jira)]

Artem Livshits created KAFKA-15370:
--

 Summary: Support Participation in 2PC (KIP-939)
 Key: KAFKA-15370
 URL: https://issues.apache.org/jira/browse/KAFKA-15370
 Project: Kafka
  Issue Type: Improvement
Reporter: Artem Livshits


This ticket tracks the implementation of KIP-939: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-939%3A+Support+Participation+in+2PC.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-960: Support interactive queries (IQv2) for versioned state stores

[Matthias J. Sax]

Thanks for updating the KIP and splitting into multiple ones. I am just 
going to reply for the single-key-single-timestamp case below.


It seems the `KeyQuery.java` code snipped is "incomplete" -- the class 
definition is missing.


At the same time, the example uses `VersionedKeyQuery` so I am not sure 
right now if you propose to re-use the existing `KeyQuery` class or 
introduce a new `VersionedKeyQuery` class?


While it was suggested that we re-use the existing `KeyQuery` class, I 
am wondering what would happen if one uses the new `asOf` method, and 
passes the query into a non-versioned store?


In the end, a non-versioned store does not know that there is an as-of 
timestamp set and thus might just do a plain lookup (it also only has a 
single value per key) and return whatever value it has stored?


I am wondering if this would be semantically questionable and/or 
confusing for users (especially for timestamped stores)? -- Because the 
non-versioned store does not know anything about the timestamp, it can 
also not even check if it's set and raise an error.



Did you try to prototype any of both approaches? Asking because I am 
wondering about generics and return types? Existing `KeyQuery` is defined as


`KeyQuery extends Query` so `V` is the result type.

However for the versioned-store we want the result type to be 
`VersionedRecord` and thus we would need to set `V = 
VersionedRecord` -- would this work or would the compiler tip over it 
(or would it work but still be confusing/complex for users to specify 
the right types)?


For `VersionedKeyQuery` we could do:

`VersionedKeyQuery extends Query>`

what seems cleaner?

Without writing code I always have a hard time to reason about generics, 
so maybe trying out both approaches might shed some light?





-Matthias


On 8/15/23 9:03 AM, Alieh Saeedi wrote:

Hi all,
thanks to all for the great points you mentioned.

Addressed reviews are listed as follows:
1. The methods are defined as composable, as Lucas suggested. Now we have
even more types of single-key_multi-timestamp queries. As Matthias
suggested in his first review, now with composable methods, queries with a
lower time bound are also possible. The meaningless combinations are
prevented by throwing exceptions.
2. I corrected and replaced asOf everywhere instead of until. I hope the
javadocs and the explanations in the KIPs are clear enough about the time
range. Matthias, Lucas, and Victoria asked about the exact time boundaries.
I assumed that if the time range is specified as [t1, t2], all the records
that have been inserted within this time range must be returned by the
query. But I think the point that all of you referred to and that Victoria
clarified very well is valid. Maybe the query must return "all the
records that are valid within the time range". Therefore, records that have
been inserted before t1 are also retuned. Now, this makes more sense to me
as a user. By the way, it seems more like a product question.
3. About the order of retuned records, I added some boolean fields to the
classes to specify them. I still do not have any clue how hard the
implementation of this will be. The question is, is the order considered
for normal range queries as well?
4. As Victoria pointed out the issue about listing tombstones, I changed
the VersionedRecord such that it can have NULL values as well. The question
is, what was the initial intention of setting the value in VersionedRecord
as NOT NULL? I am worried about breaking other parts of the code.
5. About the motivation for defining the VersionedKeyQuery and
VersionedRangeQuery classes: I think my initial intention was to
distinguish between queries that return a single record and queries that
return a set of records. On the other hand, I put both
single-key_single-timestamp queries and single-key_multi-timestamp queries
in the same class, VersionedKeyQuery. Matthias complained about it as well.
Therefore, in my new changes, I put single-key_single-timestamp queries in
the KeyQuery class and single-key_multi-timestamp queries in the
VersionedKeyQuery class. I still have kept the VersionedRangeQuery class.
If there are good arguments for kicking both VersionedKeyQuery and
VersionedRangeQuery classes out, I can change the KIPs.
6. About defining new methods in the VersionedKeyValueStore interface: I
actually have defined the required methods in the RocksDBVersionedStore
class. Since defining them for the interface requires implementing them for
all the classes that have implemented the interface.
7. I replaced Long with Instant type for timestamp.

As Matthias (explicitly) and Victoria (implicitly) suggested, I broke
KIP-960 into three different KIPs.
KIP-960: Support single-key_single-timestamp Interactive Queries (IQv2) for
Versioned State Stores

KIP-968: Support single-key_multi-timestamp Interactive Que

Re: [DISCUSS] KIP-968: Support single-key_multi-timestamp interactive queries (IQv2) for versioned state stores

[Matthias J. Sax]

Thanks for splitting this part into a separate KIP!

For `withKey()` we should be explicit that `null` is not allowed.

(Looking into existing `KeyQuery` it seems the JavaDocs don't cover this 
either -- would you like to do a tiny cleanup PR for this, or fix 
on-the-side in one of your PRs?)





The key query returns all the records that are valid in the time range starting 
from the timestamp {@code fromTimestamp}.


In the JavaDocs you use the phrase `are valid` -- I think we need to 
explain what "valid" means? It might even be worth to add some examples. 
It's annoying, but being precise if kinda important.


With regard to KIP-962, should we allow `null` for time bounds ? The 
JavaDocs should also be explicit if `null` is allowed or not and what 
the semantics are if allowed.




You are using `asOf()` however, because we are doing time-range queries, 
to me using `until()` to describe the upper bound would sound better (I 
am not a native speaker though, so maybe I am off?)




The key query returns all the records that have timestamp <= {@code 
asOfTimestamp}.


This is only correct if not lower-bound is set, right?


In your reply to KIP-960 you mentioned:


the meaningless combinations are prevented by throwing exceptions.


We should add corresponding JavaDocs like:

   @throws IllegalArgumentException if {@code fromTimestamp} is equal or
larger than {@code untilTimestamp}

Or something similar.


With regard to KIP-960: if we need to introduce a `VersionedKeyQuery` 
class for single-key-single-ts lookup, would we need to find a new name 
for the query class of this KIP, given that the return type is different?



-Matthias



On 8/16/23 10:57 AM, Alieh Saeedi wrote:

Hi all,

I splitted KIP-960

into three separate KIPs. Therefore, please continue discussions
about single-key, multi-timestamp interactive queries here. You can see all
the addressed reviews on the following page. Thanks in advance.

KIP-968: Support single-key_multi-timestamp interactive queries (IQv2) for
versioned state stores


I look forward to your feedback!

Cheers,
Alieh

[jira] [Created] (KAFKA-15371) MetadataShell is stuck when bootstrapping

[Deng Ziming (Jira)]

Deng Ziming created KAFKA-15371:
---

 Summary: MetadataShell is stuck when bootstrapping
 Key: KAFKA-15371
 URL: https://issues.apache.org/jira/browse/KAFKA-15371
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.5.1
Reporter: Deng Ziming
 Attachments: image-2023-08-17-10-35-01-039.png, 
image-2023-08-17-10-35-36-067.png

I  downloaded the 3.5.1 package and startup it, then use metadata shell to 
inspect the data

 
{code:java}
// shell
bin/kafka-metadata-shell.sh --snapshot 
/tmp/kraft-combined-logs/__cluster_metadata-0/.log  {code}
Then process will stuck at loading.

 

 

!image-2023-08-17-10-35-36-067.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-953: partition method to be overloaded to accept headers as well.

[Ismael Juma]

Hi,

Thanks for the KIP. The problem outlined here is a great example why we
should be using a record-like structure to pass the parameters to a method
like this. Then we can add more parameters without having to introduce new
methods. Have we considered this option?

Ismael

On Mon, Aug 7, 2023 at 5:26 AM Jack Tomy  wrote:

> Hey everyone.
>
> I would like to call for a vote on KIP-953: partition method to be
> overloaded to accept headers as well.
>
> KIP :
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263424937
> Discussion thread :
> https://lists.apache.org/thread/0f20kvfqkmhdqrwcb8vqgqn80szcrcdd
>
> Thanks
> --
> Best Regards
> *Jack*
>

[DISCUSS] KIP-939: Support Participation in 2PC

[Artem Livshits]

Hello,

This is a discussion thread for
https://cwiki.apache.org/confluence/display/KAFKA/KIP-939%3A+Support+Participation+in+2PC
.

The KIP proposes extending Kafka transaction support (that already uses 2PC
under the hood) to enable atomicity of dual writes to Kafka and an external
database, and helps to fix a long standing Flink issue.

An example of code that uses the dual write recipe with JDBC and should
work for most SQL databases is here
https://github.com/apache/kafka/pull/14231.

The FLIP for the sister fix in Flink is here
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=255071710

-Artem