[jira] [Created] (KAFKA-15867) Should ConsumerNetworkThread wrap the exception and notify the polling thread?

Philip Nee created KAFKA-15867: -- Summary: Should ConsumerNetworkThread wrap the exception and notify the polling thread? Key: KAFKA-15867 URL: https://issues.apache.org/jira/browse/KAFKA-15867 Project:

Re: [DISCUSS] KIP-1006: Remove SecurityManager Support

Hey Ashwin, Thanks for your question! I believe we have only removed support for two Java versions: 7: https://cwiki.apache.org/confluence/display/KAFKA/KIP-118%3A+Drop+Support+for+Java+7 in 2.0 8: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181308223 in 4.0 In both cases,

[jira] [Created] (KAFKA-15866) Refactor OffsetFetchRequestState Error handling to be more consistent with OffsetCommitRequestState

Philip Nee created KAFKA-15866: -- Summary: Refactor OffsetFetchRequestState Error handling to be more consistent with OffsetCommitRequestState Key: KAFKA-15866 URL: https://issues.apache.org/jira/browse/KAFKA-15866

Re: [VOTE] KIP-963: Additional metrics in Tiered Storage

+1 (binding) from me. Thanks for the KIP. Luke On Tue, Nov 21, 2023 at 11:53 AM Satish Duggana wrote: > +1 (binding) > Thanks for the KIP and the discussion. > > Discussion mail thread for the KIP: > https://lists.apache.org/thread/40vsyc240hyody37mf2f0pn90shkzb45 > > > > On Tue, 21 Nov 2023

Re: [DISCUSS] KIP-963: Upload and delete lag metrics in Tiered Storage

Hi Christo, Sorry for the late reply. > 3. I was thinking that the metric can be emitted while reading of those records is happening i.e. if it takes a long time then it will just gradually increase as we read. What do you think? Yes, sounds good to me. > Kamal and Luke, I agree some of the

Re: [DISCUSS] KIP-1006: Remove SecurityManager Support

Hi Greg, Thanks for writing this KIP. I agree with you that handling this now will help us react to the deprecation of SecurityManager, whenever it happens. I had a question regarding how we deprecate JDKs supported by Apache Kafka. When we drop support for JDK 17, will we set the “-target”

Re: How Kafka handle partition leader change?

I am asking this because I want to propose a change to Kafka. But looks like in certain scenario it is very hard to not loss or duplication messages. Wonder in what scenario we can accept that and where to draw the line? From: De Gao Sent: 21 November 2023 6:25

Re: How Kafka handle partition leader change?

Thanks Andrew. Sounds like the leadership change from Kafka side is a 'best effort' to avoid message duplicate or loss. Can we say that message lost is very likely during leadership change unless producer uses idempotency? Is this a generic situation that no intent to provide data integration

[jira] [Created] (KAFKA-15865) Ensure consumer.poll() execute autocommit callback

Philip Nee created KAFKA-15865: -- Summary: Ensure consumer.poll() execute autocommit callback Key: KAFKA-15865 URL: https://issues.apache.org/jira/browse/KAFKA-15865 Project: Kafka Issue Type:

[jira] [Created] (KAFKA-15864) Add more tests asserting the log-start-offset, local-log-start-offset, and HW/LSO/LEO in rolling over segments with tiered storage.

Satish Duggana created KAFKA-15864: -- Summary: Add more tests asserting the log-start-offset, local-log-start-offset, and HW/LSO/LEO in rolling over segments with tiered storage. Key: KAFKA-15864 URL:

[jira] [Created] (KAFKA-15863) Handle push telemetry throttling with quota manager

Apoorv Mittal created KAFKA-15863: - Summary: Handle push telemetry throttling with quota manager Key: KAFKA-15863 URL: https://issues.apache.org/jira/browse/KAFKA-15863 Project: Kafka Issue

Re: [DISCUSS] KIP-1005: Add EarliestLocalOffset to GetOffsetShell

Thanks Christo for starting the discussion on the KIP. As mentioned in KAFKA-15857[1], the goal is to add new entries for local-log-start-offset and tierd-offset in OffsetSpec. This will be used in AdminClient APIs and also to be added as part of GetOffsetShell. This was also raised by Kamal in

Re: [VOTE] KIP-963: Additional metrics in Tiered Storage

+1 (binding) Thanks for the KIP and the discussion. Discussion mail thread for the KIP: https://lists.apache.org/thread/40vsyc240hyody37mf2f0pn90shkzb45 On Tue, 21 Nov 2023 at 05:21, Kamal Chandraprakash wrote: > > +1 (non-binding). Thanks for the KIP! > > On Tue, Nov 21, 2023, 03:04 Divij

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2401

See Changes: -- [...truncated 435070 lines...] Gradle Test Run :streams:test > Gradle Test Executor 92 > DefaultTaskManagerTest > shouldAssignTaskThatCanBeProcessed()

[jira] [Created] (KAFKA-15862) Remove SecurityManager Support

Greg Harris created KAFKA-15862: --- Summary: Remove SecurityManager Support Key: KAFKA-15862 URL: https://issues.apache.org/jira/browse/KAFKA-15862 Project: Kafka Issue Type: New Feature

[DISCUSS] KIP-1006: Remove SecurityManager Support

Hi all, I'd like to invite you all to discuss removing SecurityManager support from Kafka. This affects the client and server SASL mechanism, Tiered Storage, and Connect classloading. Find the KIP here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1006%3A+Remove+SecurityManager+Support

[jira] [Created] (KAFKA-15861) In Kraft mode, "ssl.keystore.key" private keys are accesible to all the controllers and brokers

Jesús Cea created KAFKA-15861: - Summary: In Kraft mode, "ssl.keystore.key" private keys are accesible to all the controllers and brokers Key: KAFKA-15861 URL: https://issues.apache.org/jira/browse/KAFKA-15861

[jira] [Created] (KAFKA-15860) ControllerRegistration must be written out to the metadata image

Colin McCabe created KAFKA-15860: Summary: ControllerRegistration must be written out to the metadata image Key: KAFKA-15860 URL: https://issues.apache.org/jira/browse/KAFKA-15860 Project: Kafka

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2400

See

Re: [VOTE] KIP-963: Additional metrics in Tiered Storage

+1 (non-binding). Thanks for the KIP! On Tue, Nov 21, 2023, 03:04 Divij Vaidya wrote: > + 1 (binding) > > This Kip will greatly improve Tiered Storage troubleshooting. Thank you > Christo. > > On Mon 20. Nov 2023 at 17:21, Christo Lolov > wrote: > > > Hello all! > > > > Now that the discussion

Re: [VOTE] KIP-1000: List Client Metrics Configuration Resources

The voting for this KIP is now complete. +3 binding (Jun, Jason, Matthias) +2 non-binding (Doğuşcan, Apoorv) Thanks, Andrew > On 20 Nov 2023, at 21:42, Jason Gustafson wrote: > > +1 Thanks for the KIP! > > On Mon, Nov 20, 2023 at 9:31 AM Jun Rao wrote: > >> Hi, Andrew, >> >> Thanks for the

Re: [VOTE] KIP-1000: List Client Metrics Configuration Resources

+1 (binding) On 11/20/23 1:42 PM, Jason Gustafson wrote: +1 Thanks for the KIP! On Mon, Nov 20, 2023 at 9:31 AM Jun Rao wrote: Hi, Andrew, Thanks for the KIP. +1 Jun On Thu, Nov 16, 2023 at 9:12 AM Andrew Schofield < andrew_schofield_j...@outlook.com> wrote: Hi Apoorv, Thanks for your

Re: [VOTE] KIP-1001; CurrentControllerId Metric

The KIP makes sense. +1 On Mon, Nov 20, 2023 at 12:37 PM David Arthur wrote: > Thanks Colin, > > +1 from me > > -David > > On Tue, Nov 14, 2023 at 3:53 PM Colin McCabe wrote: > > > Hi all, > > > > I'd like to call a vote for KIP-1001: Add CurrentControllerId metric. > > > > Take a look here: >

Re: [VOTE] KIP-1000: List Client Metrics Configuration Resources

+1 Thanks for the KIP! On Mon, Nov 20, 2023 at 9:31 AM Jun Rao wrote: > Hi, Andrew, > > Thanks for the KIP. +1 > > Jun > > On Thu, Nov 16, 2023 at 9:12 AM Andrew Schofield < > andrew_schofield_j...@outlook.com> wrote: > > > Hi Apoorv, > > Thanks for your vote. > > > > Initially, I put support

Re: [VOTE] KIP-963: Additional metrics in Tiered Storage

+ 1 (binding) This Kip will greatly improve Tiered Storage troubleshooting. Thank you Christo. On Mon 20. Nov 2023 at 17:21, Christo Lolov wrote: > Hello all! > > Now that the discussion for KIP-963 has winded down, I would like to open > it for a vote targeting 3.7.0 as the release. You can

Re: [VOTE] KIP-1001; CurrentControllerId Metric

Thanks Colin, +1 from me -David On Tue, Nov 14, 2023 at 3:53 PM Colin McCabe wrote: > Hi all, > > I'd like to call a vote for KIP-1001: Add CurrentControllerId metric. > > Take a look here: > https://cwiki.apache.org/confluence/x/egyZE > > best, > Colin > -- -David

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2399

See Changes: -- [...truncated 436595 lines...] Gradle Test Run :streams:test > Gradle Test Executor 91 > DefaultTaskManagerTest > shouldNotRemoveUnlockedTask() PASSED

Re: [DISCUSS] Road to Kafka 4.0

Hi Josep, I think there is some confusion here. Quorum reconfiguration is not needed for KRaft to become production ready. Confluent runs thousands of KRaft clusters without quorum reconfiguration, and has for years. While dynamic quorum reconfiguration is a nice feature, it doesn't block

Re: [VOTE] KIP-968: Support single-key_multi-timestamp interactive queries (IQv2) for versioned state stores

Adding an enum is a good idea! Wondering if `UNORDERED` is the best name? Want to avoid bike shedding, just asking. We could also use `UNDEFINED` / `UNSPECIFIED` / `NONE` / `ANY` ? In the end, the result _might_ be ordered, we just don't guarantee any order. -Matthias On 11/20/23 9:17

Re: [DISCUSS] KIP-968: Support single-key_multi-timestamp interactive queries (IQv2) for versioned state stores

I think we should also discuss a little more about `validTo()` method? Given that "latest" version does not have a valid-to TS, should we change the return type to `Optional` and return `empty()` for "latest"? ATM the KIP uses `MAX_VALUE` for "latest" what seems to be less clean? We could

Re: [DISCUSS] Road to Kafka 4.0

Hi all, If we do a 3.8 release before 4.0 and we implement KIP-853 in 3.8, the user will be able to migrate to a KRaft cluster that supports dynamically changing the set of voters and has better support for disk failures. What are the disadvantages of adding the 3.8 release before 4.0? This

Re: [DISCUSS] KIP-1000: List Client Metrics Configuration Resources

Hi, Andrew, Thanks for the reply. LGTM. Jun On Sat, Nov 18, 2023 at 12:28 AM Andrew Schofield < andrew_schofield_j...@outlook.com> wrote: > Hi Jun, > This is an example of inconsistency between the tools. I do agree that > it would be nice to have `--list` on the `kafka-client-metrics.sh`

Re: [VOTE] KIP-1000: List Client Metrics Configuration Resources

Hi, Andrew, Thanks for the KIP. +1 Jun On Thu, Nov 16, 2023 at 9:12 AM Andrew Schofield < andrew_schofield_j...@outlook.com> wrote: > Hi Apoorv, > Thanks for your vote. > > Initially, I put support for zkBroker in order to be able to control the > error response in this case. > I have

Re: [DISCUSS] KIP-1004: Enforce tasks.max property in Kafka Connect

Hi Hector, Thanks for taking a look! I think the key difference between the proposed behavior and the rejected alternative is that the set of tasks that will be running with the former is still a complete set of tasks, whereas the set of tasks in the latter is a subset of tasks. Also noteworthy

Re: [VOTE] KIP-968: Support single-key_multi-timestamp interactive queries (IQv2) for versioned state stores

Hi all, I added the public enum `ResultOrder` to the KIP which helps with keeping three values (unordered, ascending, and descending) for the query results. Therefore the method `isAscending()` is changed to `resultOrder()` which returns either the user specified result order or `unorderd`.

Requesting permissions to contribute to Apache Kafka

I would like to request permission to contribute to Apache Kafka. wiki ID: rpradeep JIRA ID: rpradeep Thanks, Ria

[jira] [Created] (KAFKA-15859) Introduce delayed remote list offsets to make LIST_OFFSETS async

Kamal Chandraprakash created KAFKA-15859: Summary: Introduce delayed remote list offsets to make LIST_OFFSETS async Key: KAFKA-15859 URL: https://issues.apache.org/jira/browse/KAFKA-15859

[jira] [Resolved] (KAFKA-15174) Ensure the correct thread is executing the callbacks

[ https://issues.apache.org/jira/browse/KAFKA-15174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Nee resolved KAFKA-15174. Resolution: Fixed > Ensure the correct thread is executing the callbacks >

Re:[DISCUSS] KIP-1004: Enforce tasks.max property in Kafka Connect

Thanks for the KIP Chris, adding this check makes total sense. I do have one question. The second paragraph in the Public Interfaces section states: "If the connector generated excessive tasks after being reconfigured, then any existing tasks for the connector will be allowed to continue

[jira] [Created] (KAFKA-15858) Broker stays fenced until all assignments are correct

Igor Soarez created KAFKA-15858: --- Summary: Broker stays fenced until all assignments are correct Key: KAFKA-15858 URL: https://issues.apache.org/jira/browse/KAFKA-15858 Project: Kafka Issue

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2398

See Changes: -- [...truncated 440680 lines...] Gradle Test Run :core:test > Gradle Test Executor 66 > ZkMigrationClientTest > testReadMigrateAndWriteProducerId() PASSED

Re: Contributor permissions request

Hi, I've granted you permissions in Jira and the wiki. Thanks, Mickael On Mon, Nov 20, 2023 at 3:28 PM wrote: > > Requesting contributor permissions. > > JIRA username: navdeep > GitHub username: navdeepsekhon > Wiki username: navdeep > > > Thanks, > > Navdeep

Contributor permissions request

Requesting contributor permissions. JIRA username: navdeep GitHub username: navdeepsekhon Wiki username: navdeep Thanks, Navdeep

Re: [DISCUSS] KIP-994: Minor Enhancements to ListTransactions and DescribeTransactions APIs

Thanks Artem, I have made changes to the `Public Interfaces` and `Compatibility...` sections to incorporate your comment. On Mon, Nov 6, 2023 at 3:44 PM Raman Verma wrote: > I would like to start a discussion on KIP-994 > > >

Re: [DISCUSS] KIP-1005: Add EarliestLocalOffset to GetOffsetShell

Hi Christo, Thanks for the KIP! Similar to the earliest-local-log offset, can we also expose the highest-copied-remote-offset via GetOffsetShell tool? This will be useful during the debugging session. On Mon, Nov 20, 2023 at 5:38 PM Christo Lolov wrote: > Hello all! > > I would like to start

Re: [VOTE] KIP-968: Support single-key_multi-timestamp interactive queries (IQv2) for versioned state stores

Thank you, Guozhag and Bruno, for reviewing the KIP and reading the whole discussion thread. I appreciate your help:) The KIP is now corrected and updated. Cheers, Alieh On Mon, Nov 20, 2023 at 10:43 AM Bruno Cadonna wrote: > Thanks Alieh, > > I am +1 (binding). > > However, although we agreed

Re: How Kafka handle partition leader change?

Hey De Gao, The controller is the one that always elects a new leader. When that happens that metadata is changed on the controller and once committed it’s broadcast to all brokers in the cluster. In KRaft this would be via a PartitonChange record that each broker will fetch from the

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2397

See

Re: [DISCUSS] Road to Kafka 4.0

Hi all, I wanted to share my opinion regarding this topic. I know some discussions happened some time ago (over a year) but I believe it's wise to reflect and re-evaluate if those decisions are still valid. KRaft, as of Kafka 3.6.x and 3.7.x, has not yet feature parity with Zookeeper. By

[DISCUSS] KIP-1005: Add EarliestLocalOffset to GetOffsetShell

Hello all! I would like to start a discussion for https://cwiki.apache.org/confluence/display/KAFKA/KIP-1005%3A+Add+EarliestLocalOffset+to+GetOffsetShell . A new offset called local log start offset was introduced as part of KIP-405: Kafka Tiered Storage. KIP-1005 aims to expose this offset by

[VOTE] KIP-963: Additional metrics in Tiered Storage

Hello all! Now that the discussion for KIP-963 has winded down, I would like to open it for a vote targeting 3.7.0 as the release. You can find the current version of the KIP at https://cwiki.apache.org/confluence/display/KAFKA/KIP-963%3A+Additional+metrics+in+Tiered+Storage Best, Christo

[jira] [Created] (KAFKA-15857) Introduce LocalLogStartOffset and TieredOffset in OffsetSpec.

Satish Duggana created KAFKA-15857: -- Summary: Introduce LocalLogStartOffset and TieredOffset in OffsetSpec. Key: KAFKA-15857 URL: https://issues.apache.org/jira/browse/KAFKA-15857 Project: Kafka

Re: [DISCUSS] KIP-963: Upload and delete lag metrics in Tiered Storage

Hi Christo, On RemoteDeleteBytesPerSec: I think for Delete operations bytes represent the same as for Copy operations. We call copyLogSegment, but the bytes written can be different from the log segment size. We could have the same for Delete to get an idea of the amount of data delete from

Re: [DISCUSS] KIP-968: Support single-key_multi-timestamp interactive queries (IQv2) for versioned state stores

Hi Alieh, Although, I've already voted, I found a minor miss. You should also add a method isDescending() since the results could also be unordered now that we agreed that the results are unordered by default. If both -- isDescending() and isAscending -- are false neither

Re: [DISCUSS] KIP-963: Upload and delete lag metrics in Tiered Storage

Hi Christo, I think we can start the vote thread on the KIP which is updated with the finalized metrics. We can have followup KIPs with other metrics if needed in future. Thanks, Satish. On Fri, 17 Nov 2023 at 22:16, Christo Lolov wrote: > Heya all! > > I have updated the KIP so please have

Re: [VOTE] KIP-968: Support single-key_multi-timestamp interactive queries (IQv2) for versioned state stores

Thanks Alieh, I am +1 (binding). However, although we agreed on not specifying an order of the results by default, there is still the following sentence in the KIP: "The order of the returned records is by default ascending by timestamp. The method withDescendingTimestamps() can reverse

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2396

See