[jira] [Created] (KAFKA-16780) Txn consumer exerts pressure on remote storage when reading non-txn topic

Kamal Chandraprakash created KAFKA-16780: Summary: Txn consumer exerts pressure on remote storage when reading non-txn topic Key: KAFKA-16780 URL: https://issues.apache.org/jira/browse/KAFKA-16780

Re: [VOTE] KIP-950: Tiered Storage Disablement

Hi Christo, Thanks for the KIP and incorporating the review comments. Please update the KIP with the latest details. +1 (non-binding). Thanks, Kamal On Wed, May 15, 2024 at 3:18 PM Luke Chen wrote: > Hi Christo, > > In addition to the minor comments left in the discussion thread, it LGTM.

Re: [VOTE] KIP-1018: Introduce max remote fetch timeout config

rote: > > > > > +1 non binding > > > > > > Thanks > > > > > > On Thu, May 9, 2024 at 12:05 PM Luke Chen wrote: > > > > > > > > Hi Kamal, > > > > > > > > Thanks for the KIP! > >

[jira] [Created] (KAFKA-16696) Remove the in-memory implementation of RSM and RLMM

Kamal Chandraprakash created KAFKA-16696: Summary: Remove the in-memory implementation of RSM and RLMM Key: KAFKA-16696 URL: https://issues.apache.org/jira/browse/KAFKA-16696 Project: Kafka

Re: [DISCUSS] KIP-950: Tiered Storage Disablement

e. Deletion when the policy is retain happens asynchronously due to > expiration. I think that deletion when the policy is delete ought to (at > least for the initial implementation) happen synchronously. Should people > run into timeout problems we can always then have a follow-up KIP wher

Re: [DISCUSS] KIP-950: Tiered Storage Disablement

to delete is huge. We can further extend the same approach for the topic deletion requests. Also, Could you please update the state diagram about the transitions? It is not clear when to transit from DISABLING to DISABLED state? -- Kamal On Mon, May 6, 2024 at 6:55 PM Kamal Chandraprakash < k

[VOTE] KIP-1018: Introduce max remote fetch timeout config

Hi all, We would like to start a voting thread for KIP-1018: Introduce max remote fetch timeout config for DelayedRemoteFetch requests. The KIP is available on https://cwiki.apache.org/confluence/display/KAFKA/KIP-1018%3A+Introduce+max+remote+fetch+timeout+config+for+DelayedRemoteFetch+requests

Re: [DISCUSS] KIP-1018: Introduce max remote fetch timeout config

Hi all, If there are no more comments, I'll open a VOTE thread. -- Kamal On Sat, May 4, 2024 at 8:39 AM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Luke, > > Thanks for the review! > > DelayedFetch and DelayedRemoteFetch are orthogonal calls > &l

Re: [DISCUSS] KIP-1018: Introduce max remote fetch timeout config

> > possible > > > to add 2 metrics, one at the RequestMetrics level (even if it is > > > total-time-ms - (all other times)) and one at what you are proposing? > As > > an > > > operator I would find it strange to not see the metric in the > > > Request

Re: [DISCUSS] KIP-1018: Introduce max remote fetch timeout config

ge to not see the metric in the > RequestMetrics. > > Your thoughts? > > Best, > Christo > > On Sun, 28 Apr 2024 at 10:52, Kamal Chandraprakash < > kamal.chandraprak...@gmail.com> wrote: > > > Christo, > > > > Updated the KIP with the remote fetch lat

Re: [DISCUSS] KIP-1018: Introduce max remote fetch timeout config

Christo, Updated the KIP with the remote fetch latency metric. Please take another look! -- Kamal On Sun, Apr 28, 2024 at 12:23 PM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Hi Federico, > > Thanks for the suggestion! Updated the config name to " > r

Re: [DISCUSS] KIP-1018: Introduce max remote fetch timeout config

r 27, 2024 at 4:03 PM Federico Valeri wrote: > Hi Kamal, it looks like all TS configurations starts with "remote." > prefix, so I was wondering if we should name it > "remote.fetch.max.wait.ms". > > On Fri, Apr 26, 2024 at 7:07 PM Kamal Chandraprakash > wro

Re: [DISCUSS] KIP-1018: Introduce max remote fetch timeout config

Hi all, If there are no more comments, I'll start a vote thread by tomorrow. Please review the KIP. Thanks, Kamal On Sat, Mar 30, 2024 at 11:08 PM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Hi all, > > Bumping the thread. Please review this KIP. Thanks! &g

Re: [VOTE] KIP-1023: Follower fetch from tiered offset

+1 (non-binding). Thanks for the KIP! -- Kamal On Fri, Apr 26, 2024 at 11:00 AM Abhijeet Kumar wrote: > Hi All, > > I would like to start the vote for KIP-1023 - Follower fetch from tiered > offset > > The KIP is here: > >

Re: [ANNOUNCE] New committer: Igor Soarez

Congratulations, Igor! On Thu, Apr 25, 2024, 06:11 Luke Chen wrote: > Congrats, Igor! > > On Thu, Apr 25, 2024 at 6:10 AM Matthias J. Sax wrote: > > > Congrats! > > > > On 4/24/24 2:29 PM, Bill Bejeck wrote: > > > Congrats Igor! > > > > > > -Bill > > > > > > On Wed, Apr 24, 2024 at 2:37 PM Tom

[jira] [Created] (KAFKA-16605) Fix the flaky LogCleanerParameterizedIntegrationTest

Kamal Chandraprakash created KAFKA-16605: Summary: Fix the flaky LogCleanerParameterizedIntegrationTest Key: KAFKA-16605 URL: https://issues.apache.org/jira/browse/KAFKA-16605 Project: Kafka

Re: [ANNOUNCE] New Kafka PMC Member: Greg Harris

Congratulations, Greg! On Sun, Apr 14, 2024 at 8:57 AM Yash Mayya wrote: > Congrats Greg! > > On Sun, 14 Apr, 2024, 05:56 Randall Hauch, wrote: > > > Congratulations, Greg! > > > > On Sat, Apr 13, 2024 at 6:36 PM Luke Chen wrote: > > > > > Congrats, Greg! > > > > > > On Sun, Apr 14, 2024 at

Re: [DISCUSS] KIP-1026: Handling producer snapshot when upgrading from < v2.8.0 for Tiered Storage

; Thanks. > Luke > > > On Wed, Apr 3, 2024 at 11:59 PM Arpit Goyal > wrote: > >> Thanks @Kamal Chandraprakash Greg >> Harris >> I currently do not have detailed understanding on the behaviour when empty >> producer snapshot restored. I will try to

[jira] [Resolved] (KAFKA-16456) Can't stop kafka debug logs

[ https://issues.apache.org/jira/browse/KAFKA-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamal Chandraprakash resolved KAFKA-16456. -- Resolution: Not A Problem You can also dynamically change the broker loggers

[jira] [Created] (KAFKA-16454) Snapshot the state of remote log metadata for all the partitions

Kamal Chandraprakash created KAFKA-16454: Summary: Snapshot the state of remote log metadata for all the partitions Key: KAFKA-16454 URL: https://issues.apache.org/jira/browse/KAFKA-16454

Re: [DISCUSS] KIP-1007: Introduce Remote Storage Not Ready Exception

> exception is able to understand what it means. We can discuss that during > the PR review though. > > The KIP looks good to me. > > -- > Divij Vaidya > > > > On Fri, Jan 5, 2024 at 10:44 AM Satish Duggana > wrote: > > > Thanks for the KIP Kamal, LGTM. &

Re: [VOTE] KIP-1007: Introduce Remote Storage Not Ready Exception

n/java/org/apache/kafka/clients/consumer/internals/FetchCollector.java?L325 On Sun, Jan 7, 2024 at 5:36 PM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Thank you for all the votes! I'm closing the vote thread as it is open for > more than 72 hours. > The KIP h

[jira] [Created] (KAFKA-16452) Bound highwatermark offset to range b/w local-log-start-offset and log-end-offset

Kamal Chandraprakash created KAFKA-16452: Summary: Bound highwatermark offset to range b/w local-log-start-offset and log-end-offset Key: KAFKA-16452 URL: https://issues.apache.org/jira/browse/KAFKA-16452

Re: [DISCUSS] KIP-1018: Introduce max remote fetch timeout config

Hi all, Bumping the thread. Please review this KIP. Thanks! On Thu, Feb 1, 2024 at 9:11 PM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Hi Jorge, > > Thanks for the review! Added your suggestions to the KIP. PTAL. > > The `fetch.max.wait.ms` config wi

Re: [DISCUSS] KIP-1021: Allow to get last stable offset (LSO) in kafka-get-offsets.sh

Hi Ahmed, Thanks for the KIP! It will be a useful addition to know the last-stable-offset (LSO) for a partition. Motivation section is not clear to me. Can you document the scenarios on how the exposed LSO can be used for txn support/debug? -- Kamal On Thu, Mar 21, 2024 at 11:30 PM Justine

Re: [DISCUSS] KIP-950: Tiered Storage Disablement

Hi, Thanks for the KIP! Overall the KIP looks good and covered most of the items. 1. Could you explain how the brokers will handle the DisableRemoteTopic API request? 2. Who will initiate the controller interaction sequence? Does the controller listens for topic config updates and initiate the

Re: [ANNOUNCE] New committer: Christo Lolov

Congrats Christo! On Tue, Mar 26, 2024, 17:42 Bruno Cadonna wrote: > Congrats, Christo! > > Well deserved! > > Best, > Bruno > > On 3/26/24 1:04 PM, Luke Chen wrote: > > Hi, Everyone, > > > > The PMC of Apache Kafka is pleased to announce a new Kafka committer: > > Christo Lolov. > > > >

Re: [DISCUSS] KIP-1026: Handling producer snapshot when upgrading from < v2.8.0 for Tiered Storage

ired for topics migrated from < 2.8 > > version. This will not break the existing contract with the plugin. Yes, > > metrics do not make sense here as of now. > > Greg, @Kamal Chandraprakash WDYT ? > > Arpit Goyal > > 8861094754 > > > > > > On Sat, M

Re: [DISCUSS] KIP-1026: Handling producer snapshot when upgrading from < v2.8.0 for Tiered Storage

Hi Arpit, Thanks for the KIP! There is an open ticket [1] to generate the empty producer snapshot for segments which lacks one. Tiered storage is supported from IBP 2.8-IV1 which mandates that all the segments should have the producer snapshots. If we make the producer snapshot optional, then we

Re: [DISCUSS] Minimum constraint for segment.ms

One use case I see for setting the `segment.bytes` to 1 is to delete all the records from the topic. We can mention about it in the doc to use the `kafka-delete-records` API instead. On Wed, Mar 13, 2024 at 6:59 PM Divij Vaidya wrote: > + users@kafka > > Hi users of Apache Kafka > > With the

Re: [VOTE] KIP-956: Tiered Storage Quotas

+1 (non-binding), Thanks for the KIP, Abhijeet! -- Kamal On Fri, Mar 8, 2024 at 11:02 PM Jun Rao wrote: > Hi, Abhijeet, > > Thanks for the KIP. +1 > > Jun > > On Fri, Mar 8, 2024 at 3:44 AM Abhijeet Kumar > wrote: > > > Hi All, > > > > I would like to start the vote for KIP-956 - Tiered

Re: [ANNOUNCE] Apache Kafka 3.7.0

Alok Thatikunta, Alyssa Huang, Aman Singh, Andras Katona, Andrew > > > > Schofield, Anna Sophie Blee-Goldman, Anton Agestam, Apoorv Mittal, > > > > Arnout Engelen, Arpit Goyal, Artem Livshits, Ashwin Pankaj, > > > > ashwinpankaj, atu-sharm, bachmanit

Re: [DISCUSS] KIP-1018: Introduce max remote fetch timeout config

jira/browse/KAFKA-15776 > [2] https://github.com/apache/kafka/pull/14778#issuecomment-1820588080 > > On Tue, 30 Jan 2024 at 01:01, Kamal Chandraprakash < > kamal.chandraprak...@gmail.com> wrote: > > > Hi all, > > > > I have opened a KIP-1018 > > < > &

[DISCUSS] KIP-1018: Introduce max remote fetch timeout config

Hi all, I have opened a KIP-1018 to introduce dynamic max-remote-fetch-timeout broker config to give more control to the operator.

Re: [VOTE] KIP-1011: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh

+1 (non-binding) On Mon, Jan 22, 2024 at 8:34 AM ziming deng wrote: > Hello everyone, > I'd like to initiate a vote for KIP-1011. > This KIP is about replacing alterConfigs with incrementalAlterConfigs when > updating broker configs using kafka-configs.sh, this is similar to what we > have done

[jira] [Created] (KAFKA-16146) Checkpoint log-start-offset after remote log deletion

Kamal Chandraprakash created KAFKA-16146: Summary: Checkpoint log-start-offset after remote log deletion Key: KAFKA-16146 URL: https://issues.apache.org/jira/browse/KAFKA-16146 Project: Kafka

Re: [VOTE] KIP-1007: Introduce Remote Storage Not Ready Exception

will be a more explicit > > > > behaviour than what Kafka does today - I am happy with it! > > > > > > > > +1 (non-binding) from me > > > > > > > > Best, > > > > Christo > > > > > > > > On Tue, 12

Re: [VOTE] KIP-1013: Drop broker and tools support for Java 11 in Kafka 4.0 (deprecate in 3.7)

+1 (non-binding). On Wed, Jan 3, 2024 at 8:01 AM Satish Duggana wrote: > Thanks Ismael for the proposal. > > Adopting JDK 17 enhances developer productivity and has reached a > level of maturity that has led to its adoption by several other major > projects, signifying its reliability and

Re: [ANNOUNCE] New Kafka PMC Member: Divij Vaidya

Congrats Divij! On Thu, Dec 28, 2023, 07:09 Kirk True wrote: > Congrats Divij!!! > > On Wed, Dec 27, 2023, at 1:44 PM, Jorge Esteban Quilcate Otoya wrote: > > Congratulations Divij!! > > > > On Wed 27. Dec 2023 at 14.56, Tom Bentley wrote: > > > > > Congratulations! > > > > > > On Thu, 28 Dec

Re: [VOTE] KIP-1005: Expose EarliestLocalOffset and TieredOffset

+1 (non-binding). Thanks for the KIP! -- Kamal On Thu, Dec 21, 2023 at 2:23 PM Christo Lolov wrote: > Heya all! > > KIP-1005 ( > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1005%3A+Expose+EarliestLocalOffset+and+TieredOffset > ) > has been open for around a month with no further

Re: DISCUSS KIP-1011: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh

Hi Ziming, Thanks for the KIP! The proposal LGTM. I'm also inclined towards option 2 (i.e. add an explicit --enable-incremental flag in 3.X version) to avoid any incompatible change in v3.X. As mentioned in this thread, many users might be using external tools to do the topic rebalance and apply

Re: [DISCUSS] KIP-1007: Introduce Remote Storage Not Ready Exception

t; > [1] > > https://github.com/apache/kafka/blob/82808873cbf6a95611243c2e7984c4aa6ff2cfff/core/src/main/scala/kafka/log/UnifiedLog.scala#L1336 > > > -- > Divij Vaidya > > > > On Tue, Dec 12, 2023 at 9:58 AM Kamal Chandraprakash < > kamal.chandraprak...@gmail.com> wrote: > > > Thanks

[VOTE] KIP-1007: Introduce Remote Storage Not Ready Exception

Hi, I would like to call a vote for KIP-1007 . This KIP aims to introduce a new error code for retriable remote storage errors. Thanks to everyone who reviewed the KIP! -- Kamal

Re: [DISCUSS] KIP-1007: Introduce Remote Storage Not Ready Exception

Thanks Luke for reviewing this KIP! If there are no more comments from others, I'll start the VOTE since this is a minor KIP. On Mon, Dec 11, 2023 at 1:01 PM Luke Chen wrote: > Hi Kamal, > > Thanks for the KIP! > LGTM. > > Thanks. > Luke > > On Wed, Nov 22, 2023 at 7

Re: [VOTE] 3.6.1 RC0

+1 (non-binding) 1. Built the source from 3.6.1-rc0 tag in scala 2.12 and 2.13 2. Ran all the unit and integration tests. 3. Ran quickstart and verified the produce-consume on a 3 node cluster. 4. Verified the tiered storage functionality with local-tiered storage. On Tue, Nov 28, 2023 at 12:55 

[DISCUSS] KIP-1007: Introduce Remote Storage Not Ready Exception

Hi, I would like to start a discussion to introduce a new error code for retriable remote storage errors. Please take a look at the proposal: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1007%3A+Introduce+Remote+Storage+Not+Ready+Exception

[jira] [Created] (KAFKA-15876) Introduce Remote Storage Not Ready Exception

Kamal Chandraprakash created KAFKA-15876: Summary: Introduce Remote Storage Not Ready Exception Key: KAFKA-15876 URL: https://issues.apache.org/jira/browse/KAFKA-15876 Project: Kafka

Re: [VOTE] KIP-963: Additional metrics in Tiered Storage

+1 (non-binding). Thanks for the KIP! On Tue, Nov 21, 2023, 03:04 Divij Vaidya wrote: > + 1 (binding) > > This Kip will greatly improve Tiered Storage troubleshooting. Thank you > Christo. > > On Mon 20. Nov 2023 at 17:21, Christo Lolov > wrote: > > > Hello all! > > > > Now that the discussion

[jira] [Created] (KAFKA-15859) Introduce delayed remote list offsets to make LIST_OFFSETS async

Kamal Chandraprakash created KAFKA-15859: Summary: Introduce delayed remote list offsets to make LIST_OFFSETS async Key: KAFKA-15859 URL: https://issues.apache.org/jira/browse/KAFKA-15859

Re: [DISCUSS] KIP-1005: Add EarliestLocalOffset to GetOffsetShell

Hi Christo, Thanks for the KIP! Similar to the earliest-local-log offset, can we also expose the highest-copied-remote-offset via GetOffsetShell tool? This will be useful during the debugging session. On Mon, Nov 20, 2023 at 5:38 PM Christo Lolov wrote: > Hello all! > > I would like to start

Re: RemoteStorageManager.fetchLogSegment - how to deal with InterruptedException?

Hi Ivan, I've opened a relevant patch to increase the timeout to fetch data from remote storage. This will reduce the error occurrence rate: https://github.com/apache/kafka/pull/14778 I think it's better to distinguish the cases: 1) Any error that happens while reading the remote data -- this

Re: [VOTE] KIP-997: Partition-Level Throughput Metrics

+1 (non-binding). Thanks for the KIP! On Thu, Nov 16, 2023 at 9:00 AM Satish Duggana wrote: > Thanks Qichao for the KIP. > > +1 (binding) > > ~Satish. > > On Thu, 16 Nov 2023 at 02:20, Jorge Esteban Quilcate Otoya > wrote: > > > > Qichao, thanks again for leading this proposal! > > > > +1

[jira] [Resolved] (KAFKA-15376) Explore options of removing data earlier to the current leader's leader epoch lineage for topics enabled with tiered storage.

[ https://issues.apache.org/jira/browse/KAFKA-15376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamal Chandraprakash resolved KAFKA-15376. -- Resolution: Fixed This task was already addressed in the code, so closing

[jira] [Created] (KAFKA-15777) Configurable remote fetch bytes per partition from Consumer

Kamal Chandraprakash created KAFKA-15777: Summary: Configurable remote fetch bytes per partition from Consumer Key: KAFKA-15777 URL: https://issues.apache.org/jira/browse/KAFKA-15777 Project

[jira] [Created] (KAFKA-15776) Configurable delay timeout for DelayedRemoteFetch request

Kamal Chandraprakash created KAFKA-15776: Summary: Configurable delay timeout for DelayedRemoteFetch request Key: KAFKA-15776 URL: https://issues.apache.org/jira/browse/KAFKA-15776 Project

Re: [DISCUSS] KIP-963: Upload and delete lag metrics in Tiered Storage

t 2023 at 10:49, Gantigmaa Selenge < > > gsele...@redhat.com> > > > > > wrote: > > > > > > > > > >> Hi Christo, > > > > >> > > > > >> Thank you for writing the KIP. > > > > >> > &

Re: [ANNOUNCE] New Kafka PMC Member: Satish Duggana

Congratulations Satish! On Fri, Oct 27, 2023, 21:10 Jorge Esteban Quilcate Otoya < quilcate.jo...@gmail.com> wrote: > Congratulations Satish!! > > On Fri, 27 Oct 2023 at 18:38, Mickael Maison > wrote: > > > Congratulations Satish! > > > > On Fri, Oct 27, 2023 at 5:18 PM Lucas Brutschy > >

[jira] [Created] (KAFKA-15703) Update Highwatermark while building the remote log auxillary state

Kamal Chandraprakash created KAFKA-15703: Summary: Update Highwatermark while building the remote log auxillary state Key: KAFKA-15703 URL: https://issues.apache.org/jira/browse/KAFKA-15703

[jira] [Created] (KAFKA-15682) Ensure internal remote log metadata topic does not expire its segments before deleting user-topic segments

Kamal Chandraprakash created KAFKA-15682: Summary: Ensure internal remote log metadata topic does not expire its segments before deleting user-topic segments Key: KAFKA-15682 URL: https

Re: [VOTE] KIP-979 Allow independently stop KRaft processes

+1 (non-binding). Thanks for the KIP! On Mon, Oct 23, 2023, 12:55 Hailey Ni wrote: > Hi all, > > I'd like to call a vote on KIP-979 that will allow users to independently > stop KRaft processes. > > >

[jira] [Created] (KAFKA-15632) Drop the invalid remote log metadata events

Kamal Chandraprakash created KAFKA-15632: Summary: Drop the invalid remote log metadata events Key: KAFKA-15632 URL: https://issues.apache.org/jira/browse/KAFKA-15632 Project: Kafka

Re: [DISCUSS] KIP-979: Allow independently stop KRaft controllers or brokers

Hi Hailey, Thanks for working on this! This is one of the long-standing open issues. Now, users have to find the PID of the respective Kafka process to stop if more than one node is being run locally for testing purposes. The updated KIP is addressing that. LGTM. Is `node.id` and `broker.id` the

[jira] [Created] (KAFKA-15499) Fix the flaky DeleteSegmentsDueToLogStartOffsetBreachTest

Kamal Chandraprakash created KAFKA-15499: Summary: Fix the flaky DeleteSegmentsDueToLogStartOffsetBreachTest Key: KAFKA-15499 URL: https://issues.apache.org/jira/browse/KAFKA-15499 Project

Re: [ANNOUNCE] New Kafka PMC Member: Justine Olshan

Congrats Justine! On Sat, Sep 23, 2023, 13:28 Divij Vaidya wrote: > Congratulations Justine! > > On Sat 23. Sep 2023 at 07:06, Chris Egerton > wrote: > > > Congrats Justine! > > On Fri, Sep 22, 2023, 20:47 Guozhang Wang > > wrote: > > > > > Congratulations! > > > > > > On Fri, Sep 22, 2023 at

Re: [ANNOUNCE] New committer: Lucas Brutschy

Congrats Lucas! On Thu, Sep 21, 2023, 22:05 Boudjelda Mohamed Said wrote: > Congratulations, Lucas!! > > On Thu 21 Sep 2023 at 18:34, Lianet M. wrote: > > > Congratulations Lucas! > > > > On Thu, Sept 21, 2023, 11:45 a.m. Bruno Cadonna > > wrote: > > > > > Hi all, > > > > > > The PMC of

Re: [ANNOUNCE] New committer: Yash Mayya

Congratulations Yash! On Thu, Sep 21, 2023, 22:03 Bill Bejeck wrote: > Congrats Yash! > > On Thu, Sep 21, 2023 at 12:26 PM Divij Vaidya > wrote: > > > Congratulations Yash! > > > > Divij Vaidya > > > > > > On Thu, Sep 21, 2023 at 6:18 PM Sagar wrote: > > > > > > Congrats Yash ! > > > On Thu,

[jira] [Created] (KAFKA-15479) Remote log segments should be considered once during retention size breach

Kamal Chandraprakash created KAFKA-15479: Summary: Remote log segments should be considered once during retention size breach Key: KAFKA-15479 URL: https://issues.apache.org/jira/browse/KAFKA-15479

[jira] [Created] (KAFKA-15432) RLM Stop partitions should not be invoked for non-tiered storage topics

Kamal Chandraprakash created KAFKA-15432: Summary: RLM Stop partitions should not be invoked for non-tiered storage topics Key: KAFKA-15432 URL: https://issues.apache.org/jira/browse/KAFKA-15432

[jira] [Created] (KAFKA-15431) Add support to assert offloaded segment for already produced event in Tiered Storage Framework

Kamal Chandraprakash created KAFKA-15431: Summary: Add support to assert offloaded segment for already produced event in Tiered Storage Framework Key: KAFKA-15431 URL: https://issues.apache.org/jira

[jira] [Resolved] (KAFKA-15404) Failing Test DynamicBrokerReconfigurationTest#testThreadPoolResize

[ https://issues.apache.org/jira/browse/KAFKA-15404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamal Chandraprakash resolved KAFKA-15404. -- Resolution: Fixed > Failing Test DynamicBrokerReconfigurationT

[jira] [Reopened] (KAFKA-15399) Enable OffloadAndConsumeFromLeader test

[ https://issues.apache.org/jira/browse/KAFKA-15399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamal Chandraprakash reopened KAFKA-15399: -- > Enable OffloadAndConsumeFromLeader t

[jira] [Created] (KAFKA-15421) Enable DynamicBrokerReconfigurationTest#testThreadPoolResize test

Kamal Chandraprakash created KAFKA-15421: Summary: Enable DynamicBrokerReconfigurationTest#testThreadPoolResize test Key: KAFKA-15421 URL: https://issues.apache.org/jira/browse/KAFKA-15421

[jira] [Created] (KAFKA-15410) Add basic functionality integration test with tiered storage

Kamal Chandraprakash created KAFKA-15410: Summary: Add basic functionality integration test with tiered storage Key: KAFKA-15410 URL: https://issues.apache.org/jira/browse/KAFKA-15410 Project

Re: [DISCUSS] KIP-972: Add the metric of the current running version of kafka

Hi Hudeqi, Kafka already emits the version metric. Can you check whether the below metric satisfies your requirement? kafka.server:type=app-info,id=0 -- Kamal On Mon, Aug 28, 2023 at 2:29 PM hudeqi <16120...@bjtu.edu.cn> wrote: > Hi, all, I want to submit a minor kip to add a metric, which

[jira] [Created] (KAFKA-15400) Fix flaky RemoteIndexCache test

Kamal Chandraprakash created KAFKA-15400: Summary: Fix flaky RemoteIndexCache test Key: KAFKA-15400 URL: https://issues.apache.org/jira/browse/KAFKA-15400 Project: Kafka Issue Type

[jira] [Created] (KAFKA-15399) Enable OffloadAndConsumeFromLeader test

Kamal Chandraprakash created KAFKA-15399: Summary: Enable OffloadAndConsumeFromLeader test Key: KAFKA-15399 URL: https://issues.apache.org/jira/browse/KAFKA-15399 Project: Kafka

[jira] [Created] (KAFKA-15352) Ensure consistency while deleting the remote log segments

Kamal Chandraprakash created KAFKA-15352: Summary: Ensure consistency while deleting the remote log segments Key: KAFKA-15352 URL: https://issues.apache.org/jira/browse/KAFKA-15352 Project

[jira] [Created] (KAFKA-15351) Update log-start-offset after leader election for topics enabled with remote storage

Kamal Chandraprakash created KAFKA-15351: Summary: Update log-start-offset after leader election for topics enabled with remote storage Key: KAFKA-15351 URL: https://issues.apache.org/jira/browse/KAFKA

[jira] [Created] (KAFKA-15331) Handle remote log enabled topic deletion when leader is not available

Kamal Chandraprakash created KAFKA-15331: Summary: Handle remote log enabled topic deletion when leader is not available Key: KAFKA-15331 URL: https://issues.apache.org/jira/browse/KAFKA-15331

Re: [DISCUSS] KIP-963: Upload and delete lag metrics in Tiered Storage

Hi Christo, Thanks for the KIP! The proposed tiered storage metrics are useful. The unit mentioned in the KIP is the number of records. Each topic can have varying amounts of records in a segment depending on the record size. Do you think having the tier-lag by number of segments (or) size of

[jira] [Created] (KAFKA-15295) Add config validation when remote storage is enabled on a topic

Kamal Chandraprakash created KAFKA-15295: Summary: Add config validation when remote storage is enabled on a topic Key: KAFKA-15295 URL: https://issues.apache.org/jira/browse/KAFKA-15295

[jira] [Created] (KAFKA-15290) Add support to onboard existing topics to tiered storage

Kamal Chandraprakash created KAFKA-15290: Summary: Add support to onboard existing topics to tiered storage Key: KAFKA-15290 URL: https://issues.apache.org/jira/browse/KAFKA-15290 Project

[jira] [Created] (KAFKA-15272) Fix the logic which finds candidate log segments to upload it to tiered storage

Kamal Chandraprakash created KAFKA-15272: Summary: Fix the logic which finds candidate log segments to upload it to tiered storage Key: KAFKA-15272 URL: https://issues.apache.org/jira/browse/KAFKA-15272

Re: [VOTE] KIP-959 Add BooleanConverter to Kafka Connect

+1 (non-binding). Thanks for the KIP! On Tue, Jul 25, 2023 at 11:12 PM Yash Mayya wrote: > Hi Hector, > > Thanks for the KIP! > > +1 (non-binding) > > Thanks, > Yash > > On Tue, Jul 25, 2023 at 11:01 PM Andrew Schofield < > andrew_schofield_j...@outlook.com> wrote: > > > Thanks for the KIP. As

Re: [VOTE] KIP-930: Tiered Storage Metrics

+1 (non-binding) -- Kamal On Tue, Jul 25, 2023 at 11:30 AM Abhijeet Kumar wrote: > Hi All, > > I would like to start the vote for KIP-930 Tiered Storage Metrics. > > The KIP is here: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-930%3A+Tiered+Storage+Metrics > > Regards > Abhijeet.

Re: [DISCUSS] KIP-930: Tiered Storage Metrics

Hi Abhijeet, Thanks for the KIP! We are changing the metric names from what was proposed in the KIP-405 and adding new metrics for RemoteIndexCache. In the KIP, it's not clear whether we are renaming the aggregate broker level metrics for remote copy/fetch/failed-copy/failed-fetch. Are these

Re: [ANNOUNCE] Apache Kafka 3.5.1

Thanks for running the release, Divij! On Mon, Jul 24, 2023 at 3:17 PM Divij Vaidya wrote: > The Apache Kafka community is pleased to announce the release for Apache > Kafka 3.5.1 > > This release is a security patch release. It upgrades the dependency, > snappy-java, to a version which is not

[jira] [Created] (KAFKA-15194) Rename local tiered storage segment with offset as prefix for easy navigation

Kamal Chandraprakash created KAFKA-15194: Summary: Rename local tiered storage segment with offset as prefix for easy navigation Key: KAFKA-15194 URL: https://issues.apache.org/jira/browse/KAFKA-15194

[jira] [Created] (KAFKA-15168) Handle overlapping remote log segments in RemoteLogMetadata cache

Kamal Chandraprakash created KAFKA-15168: Summary: Handle overlapping remote log segments in RemoteLogMetadata cache Key: KAFKA-15168 URL: https://issues.apache.org/jira/browse/KAFKA-15168

[jira] [Created] (KAFKA-15167) Tiered Storage Test Harness Framework

Kamal Chandraprakash created KAFKA-15167: Summary: Tiered Storage Test Harness Framework Key: KAFKA-15167 URL: https://issues.apache.org/jira/browse/KAFKA-15167 Project: Kafka Issue

[jira] [Created] (KAFKA-15166) Add deletePartition API to the RemoteStorageManager

Kamal Chandraprakash created KAFKA-15166: Summary: Add deletePartition API to the RemoteStorageManager Key: KAFKA-15166 URL: https://issues.apache.org/jira/browse/KAFKA-15166 Project: Kafka

Re: [VOTE] KIP-852: Optimize calculation of size for log in remote tier

+1 (non-binding). Thanks for the KIP! — Kamal On Mon, 7 Nov 2022 at 2:20 AM, John Roesler wrote: > Hi Divij, > > Thanks for the KIP! > > I’ve read through your write-up, and it sounds reasonable to me. > > I’m +1 (binding) > > Thanks, > John > > On Tue, Nov 1, 2022, at 05:03, Divij Vaidya

Re: [DISCUSS] KIP-932: Queues for Kafka

Hi Andrew, Thank you for the KIP -- interesting read. I have some questions: 101. "The calls to KafkaConsumer.acknowledge(ConsumerRecord, AcknowledgeType) must be issued in the order in which the records appear in the ConsumerRecords object, which will be in order of increasing offset for each

Re: [DISCUSS] KIP-852 Optimize calculation of size for log in remote tier

> > The disadvantage of adding this new API is that every provider will > have to > > > implement it, agreed. But I believe that this tradeoff is worthwhile > since > > > the default implementation could be the same as you mentioned, i.e. > keeping > > > cumul

Re: [ANNOUNCE] New committer: Divij Vaidya

Congratulations, Divij!! On Wed, Jun 14, 2023, 19:49 Chris Egerton wrote: > Congrats, Divij! > > On Wed, Jun 14, 2023 at 9:06 AM Edoardo Comar > wrote: > > > Congratulations Divij ! > > > > > > On Wed, 14 Jun 2023 at 10:51, Josep Prat > > wrote: > > > > > Congrats Divij! > > > > > > On Wed,

Re: [VOTE] KIP-917: Additional custom metadata for remote log segment

+1 (non-binding). Thanks for the KIP! On Mon, Jun 12, 2023, 21:23 Divij Vaidya wrote: > I have reviewed the proposal and feel that it would be beneficial to > implement. > > Vote +1 (non-binding) > > > -- > Divij Vaidya > > > > On Mon, Jun 12, 2023 at 4:39 PM Ivan Yurchenko > wrote: > > >

Re: [ANNOUNCE] Apache Kafka 3.4.1

Thanks Luke for running this release! On Wed, Jun 7, 2023 at 8:08 AM Chia-Ping Tsai wrote: > Thank Luke for this hard work!!! > > > Chris Egerton 於 2023年6月7日 上午10:35 寫道： > > > > Thanks for running this release, Luke! > > > > On Tue, Jun 6, 2023, 22:31 Luke Chen wrote: > > > >> The Apache

Re: [DISCUSS] KIP-852 Optimize calculation of size for log in remote tier

Hi Divij, Thanks for the KIP! Sorry for the late reply. Can you explain the rejected alternative-3? Store the cumulative size of remote tier log in-memory at RemoteLogManager "*Cons*: Every time a broker starts-up, it will scan through all the segments in the remote tier to initialise the

Re: [VOTE] KIP-872: Add Serializer#serializeToByteBuffer() to reduce memory copying

+1 (non-binding). Thanks for the improvement! Thanks, Kamal On Wed, May 31, 2023 at 8:23 AM ziming deng wrote: > Hello ShunKang, > > +1(binding) from me > > -- > Thanks, > Ziming > > > On May 30, 2023, at 20:07, ShunKang Lin > wrote: > > > > Hi all, > > > > Bump this thread again and see if

Re: [VOTE] KIP-927: Improve the kafka-metadata-quorum output

+1 (non binding) On Wed, May 17, 2023 at 3:22 AM Divij Vaidya wrote: > +1 (non binding) > > Divij Vaidya > > > > On Tue, May 16, 2023 at 4:35 AM ziming deng > wrote: > > > Thanks for this improvement, +1 from me(binging) > > > > — > > Best, > > Ziming > > > > > On May 16, 2023, at 00:43,

  1   2   >