Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1432

[Apache Jenkins Server]

See 

Re: [DISCUSS] Apache Kafka 3.3.2

[José Armando García Sancio]

Thanks Chris!

I started a build for the system tests against the 3.3 branch using
Confluent's infrastructure. I'll reply here when the results are
avaialbe.

-- 
-José

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1431

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-889 Versioned State Stores

[Victoria Xia]

Hi Matthias,

Thanks for chiming in! Barring objections from anyone on this thread, I
will start the vote for this KIP on Thursday. That should be enough time to
incorporate any lingering minor changes.

> I slightly prefer to add `VersionedRecord` interface (also
like the name). I agree that it's low overhead and providing a clean
path forward for future changes seems worth it to me.

OK, that makes two of us. I updated the KIP just now to formally include
VersionedRecord as the new return type from the various
VersionedKeyValueStore methods.

> if we introduce `VersionedRecord`, I think we can keep the not-null
requirement for `ValueAndTimestamp`

Not quite. VersionedRecord is only used as a return type from read methods,
which is why VersionedRecord is able to enforce that its value is never
null. If the value being returned would have been null, then we return a
null VersionedRecord instead, rather than non-null VersionedRecord with
null value. So, there's no use case for a VersionedRecord with null value.

In contrast, even though ValueAndTimestamp is not anywhere in the public
VersionedKeyValueStore interface, ValueAndTimestamp still needs to be used
internally when representing a versioned key-value store as a
TimestampedKeyValueStore, since TimestampedKeyValueStore is used everywhere
throughout the internals of the codebase. In order to represent a versioned
key-value store as a TimestampedKeyValueStore, we have to support `put(K
key, ValueAndTimestamp value)`, which means ValueAndTimestamp needs to
support null value (with timestamp). Otherwise we cannot put a tombstone
into a versioned key-value store when using the internal
TimestampedKeyValueStore representation.

It's very much an implementation detail that ValueAndTimestamp needs to be
relaxed to allow null values. I think this is a minor enough change that is
still preferable to the alternatives (refactoring the processors to not
require TimestampedKeyValueStore, or introducing a separate workaround
`put()` method on the TimestampedKeyValueStore representation of versioned
key-value stores), so I have left it in as part of the KIP.

Best,
Victoria

On Mon, Dec 12, 2022 at 8:42 PM Matthias J. Sax  wrote:

> Thanks Victoria.
>
> I did not re-read the KIP in full on the wiki but only your email.
>
> Points (1)-(8) SGTM.
>
> About (9): I slightly prefer to add `VersionedRecord` interface (also
> like the name). I agree that it's low overhead and providing a clean
> path forward for future changes seems worth it to me. Btw: if we
> introduce `VersionedRecord`, I think we can keep the not-null
> requirement for `ValueAndTimestamp` what seems a small side benefit.
> (Btw: your code snippet in the KIP shows what `VersionedRecord` would
> have a non-null requirement for the value, but I think it would need to
> allow null as value?)
>
>
> -Matthias
>
> On 12/7/22 5:23 PM, Victoria Xia wrote:
> > Thanks for the discussion, Bruno, Sagar, and Matthias!
> >
> > It seems we've reached consensus on almost all of the discussion points.
> > I've updated the KIP with the following:
> > 1) renamed "timestampTo" in `get(key, timestampTo)` to "asOfTimestamp" to
> > clarify that this timestamp bound is inclusive, per the SQL guideline
> that
> > "AS OF " queries are inclusive. In the future, if we want to
> > introduce a timestamp range query, we can use `get(key, timestampFrom,
> > timestampTo)` and specify that timestampTo is exclusive in this method,
> > while avoiding confusing with the inclusive asOfTimestamp parameter in
> the
> > other method, given that the names are different.
> > 2) added a description of "history retention" semantics into the
> > VersionedKeyValueStore interface Javadoc, and updated the Javadoc for
> > `get(key, asOfTimestamp)` to mention explicitly that a null result is
> > returned if the provided timestamp bound is not within history retention.
> > 3) added a `delete(key, timestamp)` method (with return type
> > `ValueAndTimestamp`) to the VersionedKeyValueStore interface.
> > 4) updated the Javadoc for `segmentInterval` to clarify that the only
> > reason a user might be interested in this parameter is performance.
> >
> > Other points we discussed which did not result in updates include:
> > 5) whether to automatically update the `min.compaction.lag.ms` config on
> > changelog topics when history retention is changed -- there's support for
> > this but let's not bundle it with this KIP. We can have a separate KIP to
> > change this behavior for the existing windowed changelog topics, in
> > addition to versioned changelog topics.
> > 6) should we expose segmentInterval in this KIP -- let's go ahead and
> > expose it now since we'll almost certainly expose it (in this same
> manner)
> > in a follow-on KIP anyway, and so that poor performance for user
> workloads
> > is less likely to be a barrier for users getting started with this
> feature.
> > I updated the Javadoc for this parameter to clarify why the Javadoc
> > mentions perfo

Re: [DISCUSS] KIP-887 - Add ConfigProvider to make use of environment variables

[Chris Egerton]

Hi Roman,

Thanks for the KIP, LGTM. I should note that I might have some review
suggestions on the implementation but the design proposal (an environment
variable-driven config provider with the API described in the KIP) seems
good to me.

Cheers,

Chris

On Sat, Nov 12, 2022 at 3:47 PM Roman Schmitz 
wrote:

> Hi all,
>
> as I had this discussion with several customers and also colleagues I'd
> like to implement a ConfigProvider that makes use of env variables,
>
> Please let me know if you have any comments, questions, or suggestions!
>
>
> https://eu01.z.antigena.com/l/9tPEJkvUHxfznLkUUC0v4AMuDOEOABJpxmFl_v7_oq~euI0ygOrYSABqdIviAc~Y~pzju9k0gGz9KSr0drMghyE8Ch-4wUK0T0adZL0IIiivmg-SuWhCpE6-jMNYTpQo36CliGmbnvM242uos7WNpoKXXSTMnGY5YbQVFSOuj7oChHQGfZJxChbAseSZnruHoVneorSVSe6k_xUO6DH77DUXZF7MuCNpYZg0ePalQEJpbnT~yxVLFHzL9h
>
> Thanks,
> Roman
>

Re: [DISCUSS] Apache Kafka 3.3.2

[Chris Egerton]

Hi all (especially committers/PMC who have served as release manager
before),

I've created the first release candidate for 3.3.2 (
https://github.com/apache/kafka/tree/3.3.2-rc0) and am almost ready to
start a vote on it, but I haven't run system tests for it yet. I've tried
to kick off a run on my laptop but it's looking like that'll take over
three days to complete.

Is there a way to run system tests with Apache infra?

Cheers,

Chris

On Mon, Dec 12, 2022 at 3:38 PM Chris Egerton  wrote:

> Hi Ron,
>
> You're in luck, I was just about to start generating a release candidate!
> I'm fine with including the fix for KAFKA-14392 in 3.3.2.
>
> Cheers,
>
> Chris
>
> On Mon, Dec 12, 2022 at 3:13 PM Ron Dagostino  wrote:
>
>> Hi Chris.
>> https://eu01.z.antigena.com/l/dqC_AcOJullnCjJyPkdUEcH~mlZJ_G4H3YPEhXpPTMmRSGrk0C1f0T9~HsI-s~rWNCsHZeFaeszMvB9IcRqFW5cOurAQOujb16I131DBYS8hqSgkRzMObK~uLAEWmiiYylZ3O-dcWjZ3uoAZKBo~C4OQ_ss75P8ZD33vBZk5EPvm5v9A7m
>> "KAFKA-14392:
>> KRaft broker heartbeat timeout should not exceed
>>
>> https://eu01.z.antigena.com/l/-fHpsBqhAAaRHbr4QA2u~gvgCwjZcrATMwVm1qlTDl2fxTz3zpltpk~SIoRNlSFK44G1KBzHFM5AJTZWMcokxQVSWoBYvxEfuOoLwejDaAS~1jVupYN1YjBg8HirYS59h735N~GgNyEAuqYikpj41hwGgZbuozVJywAGGh
>> " just merged to trunk and seems like a
>> reasonable candidate for 3.3.2.  Can it be included?
>>
>> Ron
>>
>> On Fri, Nov 18, 2022 at 10:57 AM Chris Egerton 
>> wrote:
>> >
>> > Hi all,
>> >
>> > Here is the release plan for 3.3.2:
>> >
>> https://eu01.z.antigena.com/l/FwWAibomRqqG798j~nU0I_F4V5giCOjaYI4rb88d~D7v9qm9oJhFLtWDr1W24QSwgJTmKVwF4ULU6XGtuAgEN_aDONVSOSExp-rwvuJTPMW8uw6sAqi8o4tOzZLDqDDaBPz6twuQZAJGcWRioMgwIDajrwpHhryG9zXf1ex5Mx9TMjF0vuzJ0LsUKpPUeRQz_EidLaiV6AY1yfmH
>> >
>> > Currently there is one open blocker issue and one open non-blocker,
>> > non-critical issue. I plan to generate the first release candidate once
>> the
>> > blocker issue is resolved, unless other blockers are raised in the
>> meantime.
>> >
>> > Cheers,
>> >
>> > Chris
>> >
>> > On Thu, Nov 17, 2022 at 10:14 AM Bruno Cadonna 
>> wrote:
>> >
>> > > Thanks for volunteering!
>> > >
>> > > +1
>> > >
>> > > Best,
>> > > Bruno
>> > >
>> > > On 17.11.22 09:57, Luke Chen wrote:
>> > > > Thanks for volunteering!
>> > > >
>> > > > On Thu, Nov 17, 2022 at 1:07 AM José Armando García Sancio
>> > > >  wrote:
>> > > >
>> > > >> +1. Thanks for volunteering.
>> > > >>
>> > > >> --
>> > > >> -José
>> > > >>
>> > > >
>> > >
>>
>>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1430

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-14468) Refactor Commit Logic

[Philip Nee (Jira)]

Philip Nee created KAFKA-14468:
--

 Summary: Refactor Commit Logic
 Key: KAFKA-14468
 URL: https://issues.apache.org/jira/browse/KAFKA-14468
 Project: Kafka
  Issue Type: Sub-task
  Components: consumer
Reporter: Philip Nee
Assignee: Philip Nee


Refactor commit logic using the new multi-threaded coordinator construct.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1429

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-14467) Add a test to validate the replica state after processing the OFFSET_MOVED_TO_TIERED_STORAGE error, especially for the transactional state

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-14467:
--

 Summary: Add a test to validate the replica state after processing 
the OFFSET_MOVED_TO_TIERED_STORAGE error, especially for the transactional state
 Key: KAFKA-14467
 URL: https://issues.apache.org/jira/browse/KAFKA-14467
 Project: Kafka
  Issue Type: Sub-task
  Components: core
Reporter: Satish Duggana
Assignee: Satish Duggana


https://github.com/apache/kafka/pull/11390#pullrequestreview-1210993072



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1428

[Apache Jenkins Server]

See 

[jira] [Reopened] (KAFKA-13881) Add package.java for public package javadoc

[Chris Egerton (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Egerton reopened KAFKA-13881:
---

> Add package.java for public package javadoc
> ---
>
> Key: KAFKA-13881
> URL: https://issues.apache.org/jira/browse/KAFKA-13881
> Project: Kafka
>  Issue Type: Task
>Reporter: Tom Bentley
>Assignee: Greg Harris
>Priority: Trivial
> Fix For: 3.5.0
>
>
> Our public javadoc ([https://kafka.apache.org/31/javadoc/index.html)] doesn't 
> have any package descriptions, which is a bit intimidating for anyone who is 
> new to the project.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-13881) Add package.java for public package javadoc

[Chris Egerton (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Egerton resolved KAFKA-13881.
---
Fix Version/s: 3.5.0
   Resolution: Fixed

> Add package.java for public package javadoc
> ---
>
> Key: KAFKA-13881
> URL: https://issues.apache.org/jira/browse/KAFKA-13881
> Project: Kafka
>  Issue Type: Task
>Reporter: Tom Bentley
>Assignee: Greg Harris
>Priority: Trivial
> Fix For: 3.5.0
>
>
> Our public javadoc ([https://kafka.apache.org/31/javadoc/index.html)] doesn't 
> have any package descriptions, which is a bit intimidating for anyone who is 
> new to the project.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-879: Multi-level Rack Awareness

[Viktor Somogyi-Vass]

Hi Ziming,

Thanks for the reply. It may not be a public interface, yes, but it is
annotated with the @InterfaceStability.Unstable annotation from which I
presumed it'll be public at some point. Therefore in my KIP I proposed to
take it public as we've seen significant interest from our users towards
stretch clusters and this kind of replica placement I described. In fact in
our latest release we've already implemented something like this and we'd
like to contribute it back if the community also feels the need for it.
May I ask why don't you want to make this public?

Thanks,
Viktor

On Thu, Dec 8, 2022 at 3:06 PM Andrew Otto  wrote:

> FWIW, the Wikimedia Foundation would find this change really helpful.  We
> are going to soon experiment with a stretched Kafka cluster, and it would
> be nice to be able to target datacenter AND racks for replica placement.
>
> On Thu, Dec 8, 2022 at 3:37 AM ziming deng 
> wrote:
>
> > Hi Viktor,
> >
> > As far as I know, we haven't make ReplicaPlacer a public interface, and
> we
> > have no plan to make it public. I think you can submit a discussion or
> > create a JIRA ticket directly without KIP if you have ideas on improving
> > it, right?
> >
> > --
> > Best,
> > Ziming
> >
> > > On Nov 29, 2022, at 21:52, Viktor Somogyi-Vass <
> > viktor.somo...@cloudera.com.INVALID> wrote:
> > >
> > > Hi All,
> > >
> > > I'd like to bump this. I've also updated the KIP to incorporate the new
> > > KRaft changes (ReplicaPlacer). Luckily my proposals were quite similar
> to
> > > that, so mostly I've made some minor rewording, naming changes, etc.
> > >
> > > Again, the brief summary of the KIP:
> > > - expose replica placement strategies with a new config
> > > - create an admin API and protocol to expose replica placement
> > > functionality (mainly for the reassignment tool)
> > > - create a new multi-level rack awareness strategy which improves
> > > availability on stretch clusters
> > >
> > > I'm happy for any feedback.
> > >
> > > Best,
> > > Viktor
> > >
> > > On Fri, Oct 28, 2022 at 4:14 PM Viktor Somogyi-Vass <
> > > viktor.somo...@cloudera.com> wrote:
> > >
> > >> Hey all,
> > >>
> > >> I'd like to propose a new broker side replica assignment strategy and
> an
> > >> interface that generalizes replica assignment on brokers and makes
> them
> > >> pluggable.
> > >>
> > >> Briefly, the motivation for the new replica assignment strategy is
> that
> > >> more and more of our customers would want to run their clusters in a
> > >> stretched environment, where for instance a cluster is running over
> > >> multiple regions (and multiple racks inside a region). Since this
> seems
> > >> like a more common need, we'd like to contribute back our
> implementation
> > >> and also make a generalized interface, so that new strategies that
> > people
> > >> may come up with could be served better.
> > >>
> > >> I welcome any feedback on this KIP.
> > >>
> > >> The link:
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-879%3A+Multi-level+Rack+Awareness
> > >>
> > >> Best to all,
> > >> Viktor
> > >>
> >
> >
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1427

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-14466) Refactor ClassLoaderAwareRemoteStorageManager.scala to ClassLoaderAwareRemoteStorageManager.java and move it to storage module.

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-14466:
--

 Summary: Refactor ClassLoaderAwareRemoteStorageManager.scala to  
ClassLoaderAwareRemoteStorageManager.java and move it to storage module.
 Key: KAFKA-14466
 URL: https://issues.apache.org/jira/browse/KAFKA-14466
 Project: Kafka
  Issue Type: Sub-task
Reporter: Satish Duggana
Assignee: Satish Duggana


https://github.com/apache/kafka/pull/11390#discussion_r1043982906



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14285) Delete quota node in zookeeper when configs are empty

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-14285.
---
Fix Version/s: 3.4.1
   3.5.0
   Resolution: Fixed

> Delete quota node in zookeeper when configs are empty
> -
>
> Key: KAFKA-14285
> URL: https://issues.apache.org/jira/browse/KAFKA-14285
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 1.1.0, 2.7.1, 3.3.1
>Reporter: Junyang Liu
>Assignee: Junyang Liu
>Priority: Major
> Fix For: 3.4.1, 3.5.0
>
>
> When configuring flow control quota for users and clients, Kafka will add a 
> child node to /config/users or /config/clients in zookeeper. When deleting a 
> flow control quota config, the existing node in zookeeper will not be deleted 
> even if the configs in the node are empty. So if a client repeatedly 
> configure and delete flow control quota for different users and clients, the 
> node in zookeeper will increase continuously, which may leads to severe 
> problems such as memory leak.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)