date:20231002


Thanks for updating the KIP.

Not sure if I agree or not with Bruno's idea to split the query types 
further? In the end, we split them only because there is three different 
return types: single value, value-iterator, key-value-iterator.


What do we gain by splitting out single-ts-range-key? In the end, for 
range-ts-range-key the proposed class is necessary and is a superset 
(one can set both timestamps to the same value, for single-ts lookup).


The mentioned simplification might apply to "single-ts-range-key" but I 
don't see a simplification for the proposed (and necessary) query type?


On the other hand, I see an advantage of a single-ts-range-key for 
querying over the "latest version" with a range of keys. For a 
single-ts-range-key query, this it would be the default (similar to 
VersionedKeyQuery with not asOf-timestamped defined).


In the current version of the KIP, (if we agree that default should 
actually return "all versions" not "latest" -- this default was 
suggested by Bruno on KIP-968 and makes sense to me, so we would need to 
have the same default here to stay consistent), users would need to pass 
in `from(Long.MAX).to(Long.MAX)` (if I got this right) to query the 
latest point in time only, what seems to be clumsy? Or we could add a 
`lastestKeyOnly` option to `MultiVersionedRangeQuery`, but it does seems 
a little clumsy, too.





The overall order of the returned records is by Key


I assume, results are returned by timestamp for each key. The KIP should 
be explicit about it.




To be very explicit, should we rename the methods to specify the key bound?

 - withRange -> withKeyRange
 - withLowerBound -> withLowerKeyBound
 - withUpperBound -> withUpperKeyBound
 - withNoBounds -> allKeys (or withNoKeyBounds, but we use 
`allVersions` and not `noTimeBound` and should align the naming?)




-Matthias


On 9/6/23 5:25 AM, Bruno Cadonna wrote:

Hi Alieh,

Thanks for the KIP!

One high level comment/question:

I assume you separated single key queries into two classes because 
versioned key queries return a single value and multi version key 
queries return iterators. Although, range queries always return 
iterators, it would make sense to also separate range queries for 
versioned state stores into range queries that return one single version 
of the keys within a range and range queries that return multiple 
version of the keys within a range, IMO. That would reduce the 
meaningless combinations.

WDYT?

Best,
Bruno

On 8/16/23 8:01 PM, Alieh Saeedi wrote:

Hi all,

I splitted KIP-960

into
three separate KIPs. Therefore, please continue discussions about range
interactive queries here. You can see all the addressed reviews on the
following page. Thanks in advance.

KIP-969: Support range interactive queries (IQv2) for versioned state 
stores



I look forward to your feedback!

Cheers,
Alieh

Re: [DISCUSS] KIP-968: Support single-key_multi-timestamp interactive queries (IQv2) for versioned state stores


Thanks for the updated KIP. Overall I like it.

Victoria raises a very good point, and I personally tend to prefer (I 
believe so does Victoria, but it's not totally clear from her email) if 
a range query would not return any tombstones, ie, only two records in 
Victoria's example. Thus, it seems best to include a `validTo` ts-field 
to `VersionedRecord` -- otherwise, the retrieved result cannot be 
interpreted correctly.


Not sure what others think about it.

I would also be open to actually add a `includeDeletes()` (or 
`includeTombstones()`) method/flag (disabled by default) to allow users 
to get all tombstone: this would only be helpful if there are two 
consecutive tombstone though (if I got it right), so not sure if we want 
to add it or not -- it seems also possible to add it later if there is 
user demand for it, so it might be a premature addition as this point?



Nit:

the public interface ValueIterator is used 


"is used" -> "is added" (otherwise it sounds like as if `ValueIterator` 
exist already)




Should we also add a `.within(fromTs, toTs)` (or maybe some better 
name?) to allow specifying both bounds at once? The existing 
`RangeQuery` does the same for specifying the key-range, so might be 
good to add for time-range too?




-Matthias


On 9/6/23 5:01 AM, Bruno Cadonna wrote:

In my last e-mail I missed to finish a sentence.

"I think from a KIP"

should be

"I think the KIP looks good!"


On 9/6/23 1:59 PM, Bruno Cadonna wrote:

Hi Alieh,

Thanks for the KIP!

I think from a KIP

1.
I propose to throw an IllegalArgumentException or an 
IllegalStateException for meaningless combinations. In any case, the 
KIP should specify what exception is thrown.


2.
Why does not specifying a range return the latest version? I would 
expect that it returns all versions since an empty lower or upper 
limit is interpreted as no limit.


3.
I second Matthias comment about replacing "asOf" with "until" or "to".

4.
Do we need "allVersions()"? As I said above I would return all 
versions if no limits are specified. I think if we get rid of 
allVersions() there might not be any meaningless combinations anymore.
If a user applies twice the same limit like for example 
MultiVersionedKeyQuery.with(key).from(t1).from(t2) the last one wins.


5.
Could you add some more examples with time ranges to the example section?

6.
The KIP misses the test plan section.

7.
I propose to rename the class to "MultiVersionKeyQuery" since we are 
querying multiple versions of the same key.


8.
Could you also add withAscendingTimestamps()? IMO it gives users the 
possibility to make their code more readable instead of only relying 
on the default.


Best,
Bruno


On 8/17/23 4:13 AM, Matthias J. Sax wrote:

Thanks for splitting this part into a separate KIP!

For `withKey()` we should be explicit that `null` is not allowed.

(Looking into existing `KeyQuery` it seems the JavaDocs don't cover 
this either -- would you like to do a tiny cleanup PR for this, or 
fix on-the-side in one of your PRs?)




The key query returns all the records that are valid in the time 
range starting from the timestamp {@code fromTimestamp}.


In the JavaDocs you use the phrase `are valid` -- I think we need to 
explain what "valid" means? It might even be worth to add some 
examples. It's annoying, but being precise if kinda important.


With regard to KIP-962, should we allow `null` for time bounds ? The 
JavaDocs should also be explicit if `null` is allowed or not and what 
the semantics are if allowed.




You are using `asOf()` however, because we are doing time-range 
queries, to me using `until()` to describe the upper bound would 
sound better (I am not a native speaker though, so maybe I am off?)



The key query returns all the records that have timestamp <= {@code 
asOfTimestamp}.


This is only correct if not lower-bound is set, right?


In your reply to KIP-960 you mentioned:


the meaningless combinations are prevented by throwing exceptions.


We should add corresponding JavaDocs like:

    @throws IllegalArgumentException if {@code fromTimestamp} is 
equal or

 larger than {@code untilTimestamp}

Or something similar.


With regard to KIP-960: if we need to introduce a `VersionedKeyQuery` 
class for single-key-single-ts lookup, would we need to find a new 
name for the query class of this KIP, given that the return type is 
different?



-Matthias



On 8/16/23 10:57 AM, Alieh Saeedi wrote:

Hi all,

I splitted KIP-960

into three separate KIPs. Therefore, please continue discussions
about single-key, multi-timestamp interactive queries here. You can 
see all

the addressed reviews on the following page. Thanks in advance.

KIP-968: Support single-key_multi-timestamp interactive queries 
(IQv2) for

versioned state stores

Re: [DISCUSS] KIP-960: Support interactive queries (IQv2) for versioned state stores

Thanks for updating the KIP. Just re-read it. Overall LGTM.

A few nits:

single-key lookup with timestamp (upper) bound
Not sure if "(upper) bound" is the right phrasing? Personally, I find it
a little bit confusing, because the term "bound" kinda indicates a
range, but we still do a point lookup at the specified timestamp. Maybe
just remove "(upper) bound"? (I understand what you mean, and maybe it's
technically correct but IMHO it's a little hard to understand.)

In this KIP we propose a new Optional field in VersionedKeyQuery to store the
asOfTimestamp value

and

Moreover, the method asOf is added to the class in order to create key queries having asOfTimestamp value as well.

This KIP introduces `VersionedKeyQuery` but does not just add a new
option to an existing class. Guess, it was just not update correctly
from a previous version when we added a new option to existing `KeyQuery`.

I agree with Walker that we you can start a VOTE.

-Matthias

On 9/11/23 11:07 AM, Walker Carlson wrote:

Thanks for the KIP Alieh!

I don't have anything to add to the 960 discussion right now as it seems
rather straightforward. I think after you address Bruno's comments we can
bring it to a vote. I'll review the two spawned KIPs separately.

Keep it up,
Walker

On Wed, Sep 6, 2023 at 5:11 AM Bruno Cadonna wrote:

Hi Alieh,

I am sorry if I might repeat things that have been already said since I
am not sure I got all e-mails of this discussion thread.

The KIP looks good!

I just have two minor comments that I think are easily resolved.

1.
Why is defining latest() not needed? Is it because if I do not use
asOf() I get the latest value?

For example,

final VersionedKeyQuery query =
VersionedKeyQuery.withKey(1);

will return the latest version, right?

If so, that should be explicitly stated in the KIP and in the javadocs.

I assume, you wanted to say exactly that with

"Defining the latest() method is not needed since returning the latest
value has been always the default assumption."

I would propose to write something like:

"If a query is created without calling asOf() the query returns the
latest value of the key"

Adding one example in the example section for this case would also help.

2.
The KIP misses the test plan section.

Best,
Bruno

On 8/25/23 1:02 PM, Alieh Saeedi wrote:

Thank you, Matthias and Victoria.

Regarding the problem of using methods of single-key-single-ts queries

for

KeyQuery (such as asOf) and vice versa (such as skipCache()), after a
discussion, we decided to define a separate class for

single-key-single-ts

queries named VersionedKeyQuery. Subsequently, the
single-key-multi-timestamp queries (KIP-968) and range queries (KIP-969)
will be covered by the MultiVersionedKeyQuery and

MultiVersionedRangeQuery

classes, respectively.
I think the VersionedKeyQuery is type-safe since if an instance of the
VersionedKeyQuery is posed to a normal (non-versioned) state store, we

will

have the defined Kafka Streams UNKNOWN_QUERY_TYPE failure.

P.S.: The example should be correct now.

Cheers,
Alieh

On Thu, Aug 24, 2023 at 9:34 PM Victoria Xia

wrote:

Hi Alieh,

Thanks for the updates!
Some questions on the new limited-scope KIP:
1. The example in the "Examples" section shows the query type as
`KeyQuery>` and the result type as
`StateQueryResult>`. Should those have
`VersionedRecord` instead of `ValueAndTimestamp`? Also, the request

type is

currently `StateQueryRequest>>`.
Should the `ValueIterator` no longer be present, now that we are only
returning a single record?
2. Related to Matthias's question about what happens if `asOf()` is set
for a KeyQuery to a non-versioned store, what happens if `skipCache()`

set for a versioned store? And what will `isSkipCache()` return?

Versioned

stores do not support caching (at least today). I think for consistency

have to let `isSkipCache()` still default to false if `isSkipCache()` is
not set. I think that's fine, as long as it's clear to users (e.g., from
docs) that `isSkipCache()` is not relevant for queries to versioned

stores.

And some responses to your comments from earlier:

I changed the VersionedRecord such that it can have NULL values as

well.

The question is, what was the initial intention of setting the value in
VersionedRecord as NOT NULL?
We can discuss more on your other KIPs (KIP-968 and KIP-969) since this
change should only be relevant for those KIPs and not this one, but the
short answer is that today there's no situation in which VersionedRecord
would need to return a null value because if a get(key) or get(key,
asOfTimestamp) call to a versioned store were to find a null record

(i.e.,

tombstone), then a null object is returned, rather than a non-null
VersionedRecord with null value. In other words, versioned stores do not
distinguish between a tombstone having been inserted versus no record

ever

having been inserted.

About defining new methods in the VersionedKeyValueStore interface:

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2248

See

Re: [DISCUSS] KIP-986: Cross-Cluster Replication

Hey Tom,

Thanks for the high-level questions, as I am certainly approaching
this KIP differently than I've seen before.

I think that ideally this KIP will expand to include lots of
requirements and possible implementations, and that through discussion
we can narrow the scope and form a roadmap for implementation across
multiple KIPs. I don't plan to be the decision-maker for this project,
as I'm more interested in building consensus among the co-authors. I
can certainly poll that consensus and update the KIP to keep the
project moving, and any other co-author can do the same. And to set an
example, I'll clarify your questions and for anything that I agree
with, I'll ask that you make the update to the KIP, so that the KIP
captures your understanding of the problem and your requirements. If
you don't get the chance to make the changes yourself, I'll make sure
they get included eventually, as they're very good ideas :)

For your remaining questions:

M1: I was trying to draw analogies to databases, but your suggested
properties are much more compelling and informative. I'd love it if
you added some formalism here, so that we have a better grasp on what
we're trying to accomplish. +1
M2: I think the "asynchronous" problem corresponds to the goal of
"exactly once semantics" but the two are not obviously opposites. I
think the MM2 deficiencies could focus less on the architecture
(asynchronicity) and more on the user-facing effect (semantics). +1
M3: I had a "non-goals" section that ended up becoming the "rejected
alternatives" section instead. If you have some non-goals in mind,
please add them.
M4+M5: I think it's too early to nail down the assumptions directly,
but if you believe that "separate operators of source and target" is a
requirement, that would be good to write down. +1
M6: That is a concerning edge case, and I don't know how to handle it.
I was imagining that there would be a many:many relationship of
clusters and links, but I understand that the book-keeping of that
decision may be significant.
M7: I think this may be appropriate to cover in a "user story" or
"example usages". I naturally thought that the feature would describe
some minimal way of linking two topics, and the applications
(combining multiple links, performing failovers, or running
active-active, etc) would be left to users to define. I included the
regex configurations because I imagine that creating 100s or 1000s of
links would be unnecessarily tedious. The feature may also encode
those use-cases directly as first-class citizens as well.

U1: These are states that can happen in reality, and I meant for that
section to imply that we should expect these states and model them for
operations and observability.

D1: I think I may have introduced this confusion by trying to be
terse. I imagined that there will be two different topics on the
source and target, which would be synced to have the same
configuration contents, similar to MM2's implementation. This would
allow for the replication link to be permanently disconnected and the
target topic to become just a regular topic, Later, a new replication
link and new target topic (with another separate topic-id) can be
created to rebuild the replication. I also thought that it was
possible that two clusters had already chosen the same topic-id, and
that attempting to interpret one topic-id in two different clusters
was error-prone. As far as replicating __cluster_metadata: I hadn't
considered that, but that might be required depending on the semantics
we choose.
D2: Thanks, that's a good clarification. Uptime and bandwidth should
be assumed to be lower, and latency should be assumed to be higher. +1
D3: I included this restriction because it would not be transparent to
source consumers. They would need special support for connecting to
brokers from multiple clusters, with potentially distinct metadata.

Thanks so much!
Greg

On Mon, Oct 2, 2023 at 4:24 PM Tom Bentley  wrote:
>
> Hi Greg,
>
> Thanks for this KIP! It is obviously very ambitious, but it's great to have
> a conversation about it.
>
> I'll start with some general points:
>
> Do you have a plan in mind for how to proceed with elaborating this KIP?
> While I like how you're involving the community in elaborating the KIP, I
> think there is a danger, which is more likely with this inclusive approach,
> in trying to attempt too much at once.
>
> In my opinion someone needs to take the difficult decisions necessary to
> limit the initial scope (and, just as importantly, communicate that
> clearly) in order to maximise the chances of actually getting something
> accepted and implemented. Can we assume that you're that person? Defining
> the what and how of the metadata replication, and the log replication seem
> to me to be the core of what you're trying to achieve here. We should make
> anything that is not crucial to that (i.e. NAT punching) a non-goal of this
> KIP. Future KIPs can easily add those features.
>
> I also had a

Re: [DISCUSS] KIP-986: Cross-Cluster Replication

2023-10-02 Thread Tom Bentley

Hi Greg,

Thanks for this KIP! It is obviously very ambitious, but it's great to have
a conversation about it.

I'll start with some general points:

Do you have a plan in mind for how to proceed with elaborating this KIP?
While I like how you're involving the community in elaborating the KIP, I
think there is a danger, which is more likely with this inclusive approach,
in trying to attempt too much at once.

In my opinion someone needs to take the difficult decisions necessary to
limit the initial scope (and, just as importantly, communicate that
clearly) in order to maximise the chances of actually getting something
accepted and implemented. Can we assume that you're that person? Defining
the what and how of the metadata replication, and the log replication seem
to me to be the core of what you're trying to achieve here. We should make
anything that is not crucial to that (i.e. NAT punching) a non-goal of this
KIP. Future KIPs can easily add those features.

I also had a few specific points:

Motivation
M1. I don't find the "logical replication" vs "physical replication"
particularly helpful. I think one key property is "offset preserving",
which is also self-explanatory. Slightly more generally, we could define
the concept of "consumer transparency", i.e. a consumer could reconnect to
either cluster and observe the same sequences of records (same order, same
offsets, and same transaction visibility). Consumer transparency requires
synchronous replication, but offset preserving does not.
M2. In the motivation you mention that MM offers asynchronous replication,
but the Goals subsection doesn't mention support for synchronous
replication. We should be clear which (or both) we're aiming for.
M3. A Non-Goals section would be useful, especially for a KIP that's large
and ambitious like this one.
M4. It might also be worth having a list of Assumptions. Here we could list
all the things we want to assume in order to make the initial KIP feasible.
M5. For example we should be explicit about whether or not it is assumed
that the same people are operating (and thus have visibility into) both
clusters.
M6. One thing worth calling out is whether the clusters themselves are in a
leader/follower relationship (e.g. the DR scenario), or whether this is a
topic-level concern. I guess it's topic level from the topic and consumer
group regexes. But this has consequences we should explore. For example
what if a transaction includes records in topics X and Y, where X is
replicated but Y is not?
M7. I think you should be clear about whether this leader/follower
relationship can be reversed, and in what circumstances. In the user
interface section you talk about "disconnected", but not this kind of
fail-back.

User interface
U1. "Links can be temporarily or permanently disconnected." Are you
describing a fact about the network between the two clusters, or is this
disconnection something actively managed by the system, or by the operator?

Data semantics
D1. The KIP says "both cross-cluster topics and intra-cluster replicas:
Have the same configuration as their source" but you also say
"cross-cluster replicas: Have a separate topic-id", this seems like a
contradiction, on the face of it. It seems like there's a whole host of
devils in the detail behind this. It implies replication of (some of) the
__cluster_metadata, I think, but not all (since you say ACLs are not
replicated). If that's right, then what does it imply about referential
integrity between metadata records? i.e. what if metadata record A (which
is replicated) references record B (which is not)? Even if this is not
possible by design initially, how does it constrain the future evolution of
metadata record schemas? Is any such metadata replication going to be
transaction preserving? If the topic ids can differ then what is
responsible for the mapping and rewriting of metadata records which include
topic ids?
D2. "The network path between Kafka clusters is assumed to be less reliable
than the intra-cluster network," we should be explicit about whether or not
we're assuming similar network latencies and bandwidth for the
inter-cluster network links as for the in-cluster ones.
D3 "Are not eligible for fetch-from-follower on the source cluster" the
reason for this isn't immediately apparent to me.

Thanks again,

Tom

On Tue, 3 Oct 2023 at 09:37, Greg Harris 
wrote:

> Hi all,
>
> I've opened an extremely early draft of the Cross-Cluster Replication
> feature, and I would like to invite any and all co-authors to expand
> on it. Find the page here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-986%3A+Cross-Cluster+Replication
>
> This is not strictly an invitation to "review" the KIP, as the
> document has much less detail than other KIPs of similar complexity.
> But if you are knowledgeable in this area, some early sanity checks
> would be greatly appreciated.
>
> I've included a "shopping list" of properties that appear to me to be
> desirable, but I

Re: [VOTE] 3.6.0 RC2

2023-10-02 Thread Justine Olshan

I realized Luke shared the results here for RC1
https://drive.google.com/drive/folders/1S2XYd79f6_AeWj9f9qEkliRg7JtL04AC
Given we had some runs that looked reasonable, and we made a small change,
I'm ok with this. But I wouldn't be upset if we had another set of runs :)

As for the validation:

   - I've compiled from source with java 17, 2.13, run the transactional
   produce bench
   - Run unit tests
   - Validated the checksums
   - Downloaded and ran the 2.12 version of the release
   - Briefly took a look at the documentation
   - I was browsing through the site html files and I noticed the html for
   documentation.html seemed to be for 3.4. Not sure if this is a blocker, but
   wanted to flag it. This seems to be the case for the previous release
   candidates as well. (As well as 3.5 release it seems)


I will hold off on voting until we figure that part out. I will also follow
up with the documentation Divij mentioned outside this thread.

Thanks,
Justine

On Mon, Oct 2, 2023 at 3:05 PM Greg Harris 
wrote:

> Hey Satish,
>
> I verified KIP-898 functionality and the KAFKA-15473 patch.
> +1 (non-binding)
>
> Thanks!
>
> On Mon, Oct 2, 2023 at 1:28 PM Justine Olshan
>  wrote:
> >
> > Hey all -- I noticed we still have the system tests as something that
> will
> > be updated. Did we get a run for this RC?
> >
> > On Mon, Oct 2, 2023 at 1:24 PM Bill Bejeck  wrote:
> >
> > > Hi Satish,
> > >
> > > Thanks for running the release.
> > > I performed the following steps:
> > >
> > >- Validated all the checksums, signatures, and keys
> > >- Built the release from source
> > >- Ran all unit tests
> > >- Quick start validations
> > >   - ZK and Kraft
> > >   - Connect
> > >   - Kafka Streams
> > >- Spot checked java docs and documentation
> > >
> > > +1 (binding)
> > >
> > > - Bill
> > >
> > > On Mon, Oct 2, 2023 at 10:23 AM Proven Provenzano
> > >  wrote:
> > >
> > > > Hi,
> > > >
> > > > To verify the release of release 3.6.0 RC2 I did the following:
> > > >
> > > >- Downloaded the source, built and ran the tests.
> > > >- Validated SCRAM with KRaft including creating credentials with
> > > >kafka-storage.
> > > >- Validated Delegation Tokens with KRaft
> > > >
> > > > +1 (non-binding)
> > > >
> > > > --Proven
> > > >
> > > >
> > > >
> > > > On Mon, Oct 2, 2023 at 8:37 AM Divij Vaidya  >
> > > > wrote:
> > > >
> > > > > + 1 (non-binding)
> > > > >
> > > > > Verifications:
> > > > > 1. I ran a produce-consume workload with plaintext auth, JDK17,
> zstd
> > > > > compression using an open messaging benchmark and found 3.6 to be
> > > better
> > > > > than or equal to 3.5.1 across all dimensions. Notably, 3.6 had
> > > > consistently
> > > > > 6-7% lower CPU utilization, lesser spikes on P99 produce latencies
> and
> > > > > overall lower P99.8 latencies.
> > > > >
> > > > > 2. I have verified that detached signature is correct using
> > > > > https://www.apache.org/info/verification.html and the release
> manager
> > > > > public keys are available at
> > > > > https://keys.openpgp.org/search?q=F65DC3423D4CD7B9
> > > > >
> > > > > 3. I have verified that all metrics emitted in 3.5.1 (with Zk) are
> also
> > > > > being emitted in 3.6.0 (with Zk).
> > > > >
> > > > > Problems (but not blockers):
> > > > > 1. Metrics added in
> > > > >
> > > > >
> > > >
> > >
> https://github.com/apache/kafka/commit/2f71708955b293658cec3b27e9a5588d39c38d7e
> > > > > aren't available in the documentation (cc: Justine). I don't
> consider
> > > > this
> > > > > as a release blocker but we should add it as a fast follow-up.
> > > > >
> > > > > 2. Metric added in
> > > > >
> > > > >
> > > >
> > >
> https://github.com/apache/kafka/commit/a900794ace4dcf1f9dadee27fbd8b63979532a18
> > > > > isn't available in documentation (cc: David). I don't consider
> this as
> > > a
> > > > > release blocker but we should add it as a fast follow-up.
> > > > >
> > > > > --
> > > > > Divij Vaidya
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Oct 2, 2023 at 9:50 AM Federico Valeri <
> fedeval...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Satish, I did the following to verify the release:
> > > > > >
> > > > > > - Built from source with Java 17 and Scala 2.13
> > > > > > - Ran all unit and integration tests
> > > > > > - Spot checked documentation
> > > > > > - Ran custom client applications using staging artifacts on a
> 3-nodes
> > > > > > cluster
> > > > > > - Tested tiered storage with one of the available RSM
> implementations
> > > > > >
> > > > > > +1 (non binding)
> > > > > >
> > > > > > Thanks
> > > > > > Fede
> > > > > >
> > > > > > On Mon, Oct 2, 2023 at 8:50 AM Luke Chen 
> wrote:
> > > > > > >
> > > > > > > Hi Satish,
> > > > > > >
> > > > > > > I verified with:
> > > > > > > 1. Ran quick start in KRaft for scala 2.12 artifact
> > > > > > > 2. Making sure the checksum are correct
> > > > > > > 3. Browsing release notes, documents, javadocs, protocols.
> > > > > > > 4. Verified the

Re: [DISCUSS] KIP-714: Client metrics and observability


Hi,

I did not pay attention to this KIP in the past; seems it was on-hold 
for a while.


Overall it sounds very useful, and I think we should extend this with a 
follow up KIP for Kafka Streams. What is unclear to me at this point is 
the statement:



Kafka Streams applications have an application.id configured and this 
identifier should be included as the application_id metrics label.


The `application.id` is currently only used as the (main) consumer's 
`group.id` (and is part of an auto-generated `client.id` if the user 
does not set one).


This comment related to:


The following labels should be added by the client as appropriate before 
metrics are pushed.


Given that Kafka Streams uses the consumer/producer/admin client as 
"black boxes", a client does at this point not know that it's part of a 
Kafka Streams application, and thus, it won't be able to attach any such 
label to the metrics it sends. (Also producer and admin don't even know 
the value of `application.id` -- only the (main) consumer, indirectly 
via `group.id`, but also restore and global consumer don't know it, 
because they don't have `group.id` set).


While I am totally in favor of the proposal, I am wondering how we 
intent to implement it in clean way? Or would we do ok to have some 
internal client APIs that KS can use to "register" itself with the client?





While clients must support both temporalities, the broker will initially only 
send GetTelemetrySubscriptionsResponse.DeltaTemporality=True


Not sure if I can follow. How make the decision about DELTA or 
CUMULATIVE metrics? Should the broker side plugin not decide what 
metrics it what to receive in which form? So what does "initially" mean 
-- the broker won't ship with a default plugin implementation?





The following method is added to the Producer, Consumer, and Admin client 
interfaces:


Should we add anything to Kafka Streams to expose the underlying 
clients' assigned client-instance-ids programmatically? I am also 
wondering if clients should report their assigned client-instance-ids as 
metrics itself (for this case, Kafka Streams won't need to do anything, 
because we already expose all client metrics).


If we add anything programmatic, we need to make it simple, given that 
Kafka Streams has many clients per `StreamThread` and may have multiple 
threads.





enable.metrics.push
It might be worth to add this to `StreamsConfig`, too? It set via 
StreamsConfig, we would forward it to all clients automatically.





-Matthias


On 9/29/23 5:45 PM, David Jacot wrote:

Hi Andrew,

Thanks for driving this one. I haven't read all the KIP yet but I already
have an initial question. In the Threading section, it is written
"KafkaConsumer: the "background" thread (based on the consumer threading
refactor which is underway)". If I understand this correctly, it means
that KIP-714 won't work if the "old consumer" is used. Am I correct?

Cheers,
David


On Fri, Sep 22, 2023 at 12:18 PM Andrew Schofield <
andrew_schofield_j...@outlook.com> wrote:


Hi Philip,
No, I do not think it should actively search for a broker that supports
the new
RPCs. In general, either all of the brokers or none of the brokers will
support it.
In the window, where the cluster is being upgraded or client telemetry is
being
enabled, there might be a mixed situation. I wouldn’t put too much effort
into
this mixed scenario. As the client finds brokers which support the new
RPCs,
it can begin to follow the KIP-714 mechanism.

Thanks,
Andrew


On 22 Sep 2023, at 20:01, Philip Nee  wrote:

Hi Andrew -

Question on top of your answers: Do you think the client should actively
search for a broker that supports this RPC? As previously mentioned, the
broker uses the leastLoadedNode to find its first connection (am
I correct?), and what if that broker doesn't support the metric push?

P

On Fri, Sep 22, 2023 at 10:20 AM Andrew Schofield <
andrew_schofield_j...@outlook.com> wrote:


Hi Kirk,
Thanks for your question. You are correct that the presence or absence

of

the new RPCs in the
ApiVersionsResponse tells the client whether to request the telemetry
subscriptions and push
metrics.

This is of course tricky in practice. It would be conceivable, as a
cluster is upgraded to AK 3.7
or as a client metrics receiver plugin is deployed across the cluster,
that a client connects to some
brokers that support the new RPCs and some that do not.

Here’s my suggestion:
* If a client is not connected to any brokers that support in the new
RPCs, it cannot push metrics.
* If a client is only connected to brokers that support the new RPCs, it
will use the new RPCs in
accordance with the KIP.
* If a client is connected to some brokers that support the new RPCs and
some that do not, it will
use the new RPCs with the supporting subset of brokers in accordance

with

the KIP.

Comments?

Thanks,
Andrew


On 22 Sep 2023, at 16:01, Kirk True  wrote:

Hi Andrew/Jun,

I want to make sure I understand question/comment

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2247

See 


Changes:


--
[...truncated 315427 lines...]

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
onlyRemovePendingTaskToUpdateInputPartitionsShouldRemoveTaskFromPendingUpdateActions()
 STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
onlyRemovePendingTaskToUpdateInputPartitionsShouldRemoveTaskFromPendingUpdateActions()
 PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
shouldVerifyIfPendingTaskToRecycleExist() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
shouldVerifyIfPendingTaskToRecycleExist() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
shouldAddAndRemovePendingTaskToUpdateInputPartitions() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
shouldAddAndRemovePendingTaskToUpdateInputPartitions() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
onlyRemovePendingTaskToCloseDirtyShouldRemoveTaskFromPendingUpdateActions() 
STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
onlyRemovePendingTaskToCloseDirtyShouldRemoveTaskFromPendingUpdateActions() 
PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
shouldAddAndRemovePendingTaskToSuspend() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
shouldAddAndRemovePendingTaskToSuspend() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
shouldVerifyIfPendingTaskToInitExist() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
shouldVerifyIfPendingTaskToInitExist() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
onlyRemovePendingTaskToCloseCleanShouldRemoveTaskFromPendingUpdateActions() 
STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
onlyRemovePendingTaskToCloseCleanShouldRemoveTaskFromPendingUpdateActions() 
PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
shouldDrainPendingTasksToCreate() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
shouldDrainPendingTasksToCreate() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
onlyRemovePendingTaskToRecycleShouldRemoveTaskFromPendingUpdateActions() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
onlyRemovePendingTaskToRecycleShouldRemoveTaskFromPendingUpdateActions() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
shouldAddAndRemovePendingTaskToCloseClean() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
shouldAddAndRemovePendingTaskToCloseClean() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
shouldAddAndRemovePendingTaskToCloseDirty() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
shouldAddAndRemovePendingTaskToCloseDirty() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
shouldKeepAddedTasks() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
shouldKeepAddedTasks() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > StateQueryResultTest 
> More than one query result throws IllegalArgumentException STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > StateQueryResultTest 
> More than one query result throws IllegalArgumentException PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > StateQueryResultTest 
> Zero query results shouldn't error STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > StateQueryResultTest 
> Zero query results shouldn't error PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > StateQueryResultTest 
> Valid query results still works STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > StateQueryResultTest 
> Valid query results still works PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
RocksDBBlockCacheMetricsTest > shouldRecordCorrectBlockCacheUsage(RocksDBStore, 
StateStoreContext) > [1] 
org.apache.kafka.streams.state.internals.RocksDBStore@6a6a1320, 
org.apache.kafka.test.MockInternalProcessorContext@79eb85fd STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
RocksDBBlockCacheMetricsTest > shouldRecordCorrectBlockCacheUsage(RocksDBStore, 
StateStoreContext) > [1] 
org.apache.kafka.streams.state.internals.RocksDBStore@6a6a1320, 
org.apache.kafka.test.MockInternalProcessorContext@79eb85fd PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
RocksDBBlockCacheMetricsTest > shouldRecordCorrectBlockCacheUsage(RocksDBStore, 
StateStoreContext) > [2] 
org.apache.kafka.streams.state.internals.RocksDBTimestampedStore@79b66b65,

Re: [VOTE] 3.6.0 RC2

Hey Satish,

I verified KIP-898 functionality and the KAFKA-15473 patch.
+1 (non-binding)

Thanks!

On Mon, Oct 2, 2023 at 1:28 PM Justine Olshan
 wrote:
>
> Hey all -- I noticed we still have the system tests as something that will
> be updated. Did we get a run for this RC?
>
> On Mon, Oct 2, 2023 at 1:24 PM Bill Bejeck  wrote:
>
> > Hi Satish,
> >
> > Thanks for running the release.
> > I performed the following steps:
> >
> >- Validated all the checksums, signatures, and keys
> >- Built the release from source
> >- Ran all unit tests
> >- Quick start validations
> >   - ZK and Kraft
> >   - Connect
> >   - Kafka Streams
> >- Spot checked java docs and documentation
> >
> > +1 (binding)
> >
> > - Bill
> >
> > On Mon, Oct 2, 2023 at 10:23 AM Proven Provenzano
> >  wrote:
> >
> > > Hi,
> > >
> > > To verify the release of release 3.6.0 RC2 I did the following:
> > >
> > >- Downloaded the source, built and ran the tests.
> > >- Validated SCRAM with KRaft including creating credentials with
> > >kafka-storage.
> > >- Validated Delegation Tokens with KRaft
> > >
> > > +1 (non-binding)
> > >
> > > --Proven
> > >
> > >
> > >
> > > On Mon, Oct 2, 2023 at 8:37 AM Divij Vaidya 
> > > wrote:
> > >
> > > > + 1 (non-binding)
> > > >
> > > > Verifications:
> > > > 1. I ran a produce-consume workload with plaintext auth, JDK17, zstd
> > > > compression using an open messaging benchmark and found 3.6 to be
> > better
> > > > than or equal to 3.5.1 across all dimensions. Notably, 3.6 had
> > > consistently
> > > > 6-7% lower CPU utilization, lesser spikes on P99 produce latencies and
> > > > overall lower P99.8 latencies.
> > > >
> > > > 2. I have verified that detached signature is correct using
> > > > https://www.apache.org/info/verification.html and the release manager
> > > > public keys are available at
> > > > https://keys.openpgp.org/search?q=F65DC3423D4CD7B9
> > > >
> > > > 3. I have verified that all metrics emitted in 3.5.1 (with Zk) are also
> > > > being emitted in 3.6.0 (with Zk).
> > > >
> > > > Problems (but not blockers):
> > > > 1. Metrics added in
> > > >
> > > >
> > >
> > https://github.com/apache/kafka/commit/2f71708955b293658cec3b27e9a5588d39c38d7e
> > > > aren't available in the documentation (cc: Justine). I don't consider
> > > this
> > > > as a release blocker but we should add it as a fast follow-up.
> > > >
> > > > 2. Metric added in
> > > >
> > > >
> > >
> > https://github.com/apache/kafka/commit/a900794ace4dcf1f9dadee27fbd8b63979532a18
> > > > isn't available in documentation (cc: David). I don't consider this as
> > a
> > > > release blocker but we should add it as a fast follow-up.
> > > >
> > > > --
> > > > Divij Vaidya
> > > >
> > > >
> > > >
> > > > On Mon, Oct 2, 2023 at 9:50 AM Federico Valeri 
> > > > wrote:
> > > >
> > > > > Hi Satish, I did the following to verify the release:
> > > > >
> > > > > - Built from source with Java 17 and Scala 2.13
> > > > > - Ran all unit and integration tests
> > > > > - Spot checked documentation
> > > > > - Ran custom client applications using staging artifacts on a 3-nodes
> > > > > cluster
> > > > > - Tested tiered storage with one of the available RSM implementations
> > > > >
> > > > > +1 (non binding)
> > > > >
> > > > > Thanks
> > > > > Fede
> > > > >
> > > > > On Mon, Oct 2, 2023 at 8:50 AM Luke Chen  wrote:
> > > > > >
> > > > > > Hi Satish,
> > > > > >
> > > > > > I verified with:
> > > > > > 1. Ran quick start in KRaft for scala 2.12 artifact
> > > > > > 2. Making sure the checksum are correct
> > > > > > 3. Browsing release notes, documents, javadocs, protocols.
> > > > > > 4. Verified the tiered storage feature works well.
> > > > > >
> > > > > > +1 (binding).
> > > > > >
> > > > > > Thanks.
> > > > > > Luke
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Oct 2, 2023 at 5:23 AM Jakub Scholz 
> > wrote:
> > > > > >
> > > > > > > +1 (non-binding). I used the Scala 2.13 binaries and the staged
> > > Maven
> > > > > > > artifacts and run my tests. Everything seems to work fine for me.
> > > > > > >
> > > > > > > Thanks
> > > > > > > Jakub
> > > > > > >
> > > > > > > On Fri, Sep 29, 2023 at 8:17 PM Satish Duggana <
> > > > > satish.dugg...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello Kafka users, developers and client-developers,
> > > > > > > >
> > > > > > > > This is the third candidate for the release of Apache Kafka
> > > 3.6.0.
> > > > > > > > Some of the major features include:
> > > > > > > >
> > > > > > > > * KIP-405 : Kafka Tiered Storage
> > > > > > > > * KIP-868 : KRaft Metadata Transactions
> > > > > > > > * KIP-875: First-class offsets support in Kafka Connect
> > > > > > > > * KIP-898: Modernize Connect plugin discovery
> > > > > > > > * KIP-938: Add more metrics for measuring KRaft performance
> > > > > > > > * KIP-902: Upgrade Zookeeper to 3.8.1
> > > > > > > > * KIP-917: Additional custom metadata for remote log segment
> > > > > > > >
> > > > >

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.6 #82

See

[REVIEW REQUEST] ReassignPartitionCommand scala -> java

2023-10-02 Thread Николай Ижиков

Hello.

Thanks everyone for the help.
I think we come to the stage when command itself can be reviewed.

All tests and dependencies of ReassignPartitionCommand translated from scala to 
java in previous PR’s.

Please, join the review - https://github.com/apache/kafka/pull/13247

Re: Re: [DISCUSS] Cluster Linking / Cross-Cluster Replication - Call for Interest & Co-Authors

Hi everyone,

Following up on this thread, I've opened a draft KIP for everyone to
contribute to:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-986%3A+Cross-Cluster+Replication
https://lists.apache.org/thread/4563fqdcwlqwjqlm7mbj8wftx1rpo0yd

Please consider yourselves owners of this feature, and make edits to
the KIP directly when you feel it is appropriate.

Thanks!
Greg Harris

On Mon, May 22, 2023 at 4:59 AM Andrew Otto  wrote:
>
> Hello!
>
> The Wikimedia Foundation would be very interested in this feature.  It
> would make deployment of multi DC/region streaming apps much simpler.  We'd
> like to investigate more the feasibility of using a Kafka stretch cluster
> to solve
> this problem, but something built into Kafka itself that still allows for
> distinct Kafka clusters would be really great!
>
> -Andrew Otto
>
> On Sat, May 20, 2023 at 7:42 AM hudeqi <16120...@bjtu.edu.cn> wrote:
>
> > Hi, I am also very excited to see this discussion, because I also
> > implemented the "federation model" based on the kafka-0.10.2.1 version
> > before in company and got benefits from going online. It solves the problem
> > of seamlessly migrating the bytesIn/bytesOut of a topic to another kafka
> > cluster without the client being aware. Therefore, it includes the
> > cross-cluster synchronous topic data mentioned in this discussion, and
> > ensures the consistency of the offset. It also provides the global
> > coordinator service to ensure the consistency of the group offset, and the
> > metadata service to ensure that the client is visible of the real traffic
> > cluster of the topic. So I am very excited to see that the community has
> > finally discussed this feature, and I especially hope to participate in the
> > development and reviewer of this feature as a co-author.
> >
> >
> >  -原始邮件-
> >  发件人: "Chia-Ping Tsai" 
> >  发送时间: 2023-05-20 18:57:31 (星期六)
> >  收件人: dev@kafka.apache.org
> >  抄送: t...@conduktor.io.invalid
> >  主题: Re: [DISCUSS] Cluster Linking / Cross-Cluster Replication - Call
> > for Interest  Co-Authors
> > 
> >

[jira] [Created] (KAFKA-15528) KIP-986: Cross-Cluster Replication

2023-10-02 Thread Greg Harris (Jira)

Greg Harris created KAFKA-15528:
---

 Summary: KIP-986: Cross-Cluster Replication
 Key: KAFKA-15528
 URL: https://issues.apache.org/jira/browse/KAFKA-15528
 Project: Kafka
  Issue Type: New Feature
Reporter: Greg Harris


https://cwiki.apache.org/confluence/display/KAFKA/KIP-986%3A+Cross-Cluster+Replication



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[DISCUSS] KIP-986: Cross-Cluster Replication

Hi all,

I've opened an extremely early draft of the Cross-Cluster Replication
feature, and I would like to invite any and all co-authors to expand
on it. Find the page here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-986%3A+Cross-Cluster+Replication

This is not strictly an invitation to "review" the KIP, as the
document has much less detail than other KIPs of similar complexity.
But if you are knowledgeable in this area, some early sanity checks
would be greatly appreciated.

I've included a "shopping list" of properties that appear to me to be
desirable, but I don't have an implementation in mind that meets these
requirements. If you have additional requirements, an alternative UX
in mind, or wish to propose some implementation details, please edit
the KIP with your contributions.

Thanks everyone!

Greg Harris
Aiven, Inc

Re: [VOTE] 3.6.0 RC2

2023-10-02 Thread Justine Olshan

Hey all -- I noticed we still have the system tests as something that will
be updated. Did we get a run for this RC?

On Mon, Oct 2, 2023 at 1:24 PM Bill Bejeck  wrote:

> Hi Satish,
>
> Thanks for running the release.
> I performed the following steps:
>
>- Validated all the checksums, signatures, and keys
>- Built the release from source
>- Ran all unit tests
>- Quick start validations
>   - ZK and Kraft
>   - Connect
>   - Kafka Streams
>- Spot checked java docs and documentation
>
> +1 (binding)
>
> - Bill
>
> On Mon, Oct 2, 2023 at 10:23 AM Proven Provenzano
>  wrote:
>
> > Hi,
> >
> > To verify the release of release 3.6.0 RC2 I did the following:
> >
> >- Downloaded the source, built and ran the tests.
> >- Validated SCRAM with KRaft including creating credentials with
> >kafka-storage.
> >- Validated Delegation Tokens with KRaft
> >
> > +1 (non-binding)
> >
> > --Proven
> >
> >
> >
> > On Mon, Oct 2, 2023 at 8:37 AM Divij Vaidya 
> > wrote:
> >
> > > + 1 (non-binding)
> > >
> > > Verifications:
> > > 1. I ran a produce-consume workload with plaintext auth, JDK17, zstd
> > > compression using an open messaging benchmark and found 3.6 to be
> better
> > > than or equal to 3.5.1 across all dimensions. Notably, 3.6 had
> > consistently
> > > 6-7% lower CPU utilization, lesser spikes on P99 produce latencies and
> > > overall lower P99.8 latencies.
> > >
> > > 2. I have verified that detached signature is correct using
> > > https://www.apache.org/info/verification.html and the release manager
> > > public keys are available at
> > > https://keys.openpgp.org/search?q=F65DC3423D4CD7B9
> > >
> > > 3. I have verified that all metrics emitted in 3.5.1 (with Zk) are also
> > > being emitted in 3.6.0 (with Zk).
> > >
> > > Problems (but not blockers):
> > > 1. Metrics added in
> > >
> > >
> >
> https://github.com/apache/kafka/commit/2f71708955b293658cec3b27e9a5588d39c38d7e
> > > aren't available in the documentation (cc: Justine). I don't consider
> > this
> > > as a release blocker but we should add it as a fast follow-up.
> > >
> > > 2. Metric added in
> > >
> > >
> >
> https://github.com/apache/kafka/commit/a900794ace4dcf1f9dadee27fbd8b63979532a18
> > > isn't available in documentation (cc: David). I don't consider this as
> a
> > > release blocker but we should add it as a fast follow-up.
> > >
> > > --
> > > Divij Vaidya
> > >
> > >
> > >
> > > On Mon, Oct 2, 2023 at 9:50 AM Federico Valeri 
> > > wrote:
> > >
> > > > Hi Satish, I did the following to verify the release:
> > > >
> > > > - Built from source with Java 17 and Scala 2.13
> > > > - Ran all unit and integration tests
> > > > - Spot checked documentation
> > > > - Ran custom client applications using staging artifacts on a 3-nodes
> > > > cluster
> > > > - Tested tiered storage with one of the available RSM implementations
> > > >
> > > > +1 (non binding)
> > > >
> > > > Thanks
> > > > Fede
> > > >
> > > > On Mon, Oct 2, 2023 at 8:50 AM Luke Chen  wrote:
> > > > >
> > > > > Hi Satish,
> > > > >
> > > > > I verified with:
> > > > > 1. Ran quick start in KRaft for scala 2.12 artifact
> > > > > 2. Making sure the checksum are correct
> > > > > 3. Browsing release notes, documents, javadocs, protocols.
> > > > > 4. Verified the tiered storage feature works well.
> > > > >
> > > > > +1 (binding).
> > > > >
> > > > > Thanks.
> > > > > Luke
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Oct 2, 2023 at 5:23 AM Jakub Scholz 
> wrote:
> > > > >
> > > > > > +1 (non-binding). I used the Scala 2.13 binaries and the staged
> > Maven
> > > > > > artifacts and run my tests. Everything seems to work fine for me.
> > > > > >
> > > > > > Thanks
> > > > > > Jakub
> > > > > >
> > > > > > On Fri, Sep 29, 2023 at 8:17 PM Satish Duggana <
> > > > satish.dugg...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hello Kafka users, developers and client-developers,
> > > > > > >
> > > > > > > This is the third candidate for the release of Apache Kafka
> > 3.6.0.
> > > > > > > Some of the major features include:
> > > > > > >
> > > > > > > * KIP-405 : Kafka Tiered Storage
> > > > > > > * KIP-868 : KRaft Metadata Transactions
> > > > > > > * KIP-875: First-class offsets support in Kafka Connect
> > > > > > > * KIP-898: Modernize Connect plugin discovery
> > > > > > > * KIP-938: Add more metrics for measuring KRaft performance
> > > > > > > * KIP-902: Upgrade Zookeeper to 3.8.1
> > > > > > > * KIP-917: Additional custom metadata for remote log segment
> > > > > > >
> > > > > > > Release notes for the 3.6.0 release:
> > > > > > >
> > > https://home.apache.org/~satishd/kafka-3.6.0-rc2/RELEASE_NOTES.html
> > > > > > >
> > > > > > > *** Please download, test and vote by Tuesday, October 3, 12pm
> PT
> > > > > > >
> > > > > > > Kafka's KEYS file containing PGP keys we use to sign the
> release:
> > > > > > > https://kafka.apache.org/KEYS
> > > > > > >
> > > > > > > * Release artifacts to be voted upon (source and binary):
>

Re: [VOTE] 3.6.0 RC2

2023-10-02 Thread Bill Bejeck

Hi Satish,

Thanks for running the release.
I performed the following steps:

   - Validated all the checksums, signatures, and keys
   - Built the release from source
   - Ran all unit tests
   - Quick start validations
  - ZK and Kraft
  - Connect
  - Kafka Streams
   - Spot checked java docs and documentation

+1 (binding)

- Bill

On Mon, Oct 2, 2023 at 10:23 AM Proven Provenzano
 wrote:

> Hi,
>
> To verify the release of release 3.6.0 RC2 I did the following:
>
>- Downloaded the source, built and ran the tests.
>- Validated SCRAM with KRaft including creating credentials with
>kafka-storage.
>- Validated Delegation Tokens with KRaft
>
> +1 (non-binding)
>
> --Proven
>
>
>
> On Mon, Oct 2, 2023 at 8:37 AM Divij Vaidya 
> wrote:
>
> > + 1 (non-binding)
> >
> > Verifications:
> > 1. I ran a produce-consume workload with plaintext auth, JDK17, zstd
> > compression using an open messaging benchmark and found 3.6 to be better
> > than or equal to 3.5.1 across all dimensions. Notably, 3.6 had
> consistently
> > 6-7% lower CPU utilization, lesser spikes on P99 produce latencies and
> > overall lower P99.8 latencies.
> >
> > 2. I have verified that detached signature is correct using
> > https://www.apache.org/info/verification.html and the release manager
> > public keys are available at
> > https://keys.openpgp.org/search?q=F65DC3423D4CD7B9
> >
> > 3. I have verified that all metrics emitted in 3.5.1 (with Zk) are also
> > being emitted in 3.6.0 (with Zk).
> >
> > Problems (but not blockers):
> > 1. Metrics added in
> >
> >
> https://github.com/apache/kafka/commit/2f71708955b293658cec3b27e9a5588d39c38d7e
> > aren't available in the documentation (cc: Justine). I don't consider
> this
> > as a release blocker but we should add it as a fast follow-up.
> >
> > 2. Metric added in
> >
> >
> https://github.com/apache/kafka/commit/a900794ace4dcf1f9dadee27fbd8b63979532a18
> > isn't available in documentation (cc: David). I don't consider this as a
> > release blocker but we should add it as a fast follow-up.
> >
> > --
> > Divij Vaidya
> >
> >
> >
> > On Mon, Oct 2, 2023 at 9:50 AM Federico Valeri 
> > wrote:
> >
> > > Hi Satish, I did the following to verify the release:
> > >
> > > - Built from source with Java 17 and Scala 2.13
> > > - Ran all unit and integration tests
> > > - Spot checked documentation
> > > - Ran custom client applications using staging artifacts on a 3-nodes
> > > cluster
> > > - Tested tiered storage with one of the available RSM implementations
> > >
> > > +1 (non binding)
> > >
> > > Thanks
> > > Fede
> > >
> > > On Mon, Oct 2, 2023 at 8:50 AM Luke Chen  wrote:
> > > >
> > > > Hi Satish,
> > > >
> > > > I verified with:
> > > > 1. Ran quick start in KRaft for scala 2.12 artifact
> > > > 2. Making sure the checksum are correct
> > > > 3. Browsing release notes, documents, javadocs, protocols.
> > > > 4. Verified the tiered storage feature works well.
> > > >
> > > > +1 (binding).
> > > >
> > > > Thanks.
> > > > Luke
> > > >
> > > >
> > > >
> > > > On Mon, Oct 2, 2023 at 5:23 AM Jakub Scholz  wrote:
> > > >
> > > > > +1 (non-binding). I used the Scala 2.13 binaries and the staged
> Maven
> > > > > artifacts and run my tests. Everything seems to work fine for me.
> > > > >
> > > > > Thanks
> > > > > Jakub
> > > > >
> > > > > On Fri, Sep 29, 2023 at 8:17 PM Satish Duggana <
> > > satish.dugg...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hello Kafka users, developers and client-developers,
> > > > > >
> > > > > > This is the third candidate for the release of Apache Kafka
> 3.6.0.
> > > > > > Some of the major features include:
> > > > > >
> > > > > > * KIP-405 : Kafka Tiered Storage
> > > > > > * KIP-868 : KRaft Metadata Transactions
> > > > > > * KIP-875: First-class offsets support in Kafka Connect
> > > > > > * KIP-898: Modernize Connect plugin discovery
> > > > > > * KIP-938: Add more metrics for measuring KRaft performance
> > > > > > * KIP-902: Upgrade Zookeeper to 3.8.1
> > > > > > * KIP-917: Additional custom metadata for remote log segment
> > > > > >
> > > > > > Release notes for the 3.6.0 release:
> > > > > >
> > https://home.apache.org/~satishd/kafka-3.6.0-rc2/RELEASE_NOTES.html
> > > > > >
> > > > > > *** Please download, test and vote by Tuesday, October 3, 12pm PT
> > > > > >
> > > > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > > > https://kafka.apache.org/KEYS
> > > > > >
> > > > > > * Release artifacts to be voted upon (source and binary):
> > > > > > https://home.apache.org/~satishd/kafka-3.6.0-rc2/
> > > > > >
> > > > > > * Maven artifacts to be voted upon:
> > > > > >
> > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > > >
> > > > > > * Javadoc:
> > > > > > https://home.apache.org/~satishd/kafka-3.6.0-rc2/javadoc/
> > > > > >
> > > > > > * Tag to be voted upon (off 3.6 branch) is the 3.6.0-rc2 tag:
> > > > > > https://github.com/apache/kafka/releases/tag/3.6.0-rc2
> > > >

Re: [DISCUSS] KIP-979: Allow independently stop KRaft controllers or brokers

2023-10-02 Thread Hailey Ni

Hi Kamal,

I think the broker.id property has been replaced with the `node.id` property
in KRaft.  The documentation for `node.id` says it is required (
https://github.com/apache/kafka/blob/72e275f6ea867747e6b4e524c80d5ebd726ac25b/core/src/main/scala/kafka/server/KafkaConfig.scala#L741),
and the QuickStart files all use it (
https://github.com/apache/kafka/tree/72e275f6ea867747e6b4e524c80d5ebd726ac25b/config/kraft).
It is technically true that these two configs are treated as synonyms of
one another (
https://github.com/apache/kafka/blob/72e275f6ea867747e6b4e524c80d5ebd726ac25b/core/src/main/scala/kafka/server/KafkaConfig.scala#L1587-L1597),
so if you specify either one the process will still recognize it and
start.  But it makes sense to exclusively use `node.id` in KRaft because a
node isn't necessarily a broker anymore; it could be a controller (or even
a combined broker+controller).

Thanks,
Hailey

On Mon, Oct 2, 2023 at 1:17 PM Hailey Ni  wrote:

> Hi Ismeal,
>
> Thanks for the comments. I'll change the implementation to use a pair of
> mutually exclusive args --process.roles and --node.id.
>
> Thanks,
> Hailey
>
> On Mon, Oct 2, 2023 at 6:34 AM Ismael Juma  wrote:
>
>> Hi Ron,
>>
>> Yes, that's what I am proposing, yes.
>>
>> Ismael
>>
>> On Sat, Sep 30, 2023 at 2:30 PM Ron Dagostino  wrote:
>>
>> > Thanks, Ismael.  I think you are proposing a pair of mutually exclusive
>> > args --process.roles and --node.id, right?  I agree that is more
>> > user-friendly than the --required-config arg, and it comes at the
>> possible
>> > expense of generality.  So that’s the tradeoff between the two, I think.
>> > No other config comes to mind now that we’ve identified these two.  I
>> think
>> > the two specific and mutually exclusive parameters would be the way to
>> go
>> > unless someone else identifies still more options that people might
>> want.
>> >
>> > Did I get that right, or were you proposing something different?
>> >
>> > Ron
>> >
>> > > On Sep 30, 2023, at 10:42 AM, Ismael Juma  wrote:
>> > >
>> > > Hi,
>> > >
>> > > Thanks for the KIP. I think this approach based on configs is a bit
>> too
>> > > open ended and not very user friendly. Why don't we simply provide
>> flags
>> > > for the things a user may care about? So far, it seems like we have
>> two
>> > > good candidates (node id and process role). Are there any others?
>> > >
>> > > Ismael
>> > >
>> > >> On Fri, Sep 29, 2023 at 6:19 PM Hailey Ni 
>> > wrote:
>> > >>
>> > >> Hi Ron,
>> > >>
>> > >> I think you made a great point, making the "name" arbitrary instead
>> of
>> > >> hard-coding it will make the functionality much more flexible. I've
>> > updated
>> > >> the KIP and the code accordingly. Thanks for the great idea!
>> > >>
>> > >> Thanks,
>> > >> Hailey
>> > >>
>> > >>
>> > >>> On Fri, Sep 29, 2023 at 2:34 PM Ron Dagostino 
>> > wrote:
>> > >>>
>> > >>> Thanks, Hailey.  Is there a reason to restrict it to just
>> > >>> process.roles and node.id?  Someone might want to do
>> > >>> "--required-config any.name=whatever.value", for example, and at
>> first
>> > >>> glance I don't see a reason why the implementation should be any
>> > >>> different -- it seems it would probably be easier to not have to
>> worry
>> > >>> about restricting to specific cases, actually.  WDYT?
>> > >>>
>> > >>> Ron
>> > >>>
>> > >>> On Fri, Sep 29, 2023 at 5:12 PM Hailey Ni > >
>> > >>> wrote:
>> > 
>> >  Updated. Please let me know if you have any additional comments.
>> Thank
>> > >>> you!
>> > 
>> >  On Thu, Sep 21, 2023 at 3:02 PM Hailey Ni 
>> wrote:
>> > 
>> > > Hi Ron. Thanks for the response. I agree with your point. I'll
>> make
>> > >> the
>> > > corresponding changes in the KIP and KAFKA-15471
>> > > .
>> > >
>> > > On Thu, Sep 21, 2023 at 1:40 PM Ron Dagostino 
>> > >>> wrote:
>> > >
>> > >> Hi Hailey.  No, I just looked, and zookeeper-server-stop does not
>> > >> have
>> > >> any facility to be specific about which ZK nodes to signal.  So
>> > >> providing the ability in kafka-server-stop to be more specific
>> than
>> > >> just "signal all controllers" or "signal all brokers" would be a
>> > >> bonus
>> > >> and therefore not necessarily required.  But if it is easy to
>> > >> achieve
>> > >> and doesn't add any additional cognitive load -- and at first
>> glance
>> > >> it does seem so -- we should probably just support it.
>> > >>
>> > >> Ron
>> > >>
>> > >> On Wed, Sep 20, 2023 at 6:15 PM Hailey Ni
>> > > >>>
>> > >> wrote:
>> > >>>
>> > >>> Hi Ron,
>> > >>>
>> > >>> Thank you very much for the comment. I think it makes sense to
>> me
>> > >>> that
>> > >> we
>> > >>> provide an even more specific way to kill individual
>> > >> controllers/brokers.
>> > >>> I have one question: does the command line for ZooKeeper cluster
>> > >>> provide
>> > >>> such a way to

Re: [DISCUSS] KIP-979: Allow independently stop KRaft controllers or brokers

2023-10-02 Thread Hailey Ni

Hi Ismeal,

Thanks for the comments. I'll change the implementation to use a pair of
mutually exclusive args --process.roles and --node.id.

Thanks,
Hailey

On Mon, Oct 2, 2023 at 6:34 AM Ismael Juma  wrote:

> Hi Ron,
>
> Yes, that's what I am proposing, yes.
>
> Ismael
>
> On Sat, Sep 30, 2023 at 2:30 PM Ron Dagostino  wrote:
>
> > Thanks, Ismael.  I think you are proposing a pair of mutually exclusive
> > args --process.roles and --node.id, right?  I agree that is more
> > user-friendly than the --required-config arg, and it comes at the
> possible
> > expense of generality.  So that’s the tradeoff between the two, I think.
> > No other config comes to mind now that we’ve identified these two.  I
> think
> > the two specific and mutually exclusive parameters would be the way to go
> > unless someone else identifies still more options that people might want.
> >
> > Did I get that right, or were you proposing something different?
> >
> > Ron
> >
> > > On Sep 30, 2023, at 10:42 AM, Ismael Juma  wrote:
> > >
> > > Hi,
> > >
> > > Thanks for the KIP. I think this approach based on configs is a bit too
> > > open ended and not very user friendly. Why don't we simply provide
> flags
> > > for the things a user may care about? So far, it seems like we have two
> > > good candidates (node id and process role). Are there any others?
> > >
> > > Ismael
> > >
> > >> On Fri, Sep 29, 2023 at 6:19 PM Hailey Ni 
> > wrote:
> > >>
> > >> Hi Ron,
> > >>
> > >> I think you made a great point, making the "name" arbitrary instead of
> > >> hard-coding it will make the functionality much more flexible. I've
> > updated
> > >> the KIP and the code accordingly. Thanks for the great idea!
> > >>
> > >> Thanks,
> > >> Hailey
> > >>
> > >>
> > >>> On Fri, Sep 29, 2023 at 2:34 PM Ron Dagostino 
> > wrote:
> > >>>
> > >>> Thanks, Hailey.  Is there a reason to restrict it to just
> > >>> process.roles and node.id?  Someone might want to do
> > >>> "--required-config any.name=whatever.value", for example, and at
> first
> > >>> glance I don't see a reason why the implementation should be any
> > >>> different -- it seems it would probably be easier to not have to
> worry
> > >>> about restricting to specific cases, actually.  WDYT?
> > >>>
> > >>> Ron
> > >>>
> > >>> On Fri, Sep 29, 2023 at 5:12 PM Hailey Ni 
> > >>> wrote:
> > 
> >  Updated. Please let me know if you have any additional comments.
> Thank
> > >>> you!
> > 
> >  On Thu, Sep 21, 2023 at 3:02 PM Hailey Ni  wrote:
> > 
> > > Hi Ron. Thanks for the response. I agree with your point. I'll make
> > >> the
> > > corresponding changes in the KIP and KAFKA-15471
> > > .
> > >
> > > On Thu, Sep 21, 2023 at 1:40 PM Ron Dagostino 
> > >>> wrote:
> > >
> > >> Hi Hailey.  No, I just looked, and zookeeper-server-stop does not
> > >> have
> > >> any facility to be specific about which ZK nodes to signal.  So
> > >> providing the ability in kafka-server-stop to be more specific
> than
> > >> just "signal all controllers" or "signal all brokers" would be a
> > >> bonus
> > >> and therefore not necessarily required.  But if it is easy to
> > >> achieve
> > >> and doesn't add any additional cognitive load -- and at first
> glance
> > >> it does seem so -- we should probably just support it.
> > >>
> > >> Ron
> > >>
> > >> On Wed, Sep 20, 2023 at 6:15 PM Hailey Ni
>  > >>>
> > >> wrote:
> > >>>
> > >>> Hi Ron,
> > >>>
> > >>> Thank you very much for the comment. I think it makes sense to me
> > >>> that
> > >> we
> > >>> provide an even more specific way to kill individual
> > >> controllers/brokers.
> > >>> I have one question: does the command line for ZooKeeper cluster
> > >>> provide
> > >>> such a way to kill individual controllers/brokers?
> > >>>
> > >>> Thanks,
> > >>> Hailey
> > >>>
> > >>> On Tue, Sep 19, 2023 at 11:01 AM Ron Dagostino <
> rndg...@gmail.com
> > >>>
> > >> wrote:
> > >>>
> >  Thanks for the KIP, Hailey.  It will be nice to provide some
> >  fine-grained control for when people running the broker and
> > >>> controller
> >  this way want to stop just one of them.
> > 
> >  One thing that occurs to me is that in a development environment
> >  someone might want to run multiple controllers and multiple
> > >>> brokers
> >  all on the same box, and in that case they might want to
> > >> actually
> > >>> stop
> >  just one controller or just one broker instead of all of them.
> > >>> So I'm
> >  wondering if maybe instead of supporting kafka-server-stop
> >  [--process.roles ] we might want to instead support
> >  kafka-server-stop [--required-config ].  If someone
> > >>> wanted
> >  to stop any/all controllers and not touch the broker(s) they
> > >> could
> >

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2246

See 


Changes:


--
[...truncated 420469 lines...]

Gradle Test Run :streams:test > Gradle Test Executor 77 > TasksTest > 
onlyRemovePendingTaskToCloseDirtyShouldRemoveTaskFromPendingUpdateActions() 
STARTED

Gradle Test Run :streams:test > Gradle Test Executor 77 > TasksTest > 
onlyRemovePendingTaskToCloseDirtyShouldRemoveTaskFromPendingUpdateActions() 
PASSED

Gradle Test Run :streams:test > Gradle Test Executor 77 > TasksTest > 
shouldAddAndRemovePendingTaskToSuspend() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 77 > TasksTest > 
shouldAddAndRemovePendingTaskToSuspend() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 77 > TasksTest > 
shouldVerifyIfPendingTaskToInitExist() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 77 > TasksTest > 
shouldVerifyIfPendingTaskToInitExist() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 77 > TasksTest > 
onlyRemovePendingTaskToCloseCleanShouldRemoveTaskFromPendingUpdateActions() 
STARTED

Gradle Test Run :streams:test > Gradle Test Executor 77 > TasksTest > 
onlyRemovePendingTaskToCloseCleanShouldRemoveTaskFromPendingUpdateActions() 
PASSED

Gradle Test Run :streams:test > Gradle Test Executor 77 > TasksTest > 
shouldDrainPendingTasksToCreate() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 77 > TasksTest > 
shouldDrainPendingTasksToCreate() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 77 > TasksTest > 
onlyRemovePendingTaskToRecycleShouldRemoveTaskFromPendingUpdateActions() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 77 > TasksTest > 
onlyRemovePendingTaskToRecycleShouldRemoveTaskFromPendingUpdateActions() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 77 > TasksTest > 
shouldAddAndRemovePendingTaskToCloseClean() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 77 > TasksTest > 
shouldAddAndRemovePendingTaskToCloseClean() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 77 > TasksTest > 
shouldAddAndRemovePendingTaskToCloseDirty() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 77 > TasksTest > 
shouldAddAndRemovePendingTaskToCloseDirty() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 77 > TasksTest > 
shouldKeepAddedTasks() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 77 > TasksTest > 
shouldKeepAddedTasks() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 77 > StateQueryResultTest 
> More than one query result throws IllegalArgumentException STARTED

Gradle Test Run :streams:test > Gradle Test Executor 77 > StateQueryResultTest 
> More than one query result throws IllegalArgumentException PASSED

Gradle Test Run :streams:test > Gradle Test Executor 77 > StateQueryResultTest 
> Zero query results shouldn't error STARTED

Gradle Test Run :streams:test > Gradle Test Executor 77 > StateQueryResultTest 
> Zero query results shouldn't error PASSED

Gradle Test Run :streams:test > Gradle Test Executor 77 > StateQueryResultTest 
> Valid query results still works STARTED

Gradle Test Run :streams:test > Gradle Test Executor 77 > StateQueryResultTest 
> Valid query results still works PASSED

Gradle Test Run :streams:test > Gradle Test Executor 77 > 
RocksDBBlockCacheMetricsTest > shouldRecordCorrectBlockCacheUsage(RocksDBStore, 
StateStoreContext) > [1] 
org.apache.kafka.streams.state.internals.RocksDBStore@7719d8f0, 
org.apache.kafka.test.MockInternalProcessorContext@2980c3a4 STARTED

Gradle Test Run :streams:test > Gradle Test Executor 77 > 
RocksDBBlockCacheMetricsTest > shouldRecordCorrectBlockCacheUsage(RocksDBStore, 
StateStoreContext) > [1] 
org.apache.kafka.streams.state.internals.RocksDBStore@7719d8f0, 
org.apache.kafka.test.MockInternalProcessorContext@2980c3a4 PASSED

Gradle Test Run :streams:test > Gradle Test Executor 77 > 
RocksDBBlockCacheMetricsTest > shouldRecordCorrectBlockCacheUsage(RocksDBStore, 
StateStoreContext) > [2] 
org.apache.kafka.streams.state.internals.RocksDBTimestampedStore@6dd69cc5, 
org.apache.kafka.test.MockInternalProcessorContext@a4113db STARTED

Gradle Test Run :streams:test > Gradle Test Executor 77 > 
RocksDBBlockCacheMetricsTest > shouldRecordCorrectBlockCacheUsage(RocksDBStore, 
StateStoreContext) > [2] 
org.apache.kafka.streams.state.internals.RocksDBTimestampedStore@6dd69cc5, 
org.apache.kafka.test.MockInternalProcessorContext@a4113db PASSED

Gradle Test Run :streams:test > Gradle Test Executor 77 > 
RocksDBBlockCacheMetricsTest > 
shouldRecordCorrectBlockCachePinnedUsage(RocksDBStore, StateStoreContext) > [1] 
org.apache.kafka.streams.state.internals.RocksDBStore@61246cc7, 
org.apache.kafka.test.MockInternalProcessorContext@362cc174 STARTED

Gradle Test Run :streams:test > Gradle Test Executor 77 > 
RocksDBBlockCacheMetricsTest > 
shouldRecordCorrectBlockCachePinnedUsage(RocksDBStore, StateStoreContext) >

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.5 #76

See 


Changes:


--
[...truncated 2712284 lines...]
at jdk.internal.reflect.GeneratedMethodAccessor487.invoke(Unknown 
Source)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at 
org.gradle.internal.event.AbstractBroadcastDispatch.dispatch(AbstractBroadcastDispatch.java:43)
at 
org.gradle.internal.event.BroadcastDispatch$SingletonDispatch.dispatch(BroadcastDispatch.java:257)
at 
org.gradle.internal.event.BroadcastDispatch$SingletonDispatch.dispatch(BroadcastDispatch.java:164)
at 
org.gradle.internal.event.AbstractBroadcastDispatch.dispatch(AbstractBroadcastDispatch.java:83)
at 
org.gradle.internal.event.AbstractBroadcastDispatch.dispatch(AbstractBroadcastDispatch.java:69)
at 
org.gradle.internal.event.BroadcastDispatch$CompositeDispatch.dispatch(BroadcastDispatch.java:363)
at 
org.gradle.internal.event.BroadcastDispatch$CompositeDispatch.dispatch(BroadcastDispatch.java:261)
at 
org.gradle.internal.event.ListenerBroadcast.dispatch(ListenerBroadcast.java:148)
at 
org.gradle.internal.event.ListenerBroadcast.dispatch(ListenerBroadcast.java:37)
at 
org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
at com.sun.proxy.$Proxy102.output(Unknown Source)
at 
org.gradle.api.internal.tasks.testing.results.StateTrackingTestResultProcessor.output(StateTrackingTestResultProcessor.java:113)
at 
org.gradle.testretry.internal.executer.RetryTestResultProcessor.output(RetryTestResultProcessor.java:155)
at 
org.gradle.api.internal.tasks.testing.results.AttachParentTestResultProcessor.output(AttachParentTestResultProcessor.java:52)
at jdk.internal.reflect.GeneratedMethodAccessor490.invoke(Unknown 
Source)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at 
org.gradle.internal.dispatch.FailureHandlingDispatch.dispatch(FailureHandlingDispatch.java:30)
at 
org.gradle.internal.dispatch.AsyncDispatch.dispatchMessages(AsyncDispatch.java:87)
at 
org.gradle.internal.dispatch.AsyncDispatch.access$000(AsyncDispatch.java:36)
at 
org.gradle.internal.dispatch.AsyncDispatch$1.run(AsyncDispatch.java:71)
at 
org.gradle.internal.concurrent.InterruptibleRunnable.run(InterruptibleRunnable.java:42)
at 
org.gradle.internal.operations.CurrentBuildOperationPreservingRunnable.run(CurrentBuildOperationPreservingRunnable.java:42)
at 
org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
at 
org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
java.io.IOException: No space left on device
com.esotericsoftware.kryo.KryoException: java.io.IOException: No space left on 
device
at com.esotericsoftware.kryo.io.Output.flush(Output.java:165)
at com.esotericsoftware.kryo.io.Output.require(Output.java:142)
at com.esotericsoftware.kryo.io.Output.writeBoolean(Output.java:613)
at 
org.gradle.internal.serialize.kryo.KryoBackedEncoder.writeBoolean(KryoBackedEncoder.java:73)
at 
org.gradle.api.internal.tasks.testing.junit.result.TestOutputStore$Writer.onOutput(TestOutputStore.java:107)
at 
org.gradle.api.internal.tasks.testing.junit.result.TestReportDataCollector.onOutput(TestReportDataCollector.java:152)
at jdk.internal.reflect.GeneratedMethodAccessor488.invoke(Unknown 
Source)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at

[jira] [Resolved] (KAFKA-15449) Verify transactional offset commits (KIP-890 part 1)

2023-10-02 Thread Justine Olshan (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-15449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Justine Olshan resolved KAFKA-15449.

Resolution: Fixed

> Verify transactional offset commits (KIP-890 part 1)
> 
>
> Key: KAFKA-15449
> URL: https://issues.apache.org/jira/browse/KAFKA-15449
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Justine Olshan
>Assignee: Justine Olshan
>Priority: Critical
>
> We verify on produce requests but not offset commits. We should fix this to 
> avoid hanging transactions on consumer offset partitions.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-980: Allow creating connectors in a stopped state

2023-10-02 Thread Chris Egerton

Hi Yash,

Thanks for the KIP! Frankly this feels like an oversight in 875 and I'm
glad we're closing that loop ASAP.


Here are my thoughts:

1. (Nit): IMO "start.state", "initial.state", or "target.state" might be
better than just "state" for the field name in the connector creation
request.

2. Why implement this in distributed mode with two writes to the config
topic? We could augment the existing format for connector configs in the
config topic [1] with a new field instead.

3. Although standalone mode is mentioned in the KIP, there's no detail on
the Java properties file format that we support for standalone mode (i.e.,
`connect-standalone config/connect-standalone.properties
config/connect-file-source.properties
config/connect-file-sink.properties`). Do we plan on adding support for
that mode as well?

4. I suspect there will be advantages for this feature beyond offsets
migration, but if offsets migration were the only motivation for it,
wouldn't it be simpler to accept an "offsets" field in connector creation
requests? That way, users wouldn't have to start a connector in a different
state and then resume it, they could just create the connector like normal.
I think the current proposal is acceptable, but wanted to float this
alternative in case we anticipate lots of connector migrations and want to
optimize for them a bit more.

5. (NIt): We can link to our own Javadocs [2] instead of javadoc.io


[1] -
https://github.com/apache/kafka/blob/dcd8c7d05f2f22f2d815405e7ab3ad7439669239/connect/runtime/src/main/java/org/apache/kafka/connect/storage/KafkaConfigBackingStore.java#L234-L236

[2] - https://kafka.apache.org/35/javadoc/index.html?overview-summary.html


Cheers,

Chris

On Thu, Sep 21, 2023 at 2:37 AM Yash Mayya  wrote:

> Hi Ashwin,
>
> Thanks for taking a look and sharing your thoughts!
>
> 1. Yes, the request / response formats of the two APIs were intentionally
> made identical for such use-cases. [1]
>
> 2. I'm assuming you're referring to retaining the offset / config topic
> records for a connector when it is deleted by a user? Firstly, a
> connector's offsets aren't actually currently deleted when the connector is
> deleted - it was listed as potential future work in KIP-875 [2]. Secondly,
> the config topic is the source of truth for the state of a Connect cluster
> and a connector deletion is done by writing a null-valued record
> (tombstone) with the connector key to the config topic. We could
> potentially modify the delete mechanism to use a special new record
> (instead of a tombstone with the connector key) in order to retain the
> latest configuration of a connector while still deleting the actual
> connector - however, this would have its downsides and I don't see too many
> benefits. Furthermore, connector migration between different Kafka clusters
> was just used as a representational use case for creating connectors in a
> stopped state - it isn't the primary focus of this KIP as such.
>
> 3. KIP-875 goes into some more details about this [3], but the TLDR is that
> the STOPPED state will be treated like the PAUSED state on older workers
> that don't recognize the STOPPED state.
>
> Thanks,
> Yash
>
> [1] -
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect#KIP875:FirstclassoffsetssupportinKafkaConnect-Request/responseformat
>
> [2] -
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect#KIP875:FirstclassoffsetssupportinKafkaConnect-Automaticallydeleteoffsetswithconnectors
>
> [3] -
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect#KIP875:FirstclassoffsetssupportinKafkaConnect-STOPPEDtargetstate
>
> On Wed, Sep 20, 2023 at 7:24 PM Ashwin 
> wrote:
>
> > Thanks Yash! This is very useful for migrating connectors from one
> cluster
> > to another.
> >
> > I had the following comments/questions
> >
> > 1. Is the offset read using `GET /offsets` api always guaranteed to be
> in a
> > format accepted by `PATCH /offsets` ?
> > 2. I had to tackle a similar migration situation but the two connect
> > clusters in question were using the same backing Kafka cluster. The
> > challenge in this case is that when I delete the original connector, I
> want
> > to retain offsets and config topics. Do you think we should support
> > deletion of a connector without removal of these topics as part of this
> KIP
> > ?
> > 3. In the case of a downgrade, how will Connect worker handle the
> optional
> > “state” field in config topic ?
> >
> > Thanks,
> > Ashwin
> >
> >
> >
> >
> > On Sun, Sep 17, 2023 at 11:09 PM Yash Mayya 
> wrote:
> >
> > > Hi all,
> > >
> > > I'd like to begin discussion on a KIP to allow creating connectors in a
> > > stopped state -
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-980%3A+Allow+creating+connectors+in+a+stopped+state
> > >
> > >
> > > Thanks,
> > > Yash
> > >
> >
>

[jira] [Created] (KAFKA-15527) Add reverseRange and reverseAll query over kv-store in IQv2

2023-10-02 Thread Hanyu Zheng (Jira)

Hanyu Zheng created KAFKA-15527:
---

 Summary: Add reverseRange and reverseAll query over kv-store in 
IQv2
 Key: KAFKA-15527
 URL: https://issues.apache.org/jira/browse/KAFKA-15527
 Project: Kafka
  Issue Type: Improvement
Reporter: Hanyu Zheng


Add reverseRange and reverseAll query over kv-store in IQv2



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Need generate a KIP

2023-10-02 Thread Josep Prat

Done.

Thanks for showing interest in Apache Kafka!

Best,

On Mon, Oct 2, 2023 at 6:13 PM Hanyu (Peter) Zheng
 wrote:

> Thank you, Josep. Yes I also need Jira rights.
>
> On Mon, Oct 2, 2023 at 9:05 AM Josep Prat 
> wrote:
>
> > Hi Peter,
> >
> > You are set :) Please share if you also need Jira rights.
> >
> > Best,
> >
> > On Mon, Oct 2, 2023 at 5:59 PM Hanyu (Peter) Zheng
> >  wrote:
> >
> > > wiki ID: pzheng
> > >  Jira ID:  hanyuzheng
> > >
> > > --
> > >
> > > [image: Confluent] 
> > > Hanyu (Peter) Zheng he/him/his
> > > Software Engineer Intern
> > > +1 (213) 431-7193 <+1+(213)+431-7193>
> > > Follow us: [image: Blog]
> > > <
> > >
> >
> https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog
> > > >[image:
> > > Twitter] [image: LinkedIn]
> > > [image: Slack]
> > > [image: YouTube]
> > > 
> > >
> > > [image: Try Confluent Cloud for Free]
> > > <
> > >
> >
> https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic
> > > >
> > >
> >
> >
> > --
> > [image: Aiven] 
> >
> > *Josep Prat*
> > Open Source Engineering Director, *Aiven*
> > josep.p...@aiven.io   |   +491715557497
> > aiven.io    |   <
> https://www.facebook.com/aivencloud
> > >
> >      <
> > https://twitter.com/aiven_io>
> > *Aiven Deutschland GmbH*
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
>
>
> --
>
> [image: Confluent] 
> Hanyu (Peter) Zheng he/him/his
> Software Engineer Intern
> +1 (213) 431-7193 <+1+(213)+431-7193>
> Follow us: [image: Blog]
> <
> https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog
> >[image:
> Twitter] [image: LinkedIn]
> [image: Slack]
> [image: YouTube]
> 
>
> [image: Try Confluent Cloud for Free]
> <
> https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic
> >
>


-- 
[image: Aiven] 

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.p...@aiven.io   |   +491715557497
aiven.io    |   
     
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

DISCUSS KIP-984 Add pluggable compression interface to Kafka

2023-10-02 Thread Diop, Assane

https://cwiki.apache.org/confluence/display/KAFKA/KIP-984%3A+Add+pluggable+compression+interface+to+Kafka

[jira] [Created] (KAFKA-15526) Simplify LogAppendInfo parameters

2023-10-02 Thread David Mao (Jira)

David Mao created KAFKA-15526:
-

 Summary: Simplify LogAppendInfo parameters
 Key: KAFKA-15526
 URL: https://issues.apache.org/jira/browse/KAFKA-15526
 Project: Kafka
  Issue Type: Improvement
Reporter: David Mao
Assignee: David Mao


Currently LogAppendInfo is quite overloaded, carrying a bunch of redundant 
information. This makes some of the code unnecessarily complex in the log 
layer, since the log layer is unsure which fields are required to populate for 
higher layers, and higher layers are unsure which fields are required to bubble 
back to clients.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Need generate a KIP

2023-10-02 Thread Hanyu (Peter) Zheng

Thank you, Josep. Yes I also need Jira rights.

On Mon, Oct 2, 2023 at 9:05 AM Josep Prat 
wrote:

> Hi Peter,
>
> You are set :) Please share if you also need Jira rights.
>
> Best,
>
> On Mon, Oct 2, 2023 at 5:59 PM Hanyu (Peter) Zheng
>  wrote:
>
> > wiki ID: pzheng
> >  Jira ID:  hanyuzheng
> >
> > --
> >
> > [image: Confluent] 
> > Hanyu (Peter) Zheng he/him/his
> > Software Engineer Intern
> > +1 (213) 431-7193 <+1+(213)+431-7193>
> > Follow us: [image: Blog]
> > <
> >
> https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog
> > >[image:
> > Twitter] [image: LinkedIn]
> > [image: Slack]
> > [image: YouTube]
> > 
> >
> > [image: Try Confluent Cloud for Free]
> > <
> >
> https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic
> > >
> >
>
>
> --
> [image: Aiven] 
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |    >
>      <
> https://twitter.com/aiven_io>
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B
>


-- 

[image: Confluent] 
Hanyu (Peter) Zheng he/him/his
Software Engineer Intern
+1 (213) 431-7193 <+1+(213)+431-7193>
Follow us: [image: Blog]
[image:
Twitter] [image: LinkedIn]
[image: Slack]
[image: YouTube]


[image: Try Confluent Cloud for Free]

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2245

See 


Changes:


--
[...truncated 314964 lines...]

Gradle Test Run :streams:test > Gradle Test Executor 81 > TasksTest > 
shouldAddAndRemovePendingTaskToUpdateInputPartitions() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 81 > TasksTest > 
onlyRemovePendingTaskToCloseDirtyShouldRemoveTaskFromPendingUpdateActions() 
STARTED

Gradle Test Run :streams:test > Gradle Test Executor 81 > TasksTest > 
onlyRemovePendingTaskToCloseDirtyShouldRemoveTaskFromPendingUpdateActions() 
PASSED

Gradle Test Run :streams:test > Gradle Test Executor 81 > TasksTest > 
shouldAddAndRemovePendingTaskToSuspend() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 81 > TasksTest > 
shouldAddAndRemovePendingTaskToSuspend() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 81 > TasksTest > 
shouldVerifyIfPendingTaskToInitExist() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 81 > TasksTest > 
shouldVerifyIfPendingTaskToInitExist() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 81 > TasksTest > 
onlyRemovePendingTaskToCloseCleanShouldRemoveTaskFromPendingUpdateActions() 
STARTED

Gradle Test Run :streams:test > Gradle Test Executor 81 > TasksTest > 
onlyRemovePendingTaskToCloseCleanShouldRemoveTaskFromPendingUpdateActions() 
PASSED

Gradle Test Run :streams:test > Gradle Test Executor 81 > TasksTest > 
shouldDrainPendingTasksToCreate() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 81 > TasksTest > 
shouldDrainPendingTasksToCreate() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 81 > TasksTest > 
onlyRemovePendingTaskToRecycleShouldRemoveTaskFromPendingUpdateActions() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 81 > TasksTest > 
onlyRemovePendingTaskToRecycleShouldRemoveTaskFromPendingUpdateActions() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 81 > TasksTest > 
shouldAddAndRemovePendingTaskToCloseClean() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 81 > TasksTest > 
shouldAddAndRemovePendingTaskToCloseClean() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 81 > TasksTest > 
shouldAddAndRemovePendingTaskToCloseDirty() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 81 > TasksTest > 
shouldAddAndRemovePendingTaskToCloseDirty() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 81 > TasksTest > 
shouldKeepAddedTasks() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 81 > TasksTest > 
shouldKeepAddedTasks() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 81 > StateQueryResultTest 
> More than one query result throws IllegalArgumentException STARTED

Gradle Test Run :streams:test > Gradle Test Executor 81 > StateQueryResultTest 
> More than one query result throws IllegalArgumentException PASSED

Gradle Test Run :streams:test > Gradle Test Executor 81 > StateQueryResultTest 
> Zero query results shouldn't error STARTED

Gradle Test Run :streams:test > Gradle Test Executor 81 > StateQueryResultTest 
> Zero query results shouldn't error PASSED

Gradle Test Run :streams:test > Gradle Test Executor 81 > StateQueryResultTest 
> Valid query results still works STARTED

Gradle Test Run :streams:test > Gradle Test Executor 81 > StateQueryResultTest 
> Valid query results still works PASSED

Gradle Test Run :streams:test > Gradle Test Executor 81 > 
RocksDBBlockCacheMetricsTest > shouldRecordCorrectBlockCacheUsage(RocksDBStore, 
StateStoreContext) > [1] 
org.apache.kafka.streams.state.internals.RocksDBStore@59e8dca8, 
org.apache.kafka.test.MockInternalProcessorContext@6a63d6cb STARTED

Gradle Test Run :streams:test > Gradle Test Executor 81 > 
RocksDBBlockCacheMetricsTest > shouldRecordCorrectBlockCacheUsage(RocksDBStore, 
StateStoreContext) > [1] 
org.apache.kafka.streams.state.internals.RocksDBStore@59e8dca8, 
org.apache.kafka.test.MockInternalProcessorContext@6a63d6cb PASSED

Gradle Test Run :streams:test > Gradle Test Executor 81 > 
RocksDBBlockCacheMetricsTest > shouldRecordCorrectBlockCacheUsage(RocksDBStore, 
StateStoreContext) > [2] 
org.apache.kafka.streams.state.internals.RocksDBTimestampedStore@2d7d46e3, 
org.apache.kafka.test.MockInternalProcessorContext@59e87a26 STARTED

Gradle Test Run :streams:test > Gradle Test Executor 81 > 
RocksDBBlockCacheMetricsTest > shouldRecordCorrectBlockCacheUsage(RocksDBStore, 
StateStoreContext) > [2] 
org.apache.kafka.streams.state.internals.RocksDBTimestampedStore@2d7d46e3, 
org.apache.kafka.test.MockInternalProcessorContext@59e87a26 PASSED

Gradle Test Run :streams:test > Gradle Test Executor 81 > 
RocksDBBlockCacheMetricsTest > 
shouldRecordCorrectBlockCachePinnedUsage(RocksDBStore, StateStoreContext) > [1] 
org.apache.kafka.streams.state.internals.RocksDBStore@6e363d0d, 
org.apache.kafka.test.MockInternalProcessorContext@155518cd STARTED

Gradle Test Run :streams:test >

Re: Need generate a KIP

2023-10-02 Thread Josep Prat

Hi Peter,

You are set :) Please share if you also need Jira rights.

Best,

On Mon, Oct 2, 2023 at 5:59 PM Hanyu (Peter) Zheng
 wrote:

> wiki ID: pzheng
>  Jira ID:  hanyuzheng
>
> --
>
> [image: Confluent] 
> Hanyu (Peter) Zheng he/him/his
> Software Engineer Intern
> +1 (213) 431-7193 <+1+(213)+431-7193>
> Follow us: [image: Blog]
> <
> https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog
> >[image:
> Twitter] [image: LinkedIn]
> [image: Slack]
> [image: YouTube]
> 
>
> [image: Try Confluent Cloud for Free]
> <
> https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic
> >
>


-- 
[image: Aiven] 

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.p...@aiven.io   |   +491715557497
aiven.io    |   
     
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

Need generate a KIP

2023-10-02 Thread Hanyu (Peter) Zheng

wiki ID: pzheng
 Jira ID:  hanyuzheng

-- 

[image: Confluent] 
Hanyu (Peter) Zheng he/him/his
Software Engineer Intern
+1 (213) 431-7193 <+1+(213)+431-7193>
Follow us: [image: Blog]
[image:
Twitter] [image: LinkedIn]
[image: Slack]
[image: YouTube]


[image: Try Confluent Cloud for Free]

Re: Re: [DISCUSS] KIP-951: Leader discovery optimisations for the client

2023-10-02 Thread Mayank Shekhar Narula

Hi David

01 Same as Java Client's existing behaviour of requesting a full expedited
metadata-refresh done asynchronously, for error NOT_LEADER OR FENCED_EPOCH,
the KIP proposes to continue doing so. As this would help prevent similar
errors on future requests to other partitions affected by leadership
changes. Quot-ing from the KIP ->

> Even though client will receive the new leader information in the
> ProduceResponse & FetchResponse when leader changes, but same as the
> existing behaviour of the Kafka Java client, it will request expedited
> metadata-refresh done asynchronously. Since leadership change will likely
> affect many partitions, so future requests to such partitions will benefit
> from the upto date leadership information, and reduce requests going to old
> leaders.
>
Does this help?

02 Thanks for the feedback.

03 It's the latter, i.e. fields are optional even in version 16, as brokers
would only return them in specific scenarios. This was mentioned in earlier
sections of the KIP. But clarified now in the FetchResponse section too.

04 Fixed.

05 Updated.

On Thu, Sep 28, 2023 at 6:38 PM David Jacot 
wrote:

> Hi Mayank,
>
> Thanks again for the KIP and thanks for adding the new analysis. Overall, I
> am fine with it. I have a few minor comments.
>
> 01. If I understand correctly, the client will still request a metadata
> update even when it gets the new leader if the produce response or the
> fetch response. Is this correct? I think that we need this in order to get
> the full metadata. Could we elaborate a bit more about this in the KIP? I
> mean that it would be great to explain why we are doing it.
>
> 02. I was debating whether we should return the rack in the NodeEndpoints.
> I think that returning it makes sense from a consistency point of view. It
> would be weird to only update the node and the port in the metadata cache.
> However, I was thinking about the future and I was wondering how we would
> handle future metadata that we would add to the nodes, say tags. If we
> follow the same pattern, we would have to return any new fields as well. I
> suppose that it would be fine. Anyway, I think that the current approach is
> OK. I just wanted to share my thoughts.
>
> 03. In the FetchResponse Message section, it is written "NodeEndpoints is a
> tagged field, which is a minor optimisation of saving bytes on the network,
> as it won’t be set always.". Are you saying this because it won't be set
> for version prior to version 16? Or are you saying this because it may not
> be provided even in version 16? In other words, I wonder if that new
> information is actually optional or not. It is not clear in the KIP.
>
> 04. In the ProduceResponse Message section, the validVersions field misses
> in the schema.
>
> 05. The documentation of NodeEndpoints fields says "Endpoints for all
> current-leaders enumerated in PartitionData.". I suppose that this is
> incorrect, right? We will only provide endpoints
> when NOT_LEADER_OR_FOLLOWER or FENCED_LEADER_EPOCH are returned. The same
> applies to the other schema.
>
> Best,
> David
>
>
> On Wed, Sep 27, 2023 at 4:39 AM Mayank Shekhar Narula <
> mayanks.nar...@gmail.com> wrote:
>
> > Adding to Crispin. The new micro-benchmark shows improvements of 88% in
> > p99.9 with the KIP changes Vs baseline(& rejected alternate). Its
> > hypothesised improvements are seen as KIP avoids a full metadata
> refresh(Vs
> > baseline/rejected alternate), and the full metadata refresh can be slow
> due
> > to metadata reconvergence delay at the server(post leadership-change of
> > partitions). Extending this logic, KIP changes would be beneficial in
> > scenarios where full metadata refresh can be slow. Potential example
> would
> > be, metadata RPC is slowed due to head-of-line blocking by another slow
> RPC
> > in front, say a produce RPC(possibly slow due to churn in the ISR).
> >
> > This new benchmark is in addition to the previously done benchmark of
> roll
> > simulation, where improvements upto 5% were seen.
> >
> > Please do a review, as the voting thread is live.
> >
> > Thanks!
> >
> > On Wed, Sep 20, 2023 at 4:43 PM Crispin Bernier
> >  wrote:
> >
> > > Hi,
> > >
> > > I've updated the KIP with benchmark results focusing more directly on
> the
> > > redirect case, please review and +1 in the voting thread if convinced.
> > >
> > > Thank you,
> > > Crispin
> > >
> > > On Wed, Jul 26, 2023 at 11:13 PM Luke Chen  wrote:
> > >
> > > > Thanks for adding the benchmark results, Crispin!
> > > > IMO, 2~5% performance improvement is small, but given the change is
> > > small,
> > > > cost is also small (only append endpoint info when
> > > NOT_LEADER_OR_FOLLOWER..
> > > > etc error), I think it is worth doing it.
> > > >
> > > > Thank you.
> > > > Luke
> > > >
> > > > On Wed, Jul 26, 2023 at 12:33 AM Ismael Juma 
> > wrote:
> > > >
> > > > > Thanks Crispin!
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Mon, Jul 24, 2023 at 8:16 PM Crispin Bernier
> > > > >

Re: [VOTE] 3.6.0 RC2

2023-10-02 Thread Proven Provenzano

Hi,

To verify the release of release 3.6.0 RC2 I did the following:

   - Downloaded the source, built and ran the tests.
   - Validated SCRAM with KRaft including creating credentials with
   kafka-storage.
   - Validated Delegation Tokens with KRaft

+1 (non-binding)

--Proven



On Mon, Oct 2, 2023 at 8:37 AM Divij Vaidya  wrote:

> + 1 (non-binding)
>
> Verifications:
> 1. I ran a produce-consume workload with plaintext auth, JDK17, zstd
> compression using an open messaging benchmark and found 3.6 to be better
> than or equal to 3.5.1 across all dimensions. Notably, 3.6 had consistently
> 6-7% lower CPU utilization, lesser spikes on P99 produce latencies and
> overall lower P99.8 latencies.
>
> 2. I have verified that detached signature is correct using
> https://www.apache.org/info/verification.html and the release manager
> public keys are available at
> https://keys.openpgp.org/search?q=F65DC3423D4CD7B9
>
> 3. I have verified that all metrics emitted in 3.5.1 (with Zk) are also
> being emitted in 3.6.0 (with Zk).
>
> Problems (but not blockers):
> 1. Metrics added in
>
> https://github.com/apache/kafka/commit/2f71708955b293658cec3b27e9a5588d39c38d7e
> aren't available in the documentation (cc: Justine). I don't consider this
> as a release blocker but we should add it as a fast follow-up.
>
> 2. Metric added in
>
> https://github.com/apache/kafka/commit/a900794ace4dcf1f9dadee27fbd8b63979532a18
> isn't available in documentation (cc: David). I don't consider this as a
> release blocker but we should add it as a fast follow-up.
>
> --
> Divij Vaidya
>
>
>
> On Mon, Oct 2, 2023 at 9:50 AM Federico Valeri 
> wrote:
>
> > Hi Satish, I did the following to verify the release:
> >
> > - Built from source with Java 17 and Scala 2.13
> > - Ran all unit and integration tests
> > - Spot checked documentation
> > - Ran custom client applications using staging artifacts on a 3-nodes
> > cluster
> > - Tested tiered storage with one of the available RSM implementations
> >
> > +1 (non binding)
> >
> > Thanks
> > Fede
> >
> > On Mon, Oct 2, 2023 at 8:50 AM Luke Chen  wrote:
> > >
> > > Hi Satish,
> > >
> > > I verified with:
> > > 1. Ran quick start in KRaft for scala 2.12 artifact
> > > 2. Making sure the checksum are correct
> > > 3. Browsing release notes, documents, javadocs, protocols.
> > > 4. Verified the tiered storage feature works well.
> > >
> > > +1 (binding).
> > >
> > > Thanks.
> > > Luke
> > >
> > >
> > >
> > > On Mon, Oct 2, 2023 at 5:23 AM Jakub Scholz  wrote:
> > >
> > > > +1 (non-binding). I used the Scala 2.13 binaries and the staged Maven
> > > > artifacts and run my tests. Everything seems to work fine for me.
> > > >
> > > > Thanks
> > > > Jakub
> > > >
> > > > On Fri, Sep 29, 2023 at 8:17 PM Satish Duggana <
> > satish.dugg...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hello Kafka users, developers and client-developers,
> > > > >
> > > > > This is the third candidate for the release of Apache Kafka 3.6.0.
> > > > > Some of the major features include:
> > > > >
> > > > > * KIP-405 : Kafka Tiered Storage
> > > > > * KIP-868 : KRaft Metadata Transactions
> > > > > * KIP-875: First-class offsets support in Kafka Connect
> > > > > * KIP-898: Modernize Connect plugin discovery
> > > > > * KIP-938: Add more metrics for measuring KRaft performance
> > > > > * KIP-902: Upgrade Zookeeper to 3.8.1
> > > > > * KIP-917: Additional custom metadata for remote log segment
> > > > >
> > > > > Release notes for the 3.6.0 release:
> > > > >
> https://home.apache.org/~satishd/kafka-3.6.0-rc2/RELEASE_NOTES.html
> > > > >
> > > > > *** Please download, test and vote by Tuesday, October 3, 12pm PT
> > > > >
> > > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > > https://kafka.apache.org/KEYS
> > > > >
> > > > > * Release artifacts to be voted upon (source and binary):
> > > > > https://home.apache.org/~satishd/kafka-3.6.0-rc2/
> > > > >
> > > > > * Maven artifacts to be voted upon:
> > > > >
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > >
> > > > > * Javadoc:
> > > > > https://home.apache.org/~satishd/kafka-3.6.0-rc2/javadoc/
> > > > >
> > > > > * Tag to be voted upon (off 3.6 branch) is the 3.6.0-rc2 tag:
> > > > > https://github.com/apache/kafka/releases/tag/3.6.0-rc2
> > > > >
> > > > > * Documentation:
> > > > > https://kafka.apache.org/36/documentation.html
> > > > >
> > > > > * Protocol:
> > > > > https://kafka.apache.org/36/protocol.html
> > > > >
> > > > > * Successful Jenkins builds for the 3.6 branch:
> > > > > There are a few runs of unit/integration tests. You can see the
> > latest
> > > > > at https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/. We
> > will
> > > > > continue running a few more iterations.
> > > > > System tests:
> > > > > We will send an update once we have the results.
> > > > >
> > > > > Thanks,
> > > > > Satish.
> > > > >
> > > >
> >
>

Re: [DISCUSS] KIP-979: Allow independently stop KRaft controllers or brokers

2023-10-02 Thread Ismael Juma

Hi Ron,

Yes, that's what I am proposing, yes.

Ismael

On Sat, Sep 30, 2023 at 2:30 PM Ron Dagostino  wrote:

> Thanks, Ismael.  I think you are proposing a pair of mutually exclusive
> args --process.roles and --node.id, right?  I agree that is more
> user-friendly than the --required-config arg, and it comes at the possible
> expense of generality.  So that’s the tradeoff between the two, I think.
> No other config comes to mind now that we’ve identified these two.  I think
> the two specific and mutually exclusive parameters would be the way to go
> unless someone else identifies still more options that people might want.
>
> Did I get that right, or were you proposing something different?
>
> Ron
>
> > On Sep 30, 2023, at 10:42 AM, Ismael Juma  wrote:
> >
> > Hi,
> >
> > Thanks for the KIP. I think this approach based on configs is a bit too
> > open ended and not very user friendly. Why don't we simply provide flags
> > for the things a user may care about? So far, it seems like we have two
> > good candidates (node id and process role). Are there any others?
> >
> > Ismael
> >
> >> On Fri, Sep 29, 2023 at 6:19 PM Hailey Ni 
> wrote:
> >>
> >> Hi Ron,
> >>
> >> I think you made a great point, making the "name" arbitrary instead of
> >> hard-coding it will make the functionality much more flexible. I've
> updated
> >> the KIP and the code accordingly. Thanks for the great idea!
> >>
> >> Thanks,
> >> Hailey
> >>
> >>
> >>> On Fri, Sep 29, 2023 at 2:34 PM Ron Dagostino 
> wrote:
> >>>
> >>> Thanks, Hailey.  Is there a reason to restrict it to just
> >>> process.roles and node.id?  Someone might want to do
> >>> "--required-config any.name=whatever.value", for example, and at first
> >>> glance I don't see a reason why the implementation should be any
> >>> different -- it seems it would probably be easier to not have to worry
> >>> about restricting to specific cases, actually.  WDYT?
> >>>
> >>> Ron
> >>>
> >>> On Fri, Sep 29, 2023 at 5:12 PM Hailey Ni 
> >>> wrote:
> 
>  Updated. Please let me know if you have any additional comments. Thank
> >>> you!
> 
>  On Thu, Sep 21, 2023 at 3:02 PM Hailey Ni  wrote:
> 
> > Hi Ron. Thanks for the response. I agree with your point. I'll make
> >> the
> > corresponding changes in the KIP and KAFKA-15471
> > .
> >
> > On Thu, Sep 21, 2023 at 1:40 PM Ron Dagostino 
> >>> wrote:
> >
> >> Hi Hailey.  No, I just looked, and zookeeper-server-stop does not
> >> have
> >> any facility to be specific about which ZK nodes to signal.  So
> >> providing the ability in kafka-server-stop to be more specific than
> >> just "signal all controllers" or "signal all brokers" would be a
> >> bonus
> >> and therefore not necessarily required.  But if it is easy to
> >> achieve
> >> and doesn't add any additional cognitive load -- and at first glance
> >> it does seem so -- we should probably just support it.
> >>
> >> Ron
> >>
> >> On Wed, Sep 20, 2023 at 6:15 PM Hailey Ni  >>>
> >> wrote:
> >>>
> >>> Hi Ron,
> >>>
> >>> Thank you very much for the comment. I think it makes sense to me
> >>> that
> >> we
> >>> provide an even more specific way to kill individual
> >> controllers/brokers.
> >>> I have one question: does the command line for ZooKeeper cluster
> >>> provide
> >>> such a way to kill individual controllers/brokers?
> >>>
> >>> Thanks,
> >>> Hailey
> >>>
> >>> On Tue, Sep 19, 2023 at 11:01 AM Ron Dagostino  >>>
> >> wrote:
> >>>
>  Thanks for the KIP, Hailey.  It will be nice to provide some
>  fine-grained control for when people running the broker and
> >>> controller
>  this way want to stop just one of them.
> 
>  One thing that occurs to me is that in a development environment
>  someone might want to run multiple controllers and multiple
> >>> brokers
>  all on the same box, and in that case they might want to
> >> actually
> >>> stop
>  just one controller or just one broker instead of all of them.
> >>> So I'm
>  wondering if maybe instead of supporting kafka-server-stop
>  [--process.roles ] we might want to instead support
>  kafka-server-stop [--required-config ].  If someone
> >>> wanted
>  to stop any/all controllers and not touch the broker(s) they
> >> could
>  still achieve that by invoking kafka-server-stop
> >> --required-config
>  process.roles=controller.  But if they did want to stop a
> >>> particular
>  controller they could then also achieve that via
> >> kafka-server-stop
>  --required-config node.id=1 (for example).  What do you think?
> 
>  Ron
> 
>  On Thu, Sep 14, 2023 at 5:56 PM Hailey Ni
> >>> 
>  wrote:
> >
> > Hi all,
> >
> > I would like

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2244