Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #895

[Apache Jenkins Server]

See 

Re: [VOTE] 3.1.1 RC0

[Luke Chen]

Hi Dongjoon,

The Apache Kafka community doesn't recommend users to use which version of
Kafka.
The 2 releases v3.1.1 and v3.2.0 are running in parallel, and we don't
guarantee which version will be released earlier.

Thank you.
Luke




On Thu, Apr 28, 2022 at 4:54 AM Dongjoon Hyun  wrote:

> Hi, All.
>
> It seems that Apache Kafka 3.2.0 RC0 vote started already instead of
> Apache Kafka 3.1.1 release.
>
> Does Apache Kafka community recommend to use Apache Kafka 3.2.0 instead of
> Apache Kafka 3.1.1?
>
> Dongjoon.
>
> On 2022/04/14 01:00:40 Ismael Juma wrote:
> > I added a comment to that PR. Let's figure out if we need an additional
> > change before doing the next RC.
> >
> > Ismael
> >
> > On Tue, Apr 12, 2022 at 7:47 PM Luke Chen  wrote:
> >
> > > Thanks for pointing that out, David.
> > > +1 to include this PR since we've already included the first fix for
> > > KAFKA-13794, and this is a follow up fix for it.
> > >
> > > Thank you.
> > > Luke
> > >
> > > On Wed, Apr 13, 2022 at 2:31 AM David Jacot
> 
> > > wrote:
> > >
> > > > Hi Tom,
> > > >
> > > > Thanks for running the release. I wonder if we should include:
> > > >
> > > >
> > >
> https://github.com/apache/kafka/commit/134c432d6452de1bfb99d0f6b455a58c16bc626a
> > > > .
> > > >
> > > > This is a follow up of KAFKA-13794. What do you think?
> > > >
> > > > Best,
> > > > David
> > > >
> > > > On Fri, Apr 8, 2022 at 6:18 PM Tom Bentley 
> wrote:
> > > > >
> > > > > Hello Kafka users, developers and client-developers,
> > > > >
> > > > > This is the first candidate for release of Apache Kafka 3.1.1.
> > > > >
> > > > > Apache Kafka 3.1.1 is a bugfix release and 29 issues have been
> fixed
> > > > > since 3.1.0.
> > > > >
> > > > > Release notes for the 3.1.1 release:
> > > > >
> https://home.apache.org/~tombentley/kafka-3.1.1-rc0/RELEASE_NOTES.html
> > > > >
> > > > > *** Please download, test and vote by Friday 15 April, 12:00 UTC
> > > > >
> > > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > > https://kafka.apache.org/KEYS
> > > > >
> > > > > * Release artifacts to be voted upon (source and binary):
> > > > > https://home.apache.org/~tombentley/kafka-3.1.1-rc0/
> > > > >
> > > > > * Maven artifacts to be voted upon:
> > > > >
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > >
> > > > > * Javadoc:
> > > > > https://home.apache.org/~tombentley/kafka-3.1.1-rc0/javadoc/
> > > > >
> > > > > * Tag to be voted upon (off 3.1 branch) is the 3.1.1 tag:
> > > > > https://github.com/apache/kafka/releases/tag/3.1.1-rc0
> > > > >
> > > > > * Documentation:
> > > > > https://kafka.apache.org/31/documentation.html
> > > > >
> > > > > * Protocol:
> > > > > https://kafka.apache.org/31/protocol.html
> > > > >
> > > > > * Successful Jenkins builds for the 3.1 branch:
> > > > > I will share a link one the build is complete.
> > > > >
> > > > > /**
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Tom
> > > >
> > >
> >
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #894

[Apache Jenkins Server]

See 

Re: [VOTE] 3.1.1 RC0

[Dongjoon Hyun]

Hi, All.

It seems that Apache Kafka 3.2.0 RC0 vote started already instead of Apache 
Kafka 3.1.1 release.

Does Apache Kafka community recommend to use Apache Kafka 3.2.0 instead of 
Apache Kafka 3.1.1?

Dongjoon.

On 2022/04/14 01:00:40 Ismael Juma wrote:
> I added a comment to that PR. Let's figure out if we need an additional
> change before doing the next RC.
> 
> Ismael
> 
> On Tue, Apr 12, 2022 at 7:47 PM Luke Chen  wrote:
> 
> > Thanks for pointing that out, David.
> > +1 to include this PR since we've already included the first fix for
> > KAFKA-13794, and this is a follow up fix for it.
> >
> > Thank you.
> > Luke
> >
> > On Wed, Apr 13, 2022 at 2:31 AM David Jacot 
> > wrote:
> >
> > > Hi Tom,
> > >
> > > Thanks for running the release. I wonder if we should include:
> > >
> > >
> > https://github.com/apache/kafka/commit/134c432d6452de1bfb99d0f6b455a58c16bc626a
> > > .
> > >
> > > This is a follow up of KAFKA-13794. What do you think?
> > >
> > > Best,
> > > David
> > >
> > > On Fri, Apr 8, 2022 at 6:18 PM Tom Bentley  wrote:
> > > >
> > > > Hello Kafka users, developers and client-developers,
> > > >
> > > > This is the first candidate for release of Apache Kafka 3.1.1.
> > > >
> > > > Apache Kafka 3.1.1 is a bugfix release and 29 issues have been fixed
> > > > since 3.1.0.
> > > >
> > > > Release notes for the 3.1.1 release:
> > > > https://home.apache.org/~tombentley/kafka-3.1.1-rc0/RELEASE_NOTES.html
> > > >
> > > > *** Please download, test and vote by Friday 15 April, 12:00 UTC
> > > >
> > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > https://kafka.apache.org/KEYS
> > > >
> > > > * Release artifacts to be voted upon (source and binary):
> > > > https://home.apache.org/~tombentley/kafka-3.1.1-rc0/
> > > >
> > > > * Maven artifacts to be voted upon:
> > > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > >
> > > > * Javadoc:
> > > > https://home.apache.org/~tombentley/kafka-3.1.1-rc0/javadoc/
> > > >
> > > > * Tag to be voted upon (off 3.1 branch) is the 3.1.1 tag:
> > > > https://github.com/apache/kafka/releases/tag/3.1.1-rc0
> > > >
> > > > * Documentation:
> > > > https://kafka.apache.org/31/documentation.html
> > > >
> > > > * Protocol:
> > > > https://kafka.apache.org/31/protocol.html
> > > >
> > > > * Successful Jenkins builds for the 3.1 branch:
> > > > I will share a link one the build is complete.
> > > >
> > > > /**
> > > >
> > > > Thanks,
> > > >
> > > > Tom
> > >
> >
> 

Re: [DISCUSS] KIP-759: Unneeded repartition canceling

[Matthias J. Sax]

Let's wait a couple of days to give Ivan a chance to reply. If he does 
not reply, feel free to pick it up.



-Matthias

On 4/26/22 3:58 AM, Levani Kokhreidze wrote:

Hi,

Sorry, maybe I am jumping the gun here, but if by any chance this KIP becomes 
dormant, I'd be interested in picking it up.

Levani


On 23. Apr 2022, at 02:43, Matthias J. Sax  wrote:

Ivan,

are you still interested in this KIP? I think it would be a good addition.


-Matthias

On 8/16/21 5:30 PM, Matthias J. Sax wrote:

Your point about the IQ problem is an interesting one. I missed the
point that the "new key" would be a "superkey", and thus, it should
always be possible to compute the original key from the superkey. (As a
matter of fact, for windowed-table the windowed-key is also a superkey...)
I am not sure if we need to follow the "use the head idea" or if we need
a "CompositeKey" interface? It seems we can just allow for any types and
we can be agnostic to it?
KStream stream = ...
KStream stream2 =
   stream.selectKey(/*set superkey*/)
 .markAsPartitioned()
We only need a `Function` without any restrictions on the type,
to map the "superkey" to the original "partition key"?
Do you propose to provide the "revers mapper" via the
`markAsPartitioned()` method (or config object), or via the IQ methods?
Not sure which one is better?
However, I am not sure if it would solve the join problem? At least not
easily: if one has two KStream and one is properly
partitioned by `Tuple` while the other one is "marked-as-partitoned",
the join would just fail. -- Similar for a stream-table join. -- The
only fix would be to do the re-partitioning anyway, effectively ignoring
the "user hint", but it seems to defeat the purpose? Again, I would
argue that it is ok to not handle this case, but leave it as the
responsibility for the user to not mess it up.
-Matthias
On 8/9/21 2:32 PM, Ivan Ponomarev wrote:

Hi Matthias and Sophie!

==1.  markAsPartitioned vs extending `selectKey(..)` etc. with a config.==

I don't have a strong opinion here, both Sophie's and Matthias' points
look convincing for me.

I think we should estimate the following: what is the probability that
we will ever need to extend `selectKey` etc. with a config for the
purposes other than `markAsPartitioned`?

If we find this probability high, then it's just a refactoring to
deprecate overloads with `Named` and introduce overloads with dedicated
configs, and we should do it this way.

If it's low or zero, maybe it's better not to mess with the existing
APIs and to introduce a single `markAsPartitioned()` method, which
itself can be easily deprecated if we find a better solution later!


==2. The IQ problem==


it then has to be the case that



Partitioner.partition(key) == Partitioner.partition(map(key))



Sophie, you got this wrong, and Matthias already explained why.

The actual required property for the mapping function is:

\forall k1, k2 (map(k1) = map(k2) => partition(k1) = partition(k2))

or, by contraposition law,

\forall k1, k2 (partition(k1) =/= partition(k2) => map(k1) =/= map(k2) )


(look at the whiteboard photo that I attached to the KIP).

There is a big class of such mappings: key -> Tuple(key, anyValue). This
is actually what we often do before aggregation, and this mapping does
not require repartition.

But of course we can extract the original key from Tuple(key, anyValue),
and this can save IQ and joins!

This is what I'm talking about when I talk about 'CompositeKey' idea.

We can do the following:

1. implement a 'partitioner wrapper' that recognizes tuples
(CompositeKeys) and uses only the 'head' to calculate the partition,

2. implement

selectCompositeKey(BiFunction tailSelector) {
   selectKey((k, v) -> new CompositeKey(k, tailSelector.apply(k, v));
   //MARK_AS_PARTITIONED call here,
   //but this call is an implementation detail and we do not expose
   //markAsPartitioned publicly!
}

WDYT? (it's just a brainstorming idea)

09.08.2021 2:38, Matthias J. Sax пишет:

Hi,

I originally had a similar thought about `markAsPartitioned()` vs
extending `selectKey()` et al. with a config. While I agree that it
might be conceptually cleaner to use a config object, I did not propose
it as the API impact (deprecating stuff and adding new stuff) is quite
big... If we think it's an acceptable price to pay, I am ok with it
though.

I also do think, that `markAsPartitioned()` could actually be
categorized as an operator... We don't expose it in the API as
first-class citizen atm, but in fact we have two types of `KStream` -- a
"PartitionedKStream" and a "NonPartitionedKStream". Thus,
`markAsPartitioned()` can be seen as a "cast operator" that converts the
one into the other.

I also think that the raised concern about "forgetting to remove
`markAsPartitioned()`" might not be very strong though. If you have
different places in the code that link stuff together, a call to eg.
`selectKey().markAsPartitioned()` must always to together. If you have
some other place in 

Re: [kafka-clients] Re: [VOTE] 3.2.0 RC0

[Guozhang Wang]

Hi Bruno,

Could I also have this commit (
https://github.com/apache/kafka/commit/e026384ffb3170a2e71053a4163db58b9bd8fba6)
in the next release candidate? It's fixing a performance regression that
was just introduced, and not yet released in older versions.


Guozhang

On Tue, Apr 26, 2022 at 11:01 AM Jun Rao  wrote:

> Hi, Bruno.
>
> Thanks for the reply. Your understanding is correct. This is a regression
> introduced only in the 3.2 branch.
>
> Sorry for the late notice.
>
> Jun
>
> On Tue, Apr 26, 2022 at 10:04 AM Bruno Cadonna  wrote:
>
> > Hi Jun,
> >
> > Thank you for your message!
> >
> > Now I see how this issue was introduced in 3.2.0. The fix for the bug
> > described in KAFKA-12841 introduced it, right? I initially understood
> > that the PR you want to include is the fix for the bug described in
> > KAFKA-12841 which dates back to 2.6.
> >
> > I think that classifies as a regression.
> >
> > I will abort the voting and create a new release candidate.
> >
> > Best,
> > Bruno
> >
> > On 26.04.22 18:09, 'Jun Rao' via kafka-clients wrote:
> > > Hi, Bruno,
> > >
> > > Could we include https://github.com/apache/kafka/pull/12064
> > >  in 3.2.0? This fixes an
> > > issue introduced in 3.2.0 where in some of the error cases, the
> producer
> > > interceptor is called twice for the same record.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Tue, Apr 26, 2022 at 6:34 AM Bruno Cadonna  > > > wrote:
> > >
> > > Hi all,
> > >
> > > This is a gently reminder to vote for the first candidate for
> > > release of
> > > Apache Kafka 3.2.0.
> > >
> > > I added the 3.2 documentation to the kafka site. That means
> > > https://kafka.apache.org/32/documentation.html
> > >  works now.
> > >
> > > A successful system tests run can be found here:
> > > https://jenkins.confluent.io/job/system-test-kafka/job/3.2/24/
> > > 
> > >
> > > Thank you to Michal for voting on the release candidate.
> > >
> > > Best,
> > > Bruno
> > >
> > > On 15.04.22 21:05, Bruno Cadonna wrote:
> > >  > Hello Kafka users, developers and client-developers,
> > >  >
> > >  > This is the first candidate for release of Apache Kafka 3.2.0.
> > >  >
> > >  > * log4j 1.x is replaced with reload4j (KAFKA-9366)
> > >  > * StandardAuthorizer for KRaft (KIP-801)
> > >  > * Send a hint to the partition leader to recover the partition
> > > (KIP-704)
> > >  > * Top-level error code field in DescribeLogDirsResponse
> (KIP-784)
> > >  > * kafka-console-producer writes headers and null values (KIP-798
> > and
> > >  > KIP-810)
> > >  > * JoinGroupRequest and LeaveGroupRequest have a reason attached
> > > (KIP-800)
> > >  > * Static membership protocol lets the leader skip assignment
> > > (KIP-814)
> > >  > * Rack-aware standby task assignment in Kafka Streams (KIP-708)
> > >  > * Interactive Query v2 (KIP-796, KIP-805, and KIP-806)
> > >  > * Connect APIs list all connector plugins and retrieve their
> > >  > configuration (KIP-769)
> > >  > * TimestampConverter SMT supports different unix time precisions
> > > (KIP-808)
> > >  > * Connect source tasks handle producer exceptions (KIP-779)
> > >  >
> > >  > Release notes for the 3.2.0 release:
> > >  >
> > >
> https://home.apache.org/~cadonna/kafka-3.2.0-rc0/RELEASE_NOTES.html
> > > <
> https://home.apache.org/~cadonna/kafka-3.2.0-rc0/RELEASE_NOTES.html
> > >
> > >  >
> > >  > *** Please download, test and vote by Monday, April 25, 9am CEST
> > >  >
> > >  > Kafka's KEYS file containing PGP keys we use to sign the
> release:
> > >  > https://kafka.apache.org/KEYS 
> > >  >
> > >  > * Release artifacts to be voted upon (source and binary):
> > >  > https://home.apache.org/~cadonna/kafka-3.2.0-rc0/
> > > 
> > >  >
> > >  > * Maven artifacts to be voted upon:
> > >  >
> > >
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > <
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/>
> > >  >
> > >  > * Javadoc:
> > >  > https://home.apache.org/~cadonna/kafka-3.2.0-rc0/javadoc/
> > > 
> > >  >
> > >  > * Tag to be voted upon (off 3.2 branch) is the 3.2.0 tag:
> > >  > https://github.com/apache/kafka/releases/tag/3.2.0-rc0
> > > 
> > >  >
> > >  > * Documentation (not yet ported to kafka-site):
> > >  > https://kafka.apache.org/32/documentation.html
> > > 
> > >  >
> > >  > * Prot

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.0 #198

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-13859) SCRAM authentication issues with kafka-clients 3.0.1

[Oliver Payne (Jira)]

Oliver Payne created KAFKA-13859:


 Summary: SCRAM authentication issues with kafka-clients 3.0.1
 Key: KAFKA-13859
 URL: https://issues.apache.org/jira/browse/KAFKA-13859
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 3.0.1
Reporter: Oliver Payne


When attempting to produce records to Kafka using a client configured with 
SCRAM authentication, the authentication is being rejected, and the following 
exception is thrown:
{{org.apache.kafka.common.errors.ClusterAuthorizationException: Cluster 
authorization failed.}}
I am seeing this happen with a Springboot service that was recently upgraded to 
2.6.5. After looking into this, I learned that Springboot moved to 
kafka-clients 3.0.1 from 3.0.0 in that version. And sure enough, downgrading to 
kafka-clients resolved the issue, with no changes made to the configs.

I have also attempted to connect to a separate server with kafka-clients 3.0.1, 
using plaintext authentication. That works fine. So the issue appears to be 
with SCRAM authentication.

I will note that I am attempting to connect to an AWS MSK instance. We use 
SCRAM-SHA-512 as our sasl mechanism, using the basic {{ScramLoginModule.}} 



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.2 #40

[Apache Jenkins Server]

See