[jira] [Created] (KAFKA-14188) Quickstart for KRaft

[Jose Armando Garcia Sancio (Jira)]

Jose Armando Garcia Sancio created KAFKA-14188:
--

 Summary: Quickstart for KRaft
 Key: KAFKA-14188
 URL: https://issues.apache.org/jira/browse/KAFKA-14188
 Project: Kafka
  Issue Type: Task
  Components: documentation, kraft
Reporter: Jose Armando Garcia Sancio
Assignee: Jose Armando Garcia Sancio


Either:
 # Improve the quick start documentation to talk about both KRAft and ZK
 # Create a KRaft quick start that is very similar to the ZK quick start but 
uses a different startup process.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Apache Kafka 3.3.0 Release

[Colin McCabe]

Hi José,

Thanks for creating the first RC.

I found an issue where kafka-feature.sh needs some work for KRaft. So, it looks 
like we will have to sink this RC. I opened a blocker JIRA, KAFKA-14187, and 
attached a PR.

This should not block testing of other parts of the release, so hopefully we 
will still get some good testing today.

best,
Colin


On Mon, Aug 29, 2022, at 10:12, José Armando García Sancio wrote:
> The documentation and protocol links are not working. Looking into it.
>
> https://kafka.apache.org/33/documentation.html
> https://kafka.apache.org/33/protocol.html
>
> Thanks,
> -José

Re: ARM/PowerPC builds

[Colin McCabe]

Hi Amit,

I don't see why we need to run a PowerPC build on every test run. It seems 
excessive, given that most Java code should work on PowerPC without any 
changes. Why not just run it nightly?

best,
Colin


On Mon, Aug 22, 2022, at 08:05, Amit Ramswaroop Baheti wrote:
> I am planning to raise a PR for re-enabling PowerPC on the pipeline. 
>
> I will be monitoring PowerPC for failures. I hope this resolves the 
> concern about build failures on PowerPC. Let me know otherwise.
>
> Thanks,
> Amit Baheti
>
> -Original Message-
> From: Amit Ramswaroop Baheti  
> Sent: 10 August 2022 20:09
> To: dev@kafka.apache.org
> Subject: [EXTERNAL] RE: ARM/PowerPC builds
>
> I looked at the PR failures on PowerPC jenkins node & the issue was due 
> to few stuck gradle daemons. I could find that using "./gradlew 
> --status".
> After I cleaned them up using "./gradlew --stop", I was able to build & 
> test existing PR's in the jenkins workspace on that node. 
>
> I also pulled another PR #12488 manually & could execute build and 
> tests successfully on that node. Further the build and test is 
> executing within minutes. So no problem on the performance front.
>
> I am wondering if Apache infra team has any automation to clear stuck 
> gradle daemons. If not, perhaps we may look at one using cron job and 
> make such infra issues selfheal.
>
> I think it should be fine now to re-enable the PowerPC node in the pipeline. 
>
> Thanks,
> Amit Baheti
>
> -Original Message-
> From: Amit Ramswaroop Baheti 
> Sent: 05 August 2022 17:52
> To: dev@kafka.apache.org
> Subject: [EXTERNAL] RE: ARM/PowerPC builds
>
> Hi,
>
> I am looking into the failures on PowerPC. 
>
> I will share more details once I have something concrete & hopefully we 
> would be able to enable it again soon.
>
> -Amit Baheti
>
> -Original Message-
> From: Colin McCabe 
> Sent: 04 August 2022 22:39
> To: dev@kafka.apache.org
> Subject: [EXTERNAL] Re: ARM/PowerPC builds
>
> Hi Matthew,
>
> Can you open a JIRA for the test failures you have seen on M1?
>
> By the way, I have an M1 myself.
>
> best,
> Colin
>
> On Thu, Aug 4, 2022, at 04:12, Matthew Benedict de Detrich wrote:
>> Quite happy to have this change gone through since the ARM builds were 
>> constantly failing however I iterate what Divij Vaidya is saying. I 
>> just recently got a new MacBook M1 laptop that has ARM architecture 
>> and even locally the tests fail (these are the same tests that also 
>> failed in Jenkins).
>>
>> Should get to the root of the issue especially as more people will get 
>> newer Apple laptops over time.
>>
>> --
>> Matthew de Detrich
>> Aiven Deutschland GmbH
>> Immanuelkirchstraße 26, 10405 Berlin
>> Amtsgericht Charlottenburg, HRB 209739 B
>>
>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> m: +491603708037
>> w: aiven.io e: matthew.dedetr...@aiven.io On 4. Aug 2022, 12:36 +0200, 
>> Divij Vaidya , wrote:
>>> Thank you. This would greatly improve the PR experience since now, 
>>> there is higher probability for it to be green.
>>>
>>> Side question though, do we know why ARM tests are timing out? Should 
>>> we start a JIRA with Apache Infra to root cause?
>>>
>>> —
>>> Divij Vaidya
>>>
>>>
>>>
>>> On Thu, Aug 4, 2022 at 12:42 AM Colin McCabe  wrote:
>>>
>>> > Just a quick note. Today we committed INVALID URI REMOVED 
>>> > che_kafka_pull_12380=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=50nG1jjahH
>>> > 8a5uFJPfv74FQ5POP1oomXeKf8Xylvgw4=g4U2eMxnb6_oWrLqMssiRMCn_beLcy75GeHjl2_RYFce1iI2i2HfDSLLFK8gjKrV=7rXLDMIeH-7tYpHu6rFBeRVd8kqK0SMKIgb9_nvr6HI=
>>> >   , "MINOR: Remove ARM/PowerPC builds from Jenkinsfile #12380". This PR 
>>> > removes the ARM and PowerPC builds from the Jenkinsfile.
>>> >
>>> > The rationale is that these builds seem to be failing all the time, 
>>> > and this is very disruptive. I personally didn't see any successes 
>>> > in the last week or two. So I think we need to rethink this integration a 
>>> > bit.
>>> >
>>> > I'd suggest that we run these builds as nightly builds rather than 
>>> > on each commit. It's going to be rare that we make a change that 
>>> > succeeds on x86 but breaks on PowerPC or ARM. This would let us 
>>> > have very long timeouts on our ARM and PowerPC builds (they could 
>>> > take all night if necessary), hence avoiding this issue.
>>> >
>>> > best,
>>> > Colin
>>> >
>>> --
>>> Divij Vaidya

Re: ARM/PowerPC builds

[Colin McCabe]

Hi Matthew,

It seems like you are the only one who has talked about seeing M1-specific 
failures -- so far, at least. So it would be helpful if you could post what 
you've seen.

best,
Colin

On Tue, Aug 23, 2022, at 03:05, Matthew Benedict de Detrich wrote:
> Sorry I just noticed this email now, is this still necessary? I noticed 
> the failures when doing the testing round for the latest Kafka release.
>
> --
> Matthew de Detrich
> Aiven Deutschland GmbH
> Immanuelkirchstraße 26, 10405 Berlin
> Amtsgericht Charlottenburg, HRB 209739 B
>
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> m: +491603708037
> w: aiven.io e: matthew.dedetr...@aiven.io
> On 4. Aug 2022, 19:10 +0200, Colin McCabe , wrote:
>> Hi Matthew,
>>
>> Can you open a JIRA for the test failures you have seen on M1?
>>
>> By the way, I have an M1 myself.
>>
>> best,
>> Colin
>>
>> On Thu, Aug 4, 2022, at 04:12, Matthew Benedict de Detrich wrote:
>> > Quite happy to have this change gone through since the ARM builds were
>> > constantly failing however I iterate what Divij Vaidya is saying. I
>> > just recently got a new MacBook M1 laptop that has ARM architecture and
>> > even locally the tests fail (these are the same tests that also failed
>> > in Jenkins).
>> >
>> > Should get to the root of the issue especially as more people will get
>> > newer Apple laptops over time.
>> >
>> > --
>> > Matthew de Detrich
>> > Aiven Deutschland GmbH
>> > Immanuelkirchstraße 26, 10405 Berlin
>> > Amtsgericht Charlottenburg, HRB 209739 B
>> >
>> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> > m: +491603708037
>> > w: aiven.io e: matthew.dedetr...@aiven.io
>> > On 4. Aug 2022, 12:36 +0200, Divij Vaidya , wrote:
>> > > Thank you. This would greatly improve the PR experience since now, there 
>> > > is
>> > > higher probability for it to be green.
>> > >
>> > > Side question though, do we know why ARM tests are timing out? Should we
>> > > start a JIRA with Apache Infra to root cause?
>> > >
>> > > —
>> > > Divij Vaidya
>> > >
>> > >
>> > >
>> > > On Thu, Aug 4, 2022 at 12:42 AM Colin McCabe  wrote:
>> > >
>> > > > Just a quick note. Today we committed
>> > > > https://github.com/apache/kafka/pull/12380 , "MINOR: Remove ARM/PowerPC
>> > > > builds from Jenkinsfile #12380". This PR removes the ARM and PowerPC 
>> > > > builds
>> > > > from the Jenkinsfile.
>> > > >
>> > > > The rationale is that these builds seem to be failing all the time, and
>> > > > this is very disruptive. I personally didn't see any successes in the 
>> > > > last
>> > > > week or two. So I think we need to rethink this integration a bit.
>> > > >
>> > > > I'd suggest that we run these builds as nightly builds rather than on 
>> > > > each
>> > > > commit. It's going to be rare that we make a change that succeeds on 
>> > > > x86
>> > > > but breaks on PowerPC or ARM. This would let us have very long 
>> > > > timeouts on
>> > > > our ARM and PowerPC builds (they could take all night if necessary), 
>> > > > hence
>> > > > avoiding this issue.
>> > > >
>> > > > best,
>> > > > Colin
>> > > >
>> > > --
>> > > Divij Vaidya

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1184

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 339908 lines...]
[2022-08-29T19:21:51.105Z] > Task :connect:api:testSrcJar
[2022-08-29T19:21:51.105Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2022-08-29T19:21:51.105Z] > Task :connect:api:publishToMavenLocal
[2022-08-29T19:21:51.105Z] 
[2022-08-29T19:21:51.105Z] > Task :streams:javadoc
[2022-08-29T19:21:51.105Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/processor/StreamPartitioner.java:50:
 warning - Tag @link: reference not found: 
org.apache.kafka.clients.producer.internals.DefaultPartitioner
[2022-08-29T19:21:52.148Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:890:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-29T19:21:52.148Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:919:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-29T19:21:52.148Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:939:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-29T19:21:52.148Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:854:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-29T19:21:52.148Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:890:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-29T19:21:52.148Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:919:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-29T19:21:52.148Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:939:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-29T19:21:53.388Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:84:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-29T19:21:53.388Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:136:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-29T19:21:53.388Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:147:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-29T19:21:53.388Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/Repartitioned.java:101:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-29T19:21:53.388Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/Repartitioned.java:167:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-29T19:21:54.507Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/TopologyConfig.java:58:
 warning - Tag @link: missing '#': "org.apache.kafka.streams.StreamsBuilder()"
[2022-08-29T19:21:54.507Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/TopologyConfig.java:58:
 warning - Tag @link: can't find org.apache.kafka.streams.StreamsBuilder() in 
org.apache.kafka.streams.TopologyConfig
[2022-08-29T19:21:54.507Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/TopologyDescription.java:38:
 warning - Tag @link: reference not found: ProcessorContext#forward(Object, 
Object) forwards
[2022-08-29T19:21:54.507Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/Position.java:44:
 warning - Tag @link: can't find query(Query,
[2022-08-29T19:21:54.507Z]  PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-08-29T19:21:54.507Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:109:
 warning - Tag @link: reference not found: this#getResult()
[2022-08-29T19:21:54.507Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:116:
 warning - Tag @link: reference not found: this#getFailureReason()
[2022-08-29T19:21:54.507Z] 

Re: Re: [DISCUSS] KIP-710: Full support for distributed mode in dedicated MirrorMaker 2.0 clusters

[Chris Egerton]

Hi Daniel,

Yeah, I think that's the way to go. Adding multiple servers for each herder
seems like it'd be too much of a pain for users to configure, and if we
keep the API strictly internal for now, we shouldn't be painting ourselves
into too much of a corner if/when we decide to expose a public-facing REST
API for dedicated MM2 clusters.

I plan to take a look at the rest of the KIP and provide a complete review
sometime this week; I'll hold off on commenting on anything that seems like
it'll be affected by switching to an internal-only REST API until those
changes are published, but should be able to review everything else.

Cheers,

Chris

On Mon, Aug 29, 2022 at 6:57 AM Dániel Urbán  wrote:

> Hi Chris,
>
> I understand your point, sounds good to me.
> So in short, we should opt for an internal-only API, and preferably a
> single server solution. Is that right?
>
> Thanks
> Daniel
>
> Chris Egerton  ezt írta (időpont: 2022. aug. 26.,
> P, 17:36):
>
> > Hi Daniel,
> >
> > Glad to hear from you!
> >
> > With regards to the stripped-down REST API alternative, I don't see how
> > this would prevent us from introducing the fully-fledged Connect REST
> API,
> > or even an augmented variant of it, at some point down the road. If we go
> > with the internal-only API now, and want to expand later, can't we gate
> the
> > expansion behind a feature flag configuration property that by default
> > disables the new feature?
> >
> > I'm also not sure that we'd ever want to expose the raw Connect REST API
> > for dedicated MM2 clusters. If people want that capability, they can
> > already spin up a vanilla Connect cluster and run as many MM2 connectors
> as
> > they'd like on it, and as of KIP-458 [1], it's even possible to use a
> > single Connect cluster to replicate between any two Kafka clusters
> instead
> > of only targeting the Kafka cluster that the vanilla Connect cluster
> > operates on top of. I do agree that it'd be great to be able to
> dynamically
> > adjust things like topic filters without having to restart a dedicated
> MM2
> > node; I'm just not sure that the vanilla Connect REST API is the
> > appropriate way to do that, especially since the exact mechanisms that
> make
> > a single Connect cluster viable for replicating across any two Kafka
> > clusters could be abused and cause a dedicated MM2 cluster to start
> writing
> > to a completely different Kafka cluster that's not even defined in its
> > config file.
> >
> > Finally, as far as security goes--since this is essentially a bug fix,
> I'm
> > inclined to make it as easy as possible for users to adopt it. MTLS is a
> > great start for securing a REST API, but it's not sufficient on its own
> > since anyone who could issue an authenticated REST request against the
> MM2
> > cluster would still be able to make any changes they want (with the
> > exception of accessing internal endpoints, which were secured with
> > KIP-507). If we were to bring up the fully-fledged Connect REST API,
> > cluster administrators would also likely have to add some kind of
> > authorization layer to prevent people from using the REST API to mess
> with
> > the configurations of the connectors that MM2 brought up. One way of
> doing
> > that is to add a REST extension to your Connect cluster, but implementing
> > and configuring one in order to be able to run a multi-node MM2 cluster
> > without hitting this bug seems like too much work to be worth it.
> >
> > I think if we had a better picture of what a REST API for dedicated MM2
> > clusters would/should look like, then it would be easier to go along with
> > this, and we could even just add the feature flag in this KIP right now
> to
> > address any security concerns. My instinct would be to address this in a
> > follow-up KIP in order to reduce scope creep, though, and keep this KIP
> > focused on addressing the bug with multi-node dedicated MM2 clusters.
> What
> > do you think?
> >
> > Cheers,
> >
> > Chris
> >
> > [1] -
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-458%3A+Connector+Client+Config+Override+Policy
> >
> > On Thu, Aug 25, 2022 at 3:55 AM Dániel Urbán 
> > wrote:
> >
> > > Hi Chris,
> > >
> > > Thanks for bringing this up again :)
> > >
> > > 1. I think that is reasonable, though I find the current state of MM2
> to
> > be
> > > confusing. The current issue with the distributed mode is not
> documented
> > > properly, but maybe the logging will help a bit.
> > >
> > > 2. Going for an internal-only Connect REST version would lock MM2 out
> of
> > a
> > > path where the REST API can be used to dynamically reconfigure
> > > replications. For now, I agree, it would be easy to corrupt the state
> of
> > > MM2 if someone wanted to use the properties and the REST at the same
> > time,
> > > but in the future, we might have a chance to introduce a different
> config
> > > mechanism, where only the cluster connections have to be specified in
> the
> > > properties file, and everything 

Re: [VOTE] KIP-837 Allow MultiCasting a Result Record.

[Chris Egerton]

Hi all,

I think it'd be useful to be more explicit about broadcasting to all topic
partitions rather than add implicit behavior for empty cases (empty
optional, empty list, etc.). The suggested enum approach would address that
nicely.

It's also worth noting that there's no hard requirement to add sugar for
broadcasting to all topic partitions since the API already provides the
number of topic partitions available when calling a stream partitioner. If
we can't find a clean way to add this support, it might be better to leave
it out and just let people implement this themselves with a line of Java 8
streams:

return IntStream.range(0,
numPartitions).boxed().collect(Collectors.toList());

Also worth noting that there may be a bug in the default implementation for
the new StreamPartitioner::partitions method, since it doesn't appear to
correctly pick up on null return values from the partition method and
translate them into empty lists.

Cheers,

Chris

On Mon, Aug 29, 2022 at 7:32 AM Bruno Cadonna  wrote:

> Hi Sagar,
>
> I do not see an issue with using an empty list together with an empty
> Optional. A non-empty Optional that contains an empty list would just
> say that there is no partition to which the record should be sent. If
> there is no partition, the record is dropped.
>
> An empty Optional might be a way to say, broadcast to all partitions.
>
> Alternatively -- to make it more explicit -- you could return an object
> with an enum and a list of partitions. The enum could have values SOME,
> ALL, and NONE. If the value is SOME, the list of partitions contains the
> partitions to which to send the record. If the value of the enum is ALL
> or NONE, the list of partitions is not used and might be even null
> (since in that case the list should not be used and it would be a bug to
> use it).
>
> Best,
> Bruno
>
> On 24.08.22 20:11, Sagar wrote:
> > Thank you Bruno/Matthew for your comments.
> >
> > I agree using null does seem error prone. However I think using a
> singleton
> > list of [-1] might be better in terms of usability, I am saying this
> > because the KIP also has a provision to return an empty list to refer to
> > dropping the record. So, an empty optional and an empty list have totally
> > different meanings which could get confusing.
> >
> > Let me know what you think.
> >
> > Thanks!
> > Sagar.
> >
> >
> > On Wed, Aug 24, 2022 at 7:30 PM Matthew Benedict de Detrich
> >  wrote:
> >
> >> I also concur with this, having an Optional in the type makes it very
> >> clear what’s going on and better signifies an absence of value (or in
> this
> >> case the broadcast value).
> >>
> >> --
> >> Matthew de Detrich
> >> Aiven Deutschland GmbH
> >> Immanuelkirchstraße 26, 10405 Berlin
> >> Amtsgericht Charlottenburg, HRB 209739 B
> >>
> >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >> m: +491603708037
> >> w: aiven.io e: matthew.dedetr...@aiven.io
> >> On 24. Aug 2022, 14:19 +0200, dev@kafka.apache.org, wrote:
> >>>
> >>> 2.
> >>> I would prefer changing the return type of partitions() to
> >>> Optional> and using Optional.empty() as the broadcast
> >>> value. IMO, The chances that an implementation returns null due to a
> bug
> >>> is much higher than that an implementation returns an empty Optional
> due
> >>> to a bug.
> >>
> >
>

[jira] [Created] (KAFKA-14187) Fix kafka-features.sh command

[Colin McCabe (Jira)]

Colin McCabe created KAFKA-14187:


 Summary: Fix kafka-features.sh command
 Key: KAFKA-14187
 URL: https://issues.apache.org/jira/browse/KAFKA-14187
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.3.0, 3.3
Reporter: Colin McCabe
Assignee: Colin McCabe


Fix the kafka-features.sh command so that we can upgrade to the new version as 
expected.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Apache Kafka 3.3.0 Release

[José Armando García Sancio]

The documentation and protocol links are not working. Looking into it.

https://kafka.apache.org/33/documentation.html
https://kafka.apache.org/33/protocol.html

Thanks,
-José

Re: [DISCUSS] Apache Kafka 3.3.0 Release

[José Armando García Sancio]

Hello Kafka users, developers and client-developers,

This is the first candidate for release of Apache Kafka 3.3.0.

Release notes for the 3.3.0 release:
https://home.apache.org/~jsancio/kafka-3.3.0-rc1/RELEASE_NOTES.html

Please download and test.

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts (source and binary):
https://home.apache.org/~jsancio/kafka-3.3.0-rc1/

* Maven artifacts:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~jsancio/kafka-3.3.0-rc1/javadoc/

* The 3.3.0 tag:
https://github.com/apache/kafka/releases/tag/3.3.0-rc1

* Documentation:
https://kafka.apache.org/33/documentation.html

* Protocol:
https://kafka.apache.org/33/protocol.html

* Successful Jenkins builds for the 3.3 branch:
Unit/integration tests:
https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.3/50/

The system tests are failing and I am looking into the issue.

Thanks,
-- 
-José

Re: Request for Contributor access

[Mickael Maison]

Hi Prince,

I've granted you contributor access.

Thanks for your interest in Apache Kafka!
Mickael

On Mon, Aug 29, 2022 at 6:12 PM prince jha  wrote:
>
> Hi all,
>
> Can I please get the Contributor access for Kafka ? Please find my Jira ID
> below:
>
> JIRA ID: princejha
>
> Regards,
> Prince

Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol

[David Jacot]

Hi all,

The KIP states that we will re-implement the coordinator in Java. I
discussed this offline with a few folks and folks are concerned that
we could introduce many regressions in the old protocol if we do so.
Therefore, I am going to remove this statement from the KIP. It is an
implementation detail after all so it does not have to be decided at
this stage. We will likely start by trying to refactor the current
implementation as a first step.

Cheers,
David

On Mon, Aug 29, 2022 at 3:52 PM David Jacot  wrote:
>
> Hi Luke,
>
> > 1.1. I think the state machine are: "Empty, assigning, reconciling, stable,
> > dead" mentioned in Consumer Group States section, right?
>
> This sentence does not refer to those group states but rather to a
> state machine replication (SMR). This refers to the entire state of
> group coordinator which is replicated via the log layer. I will
> clarify this in the KIP.
>
> > 1.2. What do you mean "each state machine is modelled as an event loop"?
>
> The idea is to follow a model similar to the new quorum controller. We
> will have N threads to process events. Each __consumer_offsets
> partition is assigned to a unique thread and all the events (e.g.
> requests, callbacks, etc.) are processed by this thread. This simplify
> concurrency and will enable us to do simulation testing for the group
> coordinator.
>
> > 1.3. Why do we need a state machine per *__consumer_offsets* partitions?
> > Not a state machine "per consumer group" owned by a group coordinator? For
> > example, if one group coordinator owns 2 consumer groups, and both exist in
> > *__consumer_offsets-0*, will we have 1 state machine for it, or 2?
>
> See 1.1. The confusion comes from there, I think.
>
> > 1.4. I know the "*group.coordinator.threads" *is the number of threads used
> > to run the state machines. But I'm wondering if the purpose of the threads
> > is only to keep the state of each consumer group (or *__consumer_offsets*
> > partitions?), and no heavy computation, why should we need multi-threads
> > here?
>
> See 1.2. The idea is to have an ability to shard the processing as the
> computation could be heavy.
>
> > 2.1. The consumer session timeout, why does the default session timeout not
> > locate between min (45s) and max(60s)? I thought the min/max session
> > timeout is to define lower/upper bound of it, no?
> >
> > group.consumer.session.timeout.ms int 30s The timeout to detect client
> > failures when using the consumer group protocol.
> > group.consumer.min.session.timeout.ms int 45s The minimum session timeout.
> > group.consumer.max.session.timeout.ms int 60s The maximum session timeout.
>
> This is indeed a mistake. The default session timeout should be 45s
> (the current default).
>
> > 2.2. The default server side assignor are [range, uniform], which means
> > we'll default to "range" assignor. I'd like to know why not uniform one? I
> > thought usually users will choose uniform assignor (former sticky assinor)
> > for better evenly distribution. Any other reason we choose range assignor
> > as default?
> > group.consumer.assignors List range, uniform The server side assignors.
>
> The order on the server side has no influence because the client must
> chose the selector that he wants to use. There is no default in the
> current proposal. If the assignor is not specified by the client, the
> request is rejected. The default client value for
> `group.remote.assignor` is `uniform` though.
>
> Thanks for your very good comments, Luke. I hope that my answers help
> to clarify things. I will update the KIP as well based on your
> feedback.
>
> Cheers,
> David
>
> On Mon, Aug 22, 2022 at 9:29 AM Luke Chen  wrote:
> >
> > Hi David,
> >
> > Thanks for the update.
> >
> > Some more questions:
> > 1. In Group Coordinator section, you mentioned:
> > > The new group coordinator will have a state machine per
> > *__consumer_offsets* partitions, where each state machine is modelled as an
> > event loop. Those state machines will be executed in
> > *group.coordinator.threads* threads.
> >
> > 1.1. I think the state machine are: "Empty, assigning, reconciling, stable,
> > dead" mentioned in Consumer Group States section, right?
> > 1.2. What do you mean "each state machine is modelled as an event loop"?
> > 1.3. Why do we need a state machine per *__consumer_offsets* partitions?
> > Not a state machine "per consumer group" owned by a group coordinator? For
> > example, if one group coordinator owns 2 consumer groups, and both exist in
> > *__consumer_offsets-0*, will we have 1 state machine for it, or 2?
> > 1.4. I know the "*group.coordinator.threads" *is the number of threads used
> > to run the state machines. But I'm wondering if the purpose of the threads
> > is only to keep the state of each consumer group (or *__consumer_offsets*
> > partitions?), and no heavy computation, why should we need multi-threads
> > here?
> >
> > 2. For the default value in the new configs:
> > 2.1. The consumer 

Request for Contributor access

[prince jha]

Hi all,

Can I please get the Contributor access for Kafka ? Please find my Jira ID
below:

JIRA ID: princejha

Regards,
Prince

Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol

[David Jacot]

Hi Luke,

> 1.1. I think the state machine are: "Empty, assigning, reconciling, stable,
> dead" mentioned in Consumer Group States section, right?

This sentence does not refer to those group states but rather to a
state machine replication (SMR). This refers to the entire state of
group coordinator which is replicated via the log layer. I will
clarify this in the KIP.

> 1.2. What do you mean "each state machine is modelled as an event loop"?

The idea is to follow a model similar to the new quorum controller. We
will have N threads to process events. Each __consumer_offsets
partition is assigned to a unique thread and all the events (e.g.
requests, callbacks, etc.) are processed by this thread. This simplify
concurrency and will enable us to do simulation testing for the group
coordinator.

> 1.3. Why do we need a state machine per *__consumer_offsets* partitions?
> Not a state machine "per consumer group" owned by a group coordinator? For
> example, if one group coordinator owns 2 consumer groups, and both exist in
> *__consumer_offsets-0*, will we have 1 state machine for it, or 2?

See 1.1. The confusion comes from there, I think.

> 1.4. I know the "*group.coordinator.threads" *is the number of threads used
> to run the state machines. But I'm wondering if the purpose of the threads
> is only to keep the state of each consumer group (or *__consumer_offsets*
> partitions?), and no heavy computation, why should we need multi-threads
> here?

See 1.2. The idea is to have an ability to shard the processing as the
computation could be heavy.

> 2.1. The consumer session timeout, why does the default session timeout not
> locate between min (45s) and max(60s)? I thought the min/max session
> timeout is to define lower/upper bound of it, no?
>
> group.consumer.session.timeout.ms int 30s The timeout to detect client
> failures when using the consumer group protocol.
> group.consumer.min.session.timeout.ms int 45s The minimum session timeout.
> group.consumer.max.session.timeout.ms int 60s The maximum session timeout.

This is indeed a mistake. The default session timeout should be 45s
(the current default).

> 2.2. The default server side assignor are [range, uniform], which means
> we'll default to "range" assignor. I'd like to know why not uniform one? I
> thought usually users will choose uniform assignor (former sticky assinor)
> for better evenly distribution. Any other reason we choose range assignor
> as default?
> group.consumer.assignors List range, uniform The server side assignors.

The order on the server side has no influence because the client must
chose the selector that he wants to use. There is no default in the
current proposal. If the assignor is not specified by the client, the
request is rejected. The default client value for
`group.remote.assignor` is `uniform` though.

Thanks for your very good comments, Luke. I hope that my answers help
to clarify things. I will update the KIP as well based on your
feedback.

Cheers,
David

On Mon, Aug 22, 2022 at 9:29 AM Luke Chen  wrote:
>
> Hi David,
>
> Thanks for the update.
>
> Some more questions:
> 1. In Group Coordinator section, you mentioned:
> > The new group coordinator will have a state machine per
> *__consumer_offsets* partitions, where each state machine is modelled as an
> event loop. Those state machines will be executed in
> *group.coordinator.threads* threads.
>
> 1.1. I think the state machine are: "Empty, assigning, reconciling, stable,
> dead" mentioned in Consumer Group States section, right?
> 1.2. What do you mean "each state machine is modelled as an event loop"?
> 1.3. Why do we need a state machine per *__consumer_offsets* partitions?
> Not a state machine "per consumer group" owned by a group coordinator? For
> example, if one group coordinator owns 2 consumer groups, and both exist in
> *__consumer_offsets-0*, will we have 1 state machine for it, or 2?
> 1.4. I know the "*group.coordinator.threads" *is the number of threads used
> to run the state machines. But I'm wondering if the purpose of the threads
> is only to keep the state of each consumer group (or *__consumer_offsets*
> partitions?), and no heavy computation, why should we need multi-threads
> here?
>
> 2. For the default value in the new configs:
> 2.1. The consumer session timeout, why does the default session timeout not
> locate between min (45s) and max(60s)? I thought the min/max session
> timeout is to define lower/upper bound of it, no?
>
> group.consumer.session.timeout.ms int 30s The timeout to detect client
> failures when using the consumer group protocol.
> group.consumer.min.session.timeout.ms int 45s The minimum session timeout.
> group.consumer.max.session.timeout.ms int 60s The maximum session timeout.
>
>
>
> 2.2. The default server side assignor are [range, uniform], which means
> we'll default to "range" assignor. I'd like to know why not uniform one? I
> thought usually users will choose uniform assignor (former sticky assinor)
> for 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1183

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-837 Allow MultiCasting a Result Record.

[Bruno Cadonna]

Hi Sagar,

I do not see an issue with using an empty list together with an empty 
Optional. A non-empty Optional that contains an empty list would just 
say that there is no partition to which the record should be sent. If 
there is no partition, the record is dropped.


An empty Optional might be a way to say, broadcast to all partitions.

Alternatively -- to make it more explicit -- you could return an object 
with an enum and a list of partitions. The enum could have values SOME, 
ALL, and NONE. If the value is SOME, the list of partitions contains the 
partitions to which to send the record. If the value of the enum is ALL 
or NONE, the list of partitions is not used and might be even null 
(since in that case the list should not be used and it would be a bug to 
use it).


Best,
Bruno

On 24.08.22 20:11, Sagar wrote:

Thank you Bruno/Matthew for your comments.

I agree using null does seem error prone. However I think using a singleton
list of [-1] might be better in terms of usability, I am saying this
because the KIP also has a provision to return an empty list to refer to
dropping the record. So, an empty optional and an empty list have totally
different meanings which could get confusing.

Let me know what you think.

Thanks!
Sagar.


On Wed, Aug 24, 2022 at 7:30 PM Matthew Benedict de Detrich
 wrote:


I also concur with this, having an Optional in the type makes it very
clear what’s going on and better signifies an absence of value (or in this
case the broadcast value).

--
Matthew de Detrich
Aiven Deutschland GmbH
Immanuelkirchstraße 26, 10405 Berlin
Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
m: +491603708037
w: aiven.io e: matthew.dedetr...@aiven.io
On 24. Aug 2022, 14:19 +0200, dev@kafka.apache.org, wrote:


2.
I would prefer changing the return type of partitions() to
Optional> and using Optional.empty() as the broadcast
value. IMO, The chances that an implementation returns null due to a bug
is much higher than that an implementation returns an empty Optional due
to a bug.

Re: Re: [DISCUSS] KIP-710: Full support for distributed mode in dedicated MirrorMaker 2.0 clusters

[Dániel Urbán]

Hi Chris,

I understand your point, sounds good to me.
So in short, we should opt for an internal-only API, and preferably a
single server solution. Is that right?

Thanks
Daniel

Chris Egerton  ezt írta (időpont: 2022. aug. 26.,
P, 17:36):

> Hi Daniel,
>
> Glad to hear from you!
>
> With regards to the stripped-down REST API alternative, I don't see how
> this would prevent us from introducing the fully-fledged Connect REST API,
> or even an augmented variant of it, at some point down the road. If we go
> with the internal-only API now, and want to expand later, can't we gate the
> expansion behind a feature flag configuration property that by default
> disables the new feature?
>
> I'm also not sure that we'd ever want to expose the raw Connect REST API
> for dedicated MM2 clusters. If people want that capability, they can
> already spin up a vanilla Connect cluster and run as many MM2 connectors as
> they'd like on it, and as of KIP-458 [1], it's even possible to use a
> single Connect cluster to replicate between any two Kafka clusters instead
> of only targeting the Kafka cluster that the vanilla Connect cluster
> operates on top of. I do agree that it'd be great to be able to dynamically
> adjust things like topic filters without having to restart a dedicated MM2
> node; I'm just not sure that the vanilla Connect REST API is the
> appropriate way to do that, especially since the exact mechanisms that make
> a single Connect cluster viable for replicating across any two Kafka
> clusters could be abused and cause a dedicated MM2 cluster to start writing
> to a completely different Kafka cluster that's not even defined in its
> config file.
>
> Finally, as far as security goes--since this is essentially a bug fix, I'm
> inclined to make it as easy as possible for users to adopt it. MTLS is a
> great start for securing a REST API, but it's not sufficient on its own
> since anyone who could issue an authenticated REST request against the MM2
> cluster would still be able to make any changes they want (with the
> exception of accessing internal endpoints, which were secured with
> KIP-507). If we were to bring up the fully-fledged Connect REST API,
> cluster administrators would also likely have to add some kind of
> authorization layer to prevent people from using the REST API to mess with
> the configurations of the connectors that MM2 brought up. One way of doing
> that is to add a REST extension to your Connect cluster, but implementing
> and configuring one in order to be able to run a multi-node MM2 cluster
> without hitting this bug seems like too much work to be worth it.
>
> I think if we had a better picture of what a REST API for dedicated MM2
> clusters would/should look like, then it would be easier to go along with
> this, and we could even just add the feature flag in this KIP right now to
> address any security concerns. My instinct would be to address this in a
> follow-up KIP in order to reduce scope creep, though, and keep this KIP
> focused on addressing the bug with multi-node dedicated MM2 clusters. What
> do you think?
>
> Cheers,
>
> Chris
>
> [1] -
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-458%3A+Connector+Client+Config+Override+Policy
>
> On Thu, Aug 25, 2022 at 3:55 AM Dániel Urbán 
> wrote:
>
> > Hi Chris,
> >
> > Thanks for bringing this up again :)
> >
> > 1. I think that is reasonable, though I find the current state of MM2 to
> be
> > confusing. The current issue with the distributed mode is not documented
> > properly, but maybe the logging will help a bit.
> >
> > 2. Going for an internal-only Connect REST version would lock MM2 out of
> a
> > path where the REST API can be used to dynamically reconfigure
> > replications. For now, I agree, it would be easy to corrupt the state of
> > MM2 if someone wanted to use the properties and the REST at the same
> time,
> > but in the future, we might have a chance to introduce a different config
> > mechanism, where only the cluster connections have to be specified in the
> > properties file, and everything else can be configured through REST
> > (enabling replications, changing topic filters, etc.). Because of this,
> I'm
> > leaning towards a full Connect REST API. To avoid issues with conflicts
> > between the props file and the REST, we could document security best
> > practices (e.g. turn on basic auth or mTLS on the Connect REST to avoid
> > unwanted interactions).
> >
> > 3. That is a good point, and I agree, a big plus for motivation.
> >
> > I have a working version of this in which all flows spin up a dedicated
> > Connect REST, but I can give other solutions a try, too.
> >
> > Thanks,
> > Daniel
> >
> > Chris Egerton  ezt írta (időpont: 2022. aug.
> 24.,
> > Sze, 17:46):
> >
> > > Hi Daniel,
> > >
> > > I'd like to resurface this KIP in case you're still interested in
> > pursuing
> > > it. I know it's been a while since you published it, and it hasn't
> > received
> > > much attention, but I'm hoping 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1182

[Apache Jenkins Server]

See 

[GitHub] [kafka-site] SimonRenault86 commented on pull request #437: HelloSafe Kafka

[GitBox]

SimonRenault86 commented on PR #437:
URL: https://github.com/apache/kafka-site/pull/437#issuecomment-122633

   Hello @bbejeck thanks a lot for your feedbacks, i added the picture


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org