[GitHub] [kafka-site] jmalcone546 opened a new pull request, #500: Added Deep.BI - Predictive Analytics & AI Platform

[via GitHub]

jmalcone546 opened a new pull request, #500:
URL: https://github.com/apache/kafka-site/pull/500

   Hi there, I'd like to request to add Deep.BI 
(https://www.crunchbase.com/organization/deep-bi) to the powered-by list. I 
added the link to our live logo - if this should be done differently please let 
me know!
   
   You can find out more about us through our Druid Summit presentation last 
year to verify our architecture and Kafka usage: 
https://www.youtube.com/watch?v=oc3Bmi1XHX8.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1736

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-14765) Support SCRAM for brokers at bootstrap

[Proven Provenzano (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Proven Provenzano resolved KAFKA-14765.
---
Resolution: Fixed

[PR #13374|https://github.com/apache/kafka/pull/13374]includes both KIP-900 
implementation and updated integration tests.

> Support SCRAM for brokers at bootstrap
> --
>
> Key: KAFKA-14765
> URL: https://issues.apache.org/jira/browse/KAFKA-14765
> Project: Kafka
>  Issue Type: Improvement
>  Components: kraft
>Reporter: Proven Provenzano
>Assignee: Proven Provenzano
>Priority: Major
>  Labels: KIP-900
> Fix For: 3.5.0
>
>
> We want to add SCRAM support for brokers at bootstrap.
> We will support bootstrap as described in 
> [KIP-900|https://cwiki.apache.org/confluence/display/KAFKA/KIP-900%3A+KRaft+kafka-storage.sh+API+additions+to+support+SCRAM+for+Kafka+Brokers]
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14776) Update SCRAM integration tests to run with KRaft

[Proven Provenzano (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Proven Provenzano resolved KAFKA-14776.
---
Resolution: Fixed

[PR #13374|https://github.com/apache/kafka/pull/13374]includes both KIP-900 
implementation and updated integration tests.

> Update SCRAM integration tests to run with KRaft
> 
>
> Key: KAFKA-14776
> URL: https://issues.apache.org/jira/browse/KAFKA-14776
> Project: Kafka
>  Issue Type: Improvement
>  Components: kraft
>Reporter: Proven Provenzano
>Assignee: Proven Provenzano
>Priority: Major
> Fix For: 3.5.0
>
>
> I will update the SCRAM integration tests to run under both ZK and KRaft 
> quorum mode.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1735

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 459484 lines...]
[2023-04-04T18:47:21.742Z] > Task :group-coordinator:compileTestJava UP-TO-DATE
[2023-04-04T18:47:21.742Z] > Task :group-coordinator:testClasses UP-TO-DATE
[2023-04-04T18:47:21.742Z] > Task :raft:compileTestJava UP-TO-DATE
[2023-04-04T18:47:21.742Z] > Task :raft:testClasses UP-TO-DATE
[2023-04-04T18:47:21.742Z] > Task :metadata:compileTestJava UP-TO-DATE
[2023-04-04T18:47:21.742Z] > Task :metadata:testClasses UP-TO-DATE
[2023-04-04T18:47:21.742Z] > Task 
:clients:generateMetadataFileForMavenJavaPublication
[2023-04-04T18:47:21.742Z] > Task :streams:copyDependantLibs
[2023-04-04T18:47:21.742Z] > Task :streams:jar UP-TO-DATE
[2023-04-04T18:47:22.693Z] > Task 
:streams:generateMetadataFileForMavenJavaPublication
[2023-04-04T18:47:25.538Z] 
[2023-04-04T18:47:25.538Z] > Task :connect:api:javadoc
[2023-04-04T18:47:25.538Z] 
/home/jenkins/workspace/Kafka_kafka_trunk_2/connect/api/src/main/java/org/apache/kafka/connect/source/SourceRecord.java:44:
 warning - Tag @link: reference not found: org.apache.kafka.connect.data
[2023-04-04T18:47:26.492Z] 1 warning
[2023-04-04T18:47:26.492Z] 
[2023-04-04T18:47:26.492Z] > Task :connect:api:copyDependantLibs UP-TO-DATE
[2023-04-04T18:47:26.492Z] > Task :connect:api:jar UP-TO-DATE
[2023-04-04T18:47:26.492Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2023-04-04T18:47:26.492Z] > Task :connect:json:copyDependantLibs UP-TO-DATE
[2023-04-04T18:47:26.492Z] > Task :connect:json:jar UP-TO-DATE
[2023-04-04T18:47:26.492Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2023-04-04T18:47:27.444Z] > Task :connect:api:javadocJar
[2023-04-04T18:47:27.444Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2023-04-04T18:47:27.444Z] > Task :connect:api:testClasses UP-TO-DATE
[2023-04-04T18:47:27.444Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2023-04-04T18:47:27.444Z] > Task :connect:json:publishToMavenLocal
[2023-04-04T18:47:27.444Z] > Task :connect:api:testJar
[2023-04-04T18:47:27.444Z] > Task :connect:api:testSrcJar
[2023-04-04T18:47:27.444Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2023-04-04T18:47:27.444Z] > Task :connect:api:publishToMavenLocal
[2023-04-04T18:47:29.719Z] > Task :streams:javadoc
[2023-04-04T18:47:29.719Z] > Task :streams:javadocJar
[2023-04-04T18:47:29.719Z] 
[2023-04-04T18:47:29.719Z] > Task :clients:javadoc
[2023-04-04T18:47:29.719Z] 
/home/jenkins/workspace/Kafka_kafka_trunk_2/clients/src/main/java/org/apache/kafka/clients/admin/ScramMechanism.java:32:
 warning - Tag @see: missing final '>': "https://cwiki.apache.org/confluence/display/KAFKA/KIP-554%3A+Add+Broker-side+SCRAM+Config+API;>KIP-554:
 Add Broker-side SCRAM Config API
[2023-04-04T18:47:29.719Z] 
[2023-04-04T18:47:29.719Z]  This code is duplicated in 
org.apache.kafka.common.security.scram.internals.ScramMechanism.
[2023-04-04T18:47:29.719Z]  The type field in both files must match and must 
not change. The type field
[2023-04-04T18:47:29.719Z]  is used both for passing ScramCredentialUpsertion 
and for the internal
[2023-04-04T18:47:29.719Z]  UserScramCredentialRecord. Do not change the type 
field."
[2023-04-04T18:47:30.671Z] 
/home/jenkins/workspace/Kafka_kafka_trunk_2/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/package-info.java:21:
 warning - Tag @link: reference not found: 
org.apache.kafka.common.security.oauthbearer
[2023-04-04T18:47:31.623Z] 2 warnings
[2023-04-04T18:47:31.623Z] 
[2023-04-04T18:47:31.623Z] > Task :clients:javadocJar
[2023-04-04T18:47:32.677Z] > Task :clients:srcJar
[2023-04-04T18:47:32.677Z] > Task :clients:testJar
[2023-04-04T18:47:33.629Z] > Task :clients:testSrcJar
[2023-04-04T18:47:33.629Z] > Task 
:clients:publishMavenJavaPublicationToMavenLocal
[2023-04-04T18:47:33.629Z] > Task :clients:publishToMavenLocal
[2023-04-04T18:47:51.639Z] > Task :core:compileScala
[2023-04-04T18:48:51.895Z] > Task :core:classes
[2023-04-04T18:48:51.895Z] > Task :core:compileTestJava NO-SOURCE
[2023-04-04T18:49:19.167Z] > Task :core:compileTestScala
[2023-04-04T18:50:09.773Z] > Task :core:testClasses
[2023-04-04T18:50:19.834Z] > Task :streams:compileTestJava
[2023-04-04T18:50:19.834Z] > Task :streams:testClasses
[2023-04-04T18:50:21.004Z] > Task :streams:testJar
[2023-04-04T18:50:21.004Z] > Task :streams:testSrcJar
[2023-04-04T18:50:21.004Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2023-04-04T18:50:21.004Z] > Task :streams:publishToMavenLocal
[2023-04-04T18:50:21.004Z] 
[2023-04-04T18:50:21.004Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 9.0.
[2023-04-04T18:50:21.004Z] 
[2023-04-04T18:50:21.004Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.

Re: [DISCUSS] KIP-895: Dynamically refresh partition count of __consumer_offsets

[Alexandre Dupriez]

Hi Christo,

Thanks for the KIP. Apologies for the delayed review.

At a high-level, I am not sure if the KIP really solves the problem it
intends to.

More specifically, the KIP mentions that once a broker is restarted
and the group coordinator becomes aware of the new partition count of
the consumer offsets topic, the problem is mitigated. However, how do
we access the metadata and offsets recorded in a partition once it is
no longer the partition a consumer group resolves to?

Thanks,
Alexandre

Le mar. 4 avr. 2023 à 18:34, Justine Olshan
 a écrit :
>
> Hi,
>
> I'm also a bit unsure of the motivation here. Is there a need to change the
> number of partitions for this topic?
>
> Justine
>
> On Tue, Apr 4, 2023 at 10:07 AM David Jacot  wrote:
>
> > Hi,
> >
> > I am not very comfortable with the proposal of this KIP. The main issue is
> > that changing the number of partitions means that all group metadata is
> > lost because the hashing changes. I wonder if we should just disallow
> > changing the number of partitions entirely. Did we consider something like
> > this?
> >
> > Best,
> > David
> >
> > Le mar. 4 avr. 2023 à 17:57, Divij Vaidya  a
> > écrit :
> >
> > > FYI, a user faced this problem and reached out to us in the mailing list
> > > [1]. Implementation of this KIP could have reduced the downtime for these
> > > customers.
> > >
> > > Christo, would you like to create a JIRA and associate with the KIP so
> > that
> > > we can continue to collect cases in the JIRA where users have faced this
> > > problem?
> > >
> > > [1] https://lists.apache.org/thread/zoowjshvdpkh5p0p7vqjd9fq8xvkr1nd
> > >
> > > --
> > > Divij Vaidya
> > >
> > >
> > >
> > > On Wed, Jan 18, 2023 at 9:52 AM Christo Lolov 
> > > wrote:
> > >
> > > > Greetings,
> > > >
> > > > I am bumping the below DISCUSSion thread for KIP-895. The KIP presents
> > a
> > > > situation where consumer groups are in an undefined state until a
> > rolling
> > > > restart of a cluster is performed. While I have demonstrated the
> > > behaviour
> > > > using a cluster using Zookeeper I believe the same problem can be shown
> > > in
> > > > a KRaft cluster. Please let me know your opinions on the problem and
> > the
> > > > presented solution.
> > > >
> > > > Best,
> > > > Christo
> > > >
> > > > On Thursday, 29 December 2022 at 14:19:27 GMT, Christo
> > > > >  wrote:
> > > > >
> > > > >
> > > > > Hello!
> > > > > I would like to start this discussion thread on KIP-895: Dynamically
> > > > > refresh partition count of __consumer_offsets.
> > > > > The KIP proposes to alter brokers so that they refresh the partition
> > > > count
> > > > > of __consumer_offsets used to determine group coordinators without
> > > > > requiring a rolling restart of the cluster.
> > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-895%3A+Dynamically+refresh+partition+count+of+__consumer_offsets
> > > > >
> > > > > Let me know your thoughts on the matter!
> > > > > Best, Christo
> > > > >
> > > >
> > >
> >

[jira] [Created] (KAFKA-14879) Include newer versions in system tests

[Kirk True (Jira)]

Kirk True created KAFKA-14879:
-

 Summary: Include newer versions in system tests
 Key: KAFKA-14879
 URL: https://issues.apache.org/jira/browse/KAFKA-14879
 Project: Kafka
  Issue Type: Bug
  Components: clients, system tests
Reporter: Kirk True
Assignee: Kirk True


Include the following versions in the system tests:

 
 * 2.6.3
 * 2.7.2
 * 2.8.1
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-895: Dynamically refresh partition count of __consumer_offsets

[Justine Olshan]

Hi,

I'm also a bit unsure of the motivation here. Is there a need to change the
number of partitions for this topic?

Justine

On Tue, Apr 4, 2023 at 10:07 AM David Jacot  wrote:

> Hi,
>
> I am not very comfortable with the proposal of this KIP. The main issue is
> that changing the number of partitions means that all group metadata is
> lost because the hashing changes. I wonder if we should just disallow
> changing the number of partitions entirely. Did we consider something like
> this?
>
> Best,
> David
>
> Le mar. 4 avr. 2023 à 17:57, Divij Vaidya  a
> écrit :
>
> > FYI, a user faced this problem and reached out to us in the mailing list
> > [1]. Implementation of this KIP could have reduced the downtime for these
> > customers.
> >
> > Christo, would you like to create a JIRA and associate with the KIP so
> that
> > we can continue to collect cases in the JIRA where users have faced this
> > problem?
> >
> > [1] https://lists.apache.org/thread/zoowjshvdpkh5p0p7vqjd9fq8xvkr1nd
> >
> > --
> > Divij Vaidya
> >
> >
> >
> > On Wed, Jan 18, 2023 at 9:52 AM Christo Lolov 
> > wrote:
> >
> > > Greetings,
> > >
> > > I am bumping the below DISCUSSion thread for KIP-895. The KIP presents
> a
> > > situation where consumer groups are in an undefined state until a
> rolling
> > > restart of a cluster is performed. While I have demonstrated the
> > behaviour
> > > using a cluster using Zookeeper I believe the same problem can be shown
> > in
> > > a KRaft cluster. Please let me know your opinions on the problem and
> the
> > > presented solution.
> > >
> > > Best,
> > > Christo
> > >
> > > On Thursday, 29 December 2022 at 14:19:27 GMT, Christo
> > > >  wrote:
> > > >
> > > >
> > > > Hello!
> > > > I would like to start this discussion thread on KIP-895: Dynamically
> > > > refresh partition count of __consumer_offsets.
> > > > The KIP proposes to alter brokers so that they refresh the partition
> > > count
> > > > of __consumer_offsets used to determine group coordinators without
> > > > requiring a rolling restart of the cluster.
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-895%3A+Dynamically+refresh+partition+count+of+__consumer_offsets
> > > >
> > > > Let me know your thoughts on the matter!
> > > > Best, Christo
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-895: Dynamically refresh partition count of __consumer_offsets

[David Jacot]

Hi,

I am not very comfortable with the proposal of this KIP. The main issue is
that changing the number of partitions means that all group metadata is
lost because the hashing changes. I wonder if we should just disallow
changing the number of partitions entirely. Did we consider something like
this?

Best,
David

Le mar. 4 avr. 2023 à 17:57, Divij Vaidya  a
écrit :

> FYI, a user faced this problem and reached out to us in the mailing list
> [1]. Implementation of this KIP could have reduced the downtime for these
> customers.
>
> Christo, would you like to create a JIRA and associate with the KIP so that
> we can continue to collect cases in the JIRA where users have faced this
> problem?
>
> [1] https://lists.apache.org/thread/zoowjshvdpkh5p0p7vqjd9fq8xvkr1nd
>
> --
> Divij Vaidya
>
>
>
> On Wed, Jan 18, 2023 at 9:52 AM Christo Lolov 
> wrote:
>
> > Greetings,
> >
> > I am bumping the below DISCUSSion thread for KIP-895. The KIP presents a
> > situation where consumer groups are in an undefined state until a rolling
> > restart of a cluster is performed. While I have demonstrated the
> behaviour
> > using a cluster using Zookeeper I believe the same problem can be shown
> in
> > a KRaft cluster. Please let me know your opinions on the problem and the
> > presented solution.
> >
> > Best,
> > Christo
> >
> > On Thursday, 29 December 2022 at 14:19:27 GMT, Christo
> > >  wrote:
> > >
> > >
> > > Hello!
> > > I would like to start this discussion thread on KIP-895: Dynamically
> > > refresh partition count of __consumer_offsets.
> > > The KIP proposes to alter brokers so that they refresh the partition
> > count
> > > of __consumer_offsets used to determine group coordinators without
> > > requiring a rolling restart of the cluster.
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-895%3A+Dynamically+refresh+partition+count+of+__consumer_offsets
> > >
> > > Let me know your thoughts on the matter!
> > > Best, Christo
> > >
> >
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1734

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-895: Dynamically refresh partition count of __consumer_offsets

[Divij Vaidya]

FYI, a user faced this problem and reached out to us in the mailing list
[1]. Implementation of this KIP could have reduced the downtime for these
customers.

Christo, would you like to create a JIRA and associate with the KIP so that
we can continue to collect cases in the JIRA where users have faced this
problem?

[1] https://lists.apache.org/thread/zoowjshvdpkh5p0p7vqjd9fq8xvkr1nd

--
Divij Vaidya



On Wed, Jan 18, 2023 at 9:52 AM Christo Lolov 
wrote:

> Greetings,
>
> I am bumping the below DISCUSSion thread for KIP-895. The KIP presents a
> situation where consumer groups are in an undefined state until a rolling
> restart of a cluster is performed. While I have demonstrated the behaviour
> using a cluster using Zookeeper I believe the same problem can be shown in
> a KRaft cluster. Please let me know your opinions on the problem and the
> presented solution.
>
> Best,
> Christo
>
> On Thursday, 29 December 2022 at 14:19:27 GMT, Christo
> >  wrote:
> >
> >
> > Hello!
> > I would like to start this discussion thread on KIP-895: Dynamically
> > refresh partition count of __consumer_offsets.
> > The KIP proposes to alter brokers so that they refresh the partition
> count
> > of __consumer_offsets used to determine group coordinators without
> > requiring a rolling restart of the cluster.
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-895%3A+Dynamically+refresh+partition+count+of+__consumer_offsets
> >
> > Let me know your thoughts on the matter!
> > Best, Christo
> >
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.4 #104

[Apache Jenkins Server]

See 

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.3 #167

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 502272 lines...]
[2023-04-04T13:39:43.884Z] 
[2023-04-04T13:39:43.884Z] 
org.apache.kafka.streams.integration.StandbyTaskCreationIntegrationTest > 
shouldCreateStandByTasksForMaterializedAndOptimizedSourceTables STARTED
[2023-04-04T13:39:47.396Z] 
[2023-04-04T13:39:47.396Z] 
org.apache.kafka.streams.integration.StandbyTaskCreationIntegrationTest > 
shouldCreateStandByTasksForMaterializedAndOptimizedSourceTables PASSED
[2023-04-04T13:39:48.448Z] 
[2023-04-04T13:39:48.448Z] 
org.apache.kafka.streams.integration.StateDirectoryIntegrationTest > 
testNotCleanUpStateDirIfNotEmpty STARTED
[2023-04-04T13:39:50.549Z] 
[2023-04-04T13:39:50.549Z] 
org.apache.kafka.streams.integration.StateDirectoryIntegrationTest > 
testNotCleanUpStateDirIfNotEmpty PASSED
[2023-04-04T13:39:50.549Z] 
[2023-04-04T13:39:50.549Z] 
org.apache.kafka.streams.integration.StateDirectoryIntegrationTest > 
testCleanUpStateDirIfEmpty STARTED
[2023-04-04T13:39:52.653Z] 
[2023-04-04T13:39:52.653Z] 
org.apache.kafka.streams.integration.StateDirectoryIntegrationTest > 
testCleanUpStateDirIfEmpty PASSED
[2023-04-04T13:39:53.707Z] 
[2023-04-04T13:39:53.707Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQuerySpecificActivePartitionStores STARTED
[2023-04-04T13:39:58.271Z] 
[2023-04-04T13:39:58.271Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQuerySpecificActivePartitionStores PASSED
[2023-04-04T13:39:58.271Z] 
[2023-04-04T13:39:58.271Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQueryAllStalePartitionStores STARTED
[2023-04-04T13:40:03.031Z] 
[2023-04-04T13:40:03.031Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQueryAllStalePartitionStores PASSED
[2023-04-04T13:40:03.031Z] 
[2023-04-04T13:40:03.031Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreads STARTED
[2023-04-04T13:40:07.496Z] 
[2023-04-04T13:40:07.496Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreads PASSED
[2023-04-04T13:40:07.496Z] 
[2023-04-04T13:40:07.496Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStores STARTED
[2023-04-04T13:40:13.012Z] 
[2023-04-04T13:40:13.012Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStores PASSED
[2023-04-04T13:40:13.012Z] 
[2023-04-04T13:40:13.012Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQueryOnlyActivePartitionStoresByDefault STARTED
[2023-04-04T13:40:17.431Z] 
[2023-04-04T13:40:17.431Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQueryOnlyActivePartitionStoresByDefault PASSED
[2023-04-04T13:40:17.431Z] 
[2023-04-04T13:40:17.431Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQueryStoresAfterAddingAndRemovingStreamThread STARTED
[2023-04-04T13:40:24.316Z] 
[2023-04-04T13:40:24.316Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQueryStoresAfterAddingAndRemovingStreamThread PASSED
[2023-04-04T13:40:24.316Z] 
[2023-04-04T13:40:24.316Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreadsNamedTopology STARTED
[2023-04-04T13:40:29.862Z] 
[2023-04-04T13:40:29.862Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreadsNamedTopology PASSED
[2023-04-04T13:40:29.862Z] 
[2023-04-04T13:40:29.862Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testInnerRepartitioned[caching enabled = true] STARTED
[2023-04-04T13:40:35.979Z] 
[2023-04-04T13:40:35.979Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testInnerRepartitioned[caching enabled = true] PASSED
[2023-04-04T13:40:35.979Z] 
[2023-04-04T13:40:35.979Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testOuterRepartitioned[caching enabled = true] STARTED
[2023-04-04T13:40:43.865Z] 
[2023-04-04T13:40:43.865Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testOuterRepartitioned[caching enabled = true] PASSED
[2023-04-04T13:40:43.865Z] 
[2023-04-04T13:40:43.865Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testInner[caching enabled = true] STARTED
[2023-04-04T13:40:47.432Z] 
[2023-04-04T13:40:47.432Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testInner[caching enabled = true] PASSED
[2023-04-04T13:40:47.432Z] 
[2023-04-04T13:40:47.432Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testOuter[caching enabled = true] STARTED
[2023-04-04T13:40:51.857Z] 

[jira] [Created] (KAFKA-14878) Reduce assignment data size to improve Kafka Consumer scalability

[Armand (Jira)]

Armand created KAFKA-14878:
--

 Summary: Reduce assignment data size to improve Kafka Consumer 
scalability
 Key: KAFKA-14878
 URL: https://issues.apache.org/jira/browse/KAFKA-14878
 Project: Kafka
  Issue Type: Improvement
  Components: consumer
Reporter: Armand


This issue is linked to https://issues.apache.org/jira/browse/KAFKA-7149 
which describes scalability problems with Kafka streams and was resolved by 
this [PR|https://github.com/apache/kafka/pull/7185].

We have the same problem with Kafka Consumer. If we scale our solution with 50 
instances (each instance has 5 consumers, so a consumer group can contain 50 * 
5 = 250 consumers) with 5000 partitions, the leader of the group will send 
SyncGroup requests exceeding 100 MB to the group coordinator.

The only solution we found is to increase the max.message.bytes of the 
__consumer_offsets topic. Is there any plan to compress requests from the 
consumer in order to avoid sending too big requests over the network?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1733

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-14720) Tools migration guidelines

[Federico Valeri (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Federico Valeri resolved KAFKA-14720.
-
Resolution: Fixed

> Tools migration guidelines
> --
>
> Key: KAFKA-14720
> URL: https://issues.apache.org/jira/browse/KAFKA-14720
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Federico Valeri
>Assignee: Federico Valeri
>Priority: Major
>
> The tools migration effort is ongoing and being tracked in KAFKA-14525. This 
> is part of a bigger initiative to split the core module into multiple modules 
> (e.g. storage, network, security, tools), which is being tracked in 
> KAFKA-14524.
> The plan is to migrate tools and related classes in a fully compatible way 
> from kafka.tools and kafka.admin packages (core module) to 
> org.apache.kafka.tools package (tools module).
> While kicking off this activity, we identified a number of potential 
> compatibility issues:
> * Missing wrapper: some tools do not have a wrapper script. There are system 
> tests that directly refer to the tool's fully qualified class name (FQCN) and 
> expect the old package name when running on old Kafka releases. They are 
> often used for troubleshooting or automation through the kafka-run-class.sh 
> script which takes the FQCN as input parameter.
> * SPI argument: some tools have arguments for setting a custom SPI 
> implementation to be used in place of the default implementation. Any custom 
> SPI implementation depends on the old package name.
> * Broken tool: some tools do not work on supported releases.
> * Core dependency: some tools require access to non-trivial core classes that 
> should be migrated first.
> See KIP-906 for more information.
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-906%3A+Tools+migration+guidelines



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-906 Tools migration guidelines

[Federico Valeri]

Hi, the KIP is accepted with 3 binding votes (Luke, John and Tom) and
1 non binding vote (Alexandre).

Thanks everyone!

Cheers
Fede

On Tue, Apr 4, 2023 at 1:58 PM Tom Bentley  wrote:
>
> Hi Federico,
>
> Thanks for the KIP, +1 (binding).
>
> Tom
>
> On Tue, 4 Apr 2023 at 08:30, Federico Valeri  wrote:
>
> > Hi, we have 2 binding an 1 non binding votes. Thanks for tacking the
> > time to review the KIP. Can I get one more binding vote?
> >
> > Cheers
> > Fede
> >
> > On Fri, Mar 24, 2023 at 3:05 PM John Roesler  wrote:
> > >
> > > Thanks Federico,
> > >
> > > I'm +1 (binding)
> > >
> > > -John
> > >
> > > On Fri, Mar 24, 2023, at 01:11, Federico Valeri wrote:
> > > > Bumping this thread for more votes.
> > > >
> > > > Thanks.
> > > >
> > > > On Wed, Mar 15, 2023, 9:57 AM Alexandre Dupriez <
> > alexandre.dupr...@gmail.com>
> > > > wrote:
> > > >
> > > >> Hi, Frederico,
> > > >>
> > > >> Thanks for the KIP.
> > > >>
> > > >> Non-binding +1.
> > > >>
> > > >> Thanks,
> > > >> Alexandre
> > > >>
> > > >> Le mer. 15 mars 2023 à 08:28, Luke Chen  a écrit :
> > > >> >
> > > >> > +1 from me.
> > > >> >
> > > >> > Luke
> > > >> >
> > > >> > On Wed, Mar 15, 2023 at 4:11 PM Federico Valeri <
> > fedeval...@gmail.com>
> > > >> > wrote:
> > > >> >
> > > >> > > Hi everyone,
> > > >> > >
> > > >> > > I'd like to start the vote on KIP-906 Tools migration guidelines.
> > > >> > >
> > > >> > >
> > > >> > >
> > > >>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-906%3A+Tools+migration+guidelines
> > > >> > >
> > > >> > > Discussion thread:
> > > >> > > https://lists.apache.org/thread/o2ytmjj2tyc2xcy6xh5tco31yyjzvz8p
> > > >> > >
> > > >> > > Thanks
> > > >> > > Fede
> > > >> > >
> > > >>
> >
> >

Re: kafka rebalancing problem

[ziming deng]

Thank you for reporting this, if you can stably reproduce it, you can create a 
JIRA ticket to track it. You are also welcomed if you can submit a PR to fix it.

- -
Best,
Ziming.

> On Apr 4, 2023, at 18:38, 大肉丸 <382650...@qq.com.INVALID> wrote:
> 
> Excuse me, I would like to ask you a question:
> 
> I have two topic，every topic only have 1 participation, theirs name are 
> topic-A and topic B, and my broker set auto.create.topics.enable=false (its 
> important!)
> 
> 
> 
> 
> And now I start consumer-A listen topic-A, and start consumer-B listen 
> topic-B, and this perform normal. （consumer-A and consumer-B belong to the 
> same consumer group，lets say the name is group_test）
> 
> 
> When i start a consumer-C to listen a Topic that doesn't exist, and 
> consumer-C is also belong to the "group_test" consumer group,  it will always 
> be rebalancing！！！
> 
> 
> I don't know the reason, I thought one of them would not consume, but the 
> result is that it has been rebalancing, could you please tell me the reason？

Re: [VOTE] KIP-906 Tools migration guidelines

[Tom Bentley]

Hi Federico,

Thanks for the KIP, +1 (binding).

Tom

On Tue, 4 Apr 2023 at 08:30, Federico Valeri  wrote:

> Hi, we have 2 binding an 1 non binding votes. Thanks for tacking the
> time to review the KIP. Can I get one more binding vote?
>
> Cheers
> Fede
>
> On Fri, Mar 24, 2023 at 3:05 PM John Roesler  wrote:
> >
> > Thanks Federico,
> >
> > I'm +1 (binding)
> >
> > -John
> >
> > On Fri, Mar 24, 2023, at 01:11, Federico Valeri wrote:
> > > Bumping this thread for more votes.
> > >
> > > Thanks.
> > >
> > > On Wed, Mar 15, 2023, 9:57 AM Alexandre Dupriez <
> alexandre.dupr...@gmail.com>
> > > wrote:
> > >
> > >> Hi, Frederico,
> > >>
> > >> Thanks for the KIP.
> > >>
> > >> Non-binding +1.
> > >>
> > >> Thanks,
> > >> Alexandre
> > >>
> > >> Le mer. 15 mars 2023 à 08:28, Luke Chen  a écrit :
> > >> >
> > >> > +1 from me.
> > >> >
> > >> > Luke
> > >> >
> > >> > On Wed, Mar 15, 2023 at 4:11 PM Federico Valeri <
> fedeval...@gmail.com>
> > >> > wrote:
> > >> >
> > >> > > Hi everyone,
> > >> > >
> > >> > > I'd like to start the vote on KIP-906 Tools migration guidelines.
> > >> > >
> > >> > >
> > >> > >
> > >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-906%3A+Tools+migration+guidelines
> > >> > >
> > >> > > Discussion thread:
> > >> > > https://lists.apache.org/thread/o2ytmjj2tyc2xcy6xh5tco31yyjzvz8p
> > >> > >
> > >> > > Thanks
> > >> > > Fede
> > >> > >
> > >>
>
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.4 #103

[Apache Jenkins Server]

See 

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.3 #166

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 335680 lines...]
[2023-04-04T09:05:42.321Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2023-04-04T09:05:42.321Z] > Task :connect:api:publishToMavenLocal
[2023-04-04T09:05:44.101Z] 
[2023-04-04T09:05:44.101Z] > Task :streams:javadoc
[2023-04-04T09:05:44.101Z] 
/home/jenkins/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:890:
 warning - Tag @link: reference not found: DefaultPartitioner
[2023-04-04T09:05:44.101Z] 
/home/jenkins/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:919:
 warning - Tag @link: reference not found: DefaultPartitioner
[2023-04-04T09:05:44.101Z] 
/home/jenkins/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:939:
 warning - Tag @link: reference not found: DefaultPartitioner
[2023-04-04T09:05:44.101Z] 
/home/jenkins/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:854:
 warning - Tag @link: reference not found: DefaultPartitioner
[2023-04-04T09:05:44.101Z] 
/home/jenkins/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:890:
 warning - Tag @link: reference not found: DefaultPartitioner
[2023-04-04T09:05:44.101Z] 
/home/jenkins/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:919:
 warning - Tag @link: reference not found: DefaultPartitioner
[2023-04-04T09:05:44.101Z] 
/home/jenkins/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:939:
 warning - Tag @link: reference not found: DefaultPartitioner
[2023-04-04T09:05:44.101Z] 
/home/jenkins/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:84:
 warning - Tag @link: reference not found: DefaultPartitioner
[2023-04-04T09:05:44.101Z] 
/home/jenkins/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:136:
 warning - Tag @link: reference not found: DefaultPartitioner
[2023-04-04T09:05:44.101Z] 
/home/jenkins/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:147:
 warning - Tag @link: reference not found: DefaultPartitioner
[2023-04-04T09:05:44.101Z] 
/home/jenkins/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/kstream/Repartitioned.java:101:
 warning - Tag @link: reference not found: DefaultPartitioner
[2023-04-04T09:05:44.101Z] 
/home/jenkins/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/kstream/Repartitioned.java:167:
 warning - Tag @link: reference not found: DefaultPartitioner
[2023-04-04T09:05:44.101Z] 
/home/jenkins/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/TopologyConfig.java:58:
 warning - Tag @link: missing '#': "org.apache.kafka.streams.StreamsBuilder()"
[2023-04-04T09:05:44.101Z] 
/home/jenkins/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/TopologyConfig.java:58:
 warning - Tag @link: can't find org.apache.kafka.streams.StreamsBuilder() in 
org.apache.kafka.streams.TopologyConfig
[2023-04-04T09:05:44.101Z] 
/home/jenkins/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/TopologyDescription.java:38:
 warning - Tag @link: reference not found: ProcessorContext#forward(Object, 
Object) forwards
[2023-04-04T09:05:44.101Z] 
/home/jenkins/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/query/Position.java:44:
 warning - Tag @link: can't find query(Query,
[2023-04-04T09:05:44.101Z]  PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2023-04-04T09:05:44.101Z] 
/home/jenkins/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:44:
 warning - Tag @link: can't find query(Query, PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2023-04-04T09:05:44.101Z] 
/home/jenkins/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:36:
 warning - Tag @link: can't find query(Query, PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2023-04-04T09:05:44.101Z] 
/home/jenkins/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:57:
 warning - Tag @link: can't find query(Query, PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2023-04-04T09:05:44.101Z] 
/home/jenkins/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:74:
 warning - Tag @link: can't find query(Query, PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2023-04-04T09:05:44.101Z] 

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1732

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-906 Tools migration guidelines

[Federico Valeri]

Hi, we have 2 binding an 1 non binding votes. Thanks for tacking the
time to review the KIP. Can I get one more binding vote?

Cheers
Fede

On Fri, Mar 24, 2023 at 3:05 PM John Roesler  wrote:
>
> Thanks Federico,
>
> I'm +1 (binding)
>
> -John
>
> On Fri, Mar 24, 2023, at 01:11, Federico Valeri wrote:
> > Bumping this thread for more votes.
> >
> > Thanks.
> >
> > On Wed, Mar 15, 2023, 9:57 AM Alexandre Dupriez 
> > 
> > wrote:
> >
> >> Hi, Frederico,
> >>
> >> Thanks for the KIP.
> >>
> >> Non-binding +1.
> >>
> >> Thanks,
> >> Alexandre
> >>
> >> Le mer. 15 mars 2023 à 08:28, Luke Chen  a écrit :
> >> >
> >> > +1 from me.
> >> >
> >> > Luke
> >> >
> >> > On Wed, Mar 15, 2023 at 4:11 PM Federico Valeri 
> >> > wrote:
> >> >
> >> > > Hi everyone,
> >> > >
> >> > > I'd like to start the vote on KIP-906 Tools migration guidelines.
> >> > >
> >> > >
> >> > >
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-906%3A+Tools+migration+guidelines
> >> > >
> >> > > Discussion thread:
> >> > > https://lists.apache.org/thread/o2ytmjj2tyc2xcy6xh5tco31yyjzvz8p
> >> > >
> >> > > Thanks
> >> > > Fede
> >> > >
> >>

[jira] [Resolved] (KAFKA-14586) Move StreamsResetter to tools

[Sagar Rao (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sagar Rao resolved KAFKA-14586.
---
Resolution: Fixed

> Move StreamsResetter to tools
> -
>
> Key: KAFKA-14586
> URL: https://issues.apache.org/jira/browse/KAFKA-14586
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Mickael Maison
>Assignee: Sagar Rao
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)