[jira] [Created] (KAFKA-15770) org.apache.kafka.streams.integration.ConsistencyVectorIntegrationTest.shouldHaveSamePositionBoundActiveAndStandBy is flaky

[Alok Thatikunta (Jira)]

Alok Thatikunta created KAFKA-15770:
---

 Summary: 
org.apache.kafka.streams.integration.ConsistencyVectorIntegrationTest.shouldHaveSamePositionBoundActiveAndStandBy
 is flaky 
 Key: KAFKA-15770
 URL: https://issues.apache.org/jira/browse/KAFKA-15770
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: Alok Thatikunta


Test fails on CI, passes locally

[https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14607/9/testReport/junit/org.apache.kafka.streams.integration/ConsistencyVectorIntegrationTest/Build___JDK_17_and_Scala_2_13___shouldHaveSamePositionBoundActiveAndStandBy/]
{code:java}
java.lang.AssertionError: Result:SucceededQueryResult{result=<0,1698511250443>, 
executionInfo=[], position=Position{position={input-topic={0=50
Expected: is 
 but: was  {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15769) Fix wrong log with exception

[Zhongqiang Gong (Jira)]

Zhongqiang Gong created KAFKA-15769:
---

 Summary: Fix wrong log with exception
 Key: KAFKA-15769
 URL: https://issues.apache.org/jira/browse/KAFKA-15769
 Project: Kafka
  Issue Type: Improvement
  Components: logging
Affects Versions: 3.6.0
Reporter: Zhongqiang Gong
 Fix For: 3.6.0
 Attachments: image-2023-11-01-13-08-43-833.png

Like under case , log with exception is wrong.

!image-2023-11-01-13-08-43-833.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2348

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 323152 lines...]
Gradle Test Run :streams:test > Gradle Test Executor 90 > 
DefaultTaskExecutorTest > shouldSetTaskTimeoutOnTimeoutException() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 90 > 
DefaultTaskExecutorTest > shouldPunctuateSystemTime() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 90 > 
DefaultTaskExecutorTest > shouldPunctuateSystemTime() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 90 > 
DefaultTaskExecutorTest > shouldUnassignTaskWhenNotProgressing() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 90 > 
DefaultTaskExecutorTest > shouldUnassignTaskWhenNotProgressing() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 90 > 
DefaultTaskExecutorTest > shouldNotFlushOnException() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 90 > 
DefaultTaskExecutorTest > shouldNotFlushOnException() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 90 > 
DefaultTaskExecutorTest > 
shouldRespectProcessingDisabledByTaskExecutionMetadata() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 90 > 
DefaultTaskExecutorTest > 
shouldRespectProcessingDisabledByTaskExecutionMetadata() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 90 > 
DefaultTaskManagerTest > shouldLockAnEmptySetOfTasks() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 90 > 
DefaultTaskManagerTest > shouldLockAnEmptySetOfTasks() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 90 > 
DefaultTaskManagerTest > shouldAssignTasksThatCanBeSystemTimePunctuated() 
STARTED

Gradle Test Run :streams:test > Gradle Test Executor 90 > 
DefaultTaskManagerTest > shouldAssignTasksThatCanBeSystemTimePunctuated() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 90 > 
DefaultTaskManagerTest > shouldNotUnassignNotOwnedTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 90 > 
DefaultTaskManagerTest > shouldNotUnassignNotOwnedTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 90 > 
DefaultTaskManagerTest > shouldNotSetUncaughtExceptionsTwice() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 90 > 
DefaultTaskManagerTest > shouldNotSetUncaughtExceptionsTwice() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 90 > 
DefaultTaskManagerTest > shouldReturnFromAwaitOnAdding() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 90 > 
DefaultTaskManagerTest > shouldReturnFromAwaitOnAdding() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 90 > 
DefaultTaskManagerTest > shouldShutdownTaskExecutors() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 90 > 
DefaultTaskManagerTest > shouldShutdownTaskExecutors() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 90 > 
DefaultTaskManagerTest > 
shouldNotAssignTasksForPunctuationIfPunctuationDisabled() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 90 > 
DefaultTaskManagerTest > 
shouldNotAssignTasksForPunctuationIfPunctuationDisabled() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 90 > 
DefaultTaskManagerTest > shouldReturnFromAwaitOnSignalProcessableTasks() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 90 > 
DefaultTaskManagerTest > shouldReturnFromAwaitOnSignalProcessableTasks() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 90 > 
DefaultTaskManagerTest > shouldAddTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 90 > 
DefaultTaskManagerTest > shouldAddTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 90 > 
DefaultTaskManagerTest > shouldReturnFromAwaitOnUnassignment() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 90 > 
DefaultTaskManagerTest > shouldReturnFromAwaitOnUnassignment() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 90 > 
DefaultTaskManagerTest > shouldNotAssignAnyLockedTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 90 > 
DefaultTaskManagerTest > shouldNotAssignAnyLockedTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 90 > 
DefaultTaskManagerTest > shouldRemoveTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 90 > 
DefaultTaskManagerTest > shouldRemoveTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 90 > 
DefaultTaskManagerTest > shouldNotRemoveAssignedTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 90 > 
DefaultTaskManagerTest > shouldNotRemoveAssignedTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 90 > 
DefaultTaskManagerTest > shouldBlockOnAwait() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 90 > 
DefaultTaskManagerTest > shouldBlockOnAwait() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 90 > 
DefaultTaskManagerTest > 

Re: ACCESS to Apache Pony Mail

[Arpit Goyal]

I am already a committer of Apache Kafka.
On Wed, Nov 1, 2023, 05:18 Matthias J. Sax  wrote:

> Only committers can login using their ASF account.
>
> -Matthias
>
> On 10/30/23 10:19 PM, Arpit Goyal wrote:
> > Hi
> > Can anyone help me provide access to Apache Pony Mail. I tried login
> using
> > the jira credential but it didn't work.
> > Thanks and Regards
> > Arpit Goyal
> > 8861094754
> >
>

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2347

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 321056 lines...]

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldShutdownTaskExecutor() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldShutdownTaskExecutor() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldAwaitProcessableTasksIfNoneAssignable() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldAwaitProcessableTasksIfNoneAssignable() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > 
shouldRespectPunctuationDisabledByTaskExecutionMetadata() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > 
shouldRespectPunctuationDisabledByTaskExecutionMetadata() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldSetTaskTimeoutOnTimeoutException() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldSetTaskTimeoutOnTimeoutException() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldPunctuateSystemTime() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldPunctuateSystemTime() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldUnassignTaskWhenNotProgressing() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldUnassignTaskWhenNotProgressing() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldNotFlushOnException() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldNotFlushOnException() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > 
shouldRespectProcessingDisabledByTaskExecutionMetadata() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > 
shouldRespectProcessingDisabledByTaskExecutionMetadata() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskManagerTest > shouldLockAnEmptySetOfTasks() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskManagerTest > shouldLockAnEmptySetOfTasks() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskManagerTest > shouldAssignTasksThatCanBeSystemTimePunctuated() 
STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskManagerTest > shouldAssignTasksThatCanBeSystemTimePunctuated() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskManagerTest > shouldNotUnassignNotOwnedTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskManagerTest > shouldNotUnassignNotOwnedTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskManagerTest > shouldNotSetUncaughtExceptionsTwice() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskManagerTest > shouldNotSetUncaughtExceptionsTwice() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskManagerTest > shouldReturnFromAwaitOnAdding() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskManagerTest > shouldReturnFromAwaitOnAdding() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskManagerTest > shouldShutdownTaskExecutors() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskManagerTest > shouldShutdownTaskExecutors() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskManagerTest > 
shouldNotAssignTasksForPunctuationIfPunctuationDisabled() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskManagerTest > 
shouldNotAssignTasksForPunctuationIfPunctuationDisabled() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskManagerTest > shouldReturnFromAwaitOnSignalProcessableTasks() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskManagerTest > shouldReturnFromAwaitOnSignalProcessableTasks() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskManagerTest > shouldAddTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskManagerTest > shouldAddTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskManagerTest > shouldReturnFromAwaitOnUnassignment() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskManagerTest > shouldReturnFromAwaitOnUnassignment() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 

Re: ACCESS to Apache Pony Mail

[Matthias J. Sax]

Only committers can login using their ASF account.

-Matthias

On 10/30/23 10:19 PM, Arpit Goyal wrote:

Hi
Can anyone help me provide access to Apache Pony Mail. I tried login using
the jira credential but it didn't work.
Thanks and Regards
Arpit Goyal
8861094754

[jira] [Created] (KAFKA-15768) StateQueryResult#getOnlyPartitionResult should not throw for FailedQueryResult

[Matthias J. Sax (Jira)]

Matthias J. Sax created KAFKA-15768:
---

 Summary: StateQueryResult#getOnlyPartitionResult should not throw 
for FailedQueryResult
 Key: KAFKA-15768
 URL: https://issues.apache.org/jira/browse/KAFKA-15768
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: Matthias J. Sax


Calling `StateQueryResult#getOnlyPartitionResult` crashes with an incorrect 
`IllegalArgumentException` if the only result is a `FailedQueryResult`.

The issue is the internal `filter(r -> r.getResult() != 0)` step, that blindly 
(and incorrectly) calls `getResult`.

Given the semantics of `getOnlyPartitionResult` we should not care if the 
result is SuccessQueryResult or FailedQueryResult, but only check if there is a 
single result or not. (The user has not means to avoid getting an exception 
otherwise.)

Side-note: why does `FailedQueryResult#getResult` throw an 
IllegalArgumentException (there is no argument passed into the method – it 
should rather be an `IllegalStateException` – but I guess we would need a KIP 
for this fix?)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2346

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 307290 lines...]

Gradle Test Run :core:test > Gradle Test Executor 90 > 
BootstrapControllersIntegrationTest > 
testPutControllersInBootstrapBrokersConfig() SKIPPED

Gradle Test Run :core:test > Gradle Test Executor 90 > 
BootstrapControllersIntegrationTest > 
testUsingBootstrapControllersOnUnsupportedAdminApi() STARTED

Gradle Test Run :core:test > Gradle Test Executor 90 > 
BootstrapControllersIntegrationTest > 
testUsingBootstrapControllersOnUnsupportedAdminApi() PASSED

Gradle Test Run :core:test > Gradle Test Executor 90 > 
BootstrapControllersIntegrationTest > testDescribeCluster(boolean) > [1] false 
STARTED

Gradle Test Run :core:test > Gradle Test Executor 90 > 
BootstrapControllersIntegrationTest > testDescribeCluster(boolean) > [1] false 
PASSED

Gradle Test Run :core:test > Gradle Test Executor 90 > 
BootstrapControllersIntegrationTest > testDescribeCluster(boolean) > [2] true 
STARTED

Gradle Test Run :core:test > Gradle Test Executor 90 > 
BootstrapControllersIntegrationTest > testDescribeCluster(boolean) > [2] true 
PASSED

Gradle Test Run :core:test > Gradle Test Executor 90 > 
BootstrapControllersIntegrationTest > testDescribeFeatures(boolean) > [1] false 
STARTED

Gradle Test Run :core:test > Gradle Test Executor 90 > 
BootstrapControllersIntegrationTest > testDescribeFeatures(boolean) > [1] false 
PASSED

Gradle Test Run :core:test > Gradle Test Executor 90 > 
BootstrapControllersIntegrationTest > testDescribeFeatures(boolean) > [2] true 
STARTED

Gradle Test Run :core:test > Gradle Test Executor 90 > 
BootstrapControllersIntegrationTest > testDescribeFeatures(boolean) > [2] true 
PASSED

> Task :streams:test

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
KTableKTableOuterJoinTest > testSendingOldValue PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
KTableKTableOuterJoinTest > testJoin STARTED
> Task :core:compileTestScala
Cannot contact builds31: java.lang.InterruptedException
> Task :core:testClasses
> Task :streams:compileTestJava
> Task :streams:testClasses
> Task :streams:testJar
> Task :streams:testSrcJar
> Task :streams:publishMavenJavaPublicationToMavenLocal
> Task :streams:publishToMavenLocal

Deprecated Gradle features were used in this build, making it incompatible with 
Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings 
and determine if they come from your own scripts or plugins.

For more on this, please refer to 
https://docs.gradle.org/8.3/userguide/command_line_interface.html#sec:command_line_warnings
 in the Gradle documentation.

BUILD SUCCESSFUL in 8m 40s
94 actionable tasks: 41 executed, 53 up-to-date

Publishing build scan...
https://ge.apache.org/s/eocqfg3efaup2

[Pipeline] sh
+ grep ^version= gradle.properties
+ cut -d= -f 2
[Pipeline] dir
Running in /home/jenkins/workspace/Kafka_kafka_trunk/streams/quickstart
[Pipeline] {
[Pipeline] sh
+ mvn clean install -Dgpg.skip
[INFO] Scanning for projects...
[INFO] 
[INFO] Reactor Build Order:
[INFO] 
[INFO] Kafka Streams :: Quickstart[pom]
[INFO] streams-quickstart-java[maven-archetype]
[INFO] 
[INFO] < org.apache.kafka:streams-quickstart >-
[INFO] Building Kafka Streams :: Quickstart 3.7.0-SNAPSHOT[1/2]
[INFO]   from pom.xml
[INFO] [ pom ]-
[INFO] 
[INFO] --- clean:3.0.0:clean (default-clean) @ streams-quickstart ---
[INFO] 
[INFO] --- remote-resources:1.5:process (process-resource-bundles) @ 
streams-quickstart ---
[INFO] 
[INFO] --- site:3.5.1:attach-descriptor (attach-descriptor) @ 
streams-quickstart ---
[INFO] 
[INFO] --- gpg:1.6:sign (sign-artifacts) @ streams-quickstart ---
[INFO] 
[INFO] --- install:2.5.2:install (default-install) @ streams-quickstart ---
[INFO] Installing 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/quickstart/pom.xml to 
/home/jenkins/.m2/repository/org/apache/kafka/streams-quickstart/3.7.0-SNAPSHOT/streams-quickstart-3.7.0-SNAPSHOT.pom
[INFO] 
[INFO] --< org.apache.kafka:streams-quickstart-java >--
[INFO] Building streams-quickstart-java 3.7.0-SNAPSHOT[2/2]
[INFO]   from java/pom.xml
[INFO] --[ maven-archetype ]---
[INFO] 
[INFO] --- clean:3.0.0:clean (default-clean) @ streams-quickstart-java ---
[INFO] 
[INFO] --- remote-resources:1.5:process (process-resource-bundles) @ 
streams-quickstart-java ---
[INFO] 
[INFO] --- resources:2.7:resources (default-resources) @ 
streams-quickstart-java ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 6 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- 

[jira] [Created] (KAFKA-15767) Redesign TransactionManager to avoid use of ThreadLocal

[Kirk True (Jira)]

Kirk True created KAFKA-15767:
-

 Summary: Redesign TransactionManager to avoid use of ThreadLocal
 Key: KAFKA-15767
 URL: https://issues.apache.org/jira/browse/KAFKA-15767
 Project: Kafka
  Issue Type: Improvement
  Components: clients, producer 
Affects Versions: 3.6.0
Reporter: Kirk True
Assignee: Kirk True
 Fix For: 3.7.0


A {{TransactionManager}} instance is created by the {{KafkaProducer}} and 
shared with the {{Sender}} thread. The {{TransactionManager}} has internal 
states through which it transitions as part of its initialization, transaction 
management, shutdown, etc. It contains logic to ensure that those state 
transitions are valid, such that when an invalid transition is attempted, it is 
handled appropriately. 

The issue is, the definition of "handled appropriately" depends on which thread 
is making the API call that is attempting an invalid transition. The 
application thread expects that the invalid transition will generate an 
exception. However, the sender thread expects that the invalid transition will 
"poison" the entire {{TransactionManager}} instance.

So as part of the implementation of KAFKA-14831, we needed a way to change 
logic in the {{TransactionManager}} on a per-thread basis, so a {{ThreadLocal}} 
instance variable was added to the {{TransactionManager}} to keep track of 
this. However, the use of ThreadLocal instance variables is generally 
discouraged because of their tendency for memory leaks, shared state across 
multiple threads, and not working with virtual threads.

The initial implementation attempt of KAFKA-14831 used a context object, passed 
in to each method, to affect the logic. However, the number of methods that 
needed to be changed to accept this new context object grew until most of the 
methods in {{TransactionManager}} needed to be updated. Thus all the affected 
call sites needed to be updated, resulting in a much larger change than 
anticipated.

The task here is to remove the use of the {{ThreadLocal}} instance variable, 
let the application thread and {{Sender}} thread keep their desired behavior, 
but keep the change to a minimum.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15766) Possible request handler thread exhaustion in controller election

[Cameron (Jira)]

Cameron created KAFKA-15766:
---

 Summary: Possible request handler thread exhaustion in controller 
election
 Key: KAFKA-15766
 URL: https://issues.apache.org/jira/browse/KAFKA-15766
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.5.0
Reporter: Cameron


Hello,

This is my first dive into the Kafka source code, so I may be mis-interpreting 
some of the logic outlined below. Please let me know if my understanding is not 
correct.

 

--- 

 

After upgrading a relatively large Kafka cluster from IPB 2.5 to 3.5 (Kafka 
version v3.5.0), we've had numerous problems with controller failover and 
subsequent elections not being properly acknowledged by a subset of the 
cluster's brokers.  These affected brokers will temporarily become unresponsive 
due to complete request handler thread starvation via failed 
ALLOCATE_PRODUCER_IDS and ALTER_PARTITION requests to the old stopped 
controller. In the case of a disconnected response, as seen here, the 
broker-to-controller channel will place these failed requests at the beginning 
of the queue, effectively deadlocking the request handler thread pool. 
Depending on how quickly the configured queue.max.requests request queue fills, 
the new controller's UPDATE_METADATA request to notify the broker of the 
controller election result may not even be able to contact the affected broker 
to update the new controller Id. 

For context, the relevant configuration of my cluster is:
 * num.network.threads = 16
 * num.io.threads = 8
 * queued.max.requests = 500

 

I am not able to consistently repro the issue, but it happens frequently enough 
to cause major pain in our set up. The general order of events are:
 # Controller election takes place (due to intentional failover or otherwise)
 # New controller successfully registers its’ znode with Zookeeper, initiates 
UPDATE_METADATA request to notify other brokers in the cluster
 # Problematic broker, noticing it has lost connection to the old controller 
but has not yet received and/or processed the incoming UPDATE_METADATA request, 
spams (every ~50ms as hardcoded) ALLOCATE_PRODUCER_IDS and ALTER_PARTITION 
requests to the old controller, receiving disconnect responses and placing the 
request back in the front of the request queue.

 

In this scenario, the needed UPDATE_METADATA request will take much longer to 
process. In our specific cluster this will sometimes auto-resolve in 3-5m 
whereas other times it will require manual intervention to bounce the broker 
and allow it to initialize a connection with the new controller. 

 

This issue seems very similar to some previous tickets I found:
 * https://issues.apache.org/jira/browse/KAFKA-7697 
 * https://issues.apache.org/jira/browse/KAFKA-14694 

Unfortunately I have not been able to grab a threaddump because of the sporadic 
nature of the issue, but will update the ticket if I am able.

This may also affect other versions of Kafka, but we've only experienced it 
with v3.5.0 + IBP 3.5. Curiously enough we haven't ran into this issue with 
v3.5.0 + IBP 2.5, assumedly because the ALLOCATE_PRODUCER_IDS and 
ALTER_PARTITION features are not yet enabled until IBP 2.7 and 3.0, 
respectively.

 

Is there any guidance on whether this failure path is possible as described and 
what steps can be taken to mitigate the issue?

 

Thank you!

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15765) Remove task level metric "commit-latency"

[Matthias J. Sax (Jira)]

Matthias J. Sax created KAFKA-15765:
---

 Summary: Remove task level metric "commit-latency"
 Key: KAFKA-15765
 URL: https://issues.apache.org/jira/browse/KAFKA-15765
 Project: Kafka
  Issue Type: Task
  Components: streams
Reporter: Matthias J. Sax
 Fix For: 4.0.0


We stopped tracking this metric with KIP-447, but kept it for backward 
compatibility reasons. It's time to remove it completely with 4.0 release.

Cf 
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-447%3A+Producer+scalability+for+exactly+once+semantics]

And [https://github.com/apache/kafka/pull/8218/files#r390712211]

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2345

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 321601 lines...]

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
RocksDBMetricsRecorderTest > 
shouldNotRecordStatisticsBasedMetricsIfStatisticsIsNull() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
RocksDBMetricsRecorderTest > 
shouldSetStatsLevelToExceptDetailedTimersWhenValueProvidersWithStatisticsAreAdded()
 STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
RocksDBMetricsRecorderTest > 
shouldSetStatsLevelToExceptDetailedTimersWhenValueProvidersWithStatisticsAreAdded()
 PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
RocksDBMetricsRecorderTest > shouldRecordStatisticsBasedMetrics() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
RocksDBMetricsRecorderTest > shouldRecordStatisticsBasedMetrics() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
RocksDBMetricsRecorderTest > 
shouldThrowIfMetricRecorderIsReInitialisedWithDifferentStreamsMetrics() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
RocksDBMetricsRecorderTest > 
shouldThrowIfMetricRecorderIsReInitialisedWithDifferentStreamsMetrics() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
RocksDBMetricsRecorderTest > shouldInitMetricsRecorder() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
RocksDBMetricsRecorderTest > shouldInitMetricsRecorder() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
RocksDBMetricsRecorderTest > 
shouldThrowIfMetricRecorderIsReInitialisedWithDifferentTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
RocksDBMetricsRecorderTest > 
shouldThrowIfMetricRecorderIsReInitialisedWithDifferentTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
RocksDBMetricsRecorderTest > 
shouldCorrectlyHandleAvgRecordingsWithZeroSumAndCount() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
RocksDBMetricsRecorderTest > 
shouldCorrectlyHandleAvgRecordingsWithZeroSumAndCount() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
RocksDBMetricsRecorderTest > 
shouldThrowIfStatisticsToAddIsNullButExistingStatisticsAreNotNull() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
RocksDBMetricsRecorderTest > 
shouldThrowIfStatisticsToAddIsNullButExistingStatisticsAreNotNull() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
RocksDBMetricsRecorderTest > shouldNotAddItselfToRecordingTriggerWhenNotEmpty() 
STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
RocksDBMetricsRecorderTest > shouldNotAddItselfToRecordingTriggerWhenNotEmpty() 
PASSED

Gradle Test Run :streams:test > Gradle Test Executor 103 > 
KStreamRepartitionIntegrationTest > [Optimization = all] > 
shouldGoThroughRebalancingCorrectly[Optimization = all] STARTED

Gradle Test Run :streams:test > Gradle Test Executor 102 > 
SlidingWindowedKStreamIntegrationTest > [ON_WINDOW_UPDATE_cache:true] > 
shouldRestoreAfterJoinRestart[ON_WINDOW_UPDATE_cache:true] STARTED

> Task :core:test

Gradle Test Run :core:test > Gradle Test Executor 91 > 
ZkMigrationIntegrationTest > testMigrateTopicDeletions(ClusterInstance) > [1] 
Type=ZK, MetadataVersion=3.4-IV0, Security=PLAINTEXT SKIPPED

Gradle Test Run :core:test > Gradle Test Executor 91 > 
ZkMigrationIntegrationTest > testMigrateTopicDeletions(ClusterInstance) > [2] 
Type=ZK, MetadataVersion=3.5-IV2, Security=PLAINTEXT STARTED

> Task :streams:test

Gradle Test Run :streams:test > Gradle Test Executor 102 > 
SlidingWindowedKStreamIntegrationTest > [ON_WINDOW_UPDATE_cache:true] > 
shouldRestoreAfterJoinRestart[ON_WINDOW_UPDATE_cache:true] PASSED

Gradle Test Run :streams:test > Gradle Test Executor 102 > 
SlidingWindowedKStreamIntegrationTest > [ON_WINDOW_UPDATE_cache:false] > 
shouldRestoreAfterJoinRestart[ON_WINDOW_UPDATE_cache:false] STARTED

Gradle Test Run :streams:test > Gradle Test Executor 103 > 
KStreamRepartitionIntegrationTest > [Optimization = all] > 
shouldGoThroughRebalancingCorrectly[Optimization = all] PASSED

Gradle Test Run :streams:test > Gradle Test Executor 103 > 
KStreamRepartitionIntegrationTest > [Optimization = none] > 
shouldGoThroughRebalancingCorrectly[Optimization = none] STARTED

> Task :core:test

Gradle Test Run :core:test > Gradle Test Executor 91 > 
ZkMigrationIntegrationTest > testMigrateTopicDeletions(ClusterInstance) > [2] 
Type=ZK, MetadataVersion=3.5-IV2, Security=PLAINTEXT SKIPPED

Gradle Test Run :core:test > Gradle Test Executor 91 > 
ZkMigrationIntegrationTest > testMigrateTopicDeletions(ClusterInstance) > [3] 
Type=ZK, MetadataVersion=3.6-IV2, Security=PLAINTEXT STARTED

> Task :streams:test

Gradle Test Run :streams:test > Gradle Test Executor 102 > 
SlidingWindowedKStreamIntegrationTest > [ON_WINDOW_UPDATE_cache:false] > 

[jira] [Resolved] (KAFKA-15672) Add 3.6 to streams system tests

[Matthias J. Sax (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-15672.
-
Resolution: Duplicate

> Add 3.6 to streams system tests
> ---
>
> Key: KAFKA-15672
> URL: https://issues.apache.org/jira/browse/KAFKA-15672
> Project: Kafka
>  Issue Type: Test
>  Components: streams, system tests
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
>Priority: Critical
>
> 3.6.0 was released recently. We need to add `3.6.0` to the system tests (in 
> particular upgrade and broker compatibility tests)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15761) ConnectorRestartApiIntegrationTest.testMultiWorkerRestartOnlyConnector is flaky

[Chris Egerton (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Egerton resolved KAFKA-15761.
---
Resolution: Duplicate

> ConnectorRestartApiIntegrationTest.testMultiWorkerRestartOnlyConnector is 
> flaky
> ---
>
> Key: KAFKA-15761
> URL: https://issues.apache.org/jira/browse/KAFKA-15761
> Project: Kafka
>  Issue Type: Bug
>  Components: unit tests
>Reporter: Calvin Liu
>Priority: Major
>
> Build / JDK 21 and Scala 2.13 / testMultiWorkerRestartOnlyConnector – 
> org.apache.kafka.connect.integration.ConnectorRestartApiIntegrationTest
> {code:java}
> java.lang.AssertionError: Failed to stop connector and tasks within 12ms  
> at org.junit.Assert.fail(Assert.java:89)at 
> org.junit.Assert.assertTrue(Assert.java:42)  at 
> org.apache.kafka.connect.integration.ConnectorRestartApiIntegrationTest.runningConnectorAndTasksRestart(ConnectorRestartApiIntegrationTest.java:273)
>  at 
> org.apache.kafka.connect.integration.ConnectorRestartApiIntegrationTest.testMultiWorkerRestartOnlyConnector(ConnectorRestartApiIntegrationTest.java:231)
>  at 
> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
> at java.base/java.lang.reflect.Method.invoke(Method.java:580)   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
>  at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
>at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) 
>at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)  
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)  at 
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)  at 
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at 
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
>  at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
>at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
>  {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [PR] MINOR: Fix incorrect link in 3.6 announcement [kafka-site]

[via GitHub]

jolshan commented on PR #565:
URL: https://github.com/apache/kafka-site/pull/565#issuecomment-1787515153

   Thanks folks. Sorry I messed this one up.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [ANNOUNCE] New Kafka PMC Member: Satish Duggana

[Satish Duggana]

Thank you, everyone!

~Satish.

On Mon, 30 Oct 2023 at 16:14, Josep Prat  wrote:
>
> Congrats Satish!
>
> Best,
>
> ———
> Josep Prat
>
> Aiven Deutschland GmbH
>
> Alexanderufer 3-7, 10117 Berlin
>
> Amtsgericht Charlottenburg, HRB 209739 B
>
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>
> m: +491715557497
>
> w: aiven.io
>
> e: josep.p...@aiven.io
>
> On Mon, Oct 30, 2023, 11:37 Bruno Cadonna  wrote:
>
> > Congrats, Satish!
> >
> > Bruno
> >
> > On 10/29/23 2:42 PM, John Roesler wrote:
> > > Congratulations, Satish!
> > > -John
> > >
> > > On Sun, Oct 29, 2023, at 08:09, Randall Hauch wrote:
> > >> Congratulations, Satish!
> > >>
> > >> On Sun, Oct 29, 2023 at 1:47 AM Tom Bentley 
> > wrote:
> > >>
> > >>> Congratulations!
> > >>>
> > >>> On Sun, 29 Oct 2023 at 5:41 PM, Guozhang Wang <
> > guozhang.wang...@gmail.com>
> > >>> wrote:
> > >>>
> >  Congratulations Satish!
> > 
> >  On Sat, Oct 28, 2023 at 12:59 AM Luke Chen  wrote:
> > >
> > > Congrats Satish!
> > >
> > > Luke
> > >
> > > On Sat, Oct 28, 2023 at 11:16 AM ziming deng <
> > dengziming1...@gmail.com
> > 
> > > wrote:
> > >
> > >> Congratulations Satish!
> > >>
> > >>> On Oct 27, 2023, at 23:03, Jun Rao 
> > >>> wrote:
> > >>>
> > >>> Hi, Everyone,
> > >>>
> > >>> Satish Duggana has been a Kafka committer since 2022. He has been
> >  very
> > >>> instrumental to the community since becoming a committer. It's my
> > >> pleasure
> > >>> to announce that Satish is now a member of Kafka PMC.
> > >>>
> > >>> Congratulations Satish!
> > >>>
> > >>> Jun
> > >>> on behalf of Apache Kafka PMC
> > >>
> > >>
> > 
> > 
> > >>>
> >

Re: [VOTE] KIP-988 Streams StandbyUpdateListener

[Colt McNealy]

Thanks everyone. Now that everyone who commented on the discussion has
voted, I think we can close this vote as one touchdown with a missed field
goal to zero (5 binding, 1 non-binding).

Cheers,
Colt

On Tue, Oct 31, 2023 at 5:35 AM Bruno Cadonna  wrote:

> Hi Colt and Eduwer,
>
> Thanks for the KIP!
>
> +1 (binding)
>
> Best,
> Bruno
>
> On 10/26/23 7:17 PM, Matthias J. Sax wrote:
> > +1 (binding)
> >
> > On 10/25/23 4:06 PM, Sophie Blee-Goldman wrote:
> >> Happy to see this -- that's a +1 (binding) from me
> >>
> >> On Mon, Oct 23, 2023 at 6:33 AM Bill Bejeck  wrote:
> >>
> >>> This is a great addition
> >>>
> >>> +1(binding)
> >>>
> >>> -Bill
> >>>
> >>> On Fri, Oct 20, 2023 at 2:29 PM Almog Gavra 
> >>> wrote:
> >>>
>  +1 (non-binding) - great improvement, thanks Colt & Eduwer!
> 
>  On Tue, Oct 17, 2023 at 11:25 AM Guozhang Wang <
> >>> guozhang.wang...@gmail.com
> >
>  wrote:
> 
> > +1 from me.
> >
> > On Mon, Oct 16, 2023 at 1:56 AM Lucas Brutschy
> >  wrote:
> >>
> >> Hi,
> >>
> >> thanks again for the KIP!
> >>
> >> +1 (binding)
> >>
> >> Cheers,
> >> Lucas
> >>
> >>
> >>
> >> On Sun, Oct 15, 2023 at 9:13 AM Colt McNealy 
> > wrote:
> >>>
> >>> Hello there,
> >>>
> >>> I'd like to call a vote on KIP-988 (co-authored by my friend and
> > colleague
> >>> Eduwer Camacaro). We are hoping to get it in before the 3.7.0
>  release.
> >>>
> >>>
> >
> 
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-988%3A+Streams+Standby+Task+Update+Listener
> >>>
> >>> Cheers,
> >>> Colt McNealy
> >>>
> >>> *Founder, LittleHorse.dev*
> >
> 
> >>>
> >>
>

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2344

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-953: partition method to be overloaded to accept headers as well.

[Jack Tomy]

A last reminder :(

On Sun, Oct 15, 2023 at 1:25 PM Jack Tomy  wrote:

> Hey contributors,
>
> Please vote or veto the proposal.
>
> Please don't ghost. Humble request.
>
> Thanks
>
> On Thu, Sep 28, 2023 at 7:23 AM Jack Tomy  wrote:
>
>> Bumping up.
>>
>> On Sun, Sep 17, 2023 at 9:34 AM Jack Tomy  wrote:
>>
>>> Hey Ismael, Sagar and everyone,
>>>
>>> Sorry I seem to have interpreted the thread wrong. Before we go ahead
>>> with the DTO based approach I have few reasons not to go with it.
>>> a. It is not following the pattern we are following today. But here I
>>> agree that patterns are to be changed for good.
>>> b. The client is not supposed to modify the record object (This is my
>>> understanding, If this is not a necessary requirement, please call it
>>> out.), passing the entire object lets the client do that. To avoid that,
>>> there has to be a way to deep copy the record object each time, this adds
>>> unnecessary requirements on the record object to support the deepcopy
>>> implementation. I see a lot of complexity and coupling coming in here due
>>> to this N I believe it's a strong reason not to go ahead with the DTO
>>> approach.
>>>
>>> Please let me know what you think.
>>>
>>> Thanks.
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Sep 6, 2023 at 7:06 AM Sagar  wrote:
>>>
 Hey Jack,

 The way I interpreted this thread, it seems like there's more alignment
 on
 the DTO based approach. I spent some time on the suggestion that Ismael
 had
 regarding the usage of ProducerRecord. Did you get a chance to look at
 the
 reply I had posted and whether that makes sense? Also, checking out the
 AdminClient APIs examples provided by Ismael will give you more context.
 Let me know what you think.

 Thanks!
 Sagar.

 On Thu, Aug 31, 2023 at 12:49 PM Jack Tomy 
 wrote:

 > Hey everyone,
 >
 > As I see devs favouring the current style of implementation, and that
 is
 > inline with existing code. I would like to go ahead with the same
 approach
 > as mentioned in the KIP.
 > Can I get a few more votes so that I can take the KIP forward.
 >
 > Thanks
 >
 >
 >
 > On Sun, Aug 27, 2023 at 1:38 PM Sagar 
 wrote:
 >
 > > Hi Ismael,
 > >
 > > Thanks for pointing us towards the direction of a DTO based
 approach. The
 > > AdminClient examples seem very neat and extensible in that sense.
 > > Personally, I was trying to think only along the lines of how the
 current
 > > Partitioner interface has been designed, i.e having all requisite
 > > parameters as separate arguments (Topic, Key, Value etc).
 > >
 > > Regarding this question of yours:
 > >
 > > A more concrete question: did we consider having the method
 `partition`
 > > > take `ProduceRecord` as one of its parameters and `Cluster` as the
 > other?
 > >
 > >
 > > No, I don't think in the discussion thread it was brought up and as
 I
 > said
 > > it appears that could be due to an attempt to keep the new method's
 > > signature similar to the existing one within Partitioner. If I
 understood
 > > the intent of the question correctly, are you trying to hint here
 that
 > > `ProducerRecord` already contains all the arguments that the
 `partition`
 > > method accepts and also has a `headers` field within it. So,
 instead of
 > > adding another method for the `headers` field, why not create a new
 > method
 > > taking ProducerRecord directly?
 > >
 > > If my understanding is correct, then it seems like a very clean way
 of
 > > adding support for `headers`. Anyways, the partition method within
 > > KafkaProducer already takes a ProducerRecord as an argument so that
 makes
 > > it easier. Keeping that in mind, should this new method's (which
 takes a
 > > ProducerRecord as an input) default implementation invoke the
 existing
 > > method ? One challenge I see there is that the existing partition
 method
 > > expects serialized keys and values while ProducerRecord doesn't have
 > access
 > > to those (It directly operates on K, V).
 > >
 > > Thanks!
 > > Sagar.
 > >
 > >
 > > On Sun, Aug 27, 2023 at 8:51 AM Ismael Juma 
 wrote:
 > >
 > > > A more concrete question: did we consider having the method
 `partition`
 > > > take `ProduceRecord` as one of its parameters and `Cluster` as the
 > other?
 > > >
 > > > Ismael
 > > >
 > > > On Sat, Aug 26, 2023 at 12:50 PM Greg Harris
 > > >>> > > > >
 > > > wrote:
 > > >
 > > > > Hey Ismael,
 > > > >
 > > > > > The mention of "runtime" is specific to Connect. When it
 comes to
 > > > > clients,
 > > > > one typically compiles and runs with the same version or runs
 with a
 > > > newer
 > > > > version than 

Re: [VOTE] KIP-988 Streams StandbyUpdateListener

[Bruno Cadonna]

Hi Colt and Eduwer,

Thanks for the KIP!

+1 (binding)

Best,
Bruno

On 10/26/23 7:17 PM, Matthias J. Sax wrote:

+1 (binding)

On 10/25/23 4:06 PM, Sophie Blee-Goldman wrote:

Happy to see this -- that's a +1 (binding) from me

On Mon, Oct 23, 2023 at 6:33 AM Bill Bejeck  wrote:


This is a great addition

+1(binding)

-Bill

On Fri, Oct 20, 2023 at 2:29 PM Almog Gavra  
wrote:



+1 (non-binding) - great improvement, thanks Colt & Eduwer!

On Tue, Oct 17, 2023 at 11:25 AM Guozhang Wang <

guozhang.wang...@gmail.com



wrote:


+1 from me.

On Mon, Oct 16, 2023 at 1:56 AM Lucas Brutschy
 wrote:


Hi,

thanks again for the KIP!

+1 (binding)

Cheers,
Lucas



On Sun, Oct 15, 2023 at 9:13 AM Colt McNealy 

wrote:


Hello there,

I'd like to call a vote on KIP-988 (co-authored by my friend and

colleague

Eduwer Camacaro). We are hoping to get it in before the 3.7.0

release.








https://cwiki.apache.org/confluence/display/KAFKA/KIP-988%3A+Streams+Standby+Task+Update+Listener


Cheers,
Colt McNealy

*Founder, LittleHorse.dev*

Re: [PR] MINOR: Fix incorrect link in 3.6 announcement [kafka-site]

[via GitHub]

jlprat merged PR #565:
URL: https://github.com/apache/kafka-site/pull/565


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Resolved] (KAFKA-13005) Support JBOD in kraft mode

[Deng Ziming (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deng Ziming resolved KAFKA-13005.
-
Resolution: Duplicate

> Support JBOD in kraft mode
> --
>
> Key: KAFKA-13005
> URL: https://issues.apache.org/jira/browse/KAFKA-13005
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Colin McCabe
>Assignee: Deng Ziming
>Priority: Major
>  Labels: kip-500
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [PR] MINOR: Fix incorrect link in 3.6 announcement [kafka-site]

[via GitHub]

soarez commented on PR #565:
URL: https://github.com/apache/kafka-site/pull/565#issuecomment-1787104501

   cc @satishd @jlprat 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] MINOR: Fix incorrect link in 3.6 announcement [kafka-site]

[via GitHub]

soarez opened a new pull request, #565:
URL: https://github.com/apache/kafka-site/pull/565

   (no comment)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Resolved] (KAFKA-15647) Fix the different behavior in error handling between the old and new group coordinator

[David Jacot (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-15647.
-
Fix Version/s: 3.7.0
   Resolution: Fixed

> Fix the different behavior in error handling between the old and new group 
> coordinator
> --
>
> Key: KAFKA-15647
> URL: https://issues.apache.org/jira/browse/KAFKA-15647
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Dongnuo Lyu
>Assignee: Dongnuo Lyu
>Priority: Major
>  Labels: kip-848-preview
> Fix For: 3.7.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15764) Missing tests for transactions

[Divij Vaidya (Jira)]

Divij Vaidya created KAFKA-15764:


 Summary: Missing tests for transactions
 Key: KAFKA-15764
 URL: https://issues.apache.org/jira/browse/KAFKA-15764
 Project: Kafka
  Issue Type: Test
Reporter: Divij Vaidya


As part of https://issues.apache.org/jira/browse/KAFKA-15653 we found a bug 
which was not caught by any of the existing tests in Apache Kafka.

However, the bug is consistently reproducible if we use the test suite shared 
by [~twmb] at 
https://issues.apache.org/jira/browse/KAFKA-15653?focusedCommentId=1846=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-1846
 

As part of this JIRA, we want to do add relevant test in Apache Kafka which 
should have failed. We can look at the test suite that franz/go repository is 
running, more specifically the test suite that is executed by "go test -run 
Txn".(see repro.sh attached to the comment linked above).

With reference to code in repro.sh, we don't need to do docker compose down or 
up in the while loop Hence, it's ok to simply do
```
while go test -run Txn > FRANZ_FAIL; do
done
```



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15763) Group Coordinator should not deliver new assignment before previous one is acknowledged

[David Jacot (Jira)]

David Jacot created KAFKA-15763:
---

 Summary: Group Coordinator should not deliver new assignment 
before previous one is acknowledged
 Key: KAFKA-15763
 URL: https://issues.apache.org/jira/browse/KAFKA-15763
 Project: Kafka
  Issue Type: Sub-task
Reporter: David Jacot
Assignee: David Jacot


In the initial implementation of the new consumer group protocol, the group 
coordinators waits on received an acknowledgement from the consumer only when 
there are partitions to be revoked. In the case of newly assigned partitions, a 
new assignment can be delivered any time (e.g. in two subsequent heartbeats).

While implementing the state machine on the client side, we found out that this 
caused confusion because the protocol does not treat revocation and assignment 
in the same way. We also found out that changing the assignment before the 
previous one is fully processed by the member makes the client side logic more 
complicated than it should be because the consumer can't process any new 
assignment until it has completed the previous one.

In the end, it is better to change the server side to not deliver a new 
assignment before the current one is acknowledged by the consumer.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15754) The kafka-storage tool can generate UUID starting with "-"

[Paolo Patierno (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paolo Patierno resolved KAFKA-15754.

Resolution: Not A Problem

> The kafka-storage tool can generate UUID starting with "-"
> --
>
> Key: KAFKA-15754
> URL: https://issues.apache.org/jira/browse/KAFKA-15754
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 3.6.0
>Reporter: Paolo Patierno
>Assignee: Paolo Patierno
>Priority: Major
>
> Using the kafka-storage.sh tool, it seems that it can still generate a UUID 
> starting with a dash "-", which then breaks how the argparse4j library works. 
> With such an UUID (i.e. -rmdB0m4T4–Y4thlNXk4Q in my case) the tool exits with 
> the following error:
> kafka-storage: error: argument --cluster-id/-t: expected one argument
> Said that, it seems that this problem was already addressed in the 
> Uuid.randomUuid method which keeps generating a new UUID until it doesn't 
> start with "-". This is the commit addressing it 
> [https://github.com/apache/kafka/commit/5c1dd493d6f608b566fdad5ab3a896cb13622bce]
> The problem is that when the toString is called on the Uuid instance, it's 
> going to do a Base64 encoding on the generated UUID this way:
> {code:java}
> Base64.getUrlEncoder().withoutPadding().encodeToString(getBytesFromUuid()); 
> {code}
> Not sure why, but the code is using an URL (safe) encoder which, taking a 
> look at the Base64 class in Java, is using a RFC4648_URLSAFE encoder using 
> the following alphabet:
>  
> {code:java}
> private static final char[] toBase64URL = new char[]{'A', 'B', 'C', 'D', 'E', 
> 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 
> 'U', 'V', 'W', 'X', 'Y', 'Z', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 
> 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 
> 'y', 'z', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '-', '_'}; {code}
> which as you can see includes the "-" character.
> So despite the current Uuid.randomUuid is avoiding the generation of a UUID 
> containing a dash, the Base64 encoding operation can return a final UUID 
> starting with the dash instead.
>  
> I was wondering if there is any good reason for using a Base64 URL encoder 
> and not just the RFC4648 (not URL safe) which uses the common Base64 alphabet 
> not containing the "-".



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (KAFKA-15754) The kafka-storage tool can generate UUID starting with "-"

[Paolo Patierno (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paolo Patierno reopened KAFKA-15754:


> The kafka-storage tool can generate UUID starting with "-"
> --
>
> Key: KAFKA-15754
> URL: https://issues.apache.org/jira/browse/KAFKA-15754
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 3.6.0
>Reporter: Paolo Patierno
>Assignee: Paolo Patierno
>Priority: Major
>
> Using the kafka-storage.sh tool, it seems that it can still generate a UUID 
> starting with a dash "-", which then breaks how the argparse4j library works. 
> With such an UUID (i.e. -rmdB0m4T4–Y4thlNXk4Q in my case) the tool exits with 
> the following error:
> kafka-storage: error: argument --cluster-id/-t: expected one argument
> Said that, it seems that this problem was already addressed in the 
> Uuid.randomUuid method which keeps generating a new UUID until it doesn't 
> start with "-". This is the commit addressing it 
> [https://github.com/apache/kafka/commit/5c1dd493d6f608b566fdad5ab3a896cb13622bce]
> The problem is that when the toString is called on the Uuid instance, it's 
> going to do a Base64 encoding on the generated UUID this way:
> {code:java}
> Base64.getUrlEncoder().withoutPadding().encodeToString(getBytesFromUuid()); 
> {code}
> Not sure why, but the code is using an URL (safe) encoder which, taking a 
> look at the Base64 class in Java, is using a RFC4648_URLSAFE encoder using 
> the following alphabet:
>  
> {code:java}
> private static final char[] toBase64URL = new char[]{'A', 'B', 'C', 'D', 'E', 
> 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 
> 'U', 'V', 'W', 'X', 'Y', 'Z', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 
> 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 
> 'y', 'z', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '-', '_'}; {code}
> which as you can see includes the "-" character.
> So despite the current Uuid.randomUuid is avoiding the generation of a UUID 
> containing a dash, the Base64 encoding operation can return a final UUID 
> starting with the dash instead.
>  
> I was wondering if there is any good reason for using a Base64 URL encoder 
> and not just the RFC4648 (not URL safe) which uses the common Base64 alphabet 
> not containing the "-".



--
This message was sent by Atlassian Jira
(v8.20.10#820010)