Re: [PR] KAFKA-15133: only tick messageConversionsTimeMs if there are conversions [kafka]
rajinisivaram commented on code in PR #16329:
URL: https://github.com/apache/kafka/pull/16329#discussion_r1642299815
##
core/src/main/scala/kafka/network/RequestChannel.scala:
##
@@ -262,7 +262,9 @@ object RequestChannel extends Logging {
m.responseSendTimeHist.update(Math.round(responseSendTimeMs))
m.totalTimeHist.update(Math.round(totalTimeMs))
m.requestBytesHist.update(sizeOfBodyInBytes)
-
m.messageConversionsTimeHist.foreach(_.update(Math.round(messageConversionsTimeMs)))
+if (messageConversionsTimeMs > 0) {
Review Comment:
It will be better to check `messageConversionsTimeNanos` instead of the
converted `ms`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] KAFKA-16921: Migrate test of connect module to Junit5 (Runtime subpackage) [kafka]
chia7712 commented on code in PR #16350:
URL: https://github.com/apache/kafka/pull/16350#discussion_r1642307332
##
connect/runtime/src/test/java/org/apache/kafka/connect/runtime/isolation/PluginScannerTest.java:
##
@@ -17,161 +17,156 @@
package org.apache.kafka.connect.runtime.isolation;
-import org.junit.BeforeClass;
-import org.junit.Rule;
-import org.junit.Test;
-import org.junit.rules.TemporaryFolder;
-import org.junit.runner.RunWith;
-import org.junit.runners.Parameterized;
-
+import java.io.File;
import java.nio.file.Files;
import java.nio.file.Path;
-import java.util.ArrayList;
-import java.util.Collection;
import java.util.Collections;
+import java.util.HashMap;
import java.util.HashSet;
-import java.util.List;
+import java.util.Map;
import java.util.Set;
+import java.util.stream.Stream;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.io.TempDir;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertTrue;
-import static org.junit.Assert.assertFalse;
-@RunWith(Parameterized.class)
public class PluginScannerTest {
private enum ScannerType { Reflection, ServiceLoader }
-@Rule
-public TemporaryFolder pluginDir = new TemporaryFolder();
+@TempDir
+File pluginDir;
-private final PluginScanner scanner;
+private Map scannerMap;
-@Parameterized.Parameters
-public static Collection parameters() {
-List values = new ArrayList<>();
-for (ScannerType type : ScannerType.values()) {
-values.add(new Object[]{type});
-}
-return values;
+@BeforeEach
+public void setScanner() {
Review Comment:
Maybe we can parameterize scanner directly?
```java
static Stream parameters() {
return Stream.of(new ReflectionScanner(), new
ServiceLoaderScanner());
}
```
##
connect/runtime/src/test/java/org/apache/kafka/connect/runtime/distributed/WorkerGroupMemberTest.java:
##
@@ -27,25 +27,28 @@
import org.apache.kafka.connect.runtime.WorkerConfig;
import org.apache.kafka.connect.storage.ConfigBackingStore;
import org.junit.Test;
Review Comment:
please update this package
##
connect/runtime/src/test/java/org/apache/kafka/connect/runtime/rest/RestClientTest.java:
##
@@ -105,228 +106,235 @@ private static RestClient.HttpResponse
httpRequest(
);
}
-
-@RunWith(Parameterized.class)
-public static class RequestFailureParameterizedTest {
-
-@Rule
-public MockitoRule initRule =
MockitoJUnit.rule().strictness(Strictness.STRICT_STUBS);
-
-@Mock
-private HttpClient httpClient;
-
-@Parameterized.Parameter
-public Throwable requestException;
-
-@Parameterized.Parameters
-public static Collection requestExceptions() {
-return Arrays.asList(new Object[][]{
-{new InterruptedException()},
-{new ExecutionException(null)},
-{new TimeoutException()}
-});
-}
-
-private static Request buildThrowingMockRequest(Throwable t) throws
ExecutionException, InterruptedException, TimeoutException {
-Request req = mock(Request.class);
-when(req.header(anyString(), anyString())).thenReturn(req);
-when(req.send()).thenThrow(t);
-return req;
-}
-
-@Test
-public void testFailureDuringRequestCausesInternalServerError() throws
Exception {
-Request request = buildThrowingMockRequest(requestException);
-when(httpClient.newRequest(anyString())).thenReturn(request);
-ConnectRestException e = assertThrows(ConnectRestException.class,
() -> httpRequest(
-httpClient, MOCK_URL, TEST_METHOD, TEST_TYPE,
TEST_SIGNATURE_ALGORITHM
-));
-assertIsInternalServerError(e);
-assertEquals(requestException, e.getCause());
-}
+private static Stream requestExceptions() {
+return Stream.of(
+Arguments.of(new InterruptedException()),
+Arguments.of(new ExecutionException(null)),
+Arguments.of(new TimeoutException())
+);
}
+private static Request buildThrowingMockRequest(Throwable t) throws
ExecutionException, InterruptedException, TimeoutException {
+Request req = mock(Request.class);
+when(req.header(anyString(), anyString())).thenReturn(req);
+when(req.send()).thenThrow(t);
+return req;
+}
-@RunWith(MockitoJUnitRunner.Strict
[jira] [Created] (KAFKA-16973) Fix caught-up condition
David Jacot created KAFKA-16973: --- Summary: Fix caught-up condition Key: KAFKA-16973 URL: https://issues.apache.org/jira/browse/KAFKA-16973 Project: Kafka Issue Type: Sub-task Reporter: David Jacot Assignee: David Jacot When a write operation does not have any records, the coordinator runtime checked whether the state machine is caught-up to decide whether the operation should wait until the state machine is committed up to the operation point or the operation should be completed. The current implementation assumes that there will always be a pending write operation waiting in the deferred queue when the state machine is not fully caught-up yet. This is true except when the state machine is just loaded and not caught-up yet. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] KAFKA-16973; Fix caught-up condition [kafka]
dajac opened a new pull request, #16367: URL: https://github.com/apache/kafka/pull/16367 When a write operation does not have any records, the coordinator runtime checked whether the state machine is caught-up to decide whether the operation should wait until the state machine is committed up to the operation point or the operation should be completed. The current implementation assumes that there will always be a pending write operation waiting in the deferred queue when the state machine is not fully caught-up yet. This is true except when the state machine is just loaded and not caught-up yet. This patch fixes the issue by always comparing the last written offset and the last committed offset. ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-14511: extend AlterIncrementalConfigs API to support group config [kafka]
AndrewJSchofield commented on PR #15067: URL: https://github.com/apache/kafka/pull/15067#issuecomment-2172549632 @DL1231 Yes, I'll give the updated code a detailed review in the next few days. Thanks for rebasing it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINOR: Don't swallow validateReconfiguration exceptions [kafka]
rajinisivaram commented on code in PR #16346:
URL: https://github.com/apache/kafka/pull/16346#discussion_r1642352229
##
core/src/main/scala/kafka/server/DynamicBrokerConfig.scala:
##
@@ -640,8 +640,8 @@ class DynamicBrokerConfig(private val kafkaConfig:
KafkaConfig) extends Logging
reconfigurable.validateReconfiguration(newConfigs)
} catch {
case e: ConfigException => throw e
- case _: Exception =>
-throw new ConfigException(s"Validation of dynamic config update of
$updatedConfigNames failed with class ${reconfigurable.getClass}")
+ case e: Exception =>
+throw new ConfigException(s"Validation of dynamic config update of
$updatedConfigNames failed with class ${reconfigurable.getClass} due to: $e")
Review Comment:
Some config exceptions may contain sensitive data like SASL passwords or SSL
info. We need to make sure the exception string doesn't include any sensitive
information, especially if it could be sent to clients.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] KAFKA-16547: Add test for DescribeConfigsOptions#includeDocumentation [kafka]
frankvicky commented on PR #16355: URL: https://github.com/apache/kafka/pull/16355#issuecomment-2172550434 Hi @chia7712, I have refactor the test case, PTAL 😃 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16531: calculate check-quorum when leader is not in voter set [kafka]
showuon commented on code in PR #16211:
URL: https://github.com/apache/kafka/pull/16211#discussion_r1642365430
##
raft/src/main/java/org/apache/kafka/raft/LeaderState.java:
##
@@ -132,8 +133,13 @@ public long timeUntilCheckQuorumExpires(long
currentTimeMs) {
*/
public void updateCheckQuorumForFollowingVoter(int id, long currentTimeMs)
{
updateFetchedVoters(id);
-// The majority number of the voters excluding the leader. Ex: 3
voters, the value will be 1
-int majority = voterStates.size() / 2;
+// The majority number of the voters. Ex: 2 for 3 voters, 3 for 4
voters... etc.
+int majority = (int) Math.ceil((double) (voterStates.size() + 1) / 2);
+// Check if the leader is removed from the voter set
+if (voterStates.containsKey(localId)) {
+majority = majority - 1;
Review Comment:
Looks good to me. Thanks.
##
raft/src/main/java/org/apache/kafka/raft/LeaderState.java:
##
@@ -132,8 +133,13 @@ public long timeUntilCheckQuorumExpires(long
currentTimeMs) {
*/
public void updateCheckQuorumForFollowingVoter(int id, long currentTimeMs)
{
updateFetchedVoters(id);
-// The majority number of the voters excluding the leader. Ex: 3
voters, the value will be 1
-int majority = voterStates.size() / 2;
+// The majority number of the voters. Ex: 2 for 3 voters, 3 for 4
voters... etc.
+int majority = (int) Math.ceil((double) (voterStates.size() + 1) / 2);
Review Comment:
That works, too. Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] KAFKA-16531: calculate check-quorum when leader is not in voter set [kafka]
showuon commented on code in PR #16211:
URL: https://github.com/apache/kafka/pull/16211#discussion_r1642365909
##
raft/src/main/java/org/apache/kafka/raft/LeaderState.java:
##
@@ -117,9 +117,10 @@ public long timeUntilCheckQuorumExpires(long
currentTimeMs) {
long remainingMs = checkQuorumTimer.remainingMs();
if (remainingMs == 0) {
log.info(
-"Did not receive fetch request from the majority of the voters
within {}ms. Current fetched voters are {}.",
+"Did not receive fetch request from the majority of the voters
within {}ms. Current fetched voters are {}, and voters are {}",
checkQuorumTimeoutMs,
-fetchedVoters);
+fetchedVoters,
+voterStates.keySet());
Review Comment:
OK, will review https://github.com/apache/kafka/pull/16235 later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] KAFKA-16667 Avoid stale read in KRaftMigrationDriver [kafka]
soarez commented on code in PR #15918:
URL: https://github.com/apache/kafka/pull/15918#discussion_r1642377375
##
core/src/test/scala/integration/kafka/zk/ZkMigrationFailoverTest.scala:
##
@@ -274,4 +288,135 @@ class ZkMigrationFailoverTest extends Logging {
if (zkClient != null) Utils.closeQuietly(zkClient, "KafkaZkClient")
}
}
+
+ @Test
+ def testDriverSkipsEventsFromOlderEpoch(): Unit = {
+val zookeeper = new EmbeddedZookeeper()
+var zkClient: KafkaZkClient = null
+val zkConnect = s"127.0.0.1:${zookeeper.port}"
+try {
+ zkClient = KafkaZkClient(
+zkConnect,
+isSecure = false,
+3,
+6,
+1,
+Time.SYSTEM,
+name = "ZkMigrationFailoverTest",
+new ZKClientConfig)
+} catch {
+ case t: Throwable =>
+Utils.closeQuietly(zookeeper, "EmbeddedZookeeper")
+zookeeper.shutdown()
Review Comment:
I see you marked this as resolved, and addressed it for
`testDriverSkipsEventsFromOlderEpoch()` but not for
`testDriverSkipsEventsFromOlderEpoch()`. Was that intentional?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] KAFKA-16958 add STRICT_STUBS to EndToEndLatencyTest, OffsetCommitCallbackInvokerTest, ProducerPerformanceTest, and TopologyTest [kafka]
chia7712 commented on PR #16348: URL: https://github.com/apache/kafka/pull/16348#issuecomment-2172587467 @dujian0068 please fix build error -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16667 Avoid stale read in KRaftMigrationDriver [kafka]
soarez commented on code in PR #15918:
URL: https://github.com/apache/kafka/pull/15918#discussion_r1642377375
##
core/src/test/scala/integration/kafka/zk/ZkMigrationFailoverTest.scala:
##
@@ -274,4 +288,135 @@ class ZkMigrationFailoverTest extends Logging {
if (zkClient != null) Utils.closeQuietly(zkClient, "KafkaZkClient")
}
}
+
+ @Test
+ def testDriverSkipsEventsFromOlderEpoch(): Unit = {
+val zookeeper = new EmbeddedZookeeper()
+var zkClient: KafkaZkClient = null
+val zkConnect = s"127.0.0.1:${zookeeper.port}"
+try {
+ zkClient = KafkaZkClient(
+zkConnect,
+isSecure = false,
+3,
+6,
+1,
+Time.SYSTEM,
+name = "ZkMigrationFailoverTest",
+new ZKClientConfig)
+} catch {
+ case t: Throwable =>
+Utils.closeQuietly(zookeeper, "EmbeddedZookeeper")
+zookeeper.shutdown()
Review Comment:
I see you marked this as resolved, and addressed it for lines ~416 but not
for ~309. Was that intentional?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (KAFKA-14567) Kafka Streams crashes after ProducerFencedException
[
https://issues.apache.org/jira/browse/KAFKA-14567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruno Cadonna updated KAFKA-14567:
--
Fix Version/s: 3.9.0
(was: 3.8.0)
> Kafka Streams crashes after ProducerFencedException
> ---
>
> Key: KAFKA-14567
> URL: https://issues.apache.org/jira/browse/KAFKA-14567
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.7.0
>Reporter: Matthias J. Sax
>Priority: Blocker
> Labels: eos
> Fix For: 3.9.0
>
>
> Running a Kafka Streams application with EOS-v2.
> We first see a `ProducerFencedException`. After the fencing, the fenced
> thread crashed resulting in a non-recoverable error:
> {quote}[2022-12-22 13:49:13,423] ERROR [i-0c291188ec2ae17a0-StreamThread-3]
> stream-thread [i-0c291188ec2ae17a0-StreamThread-3] Failed to process stream
> task 1_2 due to the following error:
> (org.apache.kafka.streams.processor.internals.TaskExecutor)
> org.apache.kafka.streams.errors.StreamsException: Exception caught in
> process. taskId=1_2, processor=KSTREAM-SOURCE-05,
> topic=node-name-repartition, partition=2, offset=539776276,
> stacktrace=java.lang.IllegalStateException: TransactionalId
> stream-soak-test-72b6e57c-c2f5-489d-ab9f-fdbb215d2c86-3: Invalid transition
> attempted from state FATAL_ERROR to state ABORTABLE_ERROR
> at
> org.apache.kafka.clients.producer.internals.TransactionManager.transitionTo(TransactionManager.java:974)
> at
> org.apache.kafka.clients.producer.internals.TransactionManager.transitionToAbortableError(TransactionManager.java:394)
> at
> org.apache.kafka.clients.producer.internals.TransactionManager.maybeTransitionToErrorState(TransactionManager.java:620)
> at
> org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:1079)
> at
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:959)
> at
> org.apache.kafka.streams.processor.internals.StreamsProducer.send(StreamsProducer.java:257)
> at
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl.send(RecordCollectorImpl.java:207)
> at
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl.send(RecordCollectorImpl.java:162)
> at
> org.apache.kafka.streams.processor.internals.SinkNode.process(SinkNode.java:85)
> at
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forwardInternal(ProcessorContextImpl.java:290)
> at
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:269)
> at
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:228)
> at
> org.apache.kafka.streams.kstream.internals.KStreamKTableJoinProcessor.process(KStreamKTableJoinProcessor.java:88)
> at
> org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:157)
> at
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forwardInternal(ProcessorContextImpl.java:290)
> at
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:269)
> at
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:228)
> at
> org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:84)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.lambda$doProcess$1(StreamTask.java:791)
> at
> org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.maybeMeasureLatency(StreamsMetricsImpl.java:867)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.doProcess(StreamTask.java:791)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:722)
> at
> org.apache.kafka.streams.processor.internals.TaskExecutor.processTask(TaskExecutor.java:95)
> at
> org.apache.kafka.streams.processor.internals.TaskExecutor.process(TaskExecutor.java:76)
> at
> org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:1645)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:788)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:607)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:569)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:748)
> at
> org.apache.kafka.streams.processor.internals.TaskExecutor.processTask(TaskExecutor.java:95)
> at
> org.apache.kafka.streams.process
Re: [PR] KAFKA-16921: Migrate test of connect module to Junit5 (Runtime direct) [kafka]
chia7712 commented on PR #16351: URL: https://github.com/apache/kafka/pull/16351#issuecomment-2172601878 @gongxuanzhang please fix those failed tests -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16969: Log error if config conficts with MV [kafka]
soarez commented on PR #16366: URL: https://github.com/apache/kafka/pull/16366#issuecomment-2172613372 Thanks for having a look @showuon . If CI finishes without issues I'll merge this and backport to 3.7. @jlprat I think the system tests in 3.8 will also fail without this, so we probably want to backport this to 3.8 too. WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KAFKA-14567) Kafka Streams crashes after ProducerFencedException
[
https://issues.apache.org/jira/browse/KAFKA-14567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855536#comment-17855536
]
Bruno Cadonna commented on KAFKA-14567:
---
[~ableegoldman] We ([~mjsax] and I) discussed it and came to the conclusion
that is is not really a blocker for 3.8, since we also found it in 3.7. Plus,
by catching the exception in the uncaught exception handler and restarting the
thread, a work-around (although expensive) exists.
Said that, I would be more than happy to get this fixed as soon as possible. So
if you have capacity, please go ahead. We also put this ticket on our
short-term plan. Let's try to fix it from both -- your and our -- side and keep
in contact about the progress. WDYT?
> Kafka Streams crashes after ProducerFencedException
> ---
>
> Key: KAFKA-14567
> URL: https://issues.apache.org/jira/browse/KAFKA-14567
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.7.0
>Reporter: Matthias J. Sax
>Priority: Blocker
> Labels: eos
> Fix For: 3.9.0
>
>
> Running a Kafka Streams application with EOS-v2.
> We first see a `ProducerFencedException`. After the fencing, the fenced
> thread crashed resulting in a non-recoverable error:
> {quote}[2022-12-22 13:49:13,423] ERROR [i-0c291188ec2ae17a0-StreamThread-3]
> stream-thread [i-0c291188ec2ae17a0-StreamThread-3] Failed to process stream
> task 1_2 due to the following error:
> (org.apache.kafka.streams.processor.internals.TaskExecutor)
> org.apache.kafka.streams.errors.StreamsException: Exception caught in
> process. taskId=1_2, processor=KSTREAM-SOURCE-05,
> topic=node-name-repartition, partition=2, offset=539776276,
> stacktrace=java.lang.IllegalStateException: TransactionalId
> stream-soak-test-72b6e57c-c2f5-489d-ab9f-fdbb215d2c86-3: Invalid transition
> attempted from state FATAL_ERROR to state ABORTABLE_ERROR
> at
> org.apache.kafka.clients.producer.internals.TransactionManager.transitionTo(TransactionManager.java:974)
> at
> org.apache.kafka.clients.producer.internals.TransactionManager.transitionToAbortableError(TransactionManager.java:394)
> at
> org.apache.kafka.clients.producer.internals.TransactionManager.maybeTransitionToErrorState(TransactionManager.java:620)
> at
> org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:1079)
> at
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:959)
> at
> org.apache.kafka.streams.processor.internals.StreamsProducer.send(StreamsProducer.java:257)
> at
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl.send(RecordCollectorImpl.java:207)
> at
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl.send(RecordCollectorImpl.java:162)
> at
> org.apache.kafka.streams.processor.internals.SinkNode.process(SinkNode.java:85)
> at
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forwardInternal(ProcessorContextImpl.java:290)
> at
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:269)
> at
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:228)
> at
> org.apache.kafka.streams.kstream.internals.KStreamKTableJoinProcessor.process(KStreamKTableJoinProcessor.java:88)
> at
> org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:157)
> at
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forwardInternal(ProcessorContextImpl.java:290)
> at
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:269)
> at
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:228)
> at
> org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:84)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.lambda$doProcess$1(StreamTask.java:791)
> at
> org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.maybeMeasureLatency(StreamsMetricsImpl.java:867)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.doProcess(StreamTask.java:791)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:722)
> at
> org.apache.kafka.streams.processor.internals.TaskExecutor.processTask(TaskExecutor.java:95)
> at
> org.apache.kafka.streams.processor.internals.TaskExecutor.process(TaskExecutor.java:76)
> at
> org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:1645)
> at
[jira] [Created] (KAFKA-16974) KRaft support in SslAdminIntegrationTest
Mickael Maison created KAFKA-16974: -- Summary: KRaft support in SslAdminIntegrationTest Key: KAFKA-16974 URL: https://issues.apache.org/jira/browse/KAFKA-16974 Project: Kafka Issue Type: Task Components: core Reporter: Mickael Maison This class needs to be updated to support KRaft -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] KAFKA-15751, KAFKA-15752: Enable KRaft for BaseAdminIntegrationTest and SaslSslAdminIntegrationTest [kafka]
mimaison commented on code in PR #15175:
URL: https://github.com/apache/kafka/pull/15175#discussion_r1642423197
##
core/src/test/scala/integration/kafka/api/SslAdminIntegrationTest.scala:
##
@@ -42,6 +42,7 @@ object SslAdminIntegrationTest {
@volatile var lastUpdateRequestContext: Option[AuthorizableRequestContext] =
None
Review Comment:
I created https://issues.apache.org/jira/browse/KAFKA-16974 for adding KRaft
support to SslAdminIntegrationTest
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] KAFKA-10190: Set dynamic broker configs for entity default [kafka]
harryfallows commented on PR #16280: URL: https://github.com/apache/kafka/pull/16280#issuecomment-2172694775 > @harryfallows thanks for your contribution! thanks for the reviews! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KAFKA-16974) KRaft support in SslAdminIntegrationTest
[ https://issues.apache.org/jira/browse/KAFKA-16974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855541#comment-17855541 ] Kuan Po Tseng commented on KAFKA-16974: --- gentle ping [~mimaison] , may I take over this issue ? I'm willing to do this ! > KRaft support in SslAdminIntegrationTest > > > Key: KAFKA-16974 > URL: https://issues.apache.org/jira/browse/KAFKA-16974 > Project: Kafka > Issue Type: Task > Components: core >Reporter: Mickael Maison >Priority: Major > Labels: kraft-test > > This class needs to be updated to support KRaft -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16974) KRaft support in SslAdminIntegrationTest
[ https://issues.apache.org/jira/browse/KAFKA-16974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855543#comment-17855543 ] Mickael Maison commented on KAFKA-16974: Sure > KRaft support in SslAdminIntegrationTest > > > Key: KAFKA-16974 > URL: https://issues.apache.org/jira/browse/KAFKA-16974 > Project: Kafka > Issue Type: Task > Components: core >Reporter: Mickael Maison >Priority: Major > Labels: kraft-test > > This class needs to be updated to support KRaft -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] KAFKA-16968: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0 [kafka]
dajac commented on code in PR #16347:
URL: https://github.com/apache/kafka/pull/16347#discussion_r1642442783
##
server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java:
##
@@ -202,11 +202,12 @@ public enum MetadataVersion {
// Add new fetch request version for KIP-951
IBP_3_7_IV4(19, "3.7", "IV4", false),
-// Add ELR related supports (KIP-966).
-IBP_3_8_IV0(20, "3.8", "IV0", true),
+// New version for the Kafka 3.8.0 release.
+IBP_3_8_IV0(20, "3.8", "IV0", false),
+// Add ELR related supports (KIP-966).
Review Comment:
I wonder if we should put a comment here explaining that the MV below this
point are not production ready. We could also add a few words explaining that
features must be reviewed and they must moved to the next one if they are not
ready when the next MV is made production ready.
##
server-common/src/main/java/org/apache/kafka/server/common/GroupVersion.java:
##
@@ -22,7 +22,7 @@
public enum GroupVersion implements FeatureVersion {
// Version 1 enables the consumer rebalance protocol (KIP-848).
-GV_1(1, MetadataVersion.IBP_4_0_IV0, Collections.emptyMap());
+GV_1(1, MetadataVersion.IBP_3_9_IV0, Collections.emptyMap());
Review Comment:
Should we keep it attached to `IBP_4_0_IV0` as we already know that it will
be release in 4.0? If not, we should file a blocker for 3.9 to ensure that we
change it before we release 3.9.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] KAFKA-16968: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0 [kafka]
dajac commented on PR #16347: URL: https://github.com/apache/kafka/pull/16347#issuecomment-2172721012 cc @jlprat We need this on in 3.8. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16968: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0 [kafka]
jlprat commented on PR #16347: URL: https://github.com/apache/kafka/pull/16347#issuecomment-2172726481 Thanks @dajac! Let's get this to 3.8 once merged -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (KAFKA-16974) KRaft support in SslAdminIntegrationTest
[ https://issues.apache.org/jira/browse/KAFKA-16974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuan Po Tseng reassigned KAFKA-16974: - Assignee: Kuan Po Tseng > KRaft support in SslAdminIntegrationTest > > > Key: KAFKA-16974 > URL: https://issues.apache.org/jira/browse/KAFKA-16974 > Project: Kafka > Issue Type: Task > Components: core >Reporter: Mickael Maison >Assignee: Kuan Po Tseng >Priority: Major > Labels: kraft-test > > This class needs to be updated to support KRaft -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] KAFKA-16969: Log error if config conficts with MV [kafka]
jlprat commented on PR #16366: URL: https://github.com/apache/kafka/pull/16366#issuecomment-2172729234 Hi @soarez yes I think it makes sense. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (KAFKA-16975) The error message of creating `__cluster_metadata` should NOT be "Authorization failed"
Chia-Ping Tsai created KAFKA-16975: -- Summary: The error message of creating `__cluster_metadata` should NOT be "Authorization failed" Key: KAFKA-16975 URL: https://issues.apache.org/jira/browse/KAFKA-16975 Project: Kafka Issue Type: Improvement Reporter: Chia-Ping Tsai Assignee: Chia-Ping Tsai The error message of creating "__cluster_metadata" by admin is "Authorization failed.". That could confuse users that it implies you "can" create it in using root. However, the fact is that we DISALLOW users to create it as a regular topic even though you are the boss :) -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] KAFKA-16968: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0 [kafka]
AndrewJSchofield commented on code in PR #16347:
URL: https://github.com/apache/kafka/pull/16347#discussion_r1642454880
##
server-common/src/main/java/org/apache/kafka/server/common/GroupVersion.java:
##
@@ -22,7 +22,7 @@
public enum GroupVersion implements FeatureVersion {
// Version 1 enables the consumer rebalance protocol (KIP-848).
-GV_1(1, MetadataVersion.IBP_4_0_IV0, Collections.emptyMap());
+GV_1(1, MetadataVersion.IBP_3_9_IV0, Collections.emptyMap());
Review Comment:
I personally would leave it attached to `IBP_4_0_IV0`. Even if there's a
3.10, we still know GV_1 is in 4.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Reopened] (KAFKA-15420) Kafka Tiered Storage V1
[ https://issues.apache.org/jira/browse/KAFKA-15420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamal Chandraprakash reopened KAFKA-15420: -- > Kafka Tiered Storage V1 > --- > > Key: KAFKA-15420 > URL: https://issues.apache.org/jira/browse/KAFKA-15420 > Project: Kafka > Issue Type: Improvement >Affects Versions: 3.6.0 >Reporter: Satish Duggana >Assignee: Satish Duggana >Priority: Major > Labels: KIP-405 > Fix For: 3.8.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15420) Kafka Tiered Storage V1
[ https://issues.apache.org/jira/browse/KAFKA-15420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamal Chandraprakash resolved KAFKA-15420. -- Resolution: Fixed > Kafka Tiered Storage V1 > --- > > Key: KAFKA-15420 > URL: https://issues.apache.org/jira/browse/KAFKA-15420 > Project: Kafka > Issue Type: Improvement >Affects Versions: 3.6.0 >Reporter: Satish Duggana >Assignee: Satish Duggana >Priority: Major > Labels: KIP-405 > Fix For: 3.8.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] KAFKA-16968: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0 [kafka]
AndrewJSchofield commented on code in PR #16347:
URL: https://github.com/apache/kafka/pull/16347#discussion_r1642456703
##
server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java:
##
@@ -232,7 +233,7 @@ public enum MetadataVersion {
* Think carefully before you update this value. ONCE A METADATA
VERSION IS PRODUCTION,
* IT CANNOT BE CHANGED.
*/
-public static final MetadataVersion LATEST_PRODUCTION = IBP_3_7_IV4;
+public static final MetadataVersion LATEST_PRODUCTION = IBP_3_8_IV0;
Review Comment:
I guess this single line needs to be backported to 3.8.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Assigned] (KAFKA-16780) Txn consumer exerts pressure on remote storage when reading non-txn topic
[
https://issues.apache.org/jira/browse/KAFKA-16780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kamal Chandraprakash reassigned KAFKA-16780:
Assignee: Kamal Chandraprakash
> Txn consumer exerts pressure on remote storage when reading non-txn topic
> -
>
> Key: KAFKA-16780
> URL: https://issues.apache.org/jira/browse/KAFKA-16780
> Project: Kafka
> Issue Type: Task
>Reporter: Kamal Chandraprakash
>Assignee: Kamal Chandraprakash
>Priority: Major
>
> h3. Logic to read aborted txns:
> # When the consumer enables isolation_level as {{READ_COMMITTED}} and reads
> a non-txn topic, then the broker has to
> [traverse|https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/log/LocalLog.scala#L394]
> all the local log segments to collect the aborted transactions since there
> won't be any entry in the transaction index.
> # The same
> [logic|https://github.com/apache/kafka/blob/trunk/core/src/main/java/kafka/log/remote/RemoteLogManager.java#L1436]
> is applied while reading from remote storage. In this case, when the FETCH
> request is reading data from the first remote log segment, then it has to
> fetch the transaction indexes of all the remaining remote-log segments, and
> then the call lands to the local-log segments before responding to the FETCH
> request which increases the time taken to serve the requests.
> The [EoS Abort
> Index|https://docs.google.com/document/d/1Rlqizmk7QCDe8qAnVW5e5X8rGvn6m2DCR3JR2yqwVjc]
> design doc explains how the transaction index file filters out the aborted
> transaction records.
> The issue is when consumers are enabled with the {{READ_COMMITTED}} isolation
> level but read the non-txn topics. If the topic is enabled with the
> transaction, then we expect the transaction to either commit/rollback within
> 15 minutes (default transaction.max.timeout.ms = 15 mins), possibly we may
> have to search only a few remote log segments to collect the aborted txns.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-16780) Txn consumer exerts pressure on remote storage when reading non-txn topic
[
https://issues.apache.org/jira/browse/KAFKA-16780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kamal Chandraprakash updated KAFKA-16780:
-
Labels: kip (was: )
> Txn consumer exerts pressure on remote storage when reading non-txn topic
> -
>
> Key: KAFKA-16780
> URL: https://issues.apache.org/jira/browse/KAFKA-16780
> Project: Kafka
> Issue Type: Task
>Reporter: Kamal Chandraprakash
>Assignee: Kamal Chandraprakash
>Priority: Major
> Labels: kip
>
> h3. Logic to read aborted txns:
> # When the consumer enables isolation_level as {{READ_COMMITTED}} and reads
> a non-txn topic, then the broker has to
> [traverse|https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/log/LocalLog.scala#L394]
> all the local log segments to collect the aborted transactions since there
> won't be any entry in the transaction index.
> # The same
> [logic|https://github.com/apache/kafka/blob/trunk/core/src/main/java/kafka/log/remote/RemoteLogManager.java#L1436]
> is applied while reading from remote storage. In this case, when the FETCH
> request is reading data from the first remote log segment, then it has to
> fetch the transaction indexes of all the remaining remote-log segments, and
> then the call lands to the local-log segments before responding to the FETCH
> request which increases the time taken to serve the requests.
> The [EoS Abort
> Index|https://docs.google.com/document/d/1Rlqizmk7QCDe8qAnVW5e5X8rGvn6m2DCR3JR2yqwVjc]
> design doc explains how the transaction index file filters out the aborted
> transaction records.
> The issue is when consumers are enabled with the {{READ_COMMITTED}} isolation
> level but read the non-txn topics. If the topic is enabled with the
> transaction, then we expect the transaction to either commit/rollback within
> 15 minutes (default transaction.max.timeout.ms = 15 mins), possibly we may
> have to search only a few remote log segments to collect the aborted txns.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] KAFKA-15751, KAFKA-15752: Enable KRaft for BaseAdminIntegrationTest and SaslSslAdminIntegrationTest [kafka]
mimaison commented on code in PR #15175:
URL: https://github.com/apache/kafka/pull/15175#discussion_r1642463621
##
core/src/test/scala/integration/kafka/server/QuorumTestHarness.scala:
##
@@ -366,6 +366,7 @@ abstract class QuorumTestHarness extends Logging {
props.setProperty(SocketServerConfigs.LISTENERS_CONFIG,
s"CONTROLLER://localhost:0")
props.setProperty(KRaftConfigs.CONTROLLER_LISTENER_NAMES_CONFIG,
"CONTROLLER")
props.setProperty(QuorumConfig.QUORUM_VOTERS_CONFIG,
s"$nodeId@localhost:0")
+props.setProperty(ServerLogConfigs.LOG_DELETE_DELAY_MS_CONFIG, "1000")
Review Comment:
Thanks for the details. Can we add a comment in KAFKA-16211 about it? so
your findings are not lost.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] KAFKA-15751, KAFKA-15752: Enable KRaft for BaseAdminIntegrationTest and SaslSslAdminIntegrationTest [kafka]
mimaison commented on PR #15175: URL: https://github.com/apache/kafka/pull/15175#issuecomment-2172752354 The CI failed due to timeout. I kicked another build. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KAFKA-16975) The error message of creating `__cluster_metadata` should NOT be "Authorization failed"
[ https://issues.apache.org/jira/browse/KAFKA-16975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1780#comment-1780 ] Kuan Po Tseng commented on KAFKA-16975: --- +1 on this, perhaps we can should set `Errors.INVALID_REQUEST.code` and refine the error message. [~chia7712] , I'm willing to solve this if you are not working on this one, thanks ! > The error message of creating `__cluster_metadata` should NOT be > "Authorization failed" > > > Key: KAFKA-16975 > URL: https://issues.apache.org/jira/browse/KAFKA-16975 > Project: Kafka > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > > The error message of creating "__cluster_metadata" by admin is "Authorization > failed.". That could confuse users that it implies you "can" create it in > using root. However, the fact is that we DISALLOW users to create it as a > regular topic even though you are the boss :) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-16975) The error message of creating `__cluster_metadata` should NOT be "Authorization failed"
[ https://issues.apache.org/jira/browse/KAFKA-16975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai reassigned KAFKA-16975: -- Assignee: Kuan Po Tseng (was: Chia-Ping Tsai) > The error message of creating `__cluster_metadata` should NOT be > "Authorization failed" > > > Key: KAFKA-16975 > URL: https://issues.apache.org/jira/browse/KAFKA-16975 > Project: Kafka > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: Kuan Po Tseng >Priority: Minor > > The error message of creating "__cluster_metadata" by admin is "Authorization > failed.". That could confuse users that it implies you "can" create it in > using root. However, the fact is that we DISALLOW users to create it as a > regular topic even though you are the boss :) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16737) Clean up KafkaConsumerTest TODOs enabling tests for new consumer
[ https://issues.apache.org/jira/browse/KAFKA-16737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josep Prat updated KAFKA-16737: --- Fix Version/s: 3.9.0 (was: 3.8.0) > Clean up KafkaConsumerTest TODOs enabling tests for new consumer > > > Key: KAFKA-16737 > URL: https://issues.apache.org/jira/browse/KAFKA-16737 > Project: Kafka > Issue Type: Task > Components: clients, consumer >Affects Versions: 3.7.0 >Reporter: Lianet Magrans >Assignee: Lianet Magrans >Priority: Blocker > Labels: kip-848-client-support > Fix For: 3.9.0 > > > KafkaConsumerTest.java contains lots of TODOs (50+) related to tests that are > only enabled for the CLASSIC protocol and should be reviewed and enabled for > the new CONSUMER group protocol when applicable. Some tests also have TODOs > to enable them for the new consumer when certain features/bugs are addressed. > The new protocol and consumer implementation have evolved a lot since those > TODOs where added, so we should review them all, enable tests for the new > protocol when applicable and removing the TODOs from the code. Note that > there is another AsyncKafkaConsumerTest.java, testing logic specific to the > internals of the new consumer, but still many tests in the KafkaConsumerTest > apply to both the new and legacy consumer, and we should enable them for > both. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16792) Enable consumer unit tests that fail to fetch offsets only for new consumer with poll(0)
[ https://issues.apache.org/jira/browse/KAFKA-16792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1783#comment-1783 ] Josep Prat commented on KAFKA-16792: I don't think this qualifies as a blocker for 3.8.0. Please let me know if you disagree. > Enable consumer unit tests that fail to fetch offsets only for new consumer > with poll(0) > > > Key: KAFKA-16792 > URL: https://issues.apache.org/jira/browse/KAFKA-16792 > Project: Kafka > Issue Type: Test > Components: clients, consumer, unit tests >Reporter: Lianet Magrans >Assignee: Kirk True >Priority: Blocker > Labels: kip-848-client-support > Fix For: 3.8.0 > > > Enable the following unit tests for the new async consumer in > KafkaConsumerTest: > - testCurrentLag > - testFetchStableOffsetThrowInPoll > - testListOffsetShouldUpdateSubscriptions > - testPollReturnsRecords > - testResetToCommittedOffset > - testResetUsingAutoResetPolicy > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16792) Enable consumer unit tests that fail to fetch offsets only for new consumer with poll(0)
[ https://issues.apache.org/jira/browse/KAFKA-16792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josep Prat updated KAFKA-16792: --- Fix Version/s: 3.9.0 (was: 3.8.0) > Enable consumer unit tests that fail to fetch offsets only for new consumer > with poll(0) > > > Key: KAFKA-16792 > URL: https://issues.apache.org/jira/browse/KAFKA-16792 > Project: Kafka > Issue Type: Test > Components: clients, consumer, unit tests >Reporter: Lianet Magrans >Assignee: Kirk True >Priority: Blocker > Labels: kip-848-client-support > Fix For: 3.9.0 > > > Enable the following unit tests for the new async consumer in > KafkaConsumerTest: > - testCurrentLag > - testFetchStableOffsetThrowInPoll > - testListOffsetShouldUpdateSubscriptions > - testPollReturnsRecords > - testResetToCommittedOffset > - testResetUsingAutoResetPolicy > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16737) Clean up KafkaConsumerTest TODOs enabling tests for new consumer
[ https://issues.apache.org/jira/browse/KAFKA-16737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1781#comment-1781 ] Josep Prat commented on KAFKA-16737: I don't think this qualifies as a Blocker for 3.8.0. I'll move it to 3.9.0 > Clean up KafkaConsumerTest TODOs enabling tests for new consumer > > > Key: KAFKA-16737 > URL: https://issues.apache.org/jira/browse/KAFKA-16737 > Project: Kafka > Issue Type: Task > Components: clients, consumer >Affects Versions: 3.7.0 >Reporter: Lianet Magrans >Assignee: Lianet Magrans >Priority: Blocker > Labels: kip-848-client-support > Fix For: 3.8.0 > > > KafkaConsumerTest.java contains lots of TODOs (50+) related to tests that are > only enabled for the CLASSIC protocol and should be reviewed and enabled for > the new CONSUMER group protocol when applicable. Some tests also have TODOs > to enable them for the new consumer when certain features/bugs are addressed. > The new protocol and consumer implementation have evolved a lot since those > TODOs where added, so we should review them all, enable tests for the new > protocol when applicable and removing the TODOs from the code. Note that > there is another AsyncKafkaConsumerTest.java, testing logic specific to the > internals of the new consumer, but still many tests in the KafkaConsumerTest > apply to both the new and legacy consumer, and we should enable them for > both. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] KAFKA-16954: fix consumer close to release assignment in background [kafka]
jlprat commented on PR #16343: URL: https://github.com/apache/kafka/pull/16343#issuecomment-2172838030 Is there any way to have this merged before we start making RCs for 3.8.0? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KAFKA-8280) Flaky Test DynamicBrokerReconfigurationTest#testUncleanLeaderElectionEnable
[
https://issues.apache.org/jira/browse/KAFKA-8280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1786#comment-1786
]
Josep Prat commented on KAFKA-8280:
---
Pushing this to 3.9 as it's not yet worked on.
> Flaky Test DynamicBrokerReconfigurationTest#testUncleanLeaderElectionEnable
> ---
>
> Key: KAFKA-8280
> URL: https://issues.apache.org/jira/browse/KAFKA-8280
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 2.3.0
>Reporter: John Roesler
>Priority: Blocker
> Labels: flakey
> Fix For: 3.8.0
>
>
> I saw this fail again on
> https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/3979/testReport/junit/kafka.server/DynamicBrokerReconfigurationTest/testUncleanLeaderElectionEnable/
> {noformat}
> Error Message
> java.lang.AssertionError: Unclean leader not elected
> Stacktrace
> java.lang.AssertionError: Unclean leader not elected
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> kafka.server.DynamicBrokerReconfigurationTest.testUncleanLeaderElectionEnable(DynamicBrokerReconfigurationTest.scala:510)
> {noformat}
> {noformat}
> Standard Output
> Completed Updating config for entity: brokers '0'.
> Completed Updating config for entity: brokers '1'.
> Completed Updating config for entity: brokers '2'.
> [2019-04-23 01:17:11,690] ERROR [ReplicaFetcher replicaId=1, leaderId=0,
> fetcherId=1] Error for partition testtopic-6 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-23 01:17:11,690] ERROR [ReplicaFetcher replicaId=1, leaderId=0,
> fetcherId=1] Error for partition testtopic-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-23 01:17:11,722] ERROR [ReplicaFetcher replicaId=2, leaderId=1,
> fetcherId=0] Error for partition testtopic-7 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-23 01:17:11,722] ERROR [ReplicaFetcher replicaId=2, leaderId=1,
> fetcherId=0] Error for partition testtopic-1 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-23 01:17:11,760] ERROR [ReplicaFetcher replicaId=2, leaderId=0,
> fetcherId=1] Error for partition testtopic-6 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-23 01:17:11,761] ERROR [ReplicaFetcher replicaId=2, leaderId=0,
> fetcherId=1] Error for partition testtopic-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-23 01:17:11,765] ERROR [ReplicaFetcher replicaId=2, leaderId=0,
> fetcherId=0] Error for partition testtopic-3 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-23 01:17:11,765] ERROR [ReplicaFetcher replicaId=2, leaderId=0,
> fetcherId=0] Error for partition testtopic-9 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-23 01:17:13,779] ERROR [ReplicaFetcher replicaId=1, leaderId=0,
> fetcherId=1] Error for partition __consumer_offsets-8 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-23 01:17:13,780] ERROR [ReplicaFetcher replicaId=1, leaderId=0,
> fetcherId=1] Error for partition __consumer_offsets-38 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-23 01:17:13,780] ERROR [ReplicaFetcher replicaId=1, leaderId=0,
> fetcherId=1] Error for partition __consumer_offsets-2 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-23 01:17:13,780] ERROR [ReplicaFetcher replicaId=1, leaderId=0,
> fetcherId=1] Error for partition __consumer_offsets-14 at offset 0
> (kafka.server.Replica
[jira] [Updated] (KAFKA-8280) Flaky Test DynamicBrokerReconfigurationTest#testUncleanLeaderElectionEnable
[
https://issues.apache.org/jira/browse/KAFKA-8280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josep Prat updated KAFKA-8280:
--
Fix Version/s: 3.9.0
(was: 3.8.0)
> Flaky Test DynamicBrokerReconfigurationTest#testUncleanLeaderElectionEnable
> ---
>
> Key: KAFKA-8280
> URL: https://issues.apache.org/jira/browse/KAFKA-8280
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 2.3.0
>Reporter: John Roesler
>Priority: Blocker
> Labels: flakey
> Fix For: 3.9.0
>
>
> I saw this fail again on
> https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/3979/testReport/junit/kafka.server/DynamicBrokerReconfigurationTest/testUncleanLeaderElectionEnable/
> {noformat}
> Error Message
> java.lang.AssertionError: Unclean leader not elected
> Stacktrace
> java.lang.AssertionError: Unclean leader not elected
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> kafka.server.DynamicBrokerReconfigurationTest.testUncleanLeaderElectionEnable(DynamicBrokerReconfigurationTest.scala:510)
> {noformat}
> {noformat}
> Standard Output
> Completed Updating config for entity: brokers '0'.
> Completed Updating config for entity: brokers '1'.
> Completed Updating config for entity: brokers '2'.
> [2019-04-23 01:17:11,690] ERROR [ReplicaFetcher replicaId=1, leaderId=0,
> fetcherId=1] Error for partition testtopic-6 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-23 01:17:11,690] ERROR [ReplicaFetcher replicaId=1, leaderId=0,
> fetcherId=1] Error for partition testtopic-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-23 01:17:11,722] ERROR [ReplicaFetcher replicaId=2, leaderId=1,
> fetcherId=0] Error for partition testtopic-7 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-23 01:17:11,722] ERROR [ReplicaFetcher replicaId=2, leaderId=1,
> fetcherId=0] Error for partition testtopic-1 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-23 01:17:11,760] ERROR [ReplicaFetcher replicaId=2, leaderId=0,
> fetcherId=1] Error for partition testtopic-6 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-23 01:17:11,761] ERROR [ReplicaFetcher replicaId=2, leaderId=0,
> fetcherId=1] Error for partition testtopic-0 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-23 01:17:11,765] ERROR [ReplicaFetcher replicaId=2, leaderId=0,
> fetcherId=0] Error for partition testtopic-3 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-23 01:17:11,765] ERROR [ReplicaFetcher replicaId=2, leaderId=0,
> fetcherId=0] Error for partition testtopic-9 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-23 01:17:13,779] ERROR [ReplicaFetcher replicaId=1, leaderId=0,
> fetcherId=1] Error for partition __consumer_offsets-8 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-23 01:17:13,780] ERROR [ReplicaFetcher replicaId=1, leaderId=0,
> fetcherId=1] Error for partition __consumer_offsets-38 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-23 01:17:13,780] ERROR [ReplicaFetcher replicaId=1, leaderId=0,
> fetcherId=1] Error for partition __consumer_offsets-2 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> [2019-04-23 01:17:13,780] ERROR [ReplicaFetcher replicaId=1, leaderId=0,
> fetcherId=1] Error for partition __consumer_offsets-14 at offset 0
> (kafka.server.ReplicaFetcherThread:76)
> org.apache.kafka.common.
[jira] [Commented] (KAFKA-16373) Docker Official Image for Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-16373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1789#comment-1789 ] Josep Prat commented on KAFKA-16373: Hi [~krishvora] Is this completed, or are there more PRs that need to be done before closing this issue? > Docker Official Image for Apache Kafka > -- > > Key: KAFKA-16373 > URL: https://issues.apache.org/jira/browse/KAFKA-16373 > Project: Kafka > Issue Type: New Feature >Affects Versions: 3.8.0 >Reporter: Krish Vora >Assignee: Krish Vora >Priority: Major > Labels: KIP-1028 > > KIP-1028: Docker Official Image for Apache Kafka: > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Image+for+Apache+Kafka] > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-15857) Introduce LocalLogStartOffset and TieredOffset in OffsetSpec.
[ https://issues.apache.org/jira/browse/KAFKA-15857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855560#comment-17855560 ] Josep Prat commented on KAFKA-15857: Hi [~christo_lolov], I'll change the target fix version to 3.9 as it's not a blocker and not yet finished. Would you need to rollback anything, or can it stay as it is? > Introduce LocalLogStartOffset and TieredOffset in OffsetSpec. > - > > Key: KAFKA-15857 > URL: https://issues.apache.org/jira/browse/KAFKA-15857 > Project: Kafka > Issue Type: Improvement > Components: core >Reporter: Satish Duggana >Assignee: Christo Lolov >Priority: Major > Labels: need-kip, tiered-storage > Fix For: 3.8.0 > > > Introduce EarliestLocalOffset and TieredOffset in OffsetSpec which will help > in finding respective offsets while using AdminClient#listOffsets(). > EarliestLocalOffset - local log start offset of a topic partition. > TieredOffset - Highest offset up to which the segments were copied to remote > storage. > We can discuss further on naming and semantics of these offset specs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-15857) Introduce LocalLogStartOffset and TieredOffset in OffsetSpec.
[ https://issues.apache.org/jira/browse/KAFKA-15857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josep Prat updated KAFKA-15857: --- Fix Version/s: 3.9.0 (was: 3.8.0) > Introduce LocalLogStartOffset and TieredOffset in OffsetSpec. > - > > Key: KAFKA-15857 > URL: https://issues.apache.org/jira/browse/KAFKA-15857 > Project: Kafka > Issue Type: Improvement > Components: core >Reporter: Satish Duggana >Assignee: Christo Lolov >Priority: Major > Labels: need-kip, tiered-storage > Fix For: 3.9.0 > > > Introduce EarliestLocalOffset and TieredOffset in OffsetSpec which will help > in finding respective offsets while using AdminClient#listOffsets(). > EarliestLocalOffset - local log start offset of a topic partition. > TieredOffset - Highest offset up to which the segments were copied to remote > storage. > We can discuss further on naming and semantics of these offset specs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-15923) Implement changes proposed in KIP-994
[ https://issues.apache.org/jira/browse/KAFKA-15923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josep Prat updated KAFKA-15923: --- Fix Version/s: 3.9.0 > Implement changes proposed in KIP-994 > - > > Key: KAFKA-15923 > URL: https://issues.apache.org/jira/browse/KAFKA-15923 > Project: Kafka > Issue Type: Task >Reporter: Raman Verma >Assignee: Raman Verma >Priority: Minor > Fix For: 3.9.0 > > > 1. Introduce durationFilter to ListTransactionsRequest as proposed in KIP-994 > Add DurationFilter to ListTransactionsRequest and make corresponding broker > side changes. Also, make changes to `kafka-transactions.sh --list` tooling to > use this new field in the API > 2. Introduce TransactionLastUpdateTimeMs tagged field to > DescribeTransactionsResponse. Make broker side changes to send this bit of > information. Also, make changes to `kafka-transactions.sh --describe` tooling > to display this new piece of information to the output. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-15265) Remote copy/fetch quotas for tiered storage.
[ https://issues.apache.org/jira/browse/KAFKA-15265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josep Prat updated KAFKA-15265: --- Fix Version/s: 3.9.0 > Remote copy/fetch quotas for tiered storage. > > > Key: KAFKA-15265 > URL: https://issues.apache.org/jira/browse/KAFKA-15265 > Project: Kafka > Issue Type: Improvement > Components: core >Reporter: Satish Duggana >Assignee: Abhijeet Kumar >Priority: Major > Fix For: 3.9.0 > > > Related KIP: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-956+Tiered+Storage+Quotas -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-15265) Remote copy/fetch quotas for tiered storage.
[ https://issues.apache.org/jira/browse/KAFKA-15265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855564#comment-17855564 ] Josep Prat commented on KAFKA-15265: Hi [~abhijeetkumar] The way I understand this, the KIP is not completed yet, right? > Remote copy/fetch quotas for tiered storage. > > > Key: KAFKA-15265 > URL: https://issues.apache.org/jira/browse/KAFKA-15265 > Project: Kafka > Issue Type: Improvement > Components: core >Reporter: Satish Duggana >Assignee: Abhijeet Kumar >Priority: Major > > Related KIP: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-956+Tiered+Storage+Quotas -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] KAFKA-16954: fix consumer close to release assignment in background [kafka]
lucasbru commented on PR #16343: URL: https://github.com/apache/kafka/pull/16343#issuecomment-2172877792 @lianetm The general idea looks good to me, but we have a lot of failing integration tests with this change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] KAFKA-16725: Adjust share group configs to match KIP [kafka]
AndrewJSchofield opened a new pull request, #16368: URL: https://github.com/apache/kafka/pull/16368 A few of the share group configs in KIP-932 were defined with limits that do not match KIP-932. This PR corrects the limits. ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-15045: (KIP-924) Extend TaskAssignorIntegrationTest with new public TaskAssignor [kafka]
cadonna commented on PR #16349: URL: https://github.com/apache/kafka/pull/16349#issuecomment-2172907249 Should this still go into `3.8`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16673: Simplify `GroupMetadataManager#toTopicPartitions` by using `ConsumerProtocolSubscription` instead of `ConsumerPartitionAssignor.Subscription` [kafka]
dajac merged PR #16309: URL: https://github.com/apache/kafka/pull/16309 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (KAFKA-16673) Optimize toTopicPartitions with ConsumerProtocolSubscription
[ https://issues.apache.org/jira/browse/KAFKA-16673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16673. - Fix Version/s: 3.9.0 Resolution: Fixed > Optimize toTopicPartitions with ConsumerProtocolSubscription > > > Key: KAFKA-16673 > URL: https://issues.apache.org/jira/browse/KAFKA-16673 > Project: Kafka > Issue Type: Sub-task >Reporter: Dongnuo Lyu >Assignee: Dongnuo Lyu >Priority: Major > Fix For: 3.9.0 > > > https://github.com/apache/kafka/pull/15798#discussion_r1582981154 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-15265) Remote copy/fetch quotas for tiered storage.
[ https://issues.apache.org/jira/browse/KAFKA-15265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855570#comment-17855570 ] Abhijeet Kumar commented on KAFKA-15265: Hi [~jlprat] the core functionality for the KIP is done and changes are merged. There are a couple of PRs for adding metrics that are being reviewed and another PR for updating the documentation is coming soon. > Remote copy/fetch quotas for tiered storage. > > > Key: KAFKA-15265 > URL: https://issues.apache.org/jira/browse/KAFKA-15265 > Project: Kafka > Issue Type: Improvement > Components: core >Reporter: Satish Duggana >Assignee: Abhijeet Kumar >Priority: Major > Fix For: 3.9.0 > > > Related KIP: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-956+Tiered+Storage+Quotas -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] KAFKA-16970: Fix hash implementation of `ScramCredentialValue`, `ScramCredentialData`, and `ContextualRecord` [kafka]
chia7712 commented on code in PR #16359:
URL: https://github.com/apache/kafka/pull/16359#discussion_r1642536753
##
streams/src/main/java/org/apache/kafka/streams/state/internals/ContextualRecord.java:
##
@@ -64,9 +64,14 @@ public boolean equals(final Object o) {
Objects.equals(recordContext, that.recordContext);
}
+/**
+ * See {@link ProcessorRecordContext#hashCode()}
+ */
+@Deprecated
@Override
public int hashCode() {
-return Objects.hash(value, recordContext);
Review Comment:
This needs a UT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (KAFKA-15265) Remote copy/fetch quotas for tiered storage.
[ https://issues.apache.org/jira/browse/KAFKA-15265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855574#comment-17855574 ] Josep Prat commented on KAFKA-15265: As we are past code freeze, then I'd say the KIP is not completely done in 3.8. Only blockers should be added now to the 3.8 branch. Would you agree with this assessment? > Remote copy/fetch quotas for tiered storage. > > > Key: KAFKA-15265 > URL: https://issues.apache.org/jira/browse/KAFKA-15265 > Project: Kafka > Issue Type: Improvement > Components: core >Reporter: Satish Duggana >Assignee: Abhijeet Kumar >Priority: Major > Fix For: 3.9.0 > > > Related KIP: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-956+Tiered+Storage+Quotas -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] KAFKA-15265: Reapply dynamic remote configs after broker restart [kafka]
showuon commented on code in PR #16353: URL: https://github.com/apache/kafka/pull/16353#discussion_r1642569554 ## core/src/main/java/kafka/log/remote/RemoteLogManager.java: ## @@ -226,7 +227,8 @@ public RemoteLogManager(RemoteLogManagerConfig rlmConfig, rlmCopyQuotaManager = createRLMCopyQuotaManager(); rlmFetchQuotaManager = createRLMFetchQuotaManager(); -indexCache = new RemoteIndexCache(rlmConfig.remoteLogIndexFileCacheTotalSizeBytes(), remoteLogStorageManager, logDir); +RemoteLogManagerConfig rlmConfig = config.remoteLogManagerConfig(); +indexCache = new RemoteIndexCache(config.remoteLogIndexFileCacheTotalSizeBytes(), remoteLogStorageManager, logDir); Review Comment: Yes, I confirmed in KRaft, it won't have this issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-15751, KAFKA-15752: Enable KRaft for BaseAdminIntegrationTest and SaslSslAdminIntegrationTest [kafka]
tinaselenge commented on code in PR #15175:
URL: https://github.com/apache/kafka/pull/15175#discussion_r1642582379
##
core/src/test/scala/integration/kafka/api/SslAdminIntegrationTest.scala:
##
@@ -42,6 +42,7 @@ object SslAdminIntegrationTest {
@volatile var lastUpdateRequestContext: Option[AuthorizableRequestContext] =
None
Review Comment:
thanks @mimaison !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] KAFKA-16480: Bump ListOffsets version, IBP version and mark last version of ListOffsets as unstable [kafka]
clolov commented on code in PR #15673:
URL: https://github.com/apache/kafka/pull/15673#discussion_r1642582634
##
server-common/src/main/java/org/apache/kafka/server/common/TestFeatureVersion.java:
##
@@ -21,10 +21,10 @@
public enum TestFeatureVersion implements FeatureVersion {
-// TEST_1 released right before MV 3.7-IVO was released, and it has no
dependencies
+// TEST_1 released right before MV 3.7-IV0 was released, and it has no
dependencies
TEST_1(1, MetadataVersion.IBP_3_7_IV0, Collections.emptyMap()),
-// TEST_2 released right before MV 3.8-IVO was released, and it depends on
this metadata version
-TEST_2(2, MetadataVersion.IBP_3_8_IV0,
Collections.singletonMap(MetadataVersion.FEATURE_NAME,
MetadataVersion.IBP_3_8_IV0.featureLevel()));
+// TEST_2 released right before MV 4.0-IV0 was released, and it depends on
this metadata version
+TEST_2(2, MetadataVersion.IBP_4_0_IV0,
Collections.singletonMap(MetadataVersion.FEATURE_NAME,
MetadataVersion.IBP_4_0_IV0.featureLevel()));
Review Comment:
I put this to 3.8-IV1, but since 3.8-IV1 is now marked as a production
version a test (testLatestFeaturesWithOldMetadataVersion) fails with a message
like
```
test.feature.version could not be set to 2 because it depends on
metadata.version level 21
```
And two tests (testUnstableTestVersion, testUnstableFeatureThrowsError) fail
with
```
Expected java.lang.IllegalArgumentException to be thrown, but nothing was
thrown.
```
As such, I have a preference that we keep this as IBP_4_0_IV0 which appears
to exercise all scenarios successfully.
What is your opinion @jolshan? Do you want a stable version to be exercised
as part of TEST_2 right now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] KAFKA-16480: Bump ListOffsets version, IBP version and mark last version of ListOffsets as unstable [kafka]
clolov commented on PR #15673: URL: https://github.com/apache/kafka/pull/15673#issuecomment-2172995019 The latest test failures, in my opinion, are unrelated to the change -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16921: Migrate test of connect module to Junit5 (Runtime direct) [kafka]
gongxuanzhang commented on PR #16351: URL: https://github.com/apache/kafka/pull/16351#issuecomment-2172997846 I have fixed all the test cases that failed due to the changes. @chia7712 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (KAFKA-16932) Documentation changes for Native Docker image
[ https://issues.apache.org/jira/browse/KAFKA-16932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krishna Agarwal resolved KAFKA-16932. - Resolution: Done > Documentation changes for Native Docker image > - > > Key: KAFKA-16932 > URL: https://issues.apache.org/jira/browse/KAFKA-16932 > Project: Kafka > Issue Type: Task >Reporter: Krishna Agarwal >Assignee: Krishna Agarwal >Priority: Major > Labels: KIP-974 > > KIP-974 intends to release Docker image for the Native Apache Kafka. > Native Apache Kafka will be an executable binary built using using graalVM > Native-Image. The existing system tests should run on this binary as well to > ensure its functionality. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-15444) KIP-974: Docker Image for GraalVM based Native Kafka Broker
[ https://issues.apache.org/jira/browse/KAFKA-15444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krishna Agarwal updated KAFKA-15444: Fix Version/s: 3.8.0 > KIP-974: Docker Image for GraalVM based Native Kafka Broker > --- > > Key: KAFKA-15444 > URL: https://issues.apache.org/jira/browse/KAFKA-15444 > Project: Kafka > Issue Type: New Feature >Reporter: Krishna Agarwal >Assignee: Krishna Agarwal >Priority: Major > Labels: KIP-974 > Fix For: 3.8.0 > > > [KIP-974: Docker Image for GraalVM based Native Kafka > Broker|https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15444) KIP-974: Docker Image for GraalVM based Native Kafka Broker
[ https://issues.apache.org/jira/browse/KAFKA-15444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krishna Agarwal resolved KAFKA-15444. - Resolution: Done > KIP-974: Docker Image for GraalVM based Native Kafka Broker > --- > > Key: KAFKA-15444 > URL: https://issues.apache.org/jira/browse/KAFKA-15444 > Project: Kafka > Issue Type: New Feature >Reporter: Krishna Agarwal >Assignee: Krishna Agarwal >Priority: Major > Labels: KIP-974 > Fix For: 3.8.0 > > > [KIP-974: Docker Image for GraalVM based Native Kafka > Broker|https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16827) Integrate Native Apache Kafka with System tests
[ https://issues.apache.org/jira/browse/KAFKA-16827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krishna Agarwal updated KAFKA-16827: Fix Version/s: 3.8.0 > Integrate Native Apache Kafka with System tests > --- > > Key: KAFKA-16827 > URL: https://issues.apache.org/jira/browse/KAFKA-16827 > Project: Kafka > Issue Type: Task >Reporter: Krishna Agarwal >Assignee: Krishna Agarwal >Priority: Major > Labels: KIP-974 > Fix For: 3.8.0 > > > KIP-974 intends to release Docker image for the Native Apache Kafka. > Native Apache Kafka will be an executable binary built using using graalVM > Native-Image. The existing system tests should run on this binary as well to > ensure its functionality. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16932) Documentation changes for Native Docker image
[ https://issues.apache.org/jira/browse/KAFKA-16932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krishna Agarwal updated KAFKA-16932: Fix Version/s: 3.8.0 > Documentation changes for Native Docker image > - > > Key: KAFKA-16932 > URL: https://issues.apache.org/jira/browse/KAFKA-16932 > Project: Kafka > Issue Type: Task >Reporter: Krishna Agarwal >Assignee: Krishna Agarwal >Priority: Major > Labels: KIP-974 > Fix For: 3.8.0 > > > KIP-974 intends to release Docker image for the Native Apache Kafka. > Native Apache Kafka will be an executable binary built using using graalVM > Native-Image. The existing system tests should run on this binary as well to > ensure its functionality. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16713) Add new RPC definitions
[ https://issues.apache.org/jira/browse/KAFKA-16713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield updated KAFKA-16713: - Fix Version/s: 3.9.0 > Add new RPC definitions > --- > > Key: KAFKA-16713 > URL: https://issues.apache.org/jira/browse/KAFKA-16713 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > Fix For: 3.9.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16721) Add exceptions for the new error codes
[ https://issues.apache.org/jira/browse/KAFKA-16721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield updated KAFKA-16721: - Fix Version/s: 3.9.0 > Add exceptions for the new error codes > -- > > Key: KAFKA-16721 > URL: https://issues.apache.org/jira/browse/KAFKA-16721 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > Fix For: 3.9.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16724) Add new options for kafka-producer-perf-test.sh
[ https://issues.apache.org/jira/browse/KAFKA-16724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield updated KAFKA-16724: - Fix Version/s: 3.9.0 > Add new options for kafka-producer-perf-test.sh > --- > > Key: KAFKA-16724 > URL: https://issues.apache.org/jira/browse/KAFKA-16724 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Shivsundar R >Priority: Major > Fix For: 3.9.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] KAFKA-15265: Reapply dynamic remote configs after broker restart [kafka]
kamalcph commented on PR #16353: URL: https://github.com/apache/kafka/pull/16353#issuecomment-2173039397 > LGTM! Could you mention that this issue only happened in ZK mode in the PR description? Thanks. done. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16970: Fix hash implementation of `ScramCredentialValue`, `ScramCredentialData`, and `ContextualRecord` [kafka]
frankvicky commented on code in PR #16359:
URL: https://github.com/apache/kafka/pull/16359#discussion_r1642612546
##
streams/src/main/java/org/apache/kafka/streams/state/internals/ContextualRecord.java:
##
@@ -64,9 +64,14 @@ public boolean equals(final Object o) {
Objects.equals(recordContext, that.recordContext);
}
+/**
+ * See {@link ProcessorRecordContext#hashCode()}
+ */
+@Deprecated
@Override
public int hashCode() {
-return Objects.hash(value, recordContext);
Review Comment:
Ok, I will add a test for it 😃
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (KAFKA-16211) Inconsistent config values in CreateTopicsResult and DescribeConfigsResult
[ https://issues.apache.org/jira/browse/KAFKA-16211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855584#comment-17855584 ] Gantigmaa Selenge commented on KAFKA-16211: --- This issue also causes a KRaft test to fail (e.g. SaslSslAdminIntegrationTest.testCreateTopicsResponseMetadataAndConfig), if brokers and controllers have set with different configuration values, because they compare the configurations returned from create topic and describe topic requests. As a workaround, brokers and controllers have the same configuration [values|https://github.com/apache/kafka/pull/15175/files#diff-4ffa9190a8da4f602f2022e81c87bf041e79655a8ff2d5be673cd8238eced132R369] set. > Inconsistent config values in CreateTopicsResult and DescribeConfigsResult > -- > > Key: KAFKA-16211 > URL: https://issues.apache.org/jira/browse/KAFKA-16211 > Project: Kafka > Issue Type: Bug > Components: controller >Reporter: Gantigmaa Selenge >Assignee: Dung Ha >Priority: Minor > > When creating a topic in KRaft cluster, a config value returned in > CreateTopicsResult is different than what you get from describe topic > configs, if the config was set in broker.properties or controller.properties > or in both but with different values. > > For example, start a broker with `segment.bytes` set to 573741824 in the > properties file and then create a topic, the CreateTopicsResult contains: > ConfigEntry(name=segment.bytes, value=1073741824, source=DEFAULT_CONFIG, > isSensitive=false, isReadOnly=false, synonyms=[], type=INT, > documentation=null) > because the controller was started without setting this config. > However when you describe configurations for the same topic, the config value > set by the broker is returned: > Create topic configsConfigEntry(name=segment.bytes, value=573741824, > source=STATIC_BROKER_CONFIG, isSensitive=false, isReadOnly=false, > synonyms=[], type=null, documentation=null) > > Vice versa, if the controller is started with this config set to a different > value, the create topic request returns the value set by the controller and > then when you describe the config for the same topic, you get the value set > by the broker. This makes it confusing to understand which value being is > used. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16976) Improve the dynamic config handling for RemoteLogManagerConfig when a broker is restarted.
Satish Duggana created KAFKA-16976: -- Summary: Improve the dynamic config handling for RemoteLogManagerConfig when a broker is restarted. Key: KAFKA-16976 URL: https://issues.apache.org/jira/browse/KAFKA-16976 Project: Kafka Issue Type: Task Reporter: Satish Duggana Assignee: Kamal Chandraprakash Fix For: 3.9.0 This is a followup on the discussion: https://github.com/apache/kafka/pull/16353#pullrequestreview-2121953295 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] KAFKA-15265: Reapply dynamic remote configs after broker restart [kafka]
satishd commented on PR #16353: URL: https://github.com/apache/kafka/pull/16353#issuecomment-2173081649 >This require a good amount of refactoring in RemoteLogManagerConfig class. Shall we continue with this patch to backport it to 3.8 branch? Or, do we have to refactor the code and land the changes only in trunk/3.9 branch? We can make the suggested [improvements](https://github.com/apache/kafka/pull/16353#pullrequestreview-2121953295) as a followup in trunk. Filed a followup https://issues.apache.org/jira/browse/KAFKA-16976. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16803: Update ShadowJavaPlugin [kafka]
divijvaidya commented on PR #16295: URL: https://github.com/apache/kafka/pull/16295#issuecomment-2173096690 I am not a big fan of taking a dependency on a project maintained by a single developer as it's a huge supply chain risk (cue: https://xkcd.com/2347/)  Our dependency on zstd-jni falls in the same category which makes me nervous. Nevertheless, coming back to the topic, I agree that releases on https://github.com/johnrengelman/shadow seems to have stalled and hence, it makes sense to use an alternative in the short term. Using https://github.com/Goooler/shadow sounds good to me. Alternatively, can we get rid of usage of the shadow plugin? I observe that the use case for this plugin is to create a fat jar for `jmh-benchmarks`. I haven't explored this option, but from a quick look, it looks like we could use a few lines of vanilla gradle to achieve the same result (instead of relying on a dependency)? (see https://www.baeldung.com/gradle-fat-jar) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (KAFKA-16977) remote log manager dynamic configs are not available after broker restart.
Satish Duggana created KAFKA-16977: -- Summary: remote log manager dynamic configs are not available after broker restart. Key: KAFKA-16977 URL: https://issues.apache.org/jira/browse/KAFKA-16977 Project: Kafka Issue Type: Bug Reporter: Satish Duggana The below remote log configs can be configured dynamically: remote.log.manager.copy.max.bytes.per.second remote.log.manager.fetch.max.bytes.per.second and remote.log.index.file.cache.total.size.bytes If those values are dynamically configured, after the broker restart, it loads the static value from the config file instead of the dynamic value. Note that the issue happens only when running the server with ZooKeeper. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] KAFKA-16970: Fix hash implementation of `ScramCredentialValue`, `ScramCredentialData`, and `ContextualRecord` [kafka]
frankvicky commented on PR #16359: URL: https://github.com/apache/kafka/pull/16359#issuecomment-2173157896 Hi @chia7712, I have add some test cases for `ContextualRecord`, PTAL 😸 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16977: Reapply dynamic remote configs after broker restart [kafka]
chia7712 commented on code in PR #16353: URL: https://github.com/apache/kafka/pull/16353#discussion_r1642679844 ## core/src/main/java/kafka/log/remote/RemoteLogManager.java: ## @@ -226,7 +227,8 @@ public RemoteLogManager(RemoteLogManagerConfig rlmConfig, rlmCopyQuotaManager = createRLMCopyQuotaManager(); rlmFetchQuotaManager = createRLMFetchQuotaManager(); -indexCache = new RemoteIndexCache(rlmConfig.remoteLogIndexFileCacheTotalSizeBytes(), remoteLogStorageManager, logDir); +RemoteLogManagerConfig rlmConfig = config.remoteLogManagerConfig(); +indexCache = new RemoteIndexCache(config.remoteLogIndexFileCacheTotalSizeBytes(), remoteLogStorageManager, logDir); Review Comment: > Yes, I confirmed in KRaft, it won't have this issue. Sorry that I'm not sure which issue you confirmed. If we are taking about dynamic configs in starting. According to above comments, it seems to me this fix which tries to return latest (dynamic) configs works well only if kafka is in zk. In kraft, this fix is no-op as it still return static configs. Please correct me If I'm lost -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16211: Inconsistent config values in CreateTopicsResult and DescribeConfigsResult [kafka]
chia7712 commented on PR #15696: URL: https://github.com/apache/kafka/pull/15696#issuecomment-2173162636 @infantlikesprogramming any update? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16977: Reapply dynamic remote configs after broker restart [kafka]
showuon commented on code in PR #16353: URL: https://github.com/apache/kafka/pull/16353#discussion_r1642688281 ## core/src/main/java/kafka/log/remote/RemoteLogManager.java: ## @@ -226,7 +227,8 @@ public RemoteLogManager(RemoteLogManagerConfig rlmConfig, rlmCopyQuotaManager = createRLMCopyQuotaManager(); rlmFetchQuotaManager = createRLMFetchQuotaManager(); -indexCache = new RemoteIndexCache(rlmConfig.remoteLogIndexFileCacheTotalSizeBytes(), remoteLogStorageManager, logDir); +RemoteLogManagerConfig rlmConfig = config.remoteLogManagerConfig(); +indexCache = new RemoteIndexCache(config.remoteLogIndexFileCacheTotalSizeBytes(), remoteLogStorageManager, logDir); Review Comment: @chia7712 , yes, you're right! To KRaft, this fix is no-op. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-10787: Apply spotless to `stream:test-utils` and `:streams:upgrade-system-tests-xxxx` [kafka]
chia7712 merged PR #16357: URL: https://github.com/apache/kafka/pull/16357 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KAFKA-16373) Docker Official Image for Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-16373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855594#comment-17855594 ] Krish Vora commented on KAFKA-16373: Hi [~jlprat] , the PRs for this are raised. We will be adding a few more documentation PRs once we have a Docker Official Image release. As of now, we can close this Jira ticket, and raise a new ticket - for DOI for 3.7.0 . > Docker Official Image for Apache Kafka > -- > > Key: KAFKA-16373 > URL: https://issues.apache.org/jira/browse/KAFKA-16373 > Project: Kafka > Issue Type: New Feature >Affects Versions: 3.8.0 >Reporter: Krish Vora >Assignee: Krish Vora >Priority: Major > Labels: KIP-1028 > > KIP-1028: Docker Official Image for Apache Kafka: > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Image+for+Apache+Kafka] > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16978) Apache Kafka 3.7.0 release
Krish Vora created KAFKA-16978: -- Summary: Apache Kafka 3.7.0 release Key: KAFKA-16978 URL: https://issues.apache.org/jira/browse/KAFKA-16978 Project: Kafka Issue Type: Task Reporter: Krish Vora -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-16373) Docker Official Image for Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-16373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855594#comment-17855594 ] Krish Vora edited comment on KAFKA-16373 at 6/17/24 11:54 AM: -- Hi [~jlprat] , the PRs for this are raised. We will be adding a few more documentation PRs once we have a Docker Official Image release. As of now, we can close this Jira ticket, and raise a new ticket - for DOI for 3.7.0 . [Placeholder to add link to new ticket] was (Author: JIRAUSER304579): Hi [~jlprat] , the PRs for this are raised. We will be adding a few more documentation PRs once we have a Docker Official Image release. As of now, we can close this Jira ticket, and raise a new ticket - for DOI for 3.7.0 . [Placeholder to add link to new ticker] > Docker Official Image for Apache Kafka > -- > > Key: KAFKA-16373 > URL: https://issues.apache.org/jira/browse/KAFKA-16373 > Project: Kafka > Issue Type: New Feature >Affects Versions: 3.8.0 >Reporter: Krish Vora >Assignee: Krish Vora >Priority: Major > Labels: KIP-1028 > > KIP-1028: Docker Official Image for Apache Kafka: > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Image+for+Apache+Kafka] > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-16373) Docker Official Image for Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-16373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855594#comment-17855594 ] Krish Vora edited comment on KAFKA-16373 at 6/17/24 11:54 AM: -- Hi [~jlprat] , the PRs for this are raised. We will be adding a few more documentation PRs once we have a Docker Official Image release. As of now, we can close this Jira ticket, and raise a new ticket - for DOI for 3.7.0 . [Placeholder to add link to new ticker] was (Author: JIRAUSER304579): Hi [~jlprat] , the PRs for this are raised. We will be adding a few more documentation PRs once we have a Docker Official Image release. As of now, we can close this Jira ticket, and raise a new ticket - for DOI for 3.7.0 . > Docker Official Image for Apache Kafka > -- > > Key: KAFKA-16373 > URL: https://issues.apache.org/jira/browse/KAFKA-16373 > Project: Kafka > Issue Type: New Feature >Affects Versions: 3.8.0 >Reporter: Krish Vora >Assignee: Krish Vora >Priority: Major > Labels: KIP-1028 > > KIP-1028: Docker Official Image for Apache Kafka: > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Image+for+Apache+Kafka] > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16373) Docker Official Image for Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-16373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josep Prat updated KAFKA-16373: --- Fix Version/s: 3.8.0 > Docker Official Image for Apache Kafka > -- > > Key: KAFKA-16373 > URL: https://issues.apache.org/jira/browse/KAFKA-16373 > Project: Kafka > Issue Type: New Feature >Affects Versions: 3.8.0 >Reporter: Krish Vora >Assignee: Krish Vora >Priority: Major > Labels: KIP-1028 > Fix For: 3.8.0 > > > KIP-1028: Docker Official Image for Apache Kafka: > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Image+for+Apache+Kafka] > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16373) Docker Official Image for Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-16373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josep Prat resolved KAFKA-16373. Resolution: Fixed > Docker Official Image for Apache Kafka > -- > > Key: KAFKA-16373 > URL: https://issues.apache.org/jira/browse/KAFKA-16373 > Project: Kafka > Issue Type: New Feature >Affects Versions: 3.8.0 >Reporter: Krish Vora >Assignee: Krish Vora >Priority: Major > Labels: KIP-1028 > Fix For: 3.8.0 > > > KIP-1028: Docker Official Image for Apache Kafka: > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Image+for+Apache+Kafka] > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] KAFKA-16977: Reapply dynamic remote configs after broker restart [kafka]
chia7712 commented on code in PR #16353:
URL: https://github.com/apache/kafka/pull/16353#discussion_r1642698148
##
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##
@@ -412,7 +418,7 @@ public void onLeadershipChange(Set
partitionsBecomeLeader,
Map topicIds) {
LOGGER.debug("Received leadership changes for leaders: {} and
followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
-if (this.rlmConfig.isRemoteStorageSystemEnabled() &&
!isRemoteLogManagerConfigured()) {
+if (config.remoteLogManagerConfig().isRemoteStorageSystemEnabled() &&
!isRemoteLogManagerConfigured()) {
Review Comment:
As we have great `RemoteLogManagerConfig` for remote storage, could we avoid
moving configs out of `RemoteLogManagerConfig`? Especially, we are trying to
reduce the size of `KafkaConfigs`.
If this PR aims to fix it for zk mode, maybe we can make sure the
`remoteLogManagerConfig` returned by `config` always has the latest configs.
For example:
1. `config.remoteLogManagerConfig` always create new
`remoteLogManagerConfig` based on `currentConfig`
2. `KafkaConfig#updateCurrentConfig` should update inner
`_remoteLogManagerConfig` also
@kamalcph WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (KAFKA-16979) Prepare Docker Official Image Source for 3.7.0 release
Krish Vora created KAFKA-16979:
--
Summary: Prepare Docker Official Image Source for 3.7.0 release
Key: KAFKA-16979
URL: https://issues.apache.org/jira/browse/KAFKA-16979
Project: Kafka
Issue Type: Sub-task
Affects Versions: 3.7.0
Reporter: Krish Vora
Provide the image type and kafka version to {{Docker Prepare Docker Official
Image Source}} workflow. It will generate a artifact containing the static
Dockerfile and assets for that specific version. Download the same from the
workflow.
image_type: jvm
kafka_version: 3.7.0
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-16980) Extract docker official image artifact
Krish Vora created KAFKA-16980:
--
Summary: Extract docker official image artifact
Key: KAFKA-16980
URL: https://issues.apache.org/jira/browse/KAFKA-16980
Project: Kafka
Issue Type: Sub-task
Affects Versions: 3.7.0
Reporter: Krish Vora
Run the {{docker/extract_docker_official_image_artifact.py}} script, by
providing it the path to the downloaded artifact. Ensure that this creates a
new directory under {{{}docker/docker_official_images/kafka_version{}}}.
python extract_docker_official_image_artifact.py
--path_to_downloaded_artifact=path/to/downloaded/artifact
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-16981) Remove any versions for which Docker Official Images should not be supported
Krish Vora created KAFKA-16981:
--
Summary: Remove any versions for which Docker Official Images
should not be supported
Key: KAFKA-16981
URL: https://issues.apache.org/jira/browse/KAFKA-16981
Project: Kafka
Issue Type: Sub-task
Affects Versions: 3.7.0
Reporter: Krish Vora
If there any versions for which Docker Official Images should not be supported,
remove the corresponding directories under
{{{}docker/docker_official_images{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-16981) Remove any versions for which Docker Official Images should not be supported IF ANY
[
https://issues.apache.org/jira/browse/KAFKA-16981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krish Vora updated KAFKA-16981:
---
Description:
# If there any versions for which Docker Official Images should not be
supported, remove the corresponding directories under
{{{}docker/docker_official_images{}}}.
# *Commit these changes to AK trunk.*
was:
If there any versions for which Docker Official Images should not be supported,
remove the corresponding directories under
{{{}docker/docker_official_images{}}}.
Summary: Remove any versions for which Docker Official Images should
not be supported IF ANY (was: Remove any versions for which Docker Official
Images should not be supported )
> Remove any versions for which Docker Official Images should not be supported
> IF ANY
> ---
>
> Key: KAFKA-16981
> URL: https://issues.apache.org/jira/browse/KAFKA-16981
> Project: Kafka
> Issue Type: Sub-task
>Affects Versions: 3.7.0
>Reporter: Krish Vora
>Priority: Major
>
> # If there any versions for which Docker Official Images should not be
> supported, remove the corresponding directories under
> {{{}docker/docker_official_images{}}}.
> # *Commit these changes to AK trunk.*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-16978) Apache Kafka 3.7.0 Docker Official Image release
[ https://issues.apache.org/jira/browse/KAFKA-16978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krish Vora updated KAFKA-16978: --- Summary: Apache Kafka 3.7.0 Docker Official Image release (was: Apache Kafka 3.7.0 release) > Apache Kafka 3.7.0 Docker Official Image release > > > Key: KAFKA-16978 > URL: https://issues.apache.org/jira/browse/KAFKA-16978 > Project: Kafka > Issue Type: Task >Reporter: Krish Vora >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16982) Docker Official Image Build and Test
Krish Vora created KAFKA-16982:
--
Summary: Docker Official Image Build and Test
Key: KAFKA-16982
URL: https://issues.apache.org/jira/browse/KAFKA-16982
Project: Kafka
Issue Type: Sub-task
Affects Versions: 3.7.0
Reporter: Krish Vora
Provide the image type and kafka version to {{Docker Official Image Build
Test}} workflow. It will generate a test report and CVE report that can be
shared with the community, if need be,
image_type: jvm
kafka_version: 3.7.0
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-16983) Generate the PR for Docker Official Images repo
Krish Vora created KAFKA-16983:
--
Summary: Generate the PR for Docker Official Images repo
Key: KAFKA-16983
URL: https://issues.apache.org/jira/browse/KAFKA-16983
Project: Kafka
Issue Type: Sub-task
Affects Versions: 3.7.0
Reporter: Krish Vora
Run the {{docker/generate_kafka_pr_template.py}} script from trunk, by
providing it the image type. Update the existing entry.
python generate_kafka_pr_template.py --image-type=jvm
Copy this to raise a new PR in [Docker Hub's Docker Official
Repo|https://github.com/docker-library/official-images/tree/master/library/kafka]
, which modifies the exisiting entry under ```/kafka```.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-15265) Remote copy/fetch quotas for tiered storage.
[ https://issues.apache.org/jira/browse/KAFKA-15265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855598#comment-17855598 ] Abhijeet Kumar commented on KAFKA-15265: Yes, the assessment is right. > Remote copy/fetch quotas for tiered storage. > > > Key: KAFKA-15265 > URL: https://issues.apache.org/jira/browse/KAFKA-15265 > Project: Kafka > Issue Type: Improvement > Components: core >Reporter: Satish Duggana >Assignee: Abhijeet Kumar >Priority: Major > Fix For: 3.9.0 > > > Related KIP: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-956+Tiered+Storage+Quotas -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16983) Generate the PR for Docker Official Images repo
[
https://issues.apache.org/jira/browse/KAFKA-16983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krish Vora updated KAFKA-16983:
---
Description:
Run the {{docker/generate_kafka_pr_template.py}} script from trunk, by
providing it the image type. Update the existing entry.
python generate_kafka_pr_template.py --image-type=jvm
Copy this to raise a new PR in [Docker Hub's Docker Official
Repo|https://github.com/docker-library/official-images/tree/master/library/kafka]
, which modifies the exisiting entry.
was:
Run the {{docker/generate_kafka_pr_template.py}} script from trunk, by
providing it the image type. Update the existing entry.
python generate_kafka_pr_template.py --image-type=jvm
Copy this to raise a new PR in [Docker Hub's Docker Official
Repo|https://github.com/docker-library/official-images/tree/master/library/kafka]
, which modifies the exisiting entry under ```/kafka```.
> Generate the PR for Docker Official Images repo
> ---
>
> Key: KAFKA-16983
> URL: https://issues.apache.org/jira/browse/KAFKA-16983
> Project: Kafka
> Issue Type: Sub-task
>Affects Versions: 3.7.0
>Reporter: Krish Vora
>Priority: Major
>
> Run the {{docker/generate_kafka_pr_template.py}} script from trunk, by
> providing it the image type. Update the existing entry.
> python generate_kafka_pr_template.py --image-type=jvm
>
> Copy this to raise a new PR in [Docker Hub's Docker Official
> Repo|https://github.com/docker-library/official-images/tree/master/library/kafka]
> , which modifies the exisiting entry.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (KAFKA-16980) Extract docker official image artifact
[
https://issues.apache.org/jira/browse/KAFKA-16980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krish Vora reassigned KAFKA-16980:
--
Assignee: Krish Vora
> Extract docker official image artifact
> --
>
> Key: KAFKA-16980
> URL: https://issues.apache.org/jira/browse/KAFKA-16980
> Project: Kafka
> Issue Type: Sub-task
>Affects Versions: 3.7.0
>Reporter: Krish Vora
>Assignee: Krish Vora
>Priority: Major
>
> Run the {{docker/extract_docker_official_image_artifact.py}} script, by
> providing it the path to the downloaded artifact. Ensure that this creates a
> new directory under {{{}docker/docker_official_images/kafka_version{}}}.
> python extract_docker_official_image_artifact.py
> --path_to_downloaded_artifact=path/to/downloaded/artifact
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
