[jira] [Resolved] (KAFKA-15600) KIP-990: Capability to PAUSE Tasks on DeserializationException
[
https://issues.apache.org/jira/browse/KAFKA-15600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax resolved KAFKA-15600.
-
Resolution: Won't Fix
> KIP-990: Capability to PAUSE Tasks on DeserializationException
> --
>
> Key: KAFKA-15600
> URL: https://issues.apache.org/jira/browse/KAFKA-15600
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Reporter: Nicholas Telford
>Assignee: Nicholas Telford
>Priority: Minor
> Labels: kip
>
> Presently, Kafka Streams provides users with two options for handling a
> {{DeserializationException}} via the {{DeserializationExceptionHandler}}
> interface:
> # {{FAIL}} - throw an Exception that causes the stream thread to fail. This
> will either cause the whole application instance to exit, or the stream
> thread will be replaced and restarted. Either way, the failed {{Task}} will
> end up being resumed, either by the current instance or after being
> rebalanced to another, causing a cascading failure until a user intervenes to
> address the problem.
> # {{CONTINUE}} - discard the record and continue processing with the next
> record. This can cause data loss if the record triggering the
> {{DeserializationException}} should be considered a valid record. This can
> happen if an upstream producer changes the record schema in a way that is
> incompatible with the streams application, or if there is a bug in the
> {{Deserializer}} (for example, failing to handle a valid edge-case).
> The user can currently choose between data loss, or a cascading failure that
> usually causes all processing to slowly grind to a halt.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-8088) Deprecate `WindowStoreIterator` interface
[ https://issues.apache.org/jira/browse/KAFKA-8088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-8088: --- Fix Version/s: (was: 4.0.0) > Deprecate `WindowStoreIterator` interface > - > > Key: KAFKA-8088 > URL: https://issues.apache.org/jira/browse/KAFKA-8088 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Matthias J. Sax >Assignee: Matthias J. Sax >Priority: Major > Labels: kip > > The `WindowStore` interface has multiple methods to fetch() data. However, > the return types are mixed up. Two methods return `WindowStoreIterator` while > all others return `KeyValueIterator`. > We should align the return types and replace `WindowStoreIterator` with > `KeyValueIterator`. For backward compatibility reasons we can only deprecate > the interface for now and remove it only later. > KIP-439: > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-439%3A+Deprecate+Interface+WindowStoreIterator] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16925) stream-table join does not immediately forward expired records on restart
[
https://issues.apache.org/jira/browse/KAFKA-16925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17854205#comment-17854205
]
Matthias J. Sax commented on KAFKA-16925:
-
Yes, but it's tracked on a per-task basis, and thus, it could be ahead of the
time the operator tracked.
I think a better solution might be, to allow operators to "register for time
tracking" or something like this... which would be a much larger change though.
> stream-table join does not immediately forward expired records on restart
> -
>
> Key: KAFKA-16925
> URL: https://issues.apache.org/jira/browse/KAFKA-16925
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: Ayoub Omari
>Assignee: Ayoub Omari
>Priority: Major
>
> [KIP-923|https://cwiki.apache.org/confluence/display/KAFKA/KIP-923%3A+Add+A+Grace+Period+to+Stream+Table+Join]
> introduced grace period for KStreamKTableJoin. This allows to join a stream
> to a KTable backed by a Versioned state store.
> Upon receiving a record, it is put in a buffer until grace period is elapsed.
> When the grace period elapses, the record is joined with its most recent
> match from the versioned state store.
> +Late records+ are +not+ put in the buffer and are immediately joined.
>
> {code:java}
> If the grace period is non zero, the record will enter a stream buffer and
> will dequeue when the record timestamp is less than or equal to stream time
> minus the grace period. Late records, out of the grace period, will be
> executed right as they come in. (KIP-923){code}
>
> However, this is not the case today on rebalance or restart. The reason is
> that observedStreamTime is taken from the underlying state store which looses
> this information on rebalance/restart:
> [https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimeOrderedKeyValueBuffer.java#L164]
>
> If the task restarts and receives an expired record, the buffer considers
> that this record has the maximum stream time observed so far, and puts it in
> the buffer instead of immediately joining it.
>
> {*}Example{*}:
> * Grace period = 60s
> * KTable contains (key, rightValue)
>
> +Normal scenario+
> {code:java}
> streamInput1 (key, value1) <--- time = T : put in buffer
> streamInput2 (key, value2) <--- time = T - 60s : immediately joined //
> streamTime = T{code}
>
> +Scenario with rebalance+
> {code:java}
> streamInput1 (key, value1) <--- time = T : put in buffer
> // --- rebalance ---
> streamInput2 (key, value2) <--- time = T - 60s : put in buffer // streamTime
> = T - 60s{code}
>
> The processor should use currentStreamTime from Context instead. Which is
> recovered on restart.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-14567) Kafka Streams crashes after ProducerFencedException
[
https://issues.apache.org/jira/browse/KAFKA-14567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax resolved KAFKA-14567.
-
Assignee: (was: Kirk True)
Resolution: Fixed
Not 100% sure either, but I feel good enough to close this ticket for now. If
we see it again, we can reopen or create a new ticket.
> Kafka Streams crashes after ProducerFencedException
> ---
>
> Key: KAFKA-14567
> URL: https://issues.apache.org/jira/browse/KAFKA-14567
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.7.0
>Reporter: Matthias J. Sax
>Priority: Blocker
> Labels: eos
> Fix For: 3.8.0
>
>
> Running a Kafka Streams application with EOS-v2.
> We first see a `ProducerFencedException`. After the fencing, the fenced
> thread crashed resulting in a non-recoverable error:
> {quote}[2022-12-22 13:49:13,423] ERROR [i-0c291188ec2ae17a0-StreamThread-3]
> stream-thread [i-0c291188ec2ae17a0-StreamThread-3] Failed to process stream
> task 1_2 due to the following error:
> (org.apache.kafka.streams.processor.internals.TaskExecutor)
> org.apache.kafka.streams.errors.StreamsException: Exception caught in
> process. taskId=1_2, processor=KSTREAM-SOURCE-05,
> topic=node-name-repartition, partition=2, offset=539776276,
> stacktrace=java.lang.IllegalStateException: TransactionalId
> stream-soak-test-72b6e57c-c2f5-489d-ab9f-fdbb215d2c86-3: Invalid transition
> attempted from state FATAL_ERROR to state ABORTABLE_ERROR
> at
> org.apache.kafka.clients.producer.internals.TransactionManager.transitionTo(TransactionManager.java:974)
> at
> org.apache.kafka.clients.producer.internals.TransactionManager.transitionToAbortableError(TransactionManager.java:394)
> at
> org.apache.kafka.clients.producer.internals.TransactionManager.maybeTransitionToErrorState(TransactionManager.java:620)
> at
> org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:1079)
> at
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:959)
> at
> org.apache.kafka.streams.processor.internals.StreamsProducer.send(StreamsProducer.java:257)
> at
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl.send(RecordCollectorImpl.java:207)
> at
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl.send(RecordCollectorImpl.java:162)
> at
> org.apache.kafka.streams.processor.internals.SinkNode.process(SinkNode.java:85)
> at
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forwardInternal(ProcessorContextImpl.java:290)
> at
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:269)
> at
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:228)
> at
> org.apache.kafka.streams.kstream.internals.KStreamKTableJoinProcessor.process(KStreamKTableJoinProcessor.java:88)
> at
> org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:157)
> at
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forwardInternal(ProcessorContextImpl.java:290)
> at
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:269)
> at
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:228)
> at
> org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:84)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.lambda$doProcess$1(StreamTask.java:791)
> at
> org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.maybeMeasureLatency(StreamsMetricsImpl.java:867)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.doProcess(StreamTask.java:791)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:722)
> at
> org.apache.kafka.streams.processor.internals.TaskExecutor.processTask(TaskExecutor.java:95)
> at
> org.apache.kafka.streams.processor.internals.TaskExecutor.process(TaskExecutor.java:76)
> at
> org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:1645)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:788)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:607)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:569)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:748)
> at
>
[jira] [Commented] (KAFKA-16925) stream-table join does not immediately forward expired records on restart
[
https://issues.apache.org/jira/browse/KAFKA-16925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17853815#comment-17853815
]
Matthias J. Sax commented on KAFKA-16925:
-
Thanks for filing this ticket – it's a know problem, not limited to
stream-table join.
I don't think that initializing with `context.currentStreamTime()` is the right
fix though, and would in general prefer to fix this issue across the board, to
avoid building operator specific island solutions.
> stream-table join does not immediately forward expired records on restart
> -
>
> Key: KAFKA-16925
> URL: https://issues.apache.org/jira/browse/KAFKA-16925
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: Ayoub Omari
>Assignee: Ayoub Omari
>Priority: Major
>
> [KIP-923|https://cwiki.apache.org/confluence/display/KAFKA/KIP-923%3A+Add+A+Grace+Period+to+Stream+Table+Join]
> introduced grace period for KStreamKTableJoin. This allows to join a stream
> to a KTable backed by a Versioned state store.
> Upon receiving a record, it is put in a buffer until grace period is elapsed.
> When the grace period elapses, the record is joined with its most recent
> match from the versioned state store.
> +Late records+ are +not+ put in the buffer and are immediately joined.
>
> {code:java}
> If the grace period is non zero, the record will enter a stream buffer and
> will dequeue when the record timestamp is less than or equal to stream time
> minus the grace period. Late records, out of the grace period, will be
> executed right as they come in. (KIP-923){code}
>
> However, this is not the case today on rebalance or restart. The reason is
> that observedStreamTime is taken from the underlying state store which looses
> this information on rebalance/restart:
> [https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimeOrderedKeyValueBuffer.java#L164]
>
> If the task restarts and receives an expired record, the buffer considers
> that this record has the maximum stream time observed so far, and puts it in
> the buffer instead of immediately joining it.
>
> {*}Example{*}:
> * Grace period = 60s
> * KTable contains (key, rightValue)
>
> +Normal scenario+
> {code:java}
> streamInput1 (key, value1) <--- time = T : put in buffer
> streamInput2 (key, value2) <--- time = T - 60s : immediately joined //
> streamTime = T{code}
>
> +Scenario with rebalance+
> {code:java}
> streamInput1 (key, value1) <--- time = T : put in buffer
> // --- rebalance ---
> streamInput2 (key, value2) <--- time = T - 60s : put in buffer // streamTime
> = T - 60s{code}
>
> The processor should use currentStreamTime from Context instead. Which is
> recovered on restart.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16863) Consider removing `default.` prefix for exception handler config
[ https://issues.apache.org/jira/browse/KAFKA-16863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17853293#comment-17853293 ] Matthias J. Sax commented on KAFKA-16863: - Yes it needs a KIP as the label "needs-kip" suggests :) Feel free to pick it up. You understanding of the ticket is correct. > Consider removing `default.` prefix for exception handler config > > > Key: KAFKA-16863 > URL: https://issues.apache.org/jira/browse/KAFKA-16863 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Matthias J. Sax >Priority: Trivial > Labels: need-kip > > Kafka Streams has a set of configs with `default.` prefix. The intent for the > default-prefix is to make a distinction between, well the default, and > in-place overwrites in the code. Eg, users can specify ts-extractors on a > per-topic basis. > However, for the deserialization- and production-exception handlers, no such > overwrites are possible, and thus, `default.` does not really make sense, > because there is just one handler overall. Via KIP-1033 we added a new > processing-exception handler w/o a default-prefix, too. > Thus, we should consider to deprecate the two existing configs names and add > them back w/o the `default.` prefix. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16900) kafka-producer-perf-test reports error when using transaction.
[ https://issues.apache.org/jira/browse/KAFKA-16900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-16900: Affects Version/s: (was: 2.9) > kafka-producer-perf-test reports error when using transaction. > -- > > Key: KAFKA-16900 > URL: https://issues.apache.org/jira/browse/KAFKA-16900 > Project: Kafka > Issue Type: Bug > Components: producer >Reporter: Chen He >Priority: Minor > Labels: perf-test > > [https://lists.apache.org/thread/dmrbx8kzv2w5t1v0xjvyjbp5y23omlq8] > encounter the same issue as mentioned above. > Did not found the 2.13 version in affects versions so mark it as the most > latest it provided. 2.9. Please feel free to change if possible. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-12651) Connect should register REST extensions before REST resources
[ https://issues.apache.org/jira/browse/KAFKA-12651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-12651: Affects Version/s: (was: 2.9) > Connect should register REST extensions before REST resources > - > > Key: KAFKA-12651 > URL: https://issues.apache.org/jira/browse/KAFKA-12651 > Project: Kafka > Issue Type: Bug > Components: connect >Affects Versions: 2.0.0, 2.0.1, 2.1.0, 2.2.0, 2.1.1, 2.0.2, 2.3.0, 2.1.2, > 2.2.1, 2.2.2, 2.4.0, 2.3.1, 2.2.3, 2.5.0, 2.3.2, 2.4.1, 2.6.0, 2.4.2, 2.5.1, > 2.7.0, 2.5.2, 2.6.1, 2.8.0, 2.7.1, 2.6.2, 3.1.0, 2.6.3, 2.7.2, 2.8.1, 3.0.0 >Reporter: Chris Egerton >Assignee: Chris Egerton >Priority: Major > > Connect currently registers custom REST extensions after REST resources. This > can be problematic in security-conscious environments where REST extensions > are used to lock down access to the Connect REST API, as it creates a window > of opportunity for unauthenticated access to the REST API between the time > the worker's REST resources are brought up and when its REST extensions are > registered. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-12383) Get RaftClusterTest.java and other KIP-500 junit tests working
[ https://issues.apache.org/jira/browse/KAFKA-12383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-12383: Affects Version/s: (was: 2.9) > Get RaftClusterTest.java and other KIP-500 junit tests working > -- > > Key: KAFKA-12383 > URL: https://issues.apache.org/jira/browse/KAFKA-12383 > Project: Kafka > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Colin McCabe >Assignee: David Arthur >Priority: Blocker > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14187) kafka-features.sh: add support for --metadata
[ https://issues.apache.org/jira/browse/KAFKA-14187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-14187: Affects Version/s: (was: 3.3) > kafka-features.sh: add support for --metadata > - > > Key: KAFKA-14187 > URL: https://issues.apache.org/jira/browse/KAFKA-14187 > Project: Kafka > Issue Type: Bug >Affects Versions: 3.3.0 >Reporter: Colin McCabe >Assignee: Colin McCabe >Priority: Blocker > Fix For: 3.3.0 > > > Fix the kafka-features.sh command so that we can upgrade to the new version > as expected. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14200) kafka-features.sh must exit with non-zero error code on error
[ https://issues.apache.org/jira/browse/KAFKA-14200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-14200: Affects Version/s: (was: 3.3) > kafka-features.sh must exit with non-zero error code on error > - > > Key: KAFKA-14200 > URL: https://issues.apache.org/jira/browse/KAFKA-14200 > Project: Kafka > Issue Type: Bug >Affects Versions: 3.3.0 >Reporter: Colin McCabe >Assignee: Colin McCabe >Priority: Blocker > Fix For: 3.3.0 > > > kafka-features.sh must exit with a non-zero error code on error. We must do > this in order to catch regressions like KAFKA-13990. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14197) Kraft broker fails to startup after topic creation failure
[
https://issues.apache.org/jira/browse/KAFKA-14197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-14197:
Affects Version/s: 3.3.0
(was: 3.3)
> Kraft broker fails to startup after topic creation failure
> --
>
> Key: KAFKA-14197
> URL: https://issues.apache.org/jira/browse/KAFKA-14197
> Project: Kafka
> Issue Type: Bug
> Components: kraft
>Affects Versions: 3.3.0
>Reporter: Luke Chen
>Priority: Blocker
>
> In kraft ControllerWriteEvent, we start by trying to apply the record to
> controller in-memory state, then sent out the record via raft client. But if
> there is error during sending the records, there's no way to revert the
> change to controller in-memory state[1].
> The issue happened when creating topics, controller state is updated with
> topic and partition metadata (ex: broker to ISR map), but the record doesn't
> send out successfully (ex: RecordBatchTooLargeException). Then, when shutting
> down the node, the controlled shutdown will try to remove the broker from ISR
> by[2]:
> {code:java}
> generateLeaderAndIsrUpdates("enterControlledShutdown[" + brokerId + "]",
> brokerId, NO_LEADER, records,
> brokersToIsrs.partitionsWithBrokerInIsr(brokerId));{code}
>
> After we appending the partitionChangeRecords, and send to metadata topic
> successfully, it'll cause the brokers failed to "replay" these partition
> change since these topic/partitions didn't get created successfully
> previously.
> Even worse, after restarting the node, all the metadata records will replay
> again, and the same error happened again, cause the broker cannot start up
> successfully.
>
> The error and call stack is like this, basically, it complains the topic
> image can't be found
> {code:java}
> [2022-09-02 16:29:16,334] ERROR Encountered metadata loading fault: Error
> replaying metadata log record at offset 81
> (org.apache.kafka.server.fault.LoggingFaultHandler)
> java.lang.NullPointerException
> at org.apache.kafka.image.TopicDelta.replay(TopicDelta.java:69)
> at org.apache.kafka.image.TopicsDelta.replay(TopicsDelta.java:91)
> at org.apache.kafka.image.MetadataDelta.replay(MetadataDelta.java:248)
> at org.apache.kafka.image.MetadataDelta.replay(MetadataDelta.java:186)
> at
> kafka.server.metadata.BrokerMetadataListener.$anonfun$loadBatches$3(BrokerMetadataListener.scala:239)
> at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
> at
> kafka.server.metadata.BrokerMetadataListener.kafka$server$metadata$BrokerMetadataListener$$loadBatches(BrokerMetadataListener.scala:232)
> at
> kafka.server.metadata.BrokerMetadataListener$HandleCommitsEvent.run(BrokerMetadataListener.scala:113)
> at
> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:121)
> at
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:200)
> at
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:173)
> at java.base/java.lang.Thread.run(Thread.java:829)
> {code}
>
> [1]
> [https://github.com/apache/kafka/blob/ef65b6e566ef69b2f9b58038c98a5993563d7a68/metadata/src/main/java/org/apache/kafka/controller/QuorumController.java#L779-L804]
>
> [2]
> [https://github.com/apache/kafka/blob/trunk/metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java#L1270]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-14195) Fix KRaft AlterConfig policy usage for Legacy/Full case
[ https://issues.apache.org/jira/browse/KAFKA-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-14195: Affects Version/s: 3.3.0 (was: 3.3) > Fix KRaft AlterConfig policy usage for Legacy/Full case > --- > > Key: KAFKA-14195 > URL: https://issues.apache.org/jira/browse/KAFKA-14195 > Project: Kafka > Issue Type: Bug >Affects Versions: 3.3.0 >Reporter: Ron Dagostino >Assignee: Ron Dagostino >Priority: Blocker > > The fix for https://issues.apache.org/jira/browse/KAFKA-14039 adjusted the > invocation of the alter configs policy check in KRaft to match the behavior > in ZooKeeper, which is to only provide the configs that were explicitly sent > in the request. While the code was correct for the incremental alter configs > case, the code actually included the implicit deletions for the > legacy/non-incremental alter configs case, and those implicit deletions are > not included in the ZooKeeper-based invocation. The implicit deletions should > not be passed in the legacy case. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14051) KRaft remote controllers do not create metrics reporters
[ https://issues.apache.org/jira/browse/KAFKA-14051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-14051: Affects Version/s: 3.3.0 (was: 3.3) > KRaft remote controllers do not create metrics reporters > > > Key: KAFKA-14051 > URL: https://issues.apache.org/jira/browse/KAFKA-14051 > Project: Kafka > Issue Type: Bug > Components: kraft >Affects Versions: 3.3.0 >Reporter: Ron Dagostino >Assignee: Ron Dagostino >Priority: Major > > KRaft remote controllers (KRaft nodes with the configuration value > process.roles=controller) do not create the configured metrics reporters > defined by the configuration key metric.reporters. The reason is because > KRaft remote controllers are not wired up for dynamic config changes, and the > creation of the configured metric reporters actually happens during the > wiring up of the broker for dynamic reconfiguration, in the invocation of > DynamicBrokerConfig.addReconfigurables(KafkaBroker). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14214) StandardAuthorizer may transiently process ACLs out of write order
[ https://issues.apache.org/jira/browse/KAFKA-14214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-14214: Affects Version/s: 3.3.0 (was: 3.3) > StandardAuthorizer may transiently process ACLs out of write order > -- > > Key: KAFKA-14214 > URL: https://issues.apache.org/jira/browse/KAFKA-14214 > Project: Kafka > Issue Type: Bug >Affects Versions: 3.3.0 >Reporter: Akhilesh Chaganti >Assignee: Akhilesh Chaganti >Priority: Blocker > Fix For: 3.3.0 > > > The issue with StandardAuthorizer#authorize is, that it looks up > aclsByResources (which is of type ConcurrentSkipListMap)twice for every > authorize call and uses Iterator with weak consistency guarantees on top of > aclsByResources. This can cause the authorize function call to process the > concurrent writes out of order. > *Issue 1:* > When StandardAuthorizer calls into a simple authorize function, we check the > ACLs for literal/prefix matches for the resource and then make one more call > to check the ACLs for matching wildcard entries. Between the two > (checkSection) calls, let’s assume we add a DENY for resource literal and add > an ALLOW ALL wildcard. The first call to check literal/prefix rules will SKIP > DENY ACL since the writes are not yet processed and the second call would > find ALLOW wildcard entry which results in ALLOW authorization for the > resource when it is actually DENY. > *Issue: 2* > For authorization, StandardAuthorizer depends on an iterator that iterates > through the ordered set of ACLs. The iterator has weak consistency > guarantees. So when writes for two ACLs occur, one of the ACLs might be still > visible to the iterator while the other is not. > Let’s say below two ACLS are getting added in the following order to the set. > Acl1 = StandardAcl(TOPIC, foobar, LITERAL, DENY, READ, user1) > Acl2 = StandardAcl(TOPIC, foo, PREFIX, ALLOW, READ, user1) > Depending on the position of the iterator on the ordered set during the write > call, the iterator might just see Acl2 which prompts it to ALLOW the topic to > be READ even though the DENY rule was written before. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14328) KafkaAdminClient should be Changing the exception level When an exception occurs
[ https://issues.apache.org/jira/browse/KAFKA-14328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-14328: Affects Version/s: 3.3.0 (was: 3.3) > KafkaAdminClient should be Changing the exception level When an exception > occurs > > > Key: KAFKA-14328 > URL: https://issues.apache.org/jira/browse/KAFKA-14328 > Project: Kafka > Issue Type: Improvement > Components: admin >Affects Versions: 3.3.0 >Reporter: shizhenzhen >Assignee: shizhenzhen >Priority: Major > Attachments: image-2022-10-21-11-19-21-064.png, > image-2022-10-21-14-56-31-753.png, image-2022-10-21-16-54-40-588.png, > image-2022-10-21-16-56-45-448.png, image-2022-10-21-16-58-19-353.png, > image-2022-10-24-14-28-10-365.png, image-2022-10-24-14-47-30-641.png, > image-2022-10-24-14-48-27-907.png > > > > > KafkaAdminClient 的一些日志全部是 log.trace. 当遇到异常的时候根本不知道什么原因，导致排查问题非常艰难。 > > 就比如下面这里，当去请求Metadata请求的时候，如果查询到的Topic有分区Leader=-1的时候，就会抛出异常； > > 但是这个时候实际上异常是被吞掉了的，这里往上面抛出异常之后，到了下面第二张图的 Catch部分。 > 他会把这个请求重新放到到请求队列中。然后就会陷入无限读重试之后，直到达到超时时间抛出异常：Timed out waiting for a node > assignment. Call: metadata > > 无法给Metadata请求分配节点，正常情况下谁知道他真正的异常其实是 > > ``` > org.apache.kafka.common.errors.LeaderNotAvailableException: There is no > leader for this topic-partition as we are in the middle of a leadership > election. > > ``` > > > > > !https://user-images.githubusercontent.com/10442648/196944422-e11b732f-6f7f-4f77-8d9c-1f0544257461.png! > > > > 下面截图那里是我改成的warn基本的日志 > !image-2022-10-21-11-19-21-064.png! > > 所以我希望这里的log.trace 能改成 log.warn ; 给一个提醒。 > 就可以说明当前因为某个异常的原因而导致可能的重试。 > > > > > > !image-2022-10-21-14-56-31-753.png! > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16300) Wrong documentation for producer config retries
[ https://issues.apache.org/jira/browse/KAFKA-16300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-16300: Affects Version/s: 3.3.0 (was: 3.3) > Wrong documentation for producer config retries > --- > > Key: KAFKA-16300 > URL: https://issues.apache.org/jira/browse/KAFKA-16300 > Project: Kafka > Issue Type: Bug > Components: documentation >Affects Versions: 3.3.0, 3.1.2, 3.2.3, 3.4.1 >Reporter: Fede >Priority: Minor > Labels: documentation > > In documentation from version 3.1 to version 3.4, it looks like the retries > explanation has a bug related to max.in.flight.request.per.connection related > parameter and possible message reordering. > [https://kafka.apache.org/31/documentation.html#producerconfigs_retries] > [https://kafka.apache.org/32/documentation.html#producerconfigs_retries] > [https://kafka.apache.org/33/documentation.html#producerconfigs_retries] > [https://kafka.apache.org/34/documentation.html#producerconfigs_retries] > > in particular, the section > Allowing retries while setting enable.idempotence to false and > max.in.flight.requests.per.connection to 1 will potentially change the > ordering of records because if two batches are sent to a single partition, > and the first fails and is retried but the second succeeds, then the records > in the second batch may appear first. > > Is states > max.in.flight.requests.per.connection to 1 > > It should be said > max.in.flight.requests.per.connection to *greater than* 1 > > This bug has been fixed in the latest versions, but it still confuses users > using affected versions as the meaning is the opposite of what it should be. > > I created a PR , > [https://github.com/apache/kafka/pull/15413|https://github.com/apache/kafka/pull/15413)] > For version 3.2, but the build failed. Not sure why. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-15063) Throttle number of active PIDs
[
https://issues.apache.org/jira/browse/KAFKA-15063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-15063:
Affects Version/s: (was: 3.3)
> Throttle number of active PIDs
> --
>
> Key: KAFKA-15063
> URL: https://issues.apache.org/jira/browse/KAFKA-15063
> Project: Kafka
> Issue Type: New Feature
> Components: core, producer
>Affects Versions: 2.8.0, 3.1.0, 3.0.0, 3.2.0, 3.4.0
>Reporter: Omnia Ibrahim
>Priority: Major
>
> {color:#172b4d}Ticket to track KIP-936. Since KIP-679 idempotent{color}
> {color:#172b4d}producers became the default in Kafka as a result of this all
> producer instances will be assigned PID. The increase of the number of PIDs
> stored in Kafka brokers by {color}{{ProducerStateManager}}{color:#172b4d}
> exposes the broker to OOM errors if it has a high number of producers, a
> rogue or misconfigured client(s).{color}
> {color:#172b4d}The broker is still exposed to OOM{color} even after KIP-854
> introduced a separate config to expire PID from transaction IDs if there is a
> high number of PID before {{producer.id.expiration.ms}} is exceeded.
> As a result of this, the broker will keep experiencing OOM and become
> offline. The only way to recover from this is to increase the heap.
>
> {color:#172b4d}KIP-936 is proposing throttling the number of PIDs per
> KafkaPrincipal {color}
> {color:#172b4d}See the KIP-936 details here
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-936%3A+Throttle+number+of+active+PIDs]
> {color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-14250) Exception during normal operation in MirrorSourceTask causes the task to fail instead of shutting down gracefully
[ https://issues.apache.org/jira/browse/KAFKA-14250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-14250: Affects Version/s: 3.3.0 (was: 3.3) > Exception during normal operation in MirrorSourceTask causes the task to fail > instead of shutting down gracefully > - > > Key: KAFKA-14250 > URL: https://issues.apache.org/jira/browse/KAFKA-14250 > Project: Kafka > Issue Type: Bug > Components: connect, mirrormaker >Affects Versions: 3.3.0 >Reporter: Viktor Somogyi-Vass >Assignee: Viktor Somogyi-Vass >Priority: Major > Labels: cloudera > Attachments: mm2.log > > > In MirrorSourceTask we are loading offsets for the topic partitions. At this > point, while we are fetching the partitions, it is possible for the offset > reader to be stopped by a parallel thread. Stopping the reader causes a > CancellationException to be thrown, due to KAFKA-9051. > Currently this exception is not caught in MirrorSourceTask and so the > exception propagates up and causes the task to go into FAILED state. We only > need it to go to STOPPED state so that it would be restarted later. > This can be achieved by catching the exception and stopping the task directly. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14571) ZkMetadataCache.getClusterMetadata is missing rack information in aliveNodes
[ https://issues.apache.org/jira/browse/KAFKA-14571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-14571: Affects Version/s: 3.3.0 (was: 3.3) > ZkMetadataCache.getClusterMetadata is missing rack information in aliveNodes > > > Key: KAFKA-14571 > URL: https://issues.apache.org/jira/browse/KAFKA-14571 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 3.3.0 >Reporter: Edoardo Comar >Assignee: Edoardo Comar >Priority: Minor > Fix For: 3.4.0, 3.3.3 > > > ZkMetadataCache.getClusterMetadata returns a Cluster object where the > aliveNodes are missing their rack info. > when ZkMetadataCache updates the metadataSnapshot, includes the rack in > `aliveBrokers` but not in `aliveNodes` -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-13755) Broker heartbeat event should have deadline
[ https://issues.apache.org/jira/browse/KAFKA-13755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-13755: Fix Version/s: 3.3.0 (was: 3.3) > Broker heartbeat event should have deadline > --- > > Key: KAFKA-13755 > URL: https://issues.apache.org/jira/browse/KAFKA-13755 > Project: Kafka > Issue Type: Bug > Components: controller >Reporter: David Arthur >Assignee: Colin McCabe >Priority: Minor > Labels: kip-500 > Fix For: 3.3.0 > > > When we schedule the event for processing the broker heartbeat request in > QuroumController, we do not give a deadline. This means that the event will > only be processed after all other events which do have a deadline. In the > case of the controller's queue getting filled up with deadline (i.e., > "deferred") events, we may not process the heartbeat before the broker > attempts to send another one. > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-13902) Support online metadata.version upgrades for KIP-704 in KRaft
[ https://issues.apache.org/jira/browse/KAFKA-13902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-13902: Fix Version/s: 3.3.0 (was: 3.3) > Support online metadata.version upgrades for KIP-704 in KRaft > -- > > Key: KAFKA-13902 > URL: https://issues.apache.org/jira/browse/KAFKA-13902 > Project: Kafka > Issue Type: Sub-task > Components: controller >Reporter: David Arthur >Assignee: David Arthur >Priority: Minor > Labels: kip-500 > Fix For: 3.3.0 > > > KIP-704 landed in trunk before the KIP-778 metadata.version work that is now > in progress. The KIP-704 (unclean leader recovery) code uses the usual IBP > approach for enabling the feature on startup. > Once KAFKA-13830 is merged, we will need to update the KIP-704 code to deal > with online upgrades. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14275) KRaft Controllers should crash after failing to apply any metadata record
[ https://issues.apache.org/jira/browse/KAFKA-14275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-14275: Fix Version/s: 3.6.2 3.7.0 3.5.2 3.4.1 (was: 3.3) > KRaft Controllers should crash after failing to apply any metadata record > -- > > Key: KAFKA-14275 > URL: https://issues.apache.org/jira/browse/KAFKA-14275 > Project: Kafka > Issue Type: Bug > Components: kraft >Affects Versions: 3.3.1 >Reporter: Niket Goel >Assignee: Niket Goel >Priority: Major > Fix For: 3.4.1, 3.5.2, 3.7.0, 3.6.2 > > > When replaying records on a standby controller, any error encountered will > halt further processing of that batch. Currently we log an error and allow > the controller to continue normal operation. In contrast a similar error on > the active controller causes it to halt and exit the jvm. This is > inconsistent behavior as nothing prevents a standby from eventually becoming > the active controller (even when it had skipped over a record batch). We > should halt the process in the case of a standby controller as well. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14062) OAuth client token refresh fails with SASL extensions
[
https://issues.apache.org/jira/browse/KAFKA-14062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-14062:
Fix Version/s: 3.3.0
(was: 3.3)
> OAuth client token refresh fails with SASL extensions
> -
>
> Key: KAFKA-14062
> URL: https://issues.apache.org/jira/browse/KAFKA-14062
> Project: Kafka
> Issue Type: Bug
> Components: admin, clients, consumer, producer , security
>Affects Versions: 3.1.0, 3.2.0, 3.1.1, 3.3.0, 3.3
>Reporter: Kirk True
>Assignee: Kirk True
>Priority: Major
> Labels: OAuth
> Fix For: 3.3.0, 3.1.2, 3.2.1
>
>
> While testing OAuth for Connect an issue surfaced where authentication that
> was successful initially fails during token refresh. This appears to be due
> to missing SASL extensions on refresh, though those extensions were present
> on initial authentication.
> During token refresh, the Kafka client adds and removes any SASL extensions.
> If a refresh is attempted during the window when the extensions are not
> present in the subject, the refresh fails with the following error:
> {code:java}
> [2022-04-11 20:33:43,250] INFO [AdminClient clientId=adminclient-8] Failed
> authentication with / (Authentication failed: 1 extensions are
> invalid! They are: xxx: Authentication failed)
> (org.apache.kafka.common.network.Selector){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-14062) OAuth client token refresh fails with SASL extensions
[
https://issues.apache.org/jira/browse/KAFKA-14062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-14062:
Affects Version/s: (was: 3.3)
> OAuth client token refresh fails with SASL extensions
> -
>
> Key: KAFKA-14062
> URL: https://issues.apache.org/jira/browse/KAFKA-14062
> Project: Kafka
> Issue Type: Bug
> Components: admin, clients, consumer, producer , security
>Affects Versions: 3.1.0, 3.2.0, 3.1.1, 3.3.0
>Reporter: Kirk True
>Assignee: Kirk True
>Priority: Major
> Labels: OAuth
> Fix For: 3.3.0, 3.1.2, 3.2.1
>
>
> While testing OAuth for Connect an issue surfaced where authentication that
> was successful initially fails during token refresh. This appears to be due
> to missing SASL extensions on refresh, though those extensions were present
> on initial authentication.
> During token refresh, the Kafka client adds and removes any SASL extensions.
> If a refresh is attempted during the window when the extensions are not
> present in the subject, the refresh fails with the following error:
> {code:java}
> [2022-04-11 20:33:43,250] INFO [AdminClient clientId=adminclient-8] Failed
> authentication with / (Authentication failed: 1 extensions are
> invalid! They are: xxx: Authentication failed)
> (org.apache.kafka.common.network.Selector){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-13850) kafka-metadata-shell is missing some record types
[ https://issues.apache.org/jira/browse/KAFKA-13850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-13850: Fix Version/s: 3.3.0 (was: 3.3) > kafka-metadata-shell is missing some record types > - > > Key: KAFKA-13850 > URL: https://issues.apache.org/jira/browse/KAFKA-13850 > Project: Kafka > Issue Type: Bug > Components: kraft >Reporter: David Arthur >Assignee: Deng Ziming >Priority: Major > Fix For: 3.3.0 > > > Noticed while working on feature flags in KRaft, the in-memory tree of the > metadata (MetadataNodeManager) is missing support for a few of record types. > * DelegationTokenRecord > * UserScramCredentialRecord (should we include this?) > * FeatureLevelRecord > * AccessControlEntryRecord > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-13943) Fix flaky test QuorumControllerTest.testMissingInMemorySnapshot()
[
https://issues.apache.org/jira/browse/KAFKA-13943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-13943:
Fix Version/s: (was: 3.3)
> Fix flaky test QuorumControllerTest.testMissingInMemorySnapshot()
> -
>
> Key: KAFKA-13943
> URL: https://issues.apache.org/jira/browse/KAFKA-13943
> Project: Kafka
> Issue Type: Bug
> Components: unit tests
>Reporter: Divij Vaidya
>Assignee: Divij Vaidya
>Priority: Major
> Labels: flaky-test
> Fix For: 3.3.0
>
>
> Test failed at
> [https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-12197/3/tests]
>
> {noformat}
> [2022-05-27 09:34:42,382] INFO [Controller 0] Creating new QuorumController
> with clusterId wj9LhgPJTV-KYEItgqvtQA, authorizer Optional.empty.
> (org.apache.kafka.controller.QuorumController:1484)
> [2022-05-27 09:34:42,393] DEBUG [LocalLogManager 0] Node 0: running log
> check. (org.apache.kafka.metalog.LocalLogManager:479)
> [2022-05-27 09:34:42,394] DEBUG [LocalLogManager 0] initialized local log
> manager for node 0 (org.apache.kafka.metalog.LocalLogManager:622)
> [2022-05-27 09:34:42,396] INFO [LocalLogManager 0] Node 0: registered
> MetaLogListener 1774961169 (org.apache.kafka.metalog.LocalLogManager:640)
> [2022-05-27 09:34:42,397] DEBUG [LocalLogManager 0] Node 0: running log
> check. (org.apache.kafka.metalog.LocalLogManager:479)
> [2022-05-27 09:34:42,397] DEBUG [LocalLogManager 0] Node 0: Executing
> handleLeaderChange LeaderAndEpoch(leaderId=OptionalInt[0], epoch=1)
> (org.apache.kafka.metalog.LocalLogManager:520)
> [2022-05-27 09:34:42,398] DEBUG [Controller 0] Executing
> handleLeaderChange[1]. (org.apache.kafka.controller.QuorumController:438)
> [2022-05-27 09:34:42,398] INFO [Controller 0] Becoming the active controller
> at epoch 1, committed offset -1, committed epoch -1, and metadata.version 5
> (org.apache.kafka.controller.QuorumController:950)
> [2022-05-27 09:34:42,398] DEBUG [Controller 0] Creating snapshot -1
> (org.apache.kafka.timeline.SnapshotRegistry:197)
> [2022-05-27 09:34:42,399] DEBUG [Controller 0] Processed
> handleLeaderChange[1] in 951 us
> (org.apache.kafka.controller.QuorumController:385)
> [2022-05-27 09:34:42,399] INFO [Controller 0] Initializing metadata.version
> to 5 (org.apache.kafka.controller.QuorumController:926)
> [2022-05-27 09:34:42,399] INFO [Controller 0] Setting metadata.version to 5
> (org.apache.kafka.controller.FeatureControlManager:273)
> [2022-05-27 09:34:42,400] DEBUG [Controller 0] Creating snapshot
> 9223372036854775807 (org.apache.kafka.timeline.SnapshotRegistry:197)
> [2022-05-27 09:34:42,400] DEBUG [Controller 0] Read-write operation
> bootstrapMetadata(1863535402) will be completed when the log reaches offset
> 9223372036854775807. (org.apache.kafka.controller.QuorumController:725)
> [2022-05-27 09:34:42,402] DEBUG append(batch=LocalRecordBatch(leaderEpoch=1,
> appendTimestamp=10,
> records=[ApiMessageAndVersion(RegisterBrokerRecord(brokerId=0,
> incarnationId=kxAT73dKQsitIedpiPtwBw, brokerEpoch=-9223372036854775808,
> endPoints=[BrokerEndpoint(name='PLAINTEXT', host='localhost', port=9092,
> securityProtocol=0)], features=[], rack=null, fenced=true) at version 0)]),
> prevOffset=1) (org.apache.kafka.metalog.LocalLogManager$SharedLogData:247)
> [2022-05-27 09:34:42,402] INFO [Controller 0] Registered new broker:
> RegisterBrokerRecord(brokerId=0, incarnationId=kxAT73dKQsitIedpiPtwBw,
> brokerEpoch=-9223372036854775808, endPoints=[BrokerEndpoint(name='PLAINTEXT',
> host='localhost', port=9092, securityProtocol=0)], features=[], rack=null,
> fenced=true) (org.apache.kafka.controller.ClusterControlManager:368)
> [2022-05-27 09:34:42,403] WARN [Controller 0] registerBroker: failed with
> unknown server exception RuntimeException at epoch 1 in 2449 us. Reverting
> to last committed offset -1.
> (org.apache.kafka.controller.QuorumController:410)java.lang.RuntimeException:
> Can't create a new snapshot at epoch 1 because there is already a snapshot
> with epoch 9223372036854775807at
> org.apache.kafka.timeline.SnapshotRegistry.getOrCreateSnapshot(SnapshotRegistry.java:190)
> at
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:723)
> at
> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:121)
> at
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:200)
> at
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:173)
> at java.base/java.lang.Thread.run(Thread.java:833){noformat}
> {noformat}
> Full stack trace
> java.util.concurrent.ExecutionException:
>
[jira] [Updated] (KAFKA-13858) Kraft should not shutdown metadata listener until controller shutdown is finished
[ https://issues.apache.org/jira/browse/KAFKA-13858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-13858: Fix Version/s: 3.3.0 (was: 3.3) > Kraft should not shutdown metadata listener until controller shutdown is > finished > - > > Key: KAFKA-13858 > URL: https://issues.apache.org/jira/browse/KAFKA-13858 > Project: Kafka > Issue Type: Bug >Reporter: Jason Gustafson >Assignee: David Jacot >Priority: Major > Labels: kip-500 > Fix For: 3.3.0 > > > When the kraft broker begins controlled shutdown, it immediately disables the > metadata listener. This means that metadata changes as part of the controlled > shutdown do not get sent to the respective components. For partitions that > the broker is follower of, that is what we want. It prevents the follower > from being able to rejoin the ISR while still shutting down. But for > partitions that the broker is leading, it means the leader will remain active > until controlled shutdown finishes and the socket server is stopped. That > delay can be as much as 5 seconds and probably even worse. > In the zk world, we have an explicit request `StopReplica` which serves the > purpose of shutting down both follower and leader, but we don't have > something similar in kraft. For KRaft, we may not necessarily need an > explicit signal like this. We know that the broker is shutting down, so we > can treat partition changes as implicit `StopReplica` requests rather than > going through the normal `LeaderAndIsr` flow. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-13649) StandardAuthorizer should not finish loading until it is ready
[ https://issues.apache.org/jira/browse/KAFKA-13649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-13649: Fix Version/s: 3.3.0 (was: 3.3) > StandardAuthorizer should not finish loading until it is ready > -- > > Key: KAFKA-13649 > URL: https://issues.apache.org/jira/browse/KAFKA-13649 > Project: Kafka > Issue Type: Bug >Reporter: Colin McCabe >Assignee: Colin McCabe >Priority: Major > Labels: kip-500, kip-801 > Fix For: 3.3.0 > > > StandardAuthorizer should not finish loading until it reads up to the high > water mark. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14217) app-reset-tool.html should remove reference to --zookeeper flag that no longer exists
[ https://issues.apache.org/jira/browse/KAFKA-14217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-14217: Fix Version/s: 3.3.0 (was: 3.3) > app-reset-tool.html should remove reference to --zookeeper flag that no > longer exists > - > > Key: KAFKA-14217 > URL: https://issues.apache.org/jira/browse/KAFKA-14217 > Project: Kafka > Issue Type: Bug > Components: docs, documentation >Affects Versions: 3.3.0 >Reporter: Colin McCabe >Assignee: Colin McCabe >Priority: Blocker > Fix For: 3.3.0 > > > app-reset-tool.html should remove reference to --zookeeper flag that no > longer exists -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14217) app-reset-tool.html should remove reference to --zookeeper flag that no longer exists
[ https://issues.apache.org/jira/browse/KAFKA-14217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-14217: Affects Version/s: (was: 3.3) > app-reset-tool.html should remove reference to --zookeeper flag that no > longer exists > - > > Key: KAFKA-14217 > URL: https://issues.apache.org/jira/browse/KAFKA-14217 > Project: Kafka > Issue Type: Bug > Components: docs, documentation >Affects Versions: 3.3.0 >Reporter: Colin McCabe >Assignee: Colin McCabe >Priority: Blocker > Fix For: 3.3 > > > app-reset-tool.html should remove reference to --zookeeper flag that no > longer exists -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14215) KRaft forwarded requests have no quota enforcement
[ https://issues.apache.org/jira/browse/KAFKA-14215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-14215: Fix Version/s: (was: 3.3) > KRaft forwarded requests have no quota enforcement > -- > > Key: KAFKA-14215 > URL: https://issues.apache.org/jira/browse/KAFKA-14215 > Project: Kafka > Issue Type: Bug >Affects Versions: 3.3.0 >Reporter: Jason Gustafson >Assignee: Jason Gustafson >Priority: Blocker > Fix For: 3.3.0 > > > On the broker, the `BrokerMetadataPublisher` is responsible for propagating > quota changes from `ClientQuota` records to `ClientQuotaManager`. On the > controller, there is no similar logic, so no client quotas are enforced on > the controller. > On the broker side, there is no enforcement as well since the broker assumes > that the controller will be the one to do it. Basically it looks at the > throttle time returned in the response from the controller. If it is 0, then > the response is sent immediately without any throttling. > So the consequence of both of these issues is that controller-bound requests > have no throttling today. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14216) Remove ZK reference from org.apache.kafka.server.quota.ClientQuotaCallback javadoc
[ https://issues.apache.org/jira/browse/KAFKA-14216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-14216: Affects Version/s: (was: 3.3) > Remove ZK reference from org.apache.kafka.server.quota.ClientQuotaCallback > javadoc > -- > > Key: KAFKA-14216 > URL: https://issues.apache.org/jira/browse/KAFKA-14216 > Project: Kafka > Issue Type: Bug > Components: docs, documentation >Affects Versions: 3.3.0 >Reporter: Colin McCabe >Assignee: Colin McCabe >Priority: Blocker > Fix For: 3.3 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14216) Remove ZK reference from org.apache.kafka.server.quota.ClientQuotaCallback javadoc
[ https://issues.apache.org/jira/browse/KAFKA-14216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-14216: Fix Version/s: 3.3.0 (was: 3.3) > Remove ZK reference from org.apache.kafka.server.quota.ClientQuotaCallback > javadoc > -- > > Key: KAFKA-14216 > URL: https://issues.apache.org/jira/browse/KAFKA-14216 > Project: Kafka > Issue Type: Bug > Components: docs, documentation >Affects Versions: 3.3.0 >Reporter: Colin McCabe >Assignee: Colin McCabe >Priority: Blocker > Fix For: 3.3.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14170) KRaft Controller: Possible NPE when we remove topics with any offline partitions in the cluster
[
https://issues.apache.org/jira/browse/KAFKA-14170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-14170:
Fix Version/s: 3.3.0
(was: 3.3)
> KRaft Controller: Possible NPE when we remove topics with any offline
> partitions in the cluster
> ---
>
> Key: KAFKA-14170
> URL: https://issues.apache.org/jira/browse/KAFKA-14170
> Project: Kafka
> Issue Type: Bug
> Components: kraft
>Affects Versions: 3.3.0
>Reporter: Akhilesh Chaganti
>Assignee: Akhilesh Chaganti
>Priority: Blocker
> Fix For: 3.3.0
>
>
> When we remove a topic, it goes through the following function in KRaft
> Controller replay method for RemoveTopicRecord:
> {code:java}
> void removeTopicEntryForBroker(Uuid topicId, int brokerId) {
> Map topicMap = isrMembers.get(brokerId);
> if (topicMap != null) {
> if (brokerId == NO_LEADER) {
> offlinePartitionCount.set(offlinePartitionCount.get() -
> topicMap.get(topicId).length);
> }
> topicMap.remove(topicId);
> }
> } {code}
> If the broker has any offline partitions but doesn't have offline partitions
> for the topic we're deleting, the above code will run into NPE because we
> directly access the `topicMap.get(topicId).length`
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-14170) KRaft Controller: Possible NPE when we remove topics with any offline partitions in the cluster
[
https://issues.apache.org/jira/browse/KAFKA-14170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-14170:
Affects Version/s: 3.3.0
(was: 3.3)
> KRaft Controller: Possible NPE when we remove topics with any offline
> partitions in the cluster
> ---
>
> Key: KAFKA-14170
> URL: https://issues.apache.org/jira/browse/KAFKA-14170
> Project: Kafka
> Issue Type: Bug
> Components: kraft
>Affects Versions: 3.3.0
>Reporter: Akhilesh Chaganti
>Assignee: Akhilesh Chaganti
>Priority: Blocker
> Fix For: 3.3
>
>
> When we remove a topic, it goes through the following function in KRaft
> Controller replay method for RemoveTopicRecord:
> {code:java}
> void removeTopicEntryForBroker(Uuid topicId, int brokerId) {
> Map topicMap = isrMembers.get(brokerId);
> if (topicMap != null) {
> if (brokerId == NO_LEADER) {
> offlinePartitionCount.set(offlinePartitionCount.get() -
> topicMap.get(topicId).length);
> }
> topicMap.remove(topicId);
> }
> } {code}
> If the broker has any offline partitions but doesn't have offline partitions
> for the topic we're deleting, the above code will run into NPE because we
> directly access the `topicMap.get(topicId).length`
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-14215) KRaft forwarded requests have no quota enforcement
[ https://issues.apache.org/jira/browse/KAFKA-14215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-14215: Affects Version/s: (was: 3.3) > KRaft forwarded requests have no quota enforcement > -- > > Key: KAFKA-14215 > URL: https://issues.apache.org/jira/browse/KAFKA-14215 > Project: Kafka > Issue Type: Bug >Affects Versions: 3.3.0 >Reporter: Jason Gustafson >Assignee: Jason Gustafson >Priority: Blocker > Fix For: 3.3.0, 3.3 > > > On the broker, the `BrokerMetadataPublisher` is responsible for propagating > quota changes from `ClientQuota` records to `ClientQuotaManager`. On the > controller, there is no similar logic, so no client quotas are enforced on > the controller. > On the broker side, there is no enforcement as well since the broker assumes > that the controller will be the one to do it. Basically it looks at the > throttle time returned in the response from the controller. If it is 0, then > the response is sent immediately without any throttling. > So the consequence of both of these issues is that controller-bound requests > have no throttling today. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16692) InvalidRequestException: ADD_PARTITIONS_TO_TXN with version 4 which is not enabled when upgrading from kafka 3.5 to 3.6
[
https://issues.apache.org/jira/browse/KAFKA-16692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-16692:
Affects Version/s: 3.8.0
(was: 3.8)
> InvalidRequestException: ADD_PARTITIONS_TO_TXN with version 4 which is not
> enabled when upgrading from kafka 3.5 to 3.6
>
>
> Key: KAFKA-16692
> URL: https://issues.apache.org/jira/browse/KAFKA-16692
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 3.7.0, 3.6.1, 3.8.0
>Reporter: Johnson Okorie
>Assignee: Justine Olshan
>Priority: Major
> Fix For: 3.8.0, 3.7.1, 3.6.3
>
>
> We have a kafka cluster running on version 3.5.2 that we are upgrading to
> 3.6.1. This cluster has a lot of clients with exactly one semantics enabled
> and hence creating transactions. As we replaced brokers with the new
> binaries, we observed lots of clients in the cluster experiencing the
> following error:
> {code:java}
> 2024-05-07T09:08:10.039Z "tid": "" -- [Producer clientId=,
> transactionalId=] Got error produce response with
> correlation id 6402937 on topic-partition , retrying
> (2147483512 attempts left). Error: NETWORK_EXCEPTION. Error Message: The
> server disconnected before a response was received.{code}
> On inspecting the broker, we saw the following errors on brokers still
> running Kafka version 3.5.2:
>
> {code:java}
> message:
> Closing socket for because of error
> exception_exception_class:
> org.apache.kafka.common.errors.InvalidRequestException
> exception_exception_message:
> Received request api key ADD_PARTITIONS_TO_TXN with version 4 which is not
> enabled
> exception_stacktrace:
> org.apache.kafka.common.errors.InvalidRequestException: Received request api
> key ADD_PARTITIONS_TO_TXN with version 4 which is not enabled
> {code}
> On the new brokers running 3.6.1 we saw the following errors:
>
> {code:java}
> [AddPartitionsToTxnSenderThread-1055]: AddPartitionsToTxnRequest failed for
> node 1043 with a network exception.{code}
>
> I can also see this :
> {code:java}
> [AddPartitionsToTxnManager broker=1055]Cancelled in-flight
> ADD_PARTITIONS_TO_TXN request with correlation id 21120 due to node 1043
> being disconnected (elapsed time since creation: 11ms, elapsed time since
> send: 4ms, request timeout: 3ms){code}
> We started investigating this issue and digging through the changes in 3.6,
> we came across some changes introduced as part of KAFKA-14402 that we thought
> might lead to this behaviour.
> First we could see that _transaction.partition.verification.enable_ is
> enabled by default and enables a new code path that culminates in we sending
> version 4 ADD_PARTITIONS_TO_TXN requests to other brokers that are generated
> [here|https://github.com/apache/kafka/blob/29f3260a9c07e654a28620aeb93a778622a5233d/core/src/main/scala/kafka/server/AddPartitionsToTxnManager.scala#L269].
> From a
> [discussion|https://lists.apache.org/thread/4895wrd1z92kjb708zck4s1f62xq6r8x]
> on the mailing list, [~jolshan] pointed out that this scenario shouldn't be
> possible as the following code paths should prevent version 4
> ADD_PARTITIONS_TO_TXN requests being sent to other brokers:
> [https://github.com/apache/kafka/blob/525b9b1d7682ae2a527ceca83fedca44b1cba11a/clients/src/main/java/org/apache/kafka/clients/NodeApiVersions.java#L130]
>
> [https://github.com/apache/kafka/blob/525b9b1d7682ae2a527ceca83fedca44b1cba11a/core/src/main/scala/kafka/server/AddPartitionsToTxnManager.scala#L195]
> However, these requests are still sent to other brokers in our environment.
> On further inspection of the code, I am wondering if the following code path
> could lead to this issue:
> [https://github.com/apache/kafka/blob/c4deed513057c94eb502e64490d6bdc23551d8b6/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java#L500]
> In this scenario, we don't have any _NodeApiVersions_ available for the
> specified nodeId and potentially skipping the _latestUsableVersion_ check. I
> am wondering if it is possible that because _discoverBrokerVersions_ is set
> to _false_ for the network client of the {_}AddPartitionsToTxnManager{_}, it
> skips fetching {_}NodeApiVersions{_}? I can see that we create the network
> client here:
> [https://github.com/apache/kafka/blob/c4deed513057c94eb502e64490d6bdc23551d8b6/core/src/main/scala/kafka/server/KafkaServer.scala#L641]
> The _NetworkUtils.buildNetworkClient_ method seems to create a network client
> that has _discoverBrokerVersions_ set to {_}false{_}.
> I was hoping I could get some assistance debugging this issue. Happy to
> provide any additional information needed.
>
>
>
--
This message
[jira] [Resolved] (KAFKA-16911) Kafka Streams topology optimization docs incomplete
[ https://issues.apache.org/jira/browse/KAFKA-16911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-16911. - Fix Version/s: 3.8.0 Resolution: Fixed > Kafka Streams topology optimization docs incomplete > --- > > Key: KAFKA-16911 > URL: https://issues.apache.org/jira/browse/KAFKA-16911 > Project: Kafka > Issue Type: Improvement > Components: docs, streams >Affects Versions: 3.4.0 >Reporter: Matthias J. Sax >Assignee: James Galasyn >Priority: Minor > Fix For: 3.8.0 > > > The docs for topology optimization are incomplete: > [https://kafka.apache.org/37/documentation/streams/developer-guide/config-streams.html#topology-optimization] > In 3.4.0 we added a new option via > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-862%3A+Self-join+optimization+for+stream-stream+joins] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-16911) Kafka Streams topology optimization docs incomplete
[ https://issues.apache.org/jira/browse/KAFKA-16911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax reassigned KAFKA-16911: --- Assignee: James Galasyn > Kafka Streams topology optimization docs incomplete > --- > > Key: KAFKA-16911 > URL: https://issues.apache.org/jira/browse/KAFKA-16911 > Project: Kafka > Issue Type: Improvement > Components: docs, streams >Affects Versions: 3.4.0 >Reporter: Matthias J. Sax >Assignee: James Galasyn >Priority: Minor > > The docs for topology optimization are incomplete: > [https://kafka.apache.org/37/documentation/streams/developer-guide/config-streams.html#topology-optimization] > In 3.4.0 we added a new option via > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-862%3A+Self-join+optimization+for+stream-stream+joins] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-14567) Kafka Streams crashes after ProducerFencedException
[
https://issues.apache.org/jira/browse/KAFKA-14567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17853201#comment-17853201
]
Matthias J. Sax edited comment on KAFKA-14567 at 6/7/24 4:14 PM:
-
[~cadonna] – you recent fix should address this issue? ->
https://issues.apache.org/jira/browse/KAFKA-16903
Thoughts?
We don't have the logs from the fenced exception, so don't know the task-id,
but overall looks pretty much as if would have been the same issue?
was (Author: mjsax):
[~cadonna] – you recent fix should address this issue? ->
https://issues.apache.org/jira/browse/KAFKA-16903
Thoughts?
> Kafka Streams crashes after ProducerFencedException
> ---
>
> Key: KAFKA-14567
> URL: https://issues.apache.org/jira/browse/KAFKA-14567
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.7.0
>Reporter: Matthias J. Sax
>Assignee: Kirk True
>Priority: Blocker
> Labels: eos
> Fix For: 3.8.0
>
>
> Running a Kafka Streams application with EOS-v2.
> We first see a `ProducerFencedException`. After the fencing, the fenced
> thread crashed resulting in a non-recoverable error:
> {quote}[2022-12-22 13:49:13,423] ERROR [i-0c291188ec2ae17a0-StreamThread-3]
> stream-thread [i-0c291188ec2ae17a0-StreamThread-3] Failed to process stream
> task 1_2 due to the following error:
> (org.apache.kafka.streams.processor.internals.TaskExecutor)
> org.apache.kafka.streams.errors.StreamsException: Exception caught in
> process. taskId=1_2, processor=KSTREAM-SOURCE-05,
> topic=node-name-repartition, partition=2, offset=539776276,
> stacktrace=java.lang.IllegalStateException: TransactionalId
> stream-soak-test-72b6e57c-c2f5-489d-ab9f-fdbb215d2c86-3: Invalid transition
> attempted from state FATAL_ERROR to state ABORTABLE_ERROR
> at
> org.apache.kafka.clients.producer.internals.TransactionManager.transitionTo(TransactionManager.java:974)
> at
> org.apache.kafka.clients.producer.internals.TransactionManager.transitionToAbortableError(TransactionManager.java:394)
> at
> org.apache.kafka.clients.producer.internals.TransactionManager.maybeTransitionToErrorState(TransactionManager.java:620)
> at
> org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:1079)
> at
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:959)
> at
> org.apache.kafka.streams.processor.internals.StreamsProducer.send(StreamsProducer.java:257)
> at
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl.send(RecordCollectorImpl.java:207)
> at
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl.send(RecordCollectorImpl.java:162)
> at
> org.apache.kafka.streams.processor.internals.SinkNode.process(SinkNode.java:85)
> at
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forwardInternal(ProcessorContextImpl.java:290)
> at
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:269)
> at
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:228)
> at
> org.apache.kafka.streams.kstream.internals.KStreamKTableJoinProcessor.process(KStreamKTableJoinProcessor.java:88)
> at
> org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:157)
> at
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forwardInternal(ProcessorContextImpl.java:290)
> at
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:269)
> at
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:228)
> at
> org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:84)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.lambda$doProcess$1(StreamTask.java:791)
> at
> org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.maybeMeasureLatency(StreamsMetricsImpl.java:867)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.doProcess(StreamTask.java:791)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:722)
> at
> org.apache.kafka.streams.processor.internals.TaskExecutor.processTask(TaskExecutor.java:95)
> at
> org.apache.kafka.streams.processor.internals.TaskExecutor.process(TaskExecutor.java:76)
> at
> org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:1645)
> at
>
[jira] [Commented] (KAFKA-14567) Kafka Streams crashes after ProducerFencedException
[
https://issues.apache.org/jira/browse/KAFKA-14567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17853201#comment-17853201
]
Matthias J. Sax commented on KAFKA-14567:
-
[~cadonna] – you recent fix should address this issue? ->
https://issues.apache.org/jira/browse/KAFKA-16903
Thoughts?
> Kafka Streams crashes after ProducerFencedException
> ---
>
> Key: KAFKA-14567
> URL: https://issues.apache.org/jira/browse/KAFKA-14567
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.7.0
>Reporter: Matthias J. Sax
>Assignee: Kirk True
>Priority: Blocker
> Labels: eos
> Fix For: 3.8.0
>
>
> Running a Kafka Streams application with EOS-v2.
> We first see a `ProducerFencedException`. After the fencing, the fenced
> thread crashed resulting in a non-recoverable error:
> {quote}[2022-12-22 13:49:13,423] ERROR [i-0c291188ec2ae17a0-StreamThread-3]
> stream-thread [i-0c291188ec2ae17a0-StreamThread-3] Failed to process stream
> task 1_2 due to the following error:
> (org.apache.kafka.streams.processor.internals.TaskExecutor)
> org.apache.kafka.streams.errors.StreamsException: Exception caught in
> process. taskId=1_2, processor=KSTREAM-SOURCE-05,
> topic=node-name-repartition, partition=2, offset=539776276,
> stacktrace=java.lang.IllegalStateException: TransactionalId
> stream-soak-test-72b6e57c-c2f5-489d-ab9f-fdbb215d2c86-3: Invalid transition
> attempted from state FATAL_ERROR to state ABORTABLE_ERROR
> at
> org.apache.kafka.clients.producer.internals.TransactionManager.transitionTo(TransactionManager.java:974)
> at
> org.apache.kafka.clients.producer.internals.TransactionManager.transitionToAbortableError(TransactionManager.java:394)
> at
> org.apache.kafka.clients.producer.internals.TransactionManager.maybeTransitionToErrorState(TransactionManager.java:620)
> at
> org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:1079)
> at
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:959)
> at
> org.apache.kafka.streams.processor.internals.StreamsProducer.send(StreamsProducer.java:257)
> at
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl.send(RecordCollectorImpl.java:207)
> at
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl.send(RecordCollectorImpl.java:162)
> at
> org.apache.kafka.streams.processor.internals.SinkNode.process(SinkNode.java:85)
> at
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forwardInternal(ProcessorContextImpl.java:290)
> at
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:269)
> at
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:228)
> at
> org.apache.kafka.streams.kstream.internals.KStreamKTableJoinProcessor.process(KStreamKTableJoinProcessor.java:88)
> at
> org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:157)
> at
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forwardInternal(ProcessorContextImpl.java:290)
> at
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:269)
> at
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:228)
> at
> org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:84)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.lambda$doProcess$1(StreamTask.java:791)
> at
> org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.maybeMeasureLatency(StreamsMetricsImpl.java:867)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.doProcess(StreamTask.java:791)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:722)
> at
> org.apache.kafka.streams.processor.internals.TaskExecutor.processTask(TaskExecutor.java:95)
> at
> org.apache.kafka.streams.processor.internals.TaskExecutor.process(TaskExecutor.java:76)
> at
> org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:1645)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:788)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:607)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:569)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:748)
> at
>
[jira] [Updated] (KAFKA-16911) Kafka Streams topology optimization docs incomplete
[ https://issues.apache.org/jira/browse/KAFKA-16911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-16911: Description: The docs for topology optimization are incomplete: [https://kafka.apache.org/37/documentation/streams/developer-guide/config-streams.html#topology-optimization] In 3.4.0 we added a new option via [https://cwiki.apache.org/confluence/display/KAFKA/KIP-862%3A+Self-join+optimization+for+stream-stream+joins] was: The docs for topology optimization are incomplte: [https://kafka.apache.org/37/documentation/streams/developer-guide/config-streams.html#topology-optimization] In 3.4.0 we added a new option via [https://cwiki.apache.org/confluence/display/KAFKA/KIP-862%3A+Self-join+optimization+for+stream-stream+joins] > Kafka Streams topology optimization docs incomplete > --- > > Key: KAFKA-16911 > URL: https://issues.apache.org/jira/browse/KAFKA-16911 > Project: Kafka > Issue Type: Improvement > Components: docs, streams >Affects Versions: 3.4.0 >Reporter: Matthias J. Sax >Priority: Minor > > The docs for topology optimization are incomplete: > [https://kafka.apache.org/37/documentation/streams/developer-guide/config-streams.html#topology-optimization] > In 3.4.0 we added a new option via > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-862%3A+Self-join+optimization+for+stream-stream+joins] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16911) Kafka Streams topology optimization docs incomplete
Matthias J. Sax created KAFKA-16911: --- Summary: Kafka Streams topology optimization docs incomplete Key: KAFKA-16911 URL: https://issues.apache.org/jira/browse/KAFKA-16911 Project: Kafka Issue Type: Improvement Components: docs, streams Affects Versions: 3.4.0 Reporter: Matthias J. Sax The docs for topology optimization are incomplte: [https://kafka.apache.org/37/documentation/streams/developer-guide/config-streams.html#topology-optimization] In 3.4.0 we added a new option via [https://cwiki.apache.org/confluence/display/KAFKA/KIP-862%3A+Self-join+optimization+for+stream-stream+joins] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16903) Task should consider producer error previously occurred for different task
[
https://issues.apache.org/jira/browse/KAFKA-16903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-16903:
Fix Version/s: 3.8.0
> Task should consider producer error previously occurred for different task
> --
>
> Key: KAFKA-16903
> URL: https://issues.apache.org/jira/browse/KAFKA-16903
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.7.0
>Reporter: Bruno Cadonna
>Assignee: Bruno Cadonna
>Priority: Major
> Fix For: 3.8.0
>
>
> A task does not consider a producer error that occurred for a different task.
> The following log messages show the issue.
> Task {{0_2}} of a Streams app (EOSv2 enabled) crashes while sending records
> with an {{InvalidTxnStateException}}:
> {code:java}
> [2024-05-30 10:20:35,881] ERROR [kafka-producer-network-thread |
> i-0af25f5c2bd9bba31-StreamThread-1-producer] stream-thread
> [i-0af25f5c2bd9bba31-StreamThread-1] stream-task [0_2] Error encountered
> sending record to topic stream-soak-test-node-name-repartition for task 0_2
> due to:
> org.apache.kafka.common.errors.InvalidTxnStateException: The producer
> attempted a transactional operation in an invalid state.
> Exception handler choose to FAIL the processing, no more records would be
> sent. (org.apache.kafka.streams.processor.internals.RecordCollectorImpl)
> org.apache.kafka.common.errors.InvalidTxnStateException: The producer
> attempted a transactional operation in an invalid state.
> [2024-05-30 10:20:35,886] ERROR [i-0af25f5c2bd9bba31-StreamThread-1]
> stream-thread [i-0af25f5c2bd9bba31-StreamThread-1] Failed to process stream
> task 0_2 due to the following error:
> (org.apache.kafka.streams.processor.internals.TaskExecutor)
> org.apache.kafka.streams.errors.StreamsException: Error encountered sending
> record to topic stream-soak-test-node-name-repartition for task 0_2 due to:
> org.apache.kafka.common.errors.InvalidTxnStateException: The producer
> attempted a transactional operation in an invalid state.
> Exception handler choose to FAIL the processing, no more records would be
> sent.
> at
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl.recordSendError(RecordCollectorImpl.java:316)
> at
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl.lambda$send$1(RecordCollectorImpl.java:285)
> at
> org.apache.kafka.clients.producer.KafkaProducer$AppendCallbacks.onCompletion(KafkaProducer.java:1565)
> at
> org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:311)
> at
> org.apache.kafka.clients.producer.internals.ProducerBatch.done(ProducerBatch.java:272)
> at
> org.apache.kafka.clients.producer.internals.ProducerBatch.completeExceptionally(ProducerBatch.java:236)
> at
> org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:829)
> at
> org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:818)
> at
> org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:770)
> at
> org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:702)
> at
> org.apache.kafka.clients.producer.internals.Sender.lambda$null$2(Sender.java:627)
> at java.util.ArrayList.forEach(ArrayList.java:1259)
> at
> org.apache.kafka.clients.producer.internals.Sender.lambda$handleProduceResponse$3(Sender.java:612)
> at java.lang.Iterable.forEach(Iterable.java:75)
> at
> org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:612)
> at
> org.apache.kafka.clients.producer.internals.Sender.lambda$sendProduceRequest$9(Sender.java:916)
> at
> org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:154)
> at
> org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:608)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:600)
> at
> org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:348)
> at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:250)
> at java.lang.Thread.run(Thread.java:750)
> Caused by: org.apache.kafka.common.errors.InvalidTxnStateException: The
> producer attempted a transactional operation in an invalid state.
> {code}
> Just before the exception of task 0_2 also task 0_0 encountered an
> exception while producing:
> {code:java}
> [2024-05-30 10:20:35,880] ERROR [kafka-producer-network-thread |
> i-0af25f5c2bd9bba31-StreamThread-1-producer] stream-thread
> [i-0af25f5c2bd9bba31-StreamThread-1] stream-task [0_0] Error encountered
> sending record to topic stream-soak-test-network-id-repartition for
[jira] [Resolved] (KAFKA-16903) Task should consider producer error previously occurred for different task
[
https://issues.apache.org/jira/browse/KAFKA-16903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax resolved KAFKA-16903.
-
Resolution: Fixed
> Task should consider producer error previously occurred for different task
> --
>
> Key: KAFKA-16903
> URL: https://issues.apache.org/jira/browse/KAFKA-16903
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.7.0
>Reporter: Bruno Cadonna
>Assignee: Bruno Cadonna
>Priority: Major
> Fix For: 3.8.0
>
>
> A task does not consider a producer error that occurred for a different task.
> The following log messages show the issue.
> Task {{0_2}} of a Streams app (EOSv2 enabled) crashes while sending records
> with an {{InvalidTxnStateException}}:
> {code:java}
> [2024-05-30 10:20:35,881] ERROR [kafka-producer-network-thread |
> i-0af25f5c2bd9bba31-StreamThread-1-producer] stream-thread
> [i-0af25f5c2bd9bba31-StreamThread-1] stream-task [0_2] Error encountered
> sending record to topic stream-soak-test-node-name-repartition for task 0_2
> due to:
> org.apache.kafka.common.errors.InvalidTxnStateException: The producer
> attempted a transactional operation in an invalid state.
> Exception handler choose to FAIL the processing, no more records would be
> sent. (org.apache.kafka.streams.processor.internals.RecordCollectorImpl)
> org.apache.kafka.common.errors.InvalidTxnStateException: The producer
> attempted a transactional operation in an invalid state.
> [2024-05-30 10:20:35,886] ERROR [i-0af25f5c2bd9bba31-StreamThread-1]
> stream-thread [i-0af25f5c2bd9bba31-StreamThread-1] Failed to process stream
> task 0_2 due to the following error:
> (org.apache.kafka.streams.processor.internals.TaskExecutor)
> org.apache.kafka.streams.errors.StreamsException: Error encountered sending
> record to topic stream-soak-test-node-name-repartition for task 0_2 due to:
> org.apache.kafka.common.errors.InvalidTxnStateException: The producer
> attempted a transactional operation in an invalid state.
> Exception handler choose to FAIL the processing, no more records would be
> sent.
> at
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl.recordSendError(RecordCollectorImpl.java:316)
> at
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl.lambda$send$1(RecordCollectorImpl.java:285)
> at
> org.apache.kafka.clients.producer.KafkaProducer$AppendCallbacks.onCompletion(KafkaProducer.java:1565)
> at
> org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:311)
> at
> org.apache.kafka.clients.producer.internals.ProducerBatch.done(ProducerBatch.java:272)
> at
> org.apache.kafka.clients.producer.internals.ProducerBatch.completeExceptionally(ProducerBatch.java:236)
> at
> org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:829)
> at
> org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:818)
> at
> org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:770)
> at
> org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:702)
> at
> org.apache.kafka.clients.producer.internals.Sender.lambda$null$2(Sender.java:627)
> at java.util.ArrayList.forEach(ArrayList.java:1259)
> at
> org.apache.kafka.clients.producer.internals.Sender.lambda$handleProduceResponse$3(Sender.java:612)
> at java.lang.Iterable.forEach(Iterable.java:75)
> at
> org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:612)
> at
> org.apache.kafka.clients.producer.internals.Sender.lambda$sendProduceRequest$9(Sender.java:916)
> at
> org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:154)
> at
> org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:608)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:600)
> at
> org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:348)
> at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:250)
> at java.lang.Thread.run(Thread.java:750)
> Caused by: org.apache.kafka.common.errors.InvalidTxnStateException: The
> producer attempted a transactional operation in an invalid state.
> {code}
> Just before the exception of task 0_2 also task 0_0 encountered an
> exception while producing:
> {code:java}
> [2024-05-30 10:20:35,880] ERROR [kafka-producer-network-thread |
> i-0af25f5c2bd9bba31-StreamThread-1-producer] stream-thread
> [i-0af25f5c2bd9bba31-StreamThread-1] stream-task [0_0] Error encountered
> sending record to topic stream-soak-test-network-id-repartition for
[jira] [Commented] (KAFKA-16811) Punctuate Ratio metric almost impossible to track
[ https://issues.apache.org/jira/browse/KAFKA-16811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17852531#comment-17852531 ] Matthias J. Sax commented on KAFKA-16811: - Let's more the discussion on the PR > Punctuate Ratio metric almost impossible to track > - > > Key: KAFKA-16811 > URL: https://issues.apache.org/jira/browse/KAFKA-16811 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 3.7.0 >Reporter: Sebastien Viale >Assignee: Ganesh Sadanala >Priority: Minor > Time Spent: 48h > Remaining Estimate: 0h > > The Punctuate ratio metric is returned after the last record of the poll > loop. It is recomputed in every poll loop. > After a puntuate, the value is close to 1, but there is little chance that > metric is sampled at this time. > So its value is almost always 0. > A solution could be to apply a kind of "sliding window" to it and report the > value for the last x seconds. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16811) Punctuate Ratio metric almost impossible to track
[ https://issues.apache.org/jira/browse/KAFKA-16811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17852231#comment-17852231 ] Matthias J. Sax commented on KAFKA-16811: - [~ganesh_6] – WordCountProcessorDemo does not contain any punctuation calls, right? Or did you modify the code and added punctuations? > Punctuate Ratio metric almost impossible to track > - > > Key: KAFKA-16811 > URL: https://issues.apache.org/jira/browse/KAFKA-16811 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 3.7.0 >Reporter: Sebastien Viale >Assignee: Ganesh Sadanala >Priority: Minor > Time Spent: 48h > Remaining Estimate: 0h > > The Punctuate ratio metric is returned after the last record of the poll > loop. It is recomputed in every poll loop. > After a puntuate, the value is close to 1, but there is little chance that > metric is sampled at this time. > So its value is almost always 0. > A solution could be to apply a kind of "sliding window" to it and report the > value for the last x seconds. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16874) Remove old TaskAssignor interface
[ https://issues.apache.org/jira/browse/KAFKA-16874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-16874: Component/s: streams > Remove old TaskAssignor interface > - > > Key: KAFKA-16874 > URL: https://issues.apache.org/jira/browse/KAFKA-16874 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: A. Sophie Blee-Goldman >Priority: Major > > Once we have the new HAAssignor that implements the new TaskAssignor > interface, we can remove the old TaskAssignor interface. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-15905) Restarts of MirrorCheckpointTask should not permanently interrupt offset translation
[ https://issues.apache.org/jira/browse/KAFKA-15905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-15905: Fix Version/s: 3.8.0 (was: 3.8) > Restarts of MirrorCheckpointTask should not permanently interrupt offset > translation > > > Key: KAFKA-15905 > URL: https://issues.apache.org/jira/browse/KAFKA-15905 > Project: Kafka > Issue Type: Improvement > Components: mirrormaker >Affects Versions: 3.6.0 >Reporter: Greg Harris >Assignee: Edoardo Comar >Priority: Major > Fix For: 3.8.0, 3.7.1 > > > Executive summary: When the MirrorCheckpointTask restarts, it loses the state > of checkpointsPerConsumerGroup, which limits offset translation to records > mirrored after the latest restart. > For example, if 1000 records are mirrored and the OffsetSyncs are read by > MirrorCheckpointTask, the emitted checkpoints are cached, and translation can > happen at the ~500th record. If MirrorCheckpointTask restarts, and 1000 more > records are mirrored, translation can happen at the ~1500th record, but no > longer at the ~500th record. > Context: > Before KAFKA-13659, MM2 made translation decisions based on the > incompletely-initialized OffsetSyncStore, and the checkpoint could appear to > go backwards temporarily during restarts. To fix this, we forced the > OffsetSyncStore to initialize completely before translation could take place, > ensuring that the latest OffsetSync had been read, and thus providing the > most accurate translation. > Before KAFKA-14666, MM2 translated offsets only off of the latest OffsetSync. > Afterwards, an in-memory sparse cache of historical OffsetSyncs was kept, to > allow for translation of earlier offsets. This came with the caveat that the > cache's sparseness allowed translations to go backwards permanently. To > prevent this behavior, a cache of the latest Checkpoints was kept in the > MirrorCheckpointTask#checkpointsPerConsumerGroup variable, and offset > translation remained restricted to the fully-initialized OffsetSyncStore. > Effectively, the MirrorCheckpointTask ensures that it translates based on an > OffsetSync emitted during it's lifetime, to ensure that no previous > MirrorCheckpointTask emitted a later sync. If we can read the checkpoints > emitted by previous generations of MirrorCheckpointTask, we can still ensure > that checkpoints are monotonic, while allowing translation further back in > history. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16622) Mirromaker2 first Checkpoint not emitted until consumer group fully catches up once
[ https://issues.apache.org/jira/browse/KAFKA-16622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-16622: Fix Version/s: 3.8.0 (was: 3.8) > Mirromaker2 first Checkpoint not emitted until consumer group fully catches > up once > --- > > Key: KAFKA-16622 > URL: https://issues.apache.org/jira/browse/KAFKA-16622 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 3.7.0, 3.6.2, 3.8.0 >Reporter: Edoardo Comar >Assignee: Edoardo Comar >Priority: Major > Fix For: 3.8.0, 3.7.1 > > Attachments: connect.log.2024-04-26-10.zip, > edo-connect-mirror-maker-sourcetarget.properties > > > We observed an excessively delayed emission of the MM2 Checkpoint record. > It only gets created when the source consumer reaches the end of a topic. > This does not seem reasonable. > In a very simple setup : > Tested with a standalone single process MirrorMaker2 mirroring between two > single-node kafka clusters(mirromaker config attached) with quick refresh > intervals (eg 5 sec) and a small offset.lag.max (eg 10) > create a single topic in the source cluster > produce data to it (e.g. 1 records) > start a slow consumer - e.g. fetching 50records/poll and pausing 1 sec > between polls which commits after each poll > watch the Checkpoint topic in the target cluster > bin/kafka-console-consumer.sh --bootstrap-server localhost:9192 \ > --topic source.checkpoints.internal \ > --formatter org.apache.kafka.connect.mirror.formatters.CheckpointFormatter \ >--from-beginning > -> no record appears in the checkpoint topic until the consumer reaches the > end of the topic (ie its consumer group lag gets down to 0). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16757) Fix broker re-registration issues around MV 3.7-IV2
[ https://issues.apache.org/jira/browse/KAFKA-16757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-16757: Fix Version/s: 3.8.0 (was: 3.8) > Fix broker re-registration issues around MV 3.7-IV2 > --- > > Key: KAFKA-16757 > URL: https://issues.apache.org/jira/browse/KAFKA-16757 > Project: Kafka > Issue Type: Bug >Reporter: Colin McCabe >Assignee: Colin McCabe >Priority: Major > Fix For: 3.8.0, 3.7.1 > > > When upgrading from a MetadataVersion older than 3.7-IV2, we need to resend > the broker registration, so that the controller can record the storage > directories. The current code for doing this has several problems, however. > One is that it tends to trigger even in cases where we don't actually need > it. Another is that when re-registering the broker, the broker is marked as > fenced. > This PR moves the handling of the re-registration case out of > BrokerMetadataPublisher and into BrokerRegistrationTracker. The > re-registration code there will only trigger in the case where the broker > sees an existing registration for itself with no directories set. This is > much more targetted than the original code. > Additionally, in ClusterControlManager, when re-registering the same broker, > we now preserve its fencing and shutdown state, rather than clearing those. > (There isn't any good reason re-registering the same broker should clear > these things... this was purely an oversight.) Note that we can tell the > broker is "the same" because it has the same IncarnationId. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16692) InvalidRequestException: ADD_PARTITIONS_TO_TXN with version 4 which is not enabled when upgrading from kafka 3.5 to 3.6
[
https://issues.apache.org/jira/browse/KAFKA-16692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-16692:
Fix Version/s: 3.8.0
(was: 3.8)
> InvalidRequestException: ADD_PARTITIONS_TO_TXN with version 4 which is not
> enabled when upgrading from kafka 3.5 to 3.6
>
>
> Key: KAFKA-16692
> URL: https://issues.apache.org/jira/browse/KAFKA-16692
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 3.7.0, 3.6.1, 3.8
>Reporter: Johnson Okorie
>Assignee: Justine Olshan
>Priority: Major
> Fix For: 3.8.0, 3.7.1, 3.6.3
>
>
> We have a kafka cluster running on version 3.5.2 that we are upgrading to
> 3.6.1. This cluster has a lot of clients with exactly one semantics enabled
> and hence creating transactions. As we replaced brokers with the new
> binaries, we observed lots of clients in the cluster experiencing the
> following error:
> {code:java}
> 2024-05-07T09:08:10.039Z "tid": "" -- [Producer clientId=,
> transactionalId=] Got error produce response with
> correlation id 6402937 on topic-partition , retrying
> (2147483512 attempts left). Error: NETWORK_EXCEPTION. Error Message: The
> server disconnected before a response was received.{code}
> On inspecting the broker, we saw the following errors on brokers still
> running Kafka version 3.5.2:
>
> {code:java}
> message:
> Closing socket for because of error
> exception_exception_class:
> org.apache.kafka.common.errors.InvalidRequestException
> exception_exception_message:
> Received request api key ADD_PARTITIONS_TO_TXN with version 4 which is not
> enabled
> exception_stacktrace:
> org.apache.kafka.common.errors.InvalidRequestException: Received request api
> key ADD_PARTITIONS_TO_TXN with version 4 which is not enabled
> {code}
> On the new brokers running 3.6.1 we saw the following errors:
>
> {code:java}
> [AddPartitionsToTxnSenderThread-1055]: AddPartitionsToTxnRequest failed for
> node 1043 with a network exception.{code}
>
> I can also see this :
> {code:java}
> [AddPartitionsToTxnManager broker=1055]Cancelled in-flight
> ADD_PARTITIONS_TO_TXN request with correlation id 21120 due to node 1043
> being disconnected (elapsed time since creation: 11ms, elapsed time since
> send: 4ms, request timeout: 3ms){code}
> We started investigating this issue and digging through the changes in 3.6,
> we came across some changes introduced as part of KAFKA-14402 that we thought
> might lead to this behaviour.
> First we could see that _transaction.partition.verification.enable_ is
> enabled by default and enables a new code path that culminates in we sending
> version 4 ADD_PARTITIONS_TO_TXN requests to other brokers that are generated
> [here|https://github.com/apache/kafka/blob/29f3260a9c07e654a28620aeb93a778622a5233d/core/src/main/scala/kafka/server/AddPartitionsToTxnManager.scala#L269].
> From a
> [discussion|https://lists.apache.org/thread/4895wrd1z92kjb708zck4s1f62xq6r8x]
> on the mailing list, [~jolshan] pointed out that this scenario shouldn't be
> possible as the following code paths should prevent version 4
> ADD_PARTITIONS_TO_TXN requests being sent to other brokers:
> [https://github.com/apache/kafka/blob/525b9b1d7682ae2a527ceca83fedca44b1cba11a/clients/src/main/java/org/apache/kafka/clients/NodeApiVersions.java#L130]
>
> [https://github.com/apache/kafka/blob/525b9b1d7682ae2a527ceca83fedca44b1cba11a/core/src/main/scala/kafka/server/AddPartitionsToTxnManager.scala#L195]
> However, these requests are still sent to other brokers in our environment.
> On further inspection of the code, I am wondering if the following code path
> could lead to this issue:
> [https://github.com/apache/kafka/blob/c4deed513057c94eb502e64490d6bdc23551d8b6/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java#L500]
> In this scenario, we don't have any _NodeApiVersions_ available for the
> specified nodeId and potentially skipping the _latestUsableVersion_ check. I
> am wondering if it is possible that because _discoverBrokerVersions_ is set
> to _false_ for the network client of the {_}AddPartitionsToTxnManager{_}, it
> skips fetching {_}NodeApiVersions{_}? I can see that we create the network
> client here:
> [https://github.com/apache/kafka/blob/c4deed513057c94eb502e64490d6bdc23551d8b6/core/src/main/scala/kafka/server/KafkaServer.scala#L641]
> The _NetworkUtils.buildNetworkClient_ method seems to create a network client
> that has _discoverBrokerVersions_ set to {_}false{_}.
> I was hoping I could get some assistance debugging this issue. Happy to
> provide any additional information needed.
>
>
>
--
This message was sent by
[jira] [Updated] (KAFKA-15045) Move Streams task assignor to public configs
[ https://issues.apache.org/jira/browse/KAFKA-15045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-15045: Fix Version/s: 3.8.0 > Move Streams task assignor to public configs > > > Key: KAFKA-15045 > URL: https://issues.apache.org/jira/browse/KAFKA-15045 > Project: Kafka > Issue Type: New Feature > Components: streams >Reporter: A. Sophie Blee-Goldman >Assignee: A. Sophie Blee-Goldman >Priority: Major > Labels: kip > Fix For: 3.8.0 > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-924%3A+customizable+task+assignment+for+Streams -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16875) Replace ClientState with TaskAssignment when creating individual consumer Assignments
[ https://issues.apache.org/jira/browse/KAFKA-16875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-16875: Component/s: streams > Replace ClientState with TaskAssignment when creating individual consumer > Assignments > - > > Key: KAFKA-16875 > URL: https://issues.apache.org/jira/browse/KAFKA-16875 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: A. Sophie Blee-Goldman >Priority: Major > > In the initial implementation of KIP-924 in version 3.8, we converted from > the new TaskAssignor's output type (TaskAssignment) into the old > ClientState-based assignment representation. This allowed us to plug in a > custom assignor without converting all the internal mechanisms that occur > after the KafkaStreams client level assignment and process it into a consumer > level assignment. > However we ultimately want to get rid of ClientState altogether, so we need > to invert this logic so that we instead convert the ClientState into a > TaskAssignment and then use the TaskAssignment to process the assigned tasks > into consumer Assignments -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16873) Remove StreamsConfig.INTERNAL_TASK_ASSIGNOR_CLASS
[ https://issues.apache.org/jira/browse/KAFKA-16873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-16873: Component/s: streams > Remove StreamsConfig.INTERNAL_TASK_ASSIGNOR_CLASS > - > > Key: KAFKA-16873 > URL: https://issues.apache.org/jira/browse/KAFKA-16873 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: A. Sophie Blee-Goldman >Priority: Major > > Once we have all the out-of-the-box assignors implementing the new > TaskAssignor interface that corresponds to the new public task assignor > config, we can remove the old internal task assignor config altogether. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16872) Remove ClientState class
[ https://issues.apache.org/jira/browse/KAFKA-16872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-16872: Component/s: streams > Remove ClientState class > > > Key: KAFKA-16872 > URL: https://issues.apache.org/jira/browse/KAFKA-16872 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: A. Sophie Blee-Goldman >Priority: Major > > One of the end-state goals of KIP-924 is to remove the ClientState class > altogether. There are some blockers to this such as the removal of the old > internal task assignor config and the old HAAssignor, so this ticket will > probably be one of the very last KAFKA-16868 subtasks to be tackled. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16871) Clean up internal AssignmentConfigs class in Streams
[ https://issues.apache.org/jira/browse/KAFKA-16871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-16871: Component/s: streams > Clean up internal AssignmentConfigs class in Streams > > > Key: KAFKA-16871 > URL: https://issues.apache.org/jira/browse/KAFKA-16871 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: A. Sophie Blee-Goldman >Priority: Major > Labels: newbie, newbie++ > > In KIP-924 we added a new public AssignmentConfigs class to hold all of the, > you guessed it, assignment related configs. > However, there is an existing config class of the same name and largely the > same contents but that's in an internal package, specifically inside the > AssignorConfiguration class. > We should remove the old AssignmentConfigs class that's in > AssignorConfiguration and replace any usages of it with the new public > AssignmentConfigs that we added in KIP-924 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16868) Post KIP-924 StreamsPartitionAssignor code cleanup
[ https://issues.apache.org/jira/browse/KAFKA-16868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-16868: Component/s: streams > Post KIP-924 StreamsPartitionAssignor code cleanup > -- > > Key: KAFKA-16868 > URL: https://issues.apache.org/jira/browse/KAFKA-16868 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: A. Sophie Blee-Goldman >Priority: Major > > Making an umbrella task for all of the tech debt and code consolidation > cleanup work that can/should be done following the implementation of > [KIP-924: customizable task assignment for > Streams|https://cwiki.apache.org/confluence/display/KAFKA/KIP-924%3A+customizable+task+assignment+for+Streams] > Most of this revolves around deduplicating code once it's no longer needed, > including classes like the ClientState, StandbyTaskAssignor and related > elements, and the old TaskAssignor interface along with its implementations. > Note that in 3.8, the first version in which KIP-924 was released, we just > added the new public config and new TaskAssignor interface but did not get > rid of the old internal config or old TaskAssignor interface. If neither > config is set in 3.8 we still default to the old HAAssignor, as a kind of > opt-in feature flag, and internally will convert the output of the new > TaskAssignor into the old style of ClientState-based assignment tracking. We > intend to clean up all of the old code and eventually support only the new > TaskAssignor interface as well as converting everything internally from the > ClientState to the TaskAssignment/KafkaStreamsAssignment style output -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-16248) Kafka consumer should cache leader offset ranges
[
https://issues.apache.org/jira/browse/KAFKA-16248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax reassigned KAFKA-16248:
---
Assignee: Alieh Saeedi
> Kafka consumer should cache leader offset ranges
>
>
> Key: KAFKA-16248
> URL: https://issues.apache.org/jira/browse/KAFKA-16248
> Project: Kafka
> Issue Type: Bug
>Reporter: Lucas Brutschy
>Assignee: Alieh Saeedi
>Priority: Critical
>
> We noticed a streams application received an OFFSET_OUT_OF_RANGE error
> following a network partition and streams task rebalance and subsequently
> reset its offsets to the beginning.
> Inspecting the logs, we saw multiple consumer log messages like:
> {code:java}
> Setting offset for partition tp to the committed offset
> FetchPosition{offset=1234, offsetEpoch=Optional.empty...)
> {code}
> Inspecting the streams code, it looks like kafka streams calls `commitSync`
> passing through an explicit OffsetAndMetadata object but does not populate
> the offset leader epoch.
> The offset leader epoch is required in the offset commit to ensure that all
> consumers in the consumer group have coherent metadata before fetching.
> Otherwise after a consumer group rebalance, a consumer may fetch with a stale
> leader epoch with respect to the committed offset and get an offset out of
> range error from a zombie partition leader.
> An example of where this can cause issues:
> 1. We have a consumer group with consumer 1 and consumer 2. Partition P is
> assigned to consumer 1 which has up-to-date metadata for P. Consumer 2 has
> stale metadata for P.
> 2. Consumer 1 fetches partition P with offset 50, epoch 8. commits the offset
> 50 without an epoch.
> 3. The consumer group rebalances and P is now assigned to consumer 2.
> Consumer 2 has a stale leader epoch for P (let's say leader epoch 7).
> Consumer 2 will now try to fetch with leader epoch 7, offset 50. If we have a
> zombie leader due to a network partition, the zombie leader may accept
> consumer 2's fetch leader epoch and return an OFFSET_OUT_OF_RANGE to consumer
> 2.
> If in step 1, consumer 1 committed the leader epoch for the message, then
> when consumer 2 receives assignment P it would force a metadata refresh to
> discover a sufficiently new leader epoch for the committed offset.
> Kafka Streams cannot fully determine the leader epoch of the offsets it wants
> to commit - in EOS mode, streams commits the offset after the last control
> records (to avoid always having a lag of >0), but the leader epoch of the
> control record is not known to streams (since only non-control records are
> returned from Consumer.poll).
> A fix discussed with [~hachikuji] is to have the consumer cache leader epoch
> ranges, similar to how the broker maintains a leader epoch cache.
> This ticket was split from the original ticket
> https://issues.apache.org/jira/browse/KAFKA-15344 which was described as a
> streams fix, but the problem cannot be fully fixed in streams.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16863) Consider removing `default.` prefix for exception handler config
[ https://issues.apache.org/jira/browse/KAFKA-16863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17850869#comment-17850869 ] Matthias J. Sax commented on KAFKA-16863: - I think `traffic_cost` was on purpose... But I really don't feel strong about it at all. In general, I am always in favor to cleanup stuff; we also just did KIP-1020. Not sure if we should do a single KIP though. I can become very convoluted quickly. I would rather to multiple smaller KIPs? Not sure what other issue there might be? Do you have a list? > Consider removing `default.` prefix for exception handler config > > > Key: KAFKA-16863 > URL: https://issues.apache.org/jira/browse/KAFKA-16863 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Matthias J. Sax >Priority: Trivial > Labels: need-kip > > Kafka Streams has a set of configs with `default.` prefix. The intent for the > default-prefix is to make a distinction between, well the default, and > in-place overwrites in the code. Eg, users can specify ts-extractors on a > per-topic basis. > However, for the deserialization- and production-exception handlers, no such > overwrites are possible, and thus, `default.` does not really make sense, > because there is just one handler overall. Via KIP-1033 we added a new > processing-exception handler w/o a default-prefix, too. > Thus, we should consider to deprecate the two existing configs names and add > them back w/o the `default.` prefix. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16863) Consider removing `default.` prefix for exception handler config
Matthias J. Sax created KAFKA-16863: --- Summary: Consider removing `default.` prefix for exception handler config Key: KAFKA-16863 URL: https://issues.apache.org/jira/browse/KAFKA-16863 Project: Kafka Issue Type: Improvement Components: streams Reporter: Matthias J. Sax Kafka Streams has a set of configs with `default.` prefix. The intent for the default-prefix is to make a distinction between, well the default, and in-place overwrites in the code. Eg, users can specify ts-extractors on a per-topic basis. However, for the deserialization- and production-exception handlers, no such overwrites are possible, and thus, `default.` does not really make sense, because there is just one handler overall. Via KIP-1033 we added a new processing-exception handler w/o a default-prefix, too. Thus, we should consider to deprecate the two existing configs names and add them back w/o the `default.` prefix. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16586) Test TaskAssignorConvergenceTest failing
[
https://issues.apache.org/jira/browse/KAFKA-16586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848476#comment-17848476
]
Matthias J. Sax commented on KAFKA-16586:
-
I did suspect a bug, too, and did include the seed :) – Just did not know that
we also need the strategy? Given that we don't have a lot of strategies, seems
we could rerun each strategy with the given seed and one should just fail?
> Test TaskAssignorConvergenceTest failing
>
>
> Key: KAFKA-16586
> URL: https://issues.apache.org/jira/browse/KAFKA-16586
> Project: Kafka
> Issue Type: Test
> Components: streams, unit tests
>Reporter: Matthias J. Sax
>Priority: Major
>
> {code:java}
> java.lang.AssertionError: Assertion failed in randomized test. Reproduce
> with: `runRandomizedScenario(-538095696758490522)`. at
> org.apache.kafka.streams.processor.internals.assignment.TaskAssignorConvergenceTest.runRandomizedScenario(TaskAssignorConvergenceTest.java:545)
> at
> org.apache.kafka.streams.processor.internals.assignment.TaskAssignorConvergenceTest.randomClusterPerturbationsShouldConverge(TaskAssignorConvergenceTest.java:479){code}
> This might expose an actual bug (or incorrect test setup) and should be
> reproducible (did not try it myself yet).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-16586) Test TaskAssignorConvergenceTest failing
[
https://issues.apache.org/jira/browse/KAFKA-16586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-16586:
Description:
{code:java}
java.lang.AssertionError: Assertion failed in randomized test. Reproduce with:
`runRandomizedScenario(-538095696758490522)`.at
org.apache.kafka.streams.processor.internals.assignment.TaskAssignorConvergenceTest.runRandomizedScenario(TaskAssignorConvergenceTest.java:545)
at
org.apache.kafka.streams.processor.internals.assignment.TaskAssignorConvergenceTest.randomClusterPerturbationsShouldConverge(TaskAssignorConvergenceTest.java:479){code}
This might expose an actual bug (or incorrect test setup) and should be
reproducible (did not try it myself yet).
was:
{code:java}
java.lang.AssertionError: Assertion failed in randomized test. Reproduce with:
`runRandomizedScenario(-538095696758490522)`.at
org.apache.kafka.streams.processor.internals.assignment.TaskAssignorConvergenceTest.runRandomizedScenario(TaskAssignorConvergenceTest.java:545)
at
org.apache.kafka.streams.processor.internals.assignment.TaskAssignorConvergenceTest.randomClusterPerturbationsShouldConverge(TaskAssignorConvergenceTest.java:479){code}
This might expose an actual bug (or incorrect test setup) and should be
reproducible (die not try it myself yet).
> Test TaskAssignorConvergenceTest failing
>
>
> Key: KAFKA-16586
> URL: https://issues.apache.org/jira/browse/KAFKA-16586
> Project: Kafka
> Issue Type: Test
> Components: streams, unit tests
>Reporter: Matthias J. Sax
>Priority: Major
>
> {code:java}
> java.lang.AssertionError: Assertion failed in randomized test. Reproduce
> with: `runRandomizedScenario(-538095696758490522)`. at
> org.apache.kafka.streams.processor.internals.assignment.TaskAssignorConvergenceTest.runRandomizedScenario(TaskAssignorConvergenceTest.java:545)
> at
> org.apache.kafka.streams.processor.internals.assignment.TaskAssignorConvergenceTest.randomClusterPerturbationsShouldConverge(TaskAssignorConvergenceTest.java:479){code}
> This might expose an actual bug (or incorrect test setup) and should be
> reproducible (did not try it myself yet).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-15242) FixedKeyProcessor testing is unusable
[ https://issues.apache.org/jira/browse/KAFKA-15242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-15242. - Assignee: (was: Alexander Aghili) Resolution: Duplicate > FixedKeyProcessor testing is unusable > - > > Key: KAFKA-15242 > URL: https://issues.apache.org/jira/browse/KAFKA-15242 > Project: Kafka > Issue Type: Bug > Components: streams >Reporter: Zlstibor Veljkovic >Priority: Major > > Using mock processor context to get the forwarded message doesn't work. > Also there is not a well documented way for testing FixedKeyProcessors. > Please see the repo at [https://github.com/zveljkovic/kafka-repro] > but most important piece is test file with runtime and compile time errors: > [https://github.com/zveljkovic/kafka-repro/blob/master/src/test/java/com/example/demo/MyFixedKeyProcessorTest.java] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16801) Streams upgrade :test target doesn't find any junit tests
[ https://issues.apache.org/jira/browse/KAFKA-16801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848022#comment-17848022 ] Matthias J. Sax commented on KAFKA-16801: - There packages contain code for system tests. We put the code under `src/test/java/...`; there is no `src/main/java/...` and the code is not unit test code either What would be the right way to address this? > Streams upgrade :test target doesn't find any junit tests > - > > Key: KAFKA-16801 > URL: https://issues.apache.org/jira/browse/KAFKA-16801 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: Greg Harris >Priority: Major > Labels: newbie > > No test executed. This behavior has been deprecated. > This will fail with an error in Gradle 9.0. > There are test sources present but no test was executed. Please check your > test configuration. > [Documentation|https://docs.gradle.org/8.7/userguide/upgrading_version_8.html#test_task_fail_on_no_test_executed] > > 23 usages > > Task::streams:upgrade-system-tests-0100:test > Task::streams:upgrade-system-tests-0101:test > Task::streams:upgrade-system-tests-0102:test > Task::streams:upgrade-system-tests-0110:test > Task::streams:upgrade-system-tests-10:test > Task::streams:upgrade-system-tests-11:test > Task::streams:upgrade-system-tests-20:test > Task::streams:upgrade-system-tests-21:test > Task::streams:upgrade-system-tests-22:test > Task::streams:upgrade-system-tests-23:test > Task::streams:upgrade-system-tests-24:test > Task::streams:upgrade-system-tests-25:test > Task::streams:upgrade-system-tests-26:test > Task::streams:upgrade-system-tests-27:test > Task::streams:upgrade-system-tests-28:test > Task::streams:upgrade-system-tests-30:test > Task::streams:upgrade-system-tests-31:test > Task::streams:upgrade-system-tests-32:test > Task::streams:upgrade-system-tests-33:test > Task::streams:upgrade-system-tests-34:test > Task::streams:upgrade-system-tests-35:test > Task::streams:upgrade-system-tests-36:test > Task::streams:upgrade-system-tests-37:test -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16794) Can't open videos in streams documentation
[
https://issues.apache.org/jira/browse/KAFKA-16794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-16794:
Component/s: streams
> Can't open videos in streams documentation
> --
>
> Key: KAFKA-16794
> URL: https://issues.apache.org/jira/browse/KAFKA-16794
> Project: Kafka
> Issue Type: Bug
> Components: docs, streams
>Reporter: Kuan Po Tseng
>Priority: Minor
> Attachments: IMG_4445.png, image.png
>
>
> Can't open videos in page [https://kafka.apache.org/documentation/streams/]
> Open console in chrome browser and it shows error message:
> {{Refused to frame 'https://www.youtube.com/' because it violates the
> following Content Security Policy directive: "frame-src 'self'".}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-15143) MockFixedKeyProcessorContext is missing from test-utils
[
https://issues.apache.org/jira/browse/KAFKA-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848021#comment-17848021
]
Matthias J. Sax commented on KAFKA-15143:
-
As pointed out on KAFKA-15242 cf comments, we also need to do something about
`FixedKeyRecord`, because it does not have (and should not have) a public
constructor. I'll point this out on the DISCUSS thread of the KIP, too.
> MockFixedKeyProcessorContext is missing from test-utils
> ---
>
> Key: KAFKA-15143
> URL: https://issues.apache.org/jira/browse/KAFKA-15143
> Project: Kafka
> Issue Type: Bug
> Components: streams-test-utils
>Affects Versions: 3.5.0
>Reporter: Tomasz Kaszuba
>Assignee: Shashwat Pandey
>Priority: Major
> Labels: needs-kip
>
> I am trying to test a ContextualFixedKeyProcessor but it is not possible to
> call the init method from within a unit test since the MockProcessorContext
> doesn't implement
> {code:java}
> FixedKeyProcessorContext {code}
> but only
> {code:java}
> ProcessorContext
> {code}
> Shouldn't there also be a *MockFixedKeyProcessorContext* in the test utils?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-15242) FixedKeyProcessor testing is unusable
[ https://issues.apache.org/jira/browse/KAFKA-15242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848020#comment-17848020 ] Matthias J. Sax commented on KAFKA-15242: - I don't think that `TestRecord` has anything to do with it, because `TestRecord` is not used in combination with `MockProcessorContext`, but only in combination with the `TopologyTestDriver` (and corresponding `TestInputTopic` and `TestOutputTopic`). I agree though, that we need some more helper class, because `FixedKeyRecord` objects cannot be instantiated directly (no public constructor). Thanks for the call out – the KIP needs to be extended accordingly – we would have missed this... This ticket did not have this dependency in its description either though. I think we can still close it as duplicate, and add anything missing to the other ticket? > FixedKeyProcessor testing is unusable > - > > Key: KAFKA-15242 > URL: https://issues.apache.org/jira/browse/KAFKA-15242 > Project: Kafka > Issue Type: Bug > Components: streams >Reporter: Zlstibor Veljkovic >Assignee: Alexander Aghili >Priority: Major > > Using mock processor context to get the forwarded message doesn't work. > Also there is not a well documented way for testing FixedKeyProcessors. > Please see the repo at [https://github.com/zveljkovic/kafka-repro] > but most important piece is test file with runtime and compile time errors: > [https://github.com/zveljkovic/kafka-repro/blob/master/src/test/java/com/example/demo/MyFixedKeyProcessorTest.java] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16774) fix flaky StreamThreadTest#shouldCloseAllTaskProducersOnCloseIfEosEnabled
[ https://issues.apache.org/jira/browse/KAFKA-16774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17847064#comment-17847064 ] Matthias J. Sax commented on KAFKA-16774: - Ah. Nice. Glad it's not a bug. Thanks for the PR Bruno. Approved. > fix flaky StreamThreadTest#shouldCloseAllTaskProducersOnCloseIfEosEnabled > - > > Key: KAFKA-16774 > URL: https://issues.apache.org/jira/browse/KAFKA-16774 > Project: Kafka > Issue Type: Test > Components: streams, unit tests >Reporter: Chia-Ping Tsai >Assignee: Bruno Cadonna >Priority: Minor > Labels: flaky-test > > java.util.ConcurrentModificationException > at > java.base/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1720) > at > java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) > at > java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) > at > java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) > at > java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) > at > java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) > at > org.apache.kafka.streams.processor.internals.TaskManager.allTasks(TaskManager.java:1686) > at > org.apache.kafka.streams.processor.internals.TaskManager.releaseLockedUnassignedTaskDirectories(TaskManager.java:1364) > at > org.apache.kafka.streams.processor.internals.TaskManager.handleRebalanceComplete(TaskManager.java:208) > at > org.apache.kafka.streams.processor.internals.StreamsRebalanceListener.onPartitionsAssigned(StreamsRebalanceListener.java:79) > at > org.apache.kafka.streams.processor.internals.StreamThreadTest.shouldCloseAllTaskProducersOnCloseIfEosEnabled(StreamThreadTest.java:1408) > at > java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-15242) FixedKeyProcessor testing is unusable
[ https://issues.apache.org/jira/browse/KAFKA-15242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17847063#comment-17847063 ] Matthias J. Sax commented on KAFKA-15242: - The example ([https://github.com/zveljkovic/kafka-repro/blob/master/src/test/java/com/example/demo/MyFixedKeyProcessorTest.java]) tries to use `MockProcessorContext` to test `FixedKeyProcessor` – this does not work. Adding `MockFixKeyProcessorContext` should allow to test` FixKeyProcessor` using this newly added class. What other issue does this ticket include that's not covered? Can you elaborate? > FixedKeyProcessor testing is unusable > - > > Key: KAFKA-15242 > URL: https://issues.apache.org/jira/browse/KAFKA-15242 > Project: Kafka > Issue Type: Bug > Components: streams >Reporter: Zlstibor Veljkovic >Assignee: Alexander Aghili >Priority: Major > > Using mock processor context to get the forwarded message doesn't work. > Also there is not a well documented way for testing FixedKeyProcessors. > Please see the repo at [https://github.com/zveljkovic/kafka-repro] > but most important piece is test file with runtime and compile time errors: > [https://github.com/zveljkovic/kafka-repro/blob/master/src/test/java/com/example/demo/MyFixedKeyProcessorTest.java] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-16448) Add Kafka Streams exception handler for exceptions occuring during processing (KIP-1033)
[ https://issues.apache.org/jira/browse/KAFKA-16448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax reassigned KAFKA-16448: --- Assignee: Loïc Greffier > Add Kafka Streams exception handler for exceptions occuring during processing > (KIP-1033) > > > Key: KAFKA-16448 > URL: https://issues.apache.org/jira/browse/KAFKA-16448 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Damien Gasparina >Assignee: Loïc Greffier >Priority: Minor > > Jira to follow work on KIP: [KIP-1033: Add Kafka Streams exception handler > for exceptions occuring during > processing|https://cwiki.apache.org/confluence/display/KAFKA/KIP-1033%3A+Add+Kafka+Streams+exception+handler+for+exceptions+occuring+during+processing] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16333) Removed Deprecated methods KTable#join
[ https://issues.apache.org/jira/browse/KAFKA-16333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846821#comment-17846821 ] Matthias J. Sax commented on KAFKA-16333: - In general yes. But we should wait until a final decision was made if the release after 3.8 will be 3.9 or 4.0. – So we should hold off a few more weeks. > Removed Deprecated methods KTable#join > -- > > Key: KAFKA-16333 > URL: https://issues.apache.org/jira/browse/KAFKA-16333 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: Matthias J. Sax >Priority: Blocker > Fix For: 4.0.0 > > > KTable#join() methods taking a `Named` parameter got deprecated in 3.1 > release via https://issues.apache.org/jira/browse/KAFKA-13813 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16329) Remove Deprecated Task/ThreadMetadata classes and related methods
[ https://issues.apache.org/jira/browse/KAFKA-16329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846820#comment-17846820 ] Matthias J. Sax commented on KAFKA-16329: - In general yes. But we should wait until a final decision was made if the release after 3.8 will be 3.9 or 4.0. – So we should hold off a few more weeks. > Remove Deprecated Task/ThreadMetadata classes and related methods > - > > Key: KAFKA-16329 > URL: https://issues.apache.org/jira/browse/KAFKA-16329 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: Matthias J. Sax >Priority: Blocker > Fix For: 4.0.0 > > > Deprecated in AK 3.0 via > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-744%3A+Migrate+TaskMetadata+and+ThreadMetadata+to+an+interface+with+internal+implementation] > > * > org.apache.kafka.streams.processor.TaskMetadata > * org.apache.kafka.streams.processo.ThreadMetadata > * org.apache.kafka.streams.KafkaStreams#localThredMetadata > * org.apache.kafka.streams.state.StreamsMetadata > * org.apache.kafka.streams.KafkaStreams#allMetadata > * org.apache.kafka.streams.KafkaStreams#allMetadataForStore > This is related https://issues.apache.org/jira/browse/KAFKA-16330 and both > ticket should be worked on together. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16330) Remove Deprecated methods/variables from TaskId
[ https://issues.apache.org/jira/browse/KAFKA-16330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846819#comment-17846819 ] Matthias J. Sax commented on KAFKA-16330: - In general yes. But we should wait until a final decision was made if the release after 3.8 will be 3.9 or 4.0. – So we should hold off a few more weeks. > Remove Deprecated methods/variables from TaskId > --- > > Key: KAFKA-16330 > URL: https://issues.apache.org/jira/browse/KAFKA-16330 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: Matthias J. Sax >Priority: Blocker > Fix For: 4.0.0 > > > Cf > [https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181306557] > > This ticket relates to https://issues.apache.org/jira/browse/KAFKA-16329 and > both should be worked on together. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16448) Add Kafka Streams exception handler for exceptions occuring during processing (KIP-1033)
[ https://issues.apache.org/jira/browse/KAFKA-16448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846818#comment-17846818 ] Matthias J. Sax commented on KAFKA-16448: - [~muralibasani] – Thanks for you interest. This ticket is already WIP and the corresponding KIP-1033 was accepted, and there is already a PR... [~Dabz] should we assign this ticket to you or Loic (not sure if he has an Jira account – if not, we should create one so we can assign the ticket to him?) > Add Kafka Streams exception handler for exceptions occuring during processing > (KIP-1033) > > > Key: KAFKA-16448 > URL: https://issues.apache.org/jira/browse/KAFKA-16448 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Damien Gasparina >Priority: Minor > > Jira to follow work on KIP: [KIP-1033: Add Kafka Streams exception handler > for exceptions occuring during > processing|https://cwiki.apache.org/confluence/display/KAFKA/KIP-1033%3A+Add+Kafka+Streams+exception+handler+for+exceptions+occuring+during+processing] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16774) fix flaky StreamThreadTest#shouldCloseAllTaskProducersOnCloseIfEosEnabled
[ https://issues.apache.org/jira/browse/KAFKA-16774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846815#comment-17846815 ] Matthias J. Sax commented on KAFKA-16774: - Wondering if this is actually a flaky test, or if it exposes a real bug? > fix flaky StreamThreadTest#shouldCloseAllTaskProducersOnCloseIfEosEnabled > - > > Key: KAFKA-16774 > URL: https://issues.apache.org/jira/browse/KAFKA-16774 > Project: Kafka > Issue Type: Test > Components: streams, unit tests >Reporter: Chia-Ping Tsai >Priority: Minor > Labels: flaky-test > > java.util.ConcurrentModificationException > at > java.base/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1720) > at > java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) > at > java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) > at > java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) > at > java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) > at > java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) > at > org.apache.kafka.streams.processor.internals.TaskManager.allTasks(TaskManager.java:1686) > at > org.apache.kafka.streams.processor.internals.TaskManager.releaseLockedUnassignedTaskDirectories(TaskManager.java:1364) > at > org.apache.kafka.streams.processor.internals.TaskManager.handleRebalanceComplete(TaskManager.java:208) > at > org.apache.kafka.streams.processor.internals.StreamsRebalanceListener.onPartitionsAssigned(StreamsRebalanceListener.java:79) > at > org.apache.kafka.streams.processor.internals.StreamThreadTest.shouldCloseAllTaskProducersOnCloseIfEosEnabled(StreamThreadTest.java:1408) > at > java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16774) fix flaky StreamThreadTest#shouldCloseAllTaskProducersOnCloseIfEosEnabled
[ https://issues.apache.org/jira/browse/KAFKA-16774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-16774: Labels: flaky-test (was: ) > fix flaky StreamThreadTest#shouldCloseAllTaskProducersOnCloseIfEosEnabled > - > > Key: KAFKA-16774 > URL: https://issues.apache.org/jira/browse/KAFKA-16774 > Project: Kafka > Issue Type: Test > Components: streams, unit tests >Reporter: Chia-Ping Tsai >Priority: Minor > Labels: flaky-test > > java.util.ConcurrentModificationException > at > java.base/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1720) > at > java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) > at > java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) > at > java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) > at > java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) > at > java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) > at > org.apache.kafka.streams.processor.internals.TaskManager.allTasks(TaskManager.java:1686) > at > org.apache.kafka.streams.processor.internals.TaskManager.releaseLockedUnassignedTaskDirectories(TaskManager.java:1364) > at > org.apache.kafka.streams.processor.internals.TaskManager.handleRebalanceComplete(TaskManager.java:208) > at > org.apache.kafka.streams.processor.internals.StreamsRebalanceListener.onPartitionsAssigned(StreamsRebalanceListener.java:79) > at > org.apache.kafka.streams.processor.internals.StreamThreadTest.shouldCloseAllTaskProducersOnCloseIfEosEnabled(StreamThreadTest.java:1408) > at > java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16774) fix flaky StreamThreadTest#shouldCloseAllTaskProducersOnCloseIfEosEnabled
[ https://issues.apache.org/jira/browse/KAFKA-16774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-16774: Component/s: streams unit tests > fix flaky StreamThreadTest#shouldCloseAllTaskProducersOnCloseIfEosEnabled > - > > Key: KAFKA-16774 > URL: https://issues.apache.org/jira/browse/KAFKA-16774 > Project: Kafka > Issue Type: Test > Components: streams, unit tests >Reporter: Chia-Ping Tsai >Priority: Minor > > java.util.ConcurrentModificationException > at > java.base/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1720) > at > java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) > at > java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) > at > java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) > at > java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) > at > java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) > at > org.apache.kafka.streams.processor.internals.TaskManager.allTasks(TaskManager.java:1686) > at > org.apache.kafka.streams.processor.internals.TaskManager.releaseLockedUnassignedTaskDirectories(TaskManager.java:1364) > at > org.apache.kafka.streams.processor.internals.TaskManager.handleRebalanceComplete(TaskManager.java:208) > at > org.apache.kafka.streams.processor.internals.StreamsRebalanceListener.onPartitionsAssigned(StreamsRebalanceListener.java:79) > at > org.apache.kafka.streams.processor.internals.StreamThreadTest.shouldCloseAllTaskProducersOnCloseIfEosEnabled(StreamThreadTest.java:1408) > at > java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-15242) FixedKeyProcessor testing is unusable
[ https://issues.apache.org/jira/browse/KAFKA-15242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846814#comment-17846814 ] Matthias J. Sax edited comment on KAFKA-15242 at 5/16/24 5:32 AM: -- There is https://issues.apache.org/jira/browse/KAFKA-15143 which is already WIP – ie, there is KIP-1027. I believe it would cover this ticket? Wondering of we can close this ticket as duplicate? was (Author: mjsax): There is https://issues.apache.org/jira/browse/KAFKA-15143 which is already WIP – ie, there is KIP-1027. I believe it would cover this ticket? > FixedKeyProcessor testing is unusable > - > > Key: KAFKA-15242 > URL: https://issues.apache.org/jira/browse/KAFKA-15242 > Project: Kafka > Issue Type: Bug > Components: streams >Reporter: Zlstibor Veljkovic >Assignee: Alexander Aghili >Priority: Major > > Using mock processor context to get the forwarded message doesn't work. > Also there is not a well documented way for testing FixedKeyProcessors. > Please see the repo at [https://github.com/zveljkovic/kafka-repro] > but most important piece is test file with runtime and compile time errors: > [https://github.com/zveljkovic/kafka-repro/blob/master/src/test/java/com/example/demo/MyFixedKeyProcessorTest.java] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-15242) FixedKeyProcessor testing is unusable
[ https://issues.apache.org/jira/browse/KAFKA-15242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846814#comment-17846814 ] Matthias J. Sax commented on KAFKA-15242: - There is https://issues.apache.org/jira/browse/KAFKA-15143 which is already WIP – ie, there is KIP-1027. I believe it would cover this ticket? > FixedKeyProcessor testing is unusable > - > > Key: KAFKA-15242 > URL: https://issues.apache.org/jira/browse/KAFKA-15242 > Project: Kafka > Issue Type: Bug > Components: streams >Reporter: Zlstibor Veljkovic >Assignee: Alexander Aghili >Priority: Major > > Using mock processor context to get the forwarded message doesn't work. > Also there is not a well documented way for testing FixedKeyProcessors. > Please see the repo at [https://github.com/zveljkovic/kafka-repro] > but most important piece is test file with runtime and compile time errors: > [https://github.com/zveljkovic/kafka-repro/blob/master/src/test/java/com/example/demo/MyFixedKeyProcessorTest.java] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16584) Make log processing summary configurable or debug
[
https://issues.apache.org/jira/browse/KAFKA-16584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845111#comment-17845111
]
Matthias J. Sax commented on KAFKA-16584:
-
I am on the PMC and can help you. :)
After you wiki account was created, please share you wiki id and we can give
you write access to the Kafka wiki space, so you can prepare a KIP.
The goal of this ticket is, to add a new config for the logging interval, so it
should not be controversial. An example of another already approved KIP that
also added a now config is
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-811%3A+Add+config+repartition.purge.interval.ms+to+Kafka+Streams]
– This should help you to write your KIP for this ticket.
> Make log processing summary configurable or debug
> -
>
> Key: KAFKA-16584
> URL: https://issues.apache.org/jira/browse/KAFKA-16584
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.6.2
>Reporter: Andras Hatvani
>Assignee: dujian
>Priority: Major
> Labels: needs-kip, newbie
>
> Currently *every two minutes for every stream thread* statistics will be
> logged on INFO log level.
> {code}
> 2024-04-18T09:18:23.790+02:00 INFO 33178 --- [service] [-StreamThread-1]
> o.a.k.s.p.internals.StreamThread : stream-thread
> [service-149405a3-c7e3-4505-8bbd-c3bff226b115-StreamThread-1] Processed 0
> total records, ran 0 punctuators, and committed 0 total tasks since the last
> update {code}
> This is absolutely unnecessary and even harmful since it fills the logs and
> thus storage space with unwanted and useless data. Otherwise the INFO logs
> are useful and helpful, therefore it's not an option to raise the log level
> to WARN.
> Please make the logProcessingSummary
> * either to a DEBUG level log or
> * make it configurable so that it can be disabled.
> This is the relevant code:
> https://github.com/apache/kafka/blob/aee9724ee15ed539ae73c09cc2c2eda83ae3c864/streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamThread.java#L1073
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-16260) Deprecate window.size.ms and window.inner.serde.class in StreamsConfig
[
https://issues.apache.org/jira/browse/KAFKA-16260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-16260:
Component/s: clients
> Deprecate window.size.ms and window.inner.serde.class in StreamsConfig
> --
>
> Key: KAFKA-16260
> URL: https://issues.apache.org/jira/browse/KAFKA-16260
> Project: Kafka
> Issue Type: Improvement
> Components: clients, streams
>Reporter: Lucia Cerchie
>Assignee: Lucia Cerchie
>Priority: Major
> Labels: KIP
>
> {{window.size.ms}} and {{window.inner.serde.class}} are not a true
> KafkaStreams config, and are ignored when set from a KStreams application.
> Both belong on the client.
> KIP-1020:
> [https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=290982804]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-16260) Deprecate window.size.ms and window.inner.serde.class in StreamsConfig
[
https://issues.apache.org/jira/browse/KAFKA-16260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-16260:
Description:
{{window.size.ms}} and {{window.inner.serde.class}} are not a true KafkaStreams
config, and are ignored when set from a KStreams application. Both belong on
the client.
KIP-1020:
[https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=290982804]
was:
{{indwindow.size.ms}} and `is not a true KafkaStreams config, and results in an
error when set from a KStreams application. It belongs on the client.
KIP-1020:
[https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=290982804]
> Deprecate window.size.ms and window.inner.serde.class in StreamsConfig
> --
>
> Key: KAFKA-16260
> URL: https://issues.apache.org/jira/browse/KAFKA-16260
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Reporter: Lucia Cerchie
>Assignee: Lucia Cerchie
>Priority: Major
> Labels: KIP
>
> {{window.size.ms}} and {{window.inner.serde.class}} are not a true
> KafkaStreams config, and are ignored when set from a KStreams application.
> Both belong on the client.
> KIP-1020:
> [https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=290982804]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-16260) Deprecate window.size.ms and window.inner.serde.class in StreamsConfig
[
https://issues.apache.org/jira/browse/KAFKA-16260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-16260:
Description:
{{indwindow.size.ms}} and `is not a true KafkaStreams config, and results in an
error when set from a KStreams application. It belongs on the client.
KIP-1020:
[https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=290982804]
was:
{{window.size.ms}} and `is not a true KafkaStreams config, and results in an
error when set from a KStreams application. It belongs on the client.
KIP-1020:
[https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=290982804]
> Deprecate window.size.ms and window.inner.serde.class in StreamsConfig
> --
>
> Key: KAFKA-16260
> URL: https://issues.apache.org/jira/browse/KAFKA-16260
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Reporter: Lucia Cerchie
>Assignee: Lucia Cerchie
>Priority: Major
> Labels: KIP
>
> {{indwindow.size.ms}} and `is not a true KafkaStreams config, and results in
> an error when set from a KStreams application. It belongs on the client.
> KIP-1020:
> [https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=290982804]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-16260) Deprecate window.size.ms and inner.serde.class in StreamsConfig
[
https://issues.apache.org/jira/browse/KAFKA-16260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-16260:
Description:
{{window.size.ms}} and `is not a true KafkaStreams config, and results in an
error when set from a KStreams application. It belongs on the client.
KIP-1020:
[https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=290982804]
was:
{{window.size.ms}} is not a true KafkaStreams config, and results in an error
when set from a KStreams application. It belongs on the client.
[https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=290982804]
> Deprecate window.size.ms and inner.serde.class in StreamsConfig
> ---
>
> Key: KAFKA-16260
> URL: https://issues.apache.org/jira/browse/KAFKA-16260
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Reporter: Lucia Cerchie
>Assignee: Lucia Cerchie
>Priority: Major
> Labels: KIP
>
> {{window.size.ms}} and `is not a true KafkaStreams config, and results in an
> error when set from a KStreams application. It belongs on the client.
> KIP-1020:
> [https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=290982804]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-16260) Deprecate window.size.ms and window.inner.serde.class in StreamsConfig
[
https://issues.apache.org/jira/browse/KAFKA-16260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-16260:
Summary: Deprecate window.size.ms and window.inner.serde.class in
StreamsConfig (was: Deprecate window.size.ms and inner.serde.class in
StreamsConfig)
> Deprecate window.size.ms and window.inner.serde.class in StreamsConfig
> --
>
> Key: KAFKA-16260
> URL: https://issues.apache.org/jira/browse/KAFKA-16260
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Reporter: Lucia Cerchie
>Assignee: Lucia Cerchie
>Priority: Major
> Labels: KIP
>
> {{window.size.ms}} and `is not a true KafkaStreams config, and results in an
> error when set from a KStreams application. It belongs on the client.
> KIP-1020:
> [https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=290982804]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-16260) Deprecate window.size.ms and inner.serde.class in StreamsConfig
[
https://issues.apache.org/jira/browse/KAFKA-16260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-16260:
Summary: Deprecate window.size.ms and inner.serde.class in StreamsConfig
(was: Deprecate window.size.ms in StreamsConfig)
> Deprecate window.size.ms and inner.serde.class in StreamsConfig
> ---
>
> Key: KAFKA-16260
> URL: https://issues.apache.org/jira/browse/KAFKA-16260
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Reporter: Lucia Cerchie
>Assignee: Lucia Cerchie
>Priority: Major
> Labels: KIP
>
> {{window.size.ms}} is not a true KafkaStreams config, and results in an
> error when set from a KStreams application. It belongs on the client.
> [https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=290982804]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-16260) Deprecate window.size.ms in StreamsConfig
[
https://issues.apache.org/jira/browse/KAFKA-16260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-16260:
Description:
{{window.size.ms}} is not a true KafkaStreams config, and results in an error
when set from a KStreams application. It belongs on the client.
[https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=290982804]
was:{{window.size.ms}} is not a true KafkaStreams config, and results in an
error when set from a KStreams application. It belongs on the client.
> Deprecate window.size.ms in StreamsConfig
> -
>
> Key: KAFKA-16260
> URL: https://issues.apache.org/jira/browse/KAFKA-16260
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Reporter: Lucia Cerchie
>Assignee: Lucia Cerchie
>Priority: Major
> Labels: KIP
>
> {{window.size.ms}} is not a true KafkaStreams config, and results in an
> error when set from a KStreams application. It belongs on the client.
> [https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=290982804]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-16260) Deprecate window.size.ms in StreamsConfig
[
https://issues.apache.org/jira/browse/KAFKA-16260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-16260:
Labels: KIP (was: needs-kip)
> Deprecate window.size.ms in StreamsConfig
> -
>
> Key: KAFKA-16260
> URL: https://issues.apache.org/jira/browse/KAFKA-16260
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Reporter: Lucia Cerchie
>Assignee: Lucia Cerchie
>Priority: Major
> Labels: KIP
>
> {{window.size.ms}} is not a true KafkaStreams config, and results in an
> error when set from a KStreams application. It belongs on the client.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16514) Kafka Streams: stream.close(CloseOptions) does not respect options.leaveGroup flag.
[
https://issues.apache.org/jira/browse/KAFKA-16514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842805#comment-17842805
]
Matthias J. Sax commented on KAFKA-16514:
-
Thanks for the background! Makes sense.
> Kafka Streams: stream.close(CloseOptions) does not respect options.leaveGroup
> flag.
> ---
>
> Key: KAFKA-16514
> URL: https://issues.apache.org/jira/browse/KAFKA-16514
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.7.0
>Reporter: Sal Sorrentino
>Priority: Minor
>
> Working with Kafka Streams 3.7.0, but may affect earlier versions as well.
> When attempting to shutdown a streams application and leave the associated
> consumer group, the supplied `leaveGroup` option seems to have no effect.
> Sample code:
> {code:java}
> CloseOptions options = new CloseOptions().leaveGroup(true);
> stream.close(options);{code}
> The expected behavior here is that the group member would shutdown and leave
> the group, immediately triggering a consumer group rebalance. In practice,
> the rebalance happens after the appropriate timeout configuration has expired.
> I understand the default behavior in that there is an assumption that any
> associated StateStores would be persisted to disk and that in the case of a
> rolling restart/deployment, the rebalance delay may be preferable. However,
> in our application we are using in-memory state stores and standby replicas.
> There is no benefit in delaying the rebalance in this setup and we are in
> need of a way to force a member to leave the group when shutting down.
> The workaround we found is to set an undocumented internal StreamConfig to
> enforce this behavior:
> {code:java}
> props.put("internal.leave.group.on.close", true);
> {code}
> To state the obvious, this is less than ideal.
> Additional configuration details:
> {code:java}
> Properties props = new Properties();
> props.put(StreamsConfig.APPLICATION_ID_CONFIG, "someApplicationId");
> props.put(
> StreamsConfig.BOOTSTRAP_SERVERS_CONFIG,
> "localhost:9092,localhost:9093,localhost:9094");
> props.put(StreamsConfig.REPLICATION_FACTOR_CONFIG, 3);
> props.put(StreamsConfig.NUM_STANDBY_REPLICAS_CONFIG, 1);
> props.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, numProcessors);
> props.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG,
> StreamsConfig.EXACTLY_ONCE_V2);{code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-16644) FK join emits duplicate tombstone on left-side delete
[ https://issues.apache.org/jira/browse/KAFKA-16644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-16644. - Resolution: Duplicate > FK join emits duplicate tombstone on left-side delete > - > > Key: KAFKA-16644 > URL: https://issues.apache.org/jira/browse/KAFKA-16644 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 3.7.0 >Reporter: Matthias J. Sax >Priority: Major > > We introduced a regression bug in 3.7.0 release via KAFKA-14748. When a > left-hand side record is deleted, the join now emits two tombstone records > instead of one. > The problem was not detected via unit test, because the tests use a `Map` > instead of a `List` when verifying the result topic records > ([https://github.com/apache/kafka/blob/trunk/streams/src/test/java/org/apache/kafka/streams/integration/KTableKTableForeignKeyJoinIntegrationTest.java#L250-L255).] > We should update all test cases to use `List` when reading from the output > topic, and of course fix the introduced bug: The > `SubscriptionSendProcessorSupplier` is sending two subscription records > instead of just a single one: > [https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/kstream/internals/foreignkeyjoin/SubscriptionSendProcessorSupplier.java#L133-L136] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16644) FK join emits duplicate tombstone on left-side delete
[ https://issues.apache.org/jira/browse/KAFKA-16644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842514#comment-17842514 ] Matthias J. Sax commented on KAFKA-16644: - Sorry. Wrong link. Fixed -> https://issues.apache.org/jira/browse/KAFKA-14748 > FK join emits duplicate tombstone on left-side delete > - > > Key: KAFKA-16644 > URL: https://issues.apache.org/jira/browse/KAFKA-16644 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 3.7.0 >Reporter: Matthias J. Sax >Priority: Major > > We introduced a regression bug in 3.7.0 release via KAFKA-14748. When a > left-hand side record is deleted, the join now emits two tombstone records > instead of one. > The problem was not detected via unit test, because the tests use a `Map` > instead of a `List` when verifying the result topic records > ([https://github.com/apache/kafka/blob/trunk/streams/src/test/java/org/apache/kafka/streams/integration/KTableKTableForeignKeyJoinIntegrationTest.java#L250-L255).] > We should update all test cases to use `List` when reading from the output > topic, and of course fix the introduced bug: The > `SubscriptionSendProcessorSupplier` is sending two subscription records > instead of just a single one: > [https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/kstream/internals/foreignkeyjoin/SubscriptionSendProcessorSupplier.java#L133-L136] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16644) FK join emits duplicate tombstone on left-side delete
[ https://issues.apache.org/jira/browse/KAFKA-16644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-16644: Description: We introduced a regression bug in 3.7.0 release via KAFKA-14748. When a left-hand side record is deleted, the join now emits two tombstone records instead of one. The problem was not detected via unit test, because the tests use a `Map` instead of a `List` when verifying the result topic records ([https://github.com/apache/kafka/blob/trunk/streams/src/test/java/org/apache/kafka/streams/integration/KTableKTableForeignKeyJoinIntegrationTest.java#L250-L255).] We should update all test cases to use `List` when reading from the output topic, and of course fix the introduced bug: The `SubscriptionSendProcessorSupplier` is sending two subscription records instead of just a single one: [https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/kstream/internals/foreignkeyjoin/SubscriptionSendProcessorSupplier.java#L133-L136] was: We introduced a regression bug in 3.7.0 release via KAFKA-14778. When a left-hand side record is deleted, the join now emits two tombstone records instead of one. The problem was not detected via unit test, because the tests use a `Map` instead of a `List` when verifying the result topic records ([https://github.com/apache/kafka/blob/trunk/streams/src/test/java/org/apache/kafka/streams/integration/KTableKTableForeignKeyJoinIntegrationTest.java#L250-L255).] We should update all test cases to use `List` when reading from the output topic, and of course fix the introduced bug: The `SubscriptionSendProcessorSupplier` is sending two subscription records instead of just a single one: [https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/kstream/internals/foreignkeyjoin/SubscriptionSendProcessorSupplier.java#L133-L136] > FK join emits duplicate tombstone on left-side delete > - > > Key: KAFKA-16644 > URL: https://issues.apache.org/jira/browse/KAFKA-16644 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 3.7.0 >Reporter: Matthias J. Sax >Priority: Major > > We introduced a regression bug in 3.7.0 release via KAFKA-14748. When a > left-hand side record is deleted, the join now emits two tombstone records > instead of one. > The problem was not detected via unit test, because the tests use a `Map` > instead of a `List` when verifying the result topic records > ([https://github.com/apache/kafka/blob/trunk/streams/src/test/java/org/apache/kafka/streams/integration/KTableKTableForeignKeyJoinIntegrationTest.java#L250-L255).] > We should update all test cases to use `List` when reading from the output > topic, and of course fix the introduced bug: The > `SubscriptionSendProcessorSupplier` is sending two subscription records > instead of just a single one: > [https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/kstream/internals/foreignkeyjoin/SubscriptionSendProcessorSupplier.java#L133-L136] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16382) Kafka Streams drop NULL values after reset
[
https://issues.apache.org/jira/browse/KAFKA-16382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842423#comment-17842423
]
Matthias J. Sax commented on KAFKA-16382:
-
Not yet from our side... Working on other things atm. Not sure when we will be
able to pick it up, or if anybody from the community wants to take it.
> Kafka Streams drop NULL values after reset
> --
>
> Key: KAFKA-16382
> URL: https://issues.apache.org/jira/browse/KAFKA-16382
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.6.1
>Reporter: Stanislav Spiridonov
>Priority: Major
>
> Kafka Streams (KTable) drops null values after full reset.
> See
> [https://github.com/foal/Null-Issue/blob/main/src/main/java/NullProblemExample.java]
> for sample topology
> Step to reproduce (req NULL-IN, NULL-IN-AUX, NULL-OUT topics)
> # Start example - 1st round
> # Send to NULL-IN "A1:a" -> NULL-OUT "A1:anull"
> # Send to NULL-IN-AUX "A1:b" -> NULL-OUT "A1:anull, A1:ab"
> # Stop application
> # Run kafka-streams-application-reset
> {code:java}
> call bin/windows/kafka-streams-application-reset --application-id
> nullproblem-example^
> --input-topics "NULL-IN,NULL-IN-AUX"^
> --bootstrap-server "localhost:9092"
> {code}
> # Send to NULL-IN-AUX "A1:" -> NULL-OUT "A1:anull, A1:ab" - it is Ok (no app
> running yet)
> # Start example - 2nd round
> # After initialization -> NULL-OUT *still contains* 2 messages "A1:anull,
> A1:ab"
> # Expected output *3 messages* "A1:anull, A1:ab, {*}A1:{*}"
> The issue is NOT reproduced if application just restarted (skip step 5).
> The issue is NOT reproduced if internal cache is disabled.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16645) CVEs in 3.7.0 docker image
[ https://issues.apache.org/jira/browse/KAFKA-16645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842419#comment-17842419 ] Matthias J. Sax commented on KAFKA-16645: - I believe fixing these CVEs should be a blocker for 3.7.1 and 3.8.0? Thoughts? > CVEs in 3.7.0 docker image > -- > > Key: KAFKA-16645 > URL: https://issues.apache.org/jira/browse/KAFKA-16645 > Project: Kafka > Issue Type: Task >Affects Versions: 3.7.0 >Reporter: Mickael Maison >Priority: Blocker > Fix For: 3.8.0, 3.7.1 > > > Our [Docker Image CVE > Scanner|https://github.com/apache/kafka/actions/runs/874393] GitHub > action reports 2 high CVEs in our base image: > apache/kafka:3.7.0 (alpine 3.19.1) > == > Total: 2 (HIGH: 2, CRITICAL: 0) > ┌──┬┬──┬┬───┬───┬─┐ > │ Library │ Vulnerability │ Severity │ Status │ Installed Version │ Fixed > Version │Title│ > ├──┼┼──┼┼───┼───┼─┤ > │ libexpat │ CVE-2023-52425 │ HIGH │ fixed │ 2.5.0-r2 │ > 2.6.0-r0 │ expat: parsing large tokens can trigger a denial of service │ > │ ││ ││ │ > │ https://avd.aquasec.com/nvd/cve-2023-52425 │ > │ ├┤ ││ > ├───┼─┤ > │ │ CVE-2024-28757 │ ││ │ > 2.6.2-r0 │ expat: XML Entity Expansion │ > │ ││ ││ │ > │ https://avd.aquasec.com/nvd/cve-2024-28757 │ > └──┴┴──┴┴───┴───┴─┘ > Looking at the > [KIP|https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka#KIP975:DockerImageforApacheKafka-WhatifweobserveabugoracriticalCVEinthereleasedApacheKafkaDockerImage?] > that introduced the docker images, it seems we should release a bugfix when > high CVEs are detected. It would be good to investigate and assess whether > Kafka is impacted or not. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16645) CVEs in 3.7.0 docker image
[ https://issues.apache.org/jira/browse/KAFKA-16645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-16645: Priority: Blocker (was: Major) > CVEs in 3.7.0 docker image > -- > > Key: KAFKA-16645 > URL: https://issues.apache.org/jira/browse/KAFKA-16645 > Project: Kafka > Issue Type: Task >Affects Versions: 3.7.0 >Reporter: Mickael Maison >Priority: Blocker > > Our [Docker Image CVE > Scanner|https://github.com/apache/kafka/actions/runs/874393] GitHub > action reports 2 high CVEs in our base image: > apache/kafka:3.7.0 (alpine 3.19.1) > == > Total: 2 (HIGH: 2, CRITICAL: 0) > ┌──┬┬──┬┬───┬───┬─┐ > │ Library │ Vulnerability │ Severity │ Status │ Installed Version │ Fixed > Version │Title│ > ├──┼┼──┼┼───┼───┼─┤ > │ libexpat │ CVE-2023-52425 │ HIGH │ fixed │ 2.5.0-r2 │ > 2.6.0-r0 │ expat: parsing large tokens can trigger a denial of service │ > │ ││ ││ │ > │ https://avd.aquasec.com/nvd/cve-2023-52425 │ > │ ├┤ ││ > ├───┼─┤ > │ │ CVE-2024-28757 │ ││ │ > 2.6.2-r0 │ expat: XML Entity Expansion │ > │ ││ ││ │ > │ https://avd.aquasec.com/nvd/cve-2024-28757 │ > └──┴┴──┴┴───┴───┴─┘ > Looking at the > [KIP|https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka#KIP975:DockerImageforApacheKafka-WhatifweobserveabugoracriticalCVEinthereleasedApacheKafkaDockerImage?] > that introduced the docker images, it seems we should release a bugfix when > high CVEs are detected. It would be good to investigate and assess whether > Kafka is impacted or not. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16645) CVEs in 3.7.0 docker image
[ https://issues.apache.org/jira/browse/KAFKA-16645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-16645: Fix Version/s: 3.8.0 3.7.1 > CVEs in 3.7.0 docker image > -- > > Key: KAFKA-16645 > URL: https://issues.apache.org/jira/browse/KAFKA-16645 > Project: Kafka > Issue Type: Task >Affects Versions: 3.7.0 >Reporter: Mickael Maison >Priority: Blocker > Fix For: 3.8.0, 3.7.1 > > > Our [Docker Image CVE > Scanner|https://github.com/apache/kafka/actions/runs/874393] GitHub > action reports 2 high CVEs in our base image: > apache/kafka:3.7.0 (alpine 3.19.1) > == > Total: 2 (HIGH: 2, CRITICAL: 0) > ┌──┬┬──┬┬───┬───┬─┐ > │ Library │ Vulnerability │ Severity │ Status │ Installed Version │ Fixed > Version │Title│ > ├──┼┼──┼┼───┼───┼─┤ > │ libexpat │ CVE-2023-52425 │ HIGH │ fixed │ 2.5.0-r2 │ > 2.6.0-r0 │ expat: parsing large tokens can trigger a denial of service │ > │ ││ ││ │ > │ https://avd.aquasec.com/nvd/cve-2023-52425 │ > │ ├┤ ││ > ├───┼─┤ > │ │ CVE-2024-28757 │ ││ │ > 2.6.2-r0 │ expat: XML Entity Expansion │ > │ ││ ││ │ > │ https://avd.aquasec.com/nvd/cve-2024-28757 │ > └──┴┴──┴┴───┴───┴─┘ > Looking at the > [KIP|https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka#KIP975:DockerImageforApacheKafka-WhatifweobserveabugoracriticalCVEinthereleasedApacheKafkaDockerImage?] > that introduced the docker images, it seems we should release a bugfix when > high CVEs are detected. It would be good to investigate and assess whether > Kafka is impacted or not. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16644) FK join emits duplicate tombstone on left-side delete
[ https://issues.apache.org/jira/browse/KAFKA-16644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842417#comment-17842417 ] Matthias J. Sax commented on KAFKA-16644: - Thanks for the pointer. I did not have time yet to dig into the details of K16394... If that's the case, we can close this as duplicate; will take care of it after I read up on K16394. > FK join emits duplicate tombstone on left-side delete > - > > Key: KAFKA-16644 > URL: https://issues.apache.org/jira/browse/KAFKA-16644 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 3.7.0 >Reporter: Matthias J. Sax >Priority: Major > > We introduced a regression bug in 3.7.0 release via KAFKA-14778. When a > left-hand side record is deleted, the join now emits two tombstone records > instead of one. > The problem was not detected via unit test, because the tests use a `Map` > instead of a `List` when verifying the result topic records > ([https://github.com/apache/kafka/blob/trunk/streams/src/test/java/org/apache/kafka/streams/integration/KTableKTableForeignKeyJoinIntegrationTest.java#L250-L255).] > We should update all test cases to use `List` when reading from the output > topic, and of course fix the introduced bug: The > `SubscriptionSendProcessorSupplier` is sending two subscription records > instead of just a single one: > [https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/kstream/internals/foreignkeyjoin/SubscriptionSendProcessorSupplier.java#L133-L136] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16514) Kafka Streams: stream.close(CloseOptions) does not respect options.leaveGroup flag.
[
https://issues.apache.org/jira/browse/KAFKA-16514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842229#comment-17842229
]
Matthias J. Sax commented on KAFKA-16514:
-
Cool. You can find details on the wiki:
[https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals]
– if you don't get to it, also totally ok; Sophie mentioned that she might also
be able to pick it up (but most likely not for 3.8 release...)
(Btw: the wiki account creation via self-service is currently broken, but we
can create an account manually if you don't have one.)
Would you be interested to do a PR to update the JavaDocs in the meantime to
fix them?
> Kafka Streams: stream.close(CloseOptions) does not respect options.leaveGroup
> flag.
> ---
>
> Key: KAFKA-16514
> URL: https://issues.apache.org/jira/browse/KAFKA-16514
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.7.0
>Reporter: Sal Sorrentino
>Priority: Minor
>
> Working with Kafka Streams 3.7.0, but may affect earlier versions as well.
> When attempting to shutdown a streams application and leave the associated
> consumer group, the supplied `leaveGroup` option seems to have no effect.
> Sample code:
> {code:java}
> CloseOptions options = new CloseOptions().leaveGroup(true);
> stream.close(options);{code}
> The expected behavior here is that the group member would shutdown and leave
> the group, immediately triggering a consumer group rebalance. In practice,
> the rebalance happens after the appropriate timeout configuration has expired.
> I understand the default behavior in that there is an assumption that any
> associated StateStores would be persisted to disk and that in the case of a
> rolling restart/deployment, the rebalance delay may be preferable. However,
> in our application we are using in-memory state stores and standby replicas.
> There is no benefit in delaying the rebalance in this setup and we are in
> need of a way to force a member to leave the group when shutting down.
> The workaround we found is to set an undocumented internal StreamConfig to
> enforce this behavior:
> {code:java}
> props.put("internal.leave.group.on.close", true);
> {code}
> To state the obvious, this is less than ideal.
> Additional configuration details:
> {code:java}
> Properties props = new Properties();
> props.put(StreamsConfig.APPLICATION_ID_CONFIG, "someApplicationId");
> props.put(
> StreamsConfig.BOOTSTRAP_SERVERS_CONFIG,
> "localhost:9092,localhost:9093,localhost:9094");
> props.put(StreamsConfig.REPLICATION_FACTOR_CONFIG, 3);
> props.put(StreamsConfig.NUM_STANDBY_REPLICAS_CONFIG, 1);
> props.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, numProcessors);
> props.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG,
> StreamsConfig.EXACTLY_ONCE_V2);{code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
