[jira] [Commented] (KAFKA-10800) Validate the snapshot id when the state machine creates a snapshot
[
https://issues.apache.org/jira/browse/KAFKA-10800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17331149#comment-17331149
]
Haoran Xuan commented on KAFKA-10800:
-
[~jagsancio] Thanks for the clarification, now I'm clear about the validation
part. :)
About these:
{code:java}
1. Set the baseOffset to the snapshotId's endOffset - 1 for every record batch.
2. Set the relative offset for every record to 0.
3. Se the epoch of every batch to the snapshotId's epoch.{code}
I think it's easy to understand 3, but I'm curious why would we need to do 1
and 2? I guess there should be some benefit or restriction which I'm not
realized? Thanks!
> Validate the snapshot id when the state machine creates a snapshot
> --
>
> Key: KAFKA-10800
> URL: https://issues.apache.org/jira/browse/KAFKA-10800
> Project: Kafka
> Issue Type: Sub-task
> Components: replication
>Reporter: Jose Armando Garcia Sancio
>Assignee: Haoran Xuan
>Priority: Major
>
> When the state machine attempts to create a snapshot writer we should
> validate that the following is true:
> # The end offset and epoch of the snapshot is less than the high-watermark.
> # The end offset and epoch of the snapshot is valid based on the leader
> epoch cache.
> Note that this validation should not be performed when the raft client
> creates the snapshot writer because in that case the local log is out of date
> and the follower should trust the snapshot id sent by the partition leader.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [kafka] ableegoldman commented on a change in pull request #10573: KAFKA-12574: KIP-732, Deprecate eos-alpha and replace eos-beta with eos-v2
ableegoldman commented on a change in pull request #10573:
URL: https://github.com/apache/kafka/pull/10573#discussion_r619559055
##
File path:
streams/src/test/java/org/apache/kafka/streams/processor/internals/RecordCollectorTest.java
##
@@ -126,10 +129,10 @@ public void setup() {
clientSupplier.setCluster(cluster);
streamsProducer = new StreamsProducer(
config,
-"threadId",
+processId + "-StreamThread-1",
Review comment:
StreamsProducer tries to parse the thread name to get the `processId`
for the null check that's only done for eos-v2 (see above)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #10573: KAFKA-12574: KIP-732, Deprecate eos-alpha and replace eos-beta with eos-v2
ableegoldman commented on a change in pull request #10573: URL: https://github.com/apache/kafka/pull/10573#discussion_r619558964 ## File path: streams/src/test/java/org/apache/kafka/streams/processor/internals/RecordCollectorTest.java ## @@ -111,6 +113,7 @@ private final StringSerializer stringSerializer = new StringSerializer(); private final ByteArraySerializer byteArraySerializer = new ByteArraySerializer(); +private final UUID processId = UUID.randomUUID(); Review comment: StreamsProducer requires the processID to be non-null, but only for eos-v2 for some reason -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #10573: KAFKA-12574: KIP-732, Deprecate eos-alpha and replace eos-beta with eos-v2
ableegoldman commented on a change in pull request #10573:
URL: https://github.com/apache/kafka/pull/10573#discussion_r619558819
##
File path:
streams/src/main/java/org/apache/kafka/streams/processor/internals/ActiveTaskCreator.java
##
@@ -115,7 +115,7 @@ public void reInitializeThreadProducer() {
StreamsProducer streamsProducerForTask(final TaskId taskId) {
if (processingMode != EXACTLY_ONCE_ALPHA) {
-throw new IllegalStateException("Producer per thread is used.");
+throw new IllegalStateException("Expected EXACTLY_ONCE to be
enabled, but the processing mode was " + processingMode);
Review comment:
I'll do it the other way around to be consistent with how we print the
`processingMode` (ie change `eos-v2` to `EXACTLY_ONCE_V2` below instead)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #10573: KAFKA-12574: KIP-732, Deprecate eos-alpha and replace eos-beta with eos-v2
ableegoldman commented on a change in pull request #10573:
URL: https://github.com/apache/kafka/pull/10573#discussion_r619558551
##
File path:
streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamThread.java
##
@@ -603,7 +606,7 @@ boolean runLoop() {
log.error("Shutting down because the Kafka cluster seems
to be on a too old version. " +
"Setting {}=\"{}\" requires broker version 2.5
or higher.",
StreamsConfig.PROCESSING_GUARANTEE_CONFIG,
- EXACTLY_ONCE_BETA);
+ StreamsConfig.EXACTLY_ONCE_V2);
Review comment:
My rationale was that if a user is hitting this then it's presumably a
new application that they just tried to enable eos-v2 with, without upgrading
their brokers. And if it's a new 3.0 application then why would they have
chosen the deprecated eos-beta config over the eos-v2 config.
But I can try to print the actual result, it just requires a little extra
code to look this up and save it since we don't track the configs after the
thread's creation. Not the end of the world, but didn't feel necessary to me.
Thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #10573: KAFKA-12574: KIP-732, Deprecate eos-alpha and replace eos-beta with eos-v2
ableegoldman commented on a change in pull request #10573:
URL: https://github.com/apache/kafka/pull/10573#discussion_r619557786
##
File path: streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java
##
@@ -291,23 +293,35 @@
*
* Enabling exactly-once processing semantics requires broker version
0.11.0 or higher.
* If you enable this feature Kafka Streams will use more resources (like
broker connections)
- * compared to the {@link #AT_LEAST_ONCE} case.
+ * compared to {@link #AT_LEAST_ONCE "at_least_once"} and {@link
#EXACTLY_ONCE_V2 "exactly_once_v2"}.
*
- * @see #EXACTLY_ONCE_BETA
+ * @deprecated Since 3.0.0, will be removed in 4.0. Use {@link
#EXACTLY_ONCE_V2 "exactly_once_v2"} instead.
Review comment:
See comment above -- I'm ok with removing this for the
`sendOffsetsToTransaction` since all users have to do there is update one line
of code, but for eos that may require a broker upgrade I'd rather leave this in
to encourage them to do so
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #10573: KAFKA-12574: KIP-732, Deprecate eos-alpha and replace eos-beta with eos-v2
ableegoldman commented on a change in pull request #10573: URL: https://github.com/apache/kafka/pull/10573#discussion_r619557404 ## File path: docs/streams/upgrade-guide.html ## @@ -93,6 +95,12 @@ Upgrade Guide and API Changes Streams API changes in 3.0.0 + + The StreamsConfig.EXACTLY_ONCE and StreamsConfig.EXACTLY_ONCE_BETA configs have been deprecated, and a new StreamsConfig.EXACTLY_ONCE_V2 config has been + introduced. This is the same feature as eos-beta, but renamed to highlight its production-readiness. Users of exactly-once semantics should plan to migrate to the eos-v2 config and prepare for the removal of the deprecated configs in 4.0 or after at least a year + from the release of 3.0, whichever comes last. Note that eos-v2 requires broker version 2.5 or higher, like eos-beta, so users should begin to upgrade their kafka cluster if necessary. See Review comment: From my perspective, it's ok to be flexible or to backtrack and push the removal date out further than we claimed, but not the other way around. In other words we may as well be aggressive and say we plan to remove it in 4.0 now, if it hasn't been a year by the time 4.0 rolls around or we aren't ready for whatever reason then we don't have to do it. Users will actually probably be happy to have more time since let's be real, many of them will not yet have upgraded their brokers -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #10573: KAFKA-12574: KIP-732, Deprecate eos-alpha and replace eos-beta with eos-v2
ableegoldman commented on a change in pull request #10573: URL: https://github.com/apache/kafka/pull/10573#discussion_r619557012 ## File path: docs/streams/upgrade-guide.html ## @@ -93,6 +95,12 @@ Upgrade Guide and API Changes Streams API changes in 3.0.0 + + The StreamsConfig.EXACTLY_ONCE and StreamsConfig.EXACTLY_ONCE_BETA configs have been deprecated, and a new StreamsConfig.EXACTLY_ONCE_V2 config has been + introduced. This is the same feature as eos-beta, but renamed to highlight its production-readiness. Users of exactly-once semantics should plan to migrate to the eos-v2 config and prepare for the removal of the deprecated configs in 4.0 or after at least a year + from the release of 3.0, whichever comes last. Note that eos-v2 requires broker version 2.5 or higher, like eos-beta, so users should begin to upgrade their kafka cluster if necessary. See Review comment: We specifically voted on this in the KIP. Personally I was a bit hesitant when someone suggested it, but I think the motivation there was to give users some slight sense of urgency so they don't put off upgrading their code or brokers, and then complain if/when we remove it because they haven't done so yet. Not sure if you saw the WW3 thread that John started when he synced over the commit that removed all the deprecated methods in 3.0... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #10573: KAFKA-12574: KIP-732, Deprecate eos-alpha and replace eos-beta with eos-v2
ableegoldman commented on a change in pull request #10573: URL: https://github.com/apache/kafka/pull/10573#discussion_r619556206 ## File path: docs/streams/upgrade-guide.html ## @@ -93,6 +95,12 @@ Upgrade Guide and API Changes Streams API changes in 3.0.0 + + The StreamsConfig.EXACTLY_ONCE and StreamsConfig.EXACTLY_ONCE_BETA configs have been deprecated, and a new StreamsConfig.EXACTLY_ONCE_V2 config has been Review comment: I feel like it does make more sense to include the actual variable here, since that's what's being deprecated. Also that way no one has to pause and think "hm, is that in StreamsConfig or ConsumerConfig or..." -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #10573: KAFKA-12574: KIP-732, Deprecate eos-alpha and replace eos-beta with eos-v2
ableegoldman commented on a change in pull request #10573: URL: https://github.com/apache/kafka/pull/10573#discussion_r619555279 ## File path: docs/streams/upgrade-guide.html ## @@ -93,6 +95,12 @@ Upgrade Guide and API Changes Streams API changes in 3.0.0 + + The StreamsConfig.EXACTLY_ONCE and StreamsConfig.EXACTLY_ONCE_BETA configs have been deprecated, and a new StreamsConfig.EXACTLY_ONCE_V2 config has been + introduced. This is the same feature as eos-beta, but renamed to highlight its production-readiness. Users of exactly-once semantics should plan to migrate to the eos-v2 config and prepare for the removal of the deprecated configs in 4.0 or after at least a year Review comment: Hm, I thought this phrasing did imply it was already production-ready and that we just changed the name to highlight that fact. Do you have suggestions for how to word this better? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #10573: KAFKA-12574: KIP-732, Deprecate eos-alpha and replace eos-beta with eos-v2
ableegoldman commented on a change in pull request #10573: URL: https://github.com/apache/kafka/pull/10573#discussion_r619554270 ## File path: docs/streams/upgrade-guide.html ## @@ -53,17 +53,19 @@ Upgrade Guide and API Changes -Starting in Kafka Streams 2.6.x, a new processing mode is available, named EOS version 2, which is configurable by setting -processing.guarantee to "exactly_once_beta". -NOTE: The "exactly_once_beta" processing mode is ready for production (i.e., it's not "beta" software). +Starting in Kafka Streams 2.6.x, a new processing mode is available, named EOS version 2. This can be configured +by setting StreamsConfig.PROCESSING_GUARANTEE to StreamsConfig.EXACTLY_ONCE_V2 for +application versions 3.0+, or setting it to StreamsConfig.EXACTLY_ONCE_BETA for versions between 2.6 and 2.8. To use this new feature, your brokers must be on version 2.5.x or newer. -A switch from "exactly_once" to "exactly_once_beta" (or the other way around) is -only possible if the application is on version 2.6.x. -If you want to upgrade your application from an older version and enable this feature, -you first need to upgrade your application to version 2.6.x, staying on "exactly_once", -and then do second round of rolling bounces to switch to "exactly_once_beta". -For a downgrade, do the reverse: first switch the config from "exactly_once_beta" to -"exactly_once" to disable the feature in your 2.6.x application. +If you want to upgrade your EOS application from an older version and enable this feature in version 3.0+, +you first need to upgrade your application to version 3.0.x, staying on StreamsConfig.EXACTLY_ONCE, Review comment: > If users have a config properties / text file, you would use the string. I just thought it was clearer, but I didn't think about that. I'll revert this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] junrao commented on a change in pull request #10271: KAFKA-12429: Added serdes for the default implementation of RLMM based on an internal topic as storage.
junrao commented on a change in pull request #10271:
URL: https://github.com/apache/kafka/pull/10271#discussion_r619545527
##
File path:
storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/RemoteLogMetadataContext.java
##
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.server.log.remote.metadata.storage;
+
+import java.util.Objects;
+
+/**
+ * The context associated with the record in remote log metadata topic. This
contains api-key, and the payload object.
+ *
+ *
+ * For example:
+ * Remote log segment metadata record will have
+ *
+ *
+ * api key as: {@link
org.apache.kafka.server.log.remote.metadata.storage.generated.RemoteLogSegmentMetadataRecord#apiKey()}
+ * payload as: {@link
org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata}
+ *
+ *
+ *
+ *
+ * You can read more details in https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-MessageFormat;>KIP-405
+ */
+public class RemoteLogMetadataContext {
+private final short apiKey;
Review comment:
This class is now only used for holding the apiKey. We could just derive
the apiKey from the specific class like RemoteLogSegmentMetadata?
##
File path:
raft/src/main/java/org/apache/kafka/raft/metadata/AbstractMetadataRecordSerde.java
##
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.raft.metadata;
+
+import org.apache.kafka.common.errors.SerializationException;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.protocol.ObjectSerializationCache;
+import org.apache.kafka.common.protocol.Readable;
+import org.apache.kafka.common.protocol.Writable;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.raft.RecordSerde;
+
+/**
+ * This is an implementation of {@code RecordSerde} with {@code
ApiMessageAndVersion} but implementors need to implement
+ * {@link #apiMessageFor(short)} to return a {@code ApiMessage} instance for
the given {@code apiKey}.
+ *
+ * This can be used as the underlying serialization mechanism for any metadata
kind of log storage.
+ */
+public abstract class AbstractMetadataRecordSerde implements
RecordSerde {
Review comment:
(1) AbstractMetadataRecordSerde => WritableApiMessageSerde ?
(2) Also, could we remove the reference to raft in RecordSerde since it's no
longer just for raft?
##
File path:
storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/RemoteLogMetadataContext.java
##
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
[jira] [Commented] (KAFKA-12711) Add a back off option to replace thread
[ https://issues.apache.org/jira/browse/KAFKA-12711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17331098#comment-17331098 ] Guozhang Wang commented on KAFKA-12711: --- On the other hand, I think maybe we do not, at least in the first attempt, need to expose backoff to users to customize: we can consider just hard-code an exponential backoff mechanism inside Streams (we can pick some magic numbers here) to begin with, and hence no need for a KIP. > Add a back off option to replace thread > --- > > Key: KAFKA-12711 > URL: https://issues.apache.org/jira/browse/KAFKA-12711 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Walker Carlson >Priority: Major > Labels: needs-kip > > There should be a native option to set a back off period when replacing a > thread from the stream uncaught exception handler. > > Either there should be a config and a user chosen strategy or a value you can > set in the handler that causes a delay in creating the new thread. > > It might also be achieved by making the option to add a thread in N seconds > then using that in the replace thread option -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10493) KTable out-of-order updates are not being ignored
[ https://issues.apache.org/jira/browse/KAFKA-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17331097#comment-17331097 ] Guozhang Wang commented on KAFKA-10493: --- Without KIP-280, it would always be an issue; what I'm trying to say is that we should at least remove the other incorrect behaviors while we go towards the end goal. If log compaction did cause out of record to be effectively removing new records, then yes we are still in bad state, but at least we are not making things worse. > KTable out-of-order updates are not being ignored > - > > Key: KAFKA-10493 > URL: https://issues.apache.org/jira/browse/KAFKA-10493 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.6.0 >Reporter: Pedro Gontijo >Assignee: Matthias J. Sax >Priority: Blocker > Fix For: 3.0.0 > > Attachments: KTableOutOfOrderBug.java > > > On a materialized KTable, out-of-order records for a given key (records which > timestamp are older than the current value in store) are not being ignored > but used to update the local store value and also being forwarded. > I believe the bug is here: > [https://github.com/apache/kafka/blob/2.6.0/streams/src/main/java/org/apache/kafka/streams/state/internals/ValueAndTimestampSerializer.java#L77] > It should return true, not false (see javadoc) > The bug impacts here: > [https://github.com/apache/kafka/blob/2.6.0/streams/src/main/java/org/apache/kafka/streams/kstream/internals/KTableSource.java#L142-L148] > I have attached a simple stream app that shows the issue happening. > Thank you! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12711) Add a back off option to replace thread
[ https://issues.apache.org/jira/browse/KAFKA-12711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17331095#comment-17331095 ] Guozhang Wang commented on KAFKA-12711: --- I'm +1 for this feature. Generally speaking, we should make any exceptional handler implementation to be instantaneous with a decision returned from the thread, if the decision is a "delayed operation" then KS could schedule that to happen internally, instead letting the impl to get on hold of that throwing thread. To cope with that rule of thumb, maybe we can adjust our FSM as well such that, before the exception handler is triggered, the Streams instance state transit to a new state, like "pending", and only transit back to the corresponding new state when exception handler returns. We can also consider piggy-back on the "pending error" state, but that would mean that this state do not only transit to "error" but can also transit back to normal states. Personally I'm feeling okay to piggy-back on pending error so that we do not introduce a new state. > Add a back off option to replace thread > --- > > Key: KAFKA-12711 > URL: https://issues.apache.org/jira/browse/KAFKA-12711 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Walker Carlson >Priority: Major > Labels: needs-kip > > There should be a native option to set a back off period when replacing a > thread from the stream uncaught exception handler. > > Either there should be a config and a user chosen strategy or a value you can > set in the handler that causes a delay in creating the new thread. > > It might also be achieved by making the option to add a thread in N seconds > then using that in the replace thread option -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [kafka] vitojeng commented on pull request #9821: KAFKA-5876: Apply UnknownStateStoreException for Interactive Queries
vitojeng commented on pull request #9821: URL: https://github.com/apache/kafka/pull/9821#issuecomment-825981428 > Merged to trunk, ready for the next PR Will do, thanks @ableegoldman ! :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] mjsax commented on a change in pull request #10573: KAFKA-12574: KIP-732, Deprecate eos-alpha and replace eos-beta with eos-v2
mjsax commented on a change in pull request #10573:
URL: https://github.com/apache/kafka/pull/10573#discussion_r619504008
##
File path:
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java
##
@@ -642,7 +647,10 @@ public void beginTransaction() throws
ProducerFencedException {
* to the partition leader. See the exception for more details
* @throws KafkaException if the producer has encountered a previous fatal
or abortable error, or for any
* other unexpected error
+ *
+ * @deprecated Since 3.0.0, will be removed in 4.0. Use {@link
#sendOffsetsToTransaction(Map, ConsumerGroupMetadata)} instead.
Review comment:
I don't think we will be able to remove it in 4.0 and I would assume
that 4.0 is too early, and we usually keep API around for at least 1 year after
deprecation. Maybe best to just remove this part?
##
File path: docs/streams/developer-guide/config-streams.html
##
@@ -293,8 +293,9 @@ bootstrap.serversprocessing.guarantee
Medium
-The processing mode. Can be either "at_least_once"
(default),
- "exactly_once" (for EOS version 1), or "exactly_once_beta"
(for EOS version 2).
+The processing mode. Can be either "at_least_once"
(default)
+ or "exactly_once_v2" (for EOS version 2, requires broker
version 2.5+). Deprecated config options are
+ "exactly_once" (for EOS version 1) and "exactly_once_beta"
(for EOS beta, requires broker version 2.5+).
Review comment:
nit `for EOS beta` -> `for EOS version 2`
(Might be really good to get rid of the term "beta" whenever we can -- and
also make it more clear that we just renamed this config.)
##
File path: docs/streams/upgrade-guide.html
##
@@ -93,6 +95,12 @@ Upgrade Guide and API Changes
Streams API changes in 3.0.0
+
+ The StreamsConfig.EXACTLY_ONCE and
StreamsConfig.EXACTLY_ONCE_BETA configs have been deprecated, and
a new StreamsConfig.EXACTLY_ONCE_V2 config has been
+ introduced. This is the same feature as eos-beta, but renamed to
highlight its production-readiness. Users of exactly-once semantics should plan
to migrate to the eos-v2 config and prepare for the removal of the deprecated
configs in 4.0 or after at least a year
+ from the release of 3.0, whichever comes last. Note that eos-v2 requires
broker version 2.5 or higher, like eos-beta, so users should begin to upgrade
their kafka cluster if necessary. See
Review comment:
Personally, I would not talk about when we plan to remove stuff, because
the plan might change.
##
File path: streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java
##
@@ -291,23 +293,35 @@
*
* Enabling exactly-once processing semantics requires broker version
0.11.0 or higher.
* If you enable this feature Kafka Streams will use more resources (like
broker connections)
- * compared to the {@link #AT_LEAST_ONCE} case.
+ * compared to {@link #AT_LEAST_ONCE "at_least_once"} and {@link
#EXACTLY_ONCE_V2 "exactly_once_v2"}.
*
- * @see #EXACTLY_ONCE_BETA
+ * @deprecated Since 3.0.0, will be removed in 4.0. Use {@link
#EXACTLY_ONCE_V2 "exactly_once_v2"} instead.
*/
@SuppressWarnings("WeakerAccess")
+@Deprecated
public static final String EXACTLY_ONCE = "exactly_once";
/**
* Config value for parameter {@link #PROCESSING_GUARANTEE_CONFIG
"processing.guarantee"} for exactly-once processing guarantees.
*
* Enabling exactly-once (beta) requires broker version 2.5 or higher.
- * If you enable this feature Kafka Streams will use less resources (like
broker connections)
- * compare to the {@link #EXACTLY_ONCE} case.
+ * If you enable this feature Kafka Streams will use fewer resources (like
broker connections)
+ * compared to the {@link #EXACTLY_ONCE} case.
Review comment:
`{@link #EXACTLY_ONCE} (deprecated) case.`
##
File path: docs/streams/developer-guide/config-streams.html
##
@@ -667,12 +668,14 @@ probing.rebalance.interval.msprocessing.guarantee
The processing guarantee that should be used.
- Possible values are "at_least_once" (default),
- "exactly_once" (for EOS version 1),
- and "exactly_once_beta" (for EOS version 2).
- Using "exactly_once" requires broker
- version 0.11.0 or newer, while using "exactly_once_beta"
- requires broker version 2.5 or newer.
+ Possible values are "at_least_once" (default)
+ and "exactly_once_v2" (for EOS version 2).
+ Deprecated config options are "exactly_once" (for EOS alpha),
Review comment:
Fair enough.
##
File path: docs/streams/developer-guide/config-streams.html
##
@@ -667,12 +668,14 @@
[jira] [Commented] (KAFKA-10493) KTable out-of-order updates are not being ignored
[ https://issues.apache.org/jira/browse/KAFKA-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17331042#comment-17331042 ] Matthias J. Sax commented on KAFKA-10493: - Not sure if I understand the idea about "restore by timestamp" – if compaction did delete the record with larger timestamp but lower offset, the record is gone, and only the out-of-order record is left. I do agree that versioned tables would also help on this issue, but I guess the bottom line question is about timeline. We had the idea to address this ticket in 3.0.0, but given the current discussion, I am not sure any longer if we should really do it, or if we would need to wait for other ticket to be address first? > KTable out-of-order updates are not being ignored > - > > Key: KAFKA-10493 > URL: https://issues.apache.org/jira/browse/KAFKA-10493 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.6.0 >Reporter: Pedro Gontijo >Assignee: Matthias J. Sax >Priority: Blocker > Fix For: 3.0.0 > > Attachments: KTableOutOfOrderBug.java > > > On a materialized KTable, out-of-order records for a given key (records which > timestamp are older than the current value in store) are not being ignored > but used to update the local store value and also being forwarded. > I believe the bug is here: > [https://github.com/apache/kafka/blob/2.6.0/streams/src/main/java/org/apache/kafka/streams/state/internals/ValueAndTimestampSerializer.java#L77] > It should return true, not false (see javadoc) > The bug impacts here: > [https://github.com/apache/kafka/blob/2.6.0/streams/src/main/java/org/apache/kafka/streams/kstream/internals/KTableSource.java#L142-L148] > I have attached a simple stream app that shows the issue happening. > Thank you! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-12711) Add a back off option to replace thread
[ https://issues.apache.org/jira/browse/KAFKA-12711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Walker Carlson updated KAFKA-12711: --- Description: There should be a native option to set a back off period when replacing a thread from the stream uncaught exception handler. Either there should be a config and a user chosen strategy or a value you can set in the handler that causes a delay in creating the new thread. It might also be achieved by making the option to add a thread in N seconds then using that in the replace thread option was: There should be a native option to set a back off period when replacing a thread. Either there should be a config and a user chosen strategy or a value you can set in the handler that causes a delay in creating the new thread. > Add a back off option to replace thread > --- > > Key: KAFKA-12711 > URL: https://issues.apache.org/jira/browse/KAFKA-12711 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Walker Carlson >Priority: Major > Labels: needs-kip > > There should be a native option to set a back off period when replacing a > thread from the stream uncaught exception handler. > > Either there should be a config and a user chosen strategy or a value you can > set in the handler that causes a delay in creating the new thread. > > It might also be achieved by making the option to add a thread in N seconds > then using that in the replace thread option -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-12711) Add a back off option to replace thread
[ https://issues.apache.org/jira/browse/KAFKA-12711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Walker Carlson updated KAFKA-12711: --- Summary: Add a back off option to replace thread (was: Add a back off option to Replace thread) > Add a back off option to replace thread > --- > > Key: KAFKA-12711 > URL: https://issues.apache.org/jira/browse/KAFKA-12711 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Walker Carlson >Priority: Major > Labels: needs-kip > > There should be a native option to set a back off period when replacing a > thread. > > Either there should be a config and a user chosen strategy or a value you can > set in the handler that causes a delay in creating the new thread. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-12711) Add a back off option to Replace thread
[ https://issues.apache.org/jira/browse/KAFKA-12711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-12711: Labels: needs-kip (was: ) > Add a back off option to Replace thread > --- > > Key: KAFKA-12711 > URL: https://issues.apache.org/jira/browse/KAFKA-12711 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Walker Carlson >Priority: Major > Labels: needs-kip > > There should be a native option to set a back off period when replacing a > thread. > > Either there should be a config and a user chosen strategy or a value you can > set in the handler that causes a delay in creating the new thread. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [kafka] ableegoldman commented on pull request #9821: KAFKA-5876: Apply UnknownStateStoreException for Interactive Queries
ableegoldman commented on pull request #9821: URL: https://github.com/apache/kafka/pull/9821#issuecomment-825863470 Merged to trunk, ready for the next PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] ableegoldman merged pull request #9821: KAFKA-5876: Apply UnknownStateStoreException for Interactive Queries
ableegoldman merged pull request #9821: URL: https://github.com/apache/kafka/pull/9821 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] ableegoldman commented on pull request #9821: KAFKA-5876: Apply UnknownStateStoreException for Interactive Queries
ableegoldman commented on pull request #9821: URL: https://github.com/apache/kafka/pull/9821#issuecomment-825862826 Just two unrelated test failures in the build: `connect.mirror.integration.MirrorConnectorsIntegrationSSLTest.testReplication()` `kafka.integration.MetricsDuringTopicCreationDeletionTest.testMetricsDuringTopicCreateDelete()` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KAFKA-12710) Consider enabling (at least some) optimizations by default
[ https://issues.apache.org/jira/browse/KAFKA-12710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17330995#comment-17330995 ] A. Sophie Blee-Goldman commented on KAFKA-12710: Thanks, I'd forgotten about that KIP. Being able to selectively disable optimizations would make enabling (at least some) optimizations by default much more palatable, if we can do them together that would be ideal. > Consider enabling (at least some) optimizations by default > -- > > Key: KAFKA-12710 > URL: https://issues.apache.org/jira/browse/KAFKA-12710 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: A. Sophie Blee-Goldman >Priority: Major > > Topology optimizations such as the repartition consolidation and source topic > changelog are extremely useful at reducing the footprint of a Kafka Streams > application on the broker. The additional storage and resource utilization > due to changelogs and repartitions is a very real pain point, and has even > been cited as the reason for turning to other stream processing frameworks in > the past (though of course I question that judgement) > The repartition topic optimization, at the very least, should be enabled by > default. The problem is that we can't just flip the switch without breaking > existing applications during upgrade, since the location and name of such > topics in the topology may change. One possibility is to just detect this > situation and disable the optimization if we find that it would produce an > incompatible topology for an existing application. We can determine that this > is the case simply by looking for pre-existing repartition topics. If any > such topics are present, and match the set of repartition topics in the > un-optimized topology, then we know we need to switch the optimization off. > If we don't find any repartition topics, or they match the optimized > topology, then we're safe to enable it by default. > Alternatively, we could just do a KIP to indicate that we intend to change > the default in the next breaking release and that existing applications > should override this config if necessary. We should be able to implement a > fail-safe and shut down if a user misses or forgets to do so, using the > method mentioned above. > The source topic optimization is perhaps more controversial, as there have > been a few issues raised with regards to things like [restoring bad data and > asymmetric serdes|https://issues.apache.org/jira/browse/KAFKA-8037], or more > recently the bug discovered in the [emit-on-change semantics for > KTables|https://issues.apache.org/jira/browse/KAFKA-12508?focusedCommentId=17306323=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17306323]. > However for this case at least there are no compatibility concerns. It's > safe to upgrade from using a separate changelog for a source KTable to just > using that source topic directly, although the reverse is not true. We could > even automatically delete the no-longer-necessary changelog for upgrading > applications -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10493) KTable out-of-order updates are not being ignored
[ https://issues.apache.org/jira/browse/KAFKA-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17330987#comment-17330987 ] Guozhang Wang commented on KAFKA-10493: --- While thinking about https://issues.apache.org/jira/browse/KAFKA-12693, one idea is that if we restore based on timestamps instead of offsets (of course, it is not 100 percent safe in practice, since timestamps may be inaccurate as well conceptually, but it should be better than offsets), then even if there are zombies writing to changelogs, we would be much less vulnerable to such scenarios. Following that thought, and that in the long run we would go to: 1) version tables based on timestamps during normal state store materialization as well as restoration, so that we do not depend on offset ordering to update a single snapshot. 2) timestamp based compaction (KIP-280) with compaction horizon (i.e. we could choose to not compact old values which are still within the maintained old version period). Then, as of now, when we are still "keeping the latest version", to align with future, we should 1) depend on timestamps not offsets on materialization and restoration, 2) complete KIP-280 to do compaction based on timestamps. And then for 3) source changelog topics, KS would require / assume it is already configured with the right compaction policy, while on the caller level (e.g. KSQL), we would duplicate that data if necessary, e.g. if the source topic is external and won't guarantee the right configs are set. > KTable out-of-order updates are not being ignored > - > > Key: KAFKA-10493 > URL: https://issues.apache.org/jira/browse/KAFKA-10493 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.6.0 >Reporter: Pedro Gontijo >Assignee: Matthias J. Sax >Priority: Blocker > Fix For: 3.0.0 > > Attachments: KTableOutOfOrderBug.java > > > On a materialized KTable, out-of-order records for a given key (records which > timestamp are older than the current value in store) are not being ignored > but used to update the local store value and also being forwarded. > I believe the bug is here: > [https://github.com/apache/kafka/blob/2.6.0/streams/src/main/java/org/apache/kafka/streams/state/internals/ValueAndTimestampSerializer.java#L77] > It should return true, not false (see javadoc) > The bug impacts here: > [https://github.com/apache/kafka/blob/2.6.0/streams/src/main/java/org/apache/kafka/streams/kstream/internals/KTableSource.java#L142-L148] > I have attached a simple stream app that shows the issue happening. > Thank you! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-12693) Consecutive rebalances with zombie instances may cause corrupted changelogs
[ https://issues.apache.org/jira/browse/KAFKA-12693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-12693: -- Labels: streams (was: ) > Consecutive rebalances with zombie instances may cause corrupted changelogs > --- > > Key: KAFKA-12693 > URL: https://issues.apache.org/jira/browse/KAFKA-12693 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Priority: Major > Labels: streams > > When an instance (or thread within an instance) of Kafka Streams has a soft > failure and the group coordinator triggers a rebalance, that instance would > temporarily become a "zombie writer". That is, this instance does not know > there's already a new rebalance and hence its partitions have been migrated > out, until it tries to commit and then got notified of the illegal-generation > error and realize itself is the "zombie" already. During this period until > the commit, this zombie may still be writing data to the changelogs of the > migrated tasks as the new owner has already taken over and also writing to > the changelogs. > When EOS is enabled, this would not be a problem: when the zombie tries to > commit and got notified that it's fenced, its zombie appends would be > aborted. With EOS disabled, though, such shared writes would be interleaved > on the changelogs where a zombie append may arrive later after the new > writer's append, effectively overwriting that new append. > Note that such interleaving writes do not necessarily cause corrupted data: > as long as the new producer keep appending after the old zombie stops, and > all the corrupted keys are overwritten again by the new values, then it is > fine. However, if there are consecutive rebalances where right after the > changelogs are corrupted by zombie writers, and before the new writer can > overwrite them again, the task gets migrated again and needs to be restored > from changelogs, the old values would be restored instead of the new values, > effectively causing data loss. > Although this should be a rare event, we should fix it asap still. One idea > is to have producers get a PID even under ALOS: that is, we set the > transactional id in the producer config, but did not trigger any txn APIs; > when there are zombie producers, they would then be immediately fenced on > appends and hence there's no interleaved appends. I think this may require a > KIP still, since today one has to call initTxn in order to register and get > the PID. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12710) Consider enabling (at least some) optimizations by default
[ https://issues.apache.org/jira/browse/KAFKA-12710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17330979#comment-17330979 ] Guozhang Wang commented on KAFKA-12710: --- Thanks [~ableegoldman], this seems relevant to [~agavra]'s KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-705%3A+Selectively+Disable+Topology+Optimizations > Consider enabling (at least some) optimizations by default > -- > > Key: KAFKA-12710 > URL: https://issues.apache.org/jira/browse/KAFKA-12710 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: A. Sophie Blee-Goldman >Priority: Major > > Topology optimizations such as the repartition consolidation and source topic > changelog are extremely useful at reducing the footprint of a Kafka Streams > application on the broker. The additional storage and resource utilization > due to changelogs and repartitions is a very real pain point, and has even > been cited as the reason for turning to other stream processing frameworks in > the past (though of course I question that judgement) > The repartition topic optimization, at the very least, should be enabled by > default. The problem is that we can't just flip the switch without breaking > existing applications during upgrade, since the location and name of such > topics in the topology may change. One possibility is to just detect this > situation and disable the optimization if we find that it would produce an > incompatible topology for an existing application. We can determine that this > is the case simply by looking for pre-existing repartition topics. If any > such topics are present, and match the set of repartition topics in the > un-optimized topology, then we know we need to switch the optimization off. > If we don't find any repartition topics, or they match the optimized > topology, then we're safe to enable it by default. > Alternatively, we could just do a KIP to indicate that we intend to change > the default in the next breaking release and that existing applications > should override this config if necessary. We should be able to implement a > fail-safe and shut down if a user misses or forgets to do so, using the > method mentioned above. > The source topic optimization is perhaps more controversial, as there have > been a few issues raised with regards to things like [restoring bad data and > asymmetric serdes|https://issues.apache.org/jira/browse/KAFKA-8037], or more > recently the bug discovered in the [emit-on-change semantics for > KTables|https://issues.apache.org/jira/browse/KAFKA-12508?focusedCommentId=17306323=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17306323]. > However for this case at least there are no compatibility concerns. It's > safe to upgrade from using a separate changelog for a source KTable to just > using that source topic directly, although the reverse is not true. We could > even automatically delete the no-longer-necessary changelog for upgrading > applications -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [kafka] guozhangwang commented on a change in pull request #10587: KAFKA-8897: Upgrade RocksDB to 6.8.1
guozhangwang commented on a change in pull request #10587:
URL: https://github.com/apache/kafka/pull/10587#discussion_r619416799
##
File path:
streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter.java
##
@@ -99,35 +101,7 @@ public Env getEnv() {
@Override
public Options prepareForBulkLoad() {
-/* From https://github.com/facebook/rocksdb/wiki/RocksDB-FAQ
- *
- * Q: What's the fastest way to load data into RocksDB?
- *
- * A: A fast way to direct insert data to the DB:
- *
- * 1. using single writer thread and insert in sorted order
- * 2. batch hundreds of keys into one write batch
- * 3. use vector memtable
- * 4. make sure options.max_background_flushes is at least 4
- * 5. before inserting the data,
- * disable automatic compaction,
- * set options.level0_file_num_compaction_trigger,
- * options.level0_slowdown_writes_trigger
- * and options.level0_stop_writes_trigger to very large.
- * After inserting all the data, issue a manual compaction.
- *
- * 3-5 will be automatically done if you call
Options::PrepareForBulkLoad() to your option
- */
-// (1) not in our control
-// (2) is done via bulk-loading API
-// (3) skipping because, not done in actual PrepareForBulkLoad() code
in https://github.com/facebook/rocksdb/blob/master/options/options.cc
-//columnFamilyOptions.setMemTableConfig(new VectorMemTableConfig());
-// (4-5) below:
-dbOptions.setMaxBackgroundFlushes(4);
-columnFamilyOptions.setDisableAutoCompactions(true);
-columnFamilyOptions.setLevel0FileNumCompactionTrigger(1 << 30);
-columnFamilyOptions.setLevel0SlowdownWritesTrigger(1 << 30);
-columnFamilyOptions.setLevel0StopWritesTrigger(1 << 30);
+super.prepareForBulkLoad();
Review comment:
Maybe still valuable to leave a comment on which rocksDB version the
bulk loading behavior is updated to meet our needs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (KAFKA-12712) KRaft: Missing controller.quorom.voters config not properly handled
[
https://issues.apache.org/jira/browse/KAFKA-12712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismael Juma updated KAFKA-12712:
Fix Version/s: 3.0.0
> KRaft: Missing controller.quorom.voters config not properly handled
> ---
>
> Key: KAFKA-12712
> URL: https://issues.apache.org/jira/browse/KAFKA-12712
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 2.8.0
>Reporter: Magnus Edenhill
>Priority: Major
> Fix For: 3.0.0
>
>
> When trying out KRaft in 2.8 I mispelled controller.quorum.voters as
> controller.quorum.voters, but the broker did not fail to start, nor did it
> print any warning.
>
> Instead it raised this error:
>
> {code:java}
> [2021-04-23 18:25:13,484] INFO Starting controller
> (kafka.server.ControllerServer)[2021-04-23 18:25:13,484] INFO Starting
> controller (kafka.server.ControllerServer)[2021-04-23 18:25:13,485] ERROR
> [kafka-raft-io-thread]: Error due to
> (kafka.raft.KafkaRaftManager$RaftIoThread)java.lang.IllegalArgumentException:
> bound must be positive at java.util.Random.nextInt(Random.java:388) at
> org.apache.kafka.raft.RequestManager.findReadyVoter(RequestManager.java:57)
> at
> org.apache.kafka.raft.KafkaRaftClient.maybeSendAnyVoterFetch(KafkaRaftClient.java:1778)
> at
> org.apache.kafka.raft.KafkaRaftClient.pollUnattachedAsObserver(KafkaRaftClient.java:2080)
> at
> org.apache.kafka.raft.KafkaRaftClient.pollUnattached(KafkaRaftClient.java:2061)
> at
> org.apache.kafka.raft.KafkaRaftClient.pollCurrentState(KafkaRaftClient.java:2096)
> at org.apache.kafka.raft.KafkaRaftClient.poll(KafkaRaftClient.java:2181) at
> kafka.raft.KafkaRaftManager$RaftIoThread.doWork(RaftManager.scala:53) at
> kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
> {code}
> which I guess eventually (1 minute later) lead to this error which terminated
> the broker:
> {code:java}
> [2021-04-23 18:26:14,435] ERROR [BrokerLifecycleManager id=2] Shutting down
> because we were unable to register with the controller quorum.
> (kafka.server.BrokerLifecycleManager)[2021-04-23 18:26:14,435] ERROR
> [BrokerLifecycleManager id=2] Shutting down because we were unable to
> register with the controller quorum.
> (kafka.server.BrokerLifecycleManager)[2021-04-23 18:26:14,436] INFO
> [BrokerLifecycleManager id=2] registrationTimeout: shutting down event queue.
> (org.apache.kafka.queue.KafkaEventQueue)[2021-04-23 18:26:14,437] INFO
> [BrokerLifecycleManager id=2] Transitioning from STARTING to SHUTTING_DOWN.
> (kafka.server.BrokerLifecycleManager)[2021-04-23 18:26:14,437] INFO
> [broker-2-to-controller-send-thread]: Shutting down
> (kafka.server.BrokerToControllerRequestThread)[2021-04-23 18:26:14,438] INFO
> [broker-2-to-controller-send-thread]: Stopped
> (kafka.server.BrokerToControllerRequestThread)[2021-04-23 18:26:14,438] INFO
> [broker-2-to-controller-send-thread]: Shutdown completed
> (kafka.server.BrokerToControllerRequestThread)[2021-04-23 18:26:14,441] ERROR
> [BrokerServer id=2] Fatal error during broker startup. Prepare to shutdown
> (kafka.server.BrokerServer)java.util.concurrent.CancellationException at
> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276) at
> kafka.server.BrokerLifecycleManager$ShutdownEvent.run(BrokerLifecycleManager.scala:474)
> at
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:174)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> But since the client listeners were made available prior to shutting down,
> the broker was deemed up and operational by the (naiive) monitoring tool.
> So..:
> - Broker should fail on startup on invalid/unknown config properties. I
> understand this is tehcnically tricky, so at least a warning log should be
> printed.
> - Perhaps not create client listeners before control plane is somewhat happy.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-10619) Producer will enable EOS by default
[ https://issues.apache.org/jira/browse/KAFKA-10619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329915#comment-17329915 ] Soumyajit Sahu edited comment on KAFKA-10619 at 4/23/21, 5:47 PM: -- Hi [~d8tltanc] , I would like to raise an issue with making enable.idempotence=true by default. Recently, we hit an issue where a producer app of ours was using enable.idempotence=true, but was creating a producer object for each message. We ended up in having thousands of producer ids for our log segment, and the broker crashed with OutOfMemory exception while trying to take a snapshot. I am not aware of any way to monitor this or check on such behavior. Hence, setting this to true could break many clusters with a surprise. It would be ideal to come up with a monitoring or controlling mechanism for this before we set it to true by default. was (Author: soumyajitsahu): Hi [~d8tltanc] , I would like to raise an issue with making enable.idempotence=true by default. Recently, we hit an issue where a producer app of ours was using enable.idempotence=true, but was creating a producer object for each message. We ended up in having thousands of producer ids for our log segment, and the broker crashed with OutOfMemory exception while trying to take a snapshot. I am not aware of any way to monitor this or check on such behavior. Hence, setting this to true could break many clusters with a surprise. It would be ideal to come up with a monitoring or controlling mechanism for this before we set it to true by default. > Producer will enable EOS by default > --- > > Key: KAFKA-10619 > URL: https://issues.apache.org/jira/browse/KAFKA-10619 > Project: Kafka > Issue Type: Improvement >Reporter: Cheng Tan >Assignee: Cheng Tan >Priority: Major > Fix For: 3.0.0 > > > This is an after-work for KIP-185. > In the producer config, > # the default value of `acks` will change to `all` > # `enable.idempotence` will change to `true` > [An analysis of the impact of max.in.flight.requests.per.connection and acks > on Producer > performance|https://cwiki.apache.org/confluence/display/KAFKA/An+analysis+of+the+impact+of+max.in.flight.requests.per.connection+and+acks+on+Producer+performance] > indicates that changing `acks` from `1` to `all` won't increase the latency > and decrease the throughput in a significant way. > > KIP: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-679%3A+Producer+will+enable+the+strongest+delivery+guarantee+by+default -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-10619) Producer will enable EOS by default
[ https://issues.apache.org/jira/browse/KAFKA-10619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329915#comment-17329915 ] Soumyajit Sahu edited comment on KAFKA-10619 at 4/23/21, 5:45 PM: -- Hi [~d8tltanc] , I would like to raise an issue with making enable.idempotence=true by default. Recently, we hit an issue where a producer app of ours was using enable.idempotence=true, but was creating a producer object for each message. We ended up in having thousands of producer ids for our log segment, and the broker crashed with OutOfMemory exception while trying to take a snapshot. I am not aware of any way to monitor this or check on such behavior. Hence, setting this to true could break many clusters with a surprise. It would be ideal to come up with a monitoring or controlling mechanism for this before we set it to true by default. was (Author: soumyajitsahu): Hi Cheng, I would like to raise an issue with making enable.idempotence=true by default. Recently, we hit an issue where a producer app of ours was using enable.idempotence=true, but was creating a producer object for each message. We ended up in having thousands of producer ids for our log segment, and the broker crashed with OutOfMemory exception while trying to take a snapshot. I am not aware of any way to monitor this or check on such behavior. Hence, setting this to true could break many clusters with a surprise. It would be ideal to come up with a monitoring or controlling mechanism for this before we set it to true by default. > Producer will enable EOS by default > --- > > Key: KAFKA-10619 > URL: https://issues.apache.org/jira/browse/KAFKA-10619 > Project: Kafka > Issue Type: Improvement >Reporter: Cheng Tan >Assignee: Cheng Tan >Priority: Major > Fix For: 3.0.0 > > > This is an after-work for KIP-185. > In the producer config, > # the default value of `acks` will change to `all` > # `enable.idempotence` will change to `true` > [An analysis of the impact of max.in.flight.requests.per.connection and acks > on Producer > performance|https://cwiki.apache.org/confluence/display/KAFKA/An+analysis+of+the+impact+of+max.in.flight.requests.per.connection+and+acks+on+Producer+performance] > indicates that changing `acks` from `1` to `all` won't increase the latency > and decrease the throughput in a significant way. > > KIP: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-679%3A+Producer+will+enable+the+strongest+delivery+guarantee+by+default -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-12698) CVE-2019-25013 high priority vulnerability reported in Kafka
[ https://issues.apache.org/jira/browse/KAFKA-12698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17330927#comment-17330927 ] Andrew May edited comment on KAFKA-12698 at 4/23/21, 5:14 PM: -- That vulnerability is a glibc one, so I'm not sure why it's being reported as a vulnerability in a Java library (unless I'm missing something - I'm not an expert). N.B. at about the same time that this issue was created, the CVE linked above was updated to say that kafka 2.6.0 was vulnerable (& the CVE links to emails generated off the back of updates to this Jira issue) This is now causing any dependency-checker tools that use the NIST NVD database to report Kafka 2.6.0 as vulnerable. Could it be that what has caused this report is that a container scanner 'clair' has scanned a container made up of ['version of linux with a vulnerable version of glibc' + 'some JVM' + 'kafka 2.6.0']? was (Author: andymay): That vulnerability is a glibc one, so I'm not sure why it's being reported as a vulnerability in a Java library. N.B. at about the same time that this issue was created, the CVE linked above was updated to say that kafka 2.6.0 was vulnerable (& the CVE links to emails generated off the back of updates to this Jira issue) This is now causing any dependency-checker tools that use the NIST NVD database to report Kafka 2.6.0 as vulnerable. Could it be that what has caused this report is that a container scanner 'clair' has scanned a container made up of ['version of linux with a vulnerable version of glibc' + 'some JVM' + 'kafka 2.6.0']? > CVE-2019-25013 high priority vulnerability reported in Kafka > - > > Key: KAFKA-12698 > URL: https://issues.apache.org/jira/browse/KAFKA-12698 > Project: Kafka > Issue Type: Bug >Reporter: priya Vijay >Priority: Major > > On running clair scanner for kafka version 2.6, the following high priority > vulnerability is reported for Kafka. > CVE-2019-25013 [https://nvd.nist.gov/vuln/detail/CVE-2019-25013] > details: The iconv feature in the GNU C Library (aka glibc or libc6) through > 2.32, when processing invalid multi-byte input sequences in the EUC-KR > encoding, may have a buffer over-read -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12698) CVE-2019-25013 high priority vulnerability reported in Kafka
[ https://issues.apache.org/jira/browse/KAFKA-12698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17330927#comment-17330927 ] Andrew May commented on KAFKA-12698: That vulnerability is a glibc one, so I'm not sure why it's being reported as a vulnerability in a Java library. N.B. at about the same time that this issue was created, the CVE linked above was updated to say that kafka 2.6.0 was vulnerable (& the CVE links to emails generated off the back of updates to this Jira issue) This is now causing any dependency-checker tools that use the NIST NVD database to report Kafka 2.6.0 as vulnerable. Could it be that what has caused this report is that a container scanner 'clair' has scanned a container made up of ['version of linux with a vulnerable version of glibc' + 'some JVM' + 'kafka 2.6.0']? > CVE-2019-25013 high priority vulnerability reported in Kafka > - > > Key: KAFKA-12698 > URL: https://issues.apache.org/jira/browse/KAFKA-12698 > Project: Kafka > Issue Type: Bug >Reporter: priya Vijay >Priority: Major > > On running clair scanner for kafka version 2.6, the following high priority > vulnerability is reported for Kafka. > CVE-2019-25013 [https://nvd.nist.gov/vuln/detail/CVE-2019-25013] > details: The iconv feature in the GNU C Library (aka glibc or libc6) through > 2.32, when processing invalid multi-byte input sequences in the EUC-KR > encoding, may have a buffer over-read -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [kafka] guozhangwang merged pull request #9640: KAFKA-10283; Consolidate client-level and consumer-level assignment within ClientState
guozhangwang merged pull request #9640: URL: https://github.com/apache/kafka/pull/9640 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] guozhangwang commented on pull request #9640: KAFKA-10283; Consolidate client-level and consumer-level assignment within ClientState
guozhangwang commented on pull request #9640: URL: https://github.com/apache/kafka/pull/9640#issuecomment-825795078 LGTM! And merged to trunk. Thanks @highluck -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (KAFKA-12712) KRaft: Missing controller.quorom.voters config not properly handled
Magnus Edenhill created KAFKA-12712:
---
Summary: KRaft: Missing controller.quorom.voters config not
properly handled
Key: KAFKA-12712
URL: https://issues.apache.org/jira/browse/KAFKA-12712
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 2.8.0
Reporter: Magnus Edenhill
When trying out KRaft in 2.8 I mispelled controller.quorum.voters as
controller.quorum.voters, but the broker did not fail to start, nor did it
print any warning.
Instead it raised this error:
{code:java}
[2021-04-23 18:25:13,484] INFO Starting controller
(kafka.server.ControllerServer)[2021-04-23 18:25:13,484] INFO Starting
controller (kafka.server.ControllerServer)[2021-04-23 18:25:13,485] ERROR
[kafka-raft-io-thread]: Error due to
(kafka.raft.KafkaRaftManager$RaftIoThread)java.lang.IllegalArgumentException:
bound must be positive at java.util.Random.nextInt(Random.java:388) at
org.apache.kafka.raft.RequestManager.findReadyVoter(RequestManager.java:57) at
org.apache.kafka.raft.KafkaRaftClient.maybeSendAnyVoterFetch(KafkaRaftClient.java:1778)
at
org.apache.kafka.raft.KafkaRaftClient.pollUnattachedAsObserver(KafkaRaftClient.java:2080)
at
org.apache.kafka.raft.KafkaRaftClient.pollUnattached(KafkaRaftClient.java:2061)
at
org.apache.kafka.raft.KafkaRaftClient.pollCurrentState(KafkaRaftClient.java:2096)
at org.apache.kafka.raft.KafkaRaftClient.poll(KafkaRaftClient.java:2181) at
kafka.raft.KafkaRaftManager$RaftIoThread.doWork(RaftManager.scala:53) at
kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
{code}
which I guess eventually (1 minute later) lead to this error which terminated
the broker:
{code:java}
[2021-04-23 18:26:14,435] ERROR [BrokerLifecycleManager id=2] Shutting down
because we were unable to register with the controller quorum.
(kafka.server.BrokerLifecycleManager)[2021-04-23 18:26:14,435] ERROR
[BrokerLifecycleManager id=2] Shutting down because we were unable to register
with the controller quorum. (kafka.server.BrokerLifecycleManager)[2021-04-23
18:26:14,436] INFO [BrokerLifecycleManager id=2] registrationTimeout: shutting
down event queue. (org.apache.kafka.queue.KafkaEventQueue)[2021-04-23
18:26:14,437] INFO [BrokerLifecycleManager id=2] Transitioning from STARTING to
SHUTTING_DOWN. (kafka.server.BrokerLifecycleManager)[2021-04-23 18:26:14,437]
INFO [broker-2-to-controller-send-thread]: Shutting down
(kafka.server.BrokerToControllerRequestThread)[2021-04-23 18:26:14,438] INFO
[broker-2-to-controller-send-thread]: Stopped
(kafka.server.BrokerToControllerRequestThread)[2021-04-23 18:26:14,438] INFO
[broker-2-to-controller-send-thread]: Shutdown completed
(kafka.server.BrokerToControllerRequestThread)[2021-04-23 18:26:14,441] ERROR
[BrokerServer id=2] Fatal error during broker startup. Prepare to shutdown
(kafka.server.BrokerServer)java.util.concurrent.CancellationException at
java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276) at
kafka.server.BrokerLifecycleManager$ShutdownEvent.run(BrokerLifecycleManager.scala:474)
at
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:174)
at java.lang.Thread.run(Thread.java:748)
{code}
But since the client listeners were made available prior to shutting down, the
broker was deemed up and operational by the (naiive) monitoring tool.
So..:
- Broker should fail on startup on invalid/unknown config properties. I
understand this is tehcnically tricky, so at least a warning log should be
printed.
- Perhaps not create client listeners before control plane is somewhat happy.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (KAFKA-12691) TaskMetadata timeSinceIdlingStarted not reporting correctly
[ https://issues.apache.org/jira/browse/KAFKA-12691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Walker Carlson closed KAFKA-12691. -- > TaskMetadata timeSinceIdlingStarted not reporting correctly > --- > > Key: KAFKA-12691 > URL: https://issues.apache.org/jira/browse/KAFKA-12691 > Project: Kafka > Issue Type: Bug > Components: streams >Reporter: Walker Carlson >Assignee: Walker Carlson >Priority: Major > Fix For: 3.0.0 > > > TaskMetadata timeSinceIdlingStarted not reporting correctly. It takes into > account suspended but not the call to is processable. To fix this we need to > record when the first time it is not processable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-12711) Add a back off option to Replace thread
[ https://issues.apache.org/jira/browse/KAFKA-12711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Walker Carlson updated KAFKA-12711: --- Description: There should be a native option to set a back off period when replacing a thread. Either there should be a config and a user chosen strategy or a value you can set in the handler that causes a delay in creating the new thread. was: There should be a native option to set a back off when replacing a thread. Either there should be a config and a user chosen strategy or a value you can set in the handler that causes a delay in creating the new thread. > Add a back off option to Replace thread > --- > > Key: KAFKA-12711 > URL: https://issues.apache.org/jira/browse/KAFKA-12711 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Walker Carlson >Priority: Major > > There should be a native option to set a back off period when replacing a > thread. > > Either there should be a config and a user chosen strategy or a value you can > set in the handler that causes a delay in creating the new thread. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-12711) Add a back off option to Replace thread
Walker Carlson created KAFKA-12711: -- Summary: Add a back off option to Replace thread Key: KAFKA-12711 URL: https://issues.apache.org/jira/browse/KAFKA-12711 Project: Kafka Issue Type: Improvement Components: streams Reporter: Walker Carlson There should be a native option to set a back off when replacing a thread. Either there should be a config and a user chosen strategy or a value you can set in the handler that causes a delay in creating the new thread. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [kafka] bruto1 opened a new pull request #10590: KAFKA-5761: support ByteBuffer as value in ProducerRecord and avoid redundant serialization when it's used
bruto1 opened a new pull request #10590: URL: https://github.com/apache/kafka/pull/10590 Allows to use pooled byte buffer implementations to serialize record value before calling KafkaProducer.send and reclaim the buffer afterwards, thus reducing unnecessary allocations and GC pressure Is a new take on https://cwiki.apache.org/confluence/display/KAFKA/KIP-646+Serializer+API+should+support+ByteBuffer -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vamossagar12 commented on pull request #10542: KAFKA-12313: Streamling windowed Deserialiser configs.
vamossagar12 commented on pull request #10542: URL: https://github.com/apache/kafka/pull/10542#issuecomment-825558068 Thanks @ableegoldman , actually i think i had updated the ticket as well that we need to include serialisers as well. i have updated the KIP. and sent it again for discussion. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KAFKA-10493) KTable out-of-order updates are not being ignored
[
https://issues.apache.org/jira/browse/KAFKA-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17330116#comment-17330116
]
Matthias J. Sax commented on KAFKA-10493:
-
{quote}What is worse not dropping out-of-order records or not having the source
topic optimization? It seems like they are not compatible.
{quote}
Very good question. Don't know.
{quote}Streams does not have control over the source topic. So even if KIP-280
is done, Streams cannot impose a compaction strategy on the source topic.
{quote}
Sure, but at least with KIP-280 it will be possible to use both. (Of course, if
we enable dropping out-of-order in KS, users would need to have up-to-data
broker, too, to enjoy both featured enabled...)
{quote}Imo we should find a way to fix this that doesn't prevent users from
leveraging the source topic optimization.
{quote}
IMHO, only KIP-280 provides a solution for this?
{quote}but also disable compaction entirely IIUC.
{quote}
Why that? If we use a dedicated changelog and drop out-or-order input data, we
ensure that we only write in-order data into the changelog and thus we can
still configure the changelog with compaction.
{quote}I haven't fully digested this current discussion about the impact of
dropping out-of-order updates with a compacted changelog, but perhaps we could
store some information in the committed offset metadata to help us here.
{quote}
As mentioned above: from my understanding we have only two options to make it
work, either KIP-280, or a dedicated changelog topic.
> KTable out-of-order updates are not being ignored
> -
>
> Key: KAFKA-10493
> URL: https://issues.apache.org/jira/browse/KAFKA-10493
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.6.0
>Reporter: Pedro Gontijo
>Assignee: Matthias J. Sax
>Priority: Blocker
> Fix For: 3.0.0
>
> Attachments: KTableOutOfOrderBug.java
>
>
> On a materialized KTable, out-of-order records for a given key (records which
> timestamp are older than the current value in store) are not being ignored
> but used to update the local store value and also being forwarded.
> I believe the bug is here:
> [https://github.com/apache/kafka/blob/2.6.0/streams/src/main/java/org/apache/kafka/streams/state/internals/ValueAndTimestampSerializer.java#L77]
> It should return true, not false (see javadoc)
> The bug impacts here:
> [https://github.com/apache/kafka/blob/2.6.0/streams/src/main/java/org/apache/kafka/streams/kstream/internals/KTableSource.java#L142-L148]
> I have attached a simple stream app that shows the issue happening.
> Thank you!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [kafka] showuon commented on pull request #9627: KAFKA-10746: Change to Warn logs when necessary to notify users
showuon commented on pull request #9627: URL: https://github.com/apache/kafka/pull/9627#issuecomment-825451248 @chia7712 , please take a look. Thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] showuon edited a comment on pull request #9627: KAFKA-10746: Change to Warn logs when necessary to notify users
showuon edited a comment on pull request #9627: URL: https://github.com/apache/kafka/pull/9627#issuecomment-825382213 Failed test cases are unrelated and flaky (all passed in my local env). Thanks. `tests/Build/JDK 11 and Scala 2.13` --> all tests passed ``` Build / JDK 15 and Scala 2.13 / kafka.network.DynamicConnectionQuotaTest.testDynamicListenerConnectionCreationRateQuota() Build / JDK 15 and Scala 2.13 / kafka.server.RaftClusterTest.testCreateClusterAndCreateAndManyTopicsWithManyPartitions() Build / JDK 15 and Scala 2.13 / kafka.server.RaftClusterTest.testCreateClusterAndCreateAndManyTopicsWithManyPartitions() Build / JDK 8 and Scala 2.12 / org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationTest.testReplication() Build / JDK 8 and Scala 2.12 / kafka.admin.TopicCommandWithAdminClientTest.testDescribeUnderMinIsrPartitionsMixed() ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KAFKA-12698) CVE-2019-25013 high priority vulnerability reported in Kafka
[ https://issues.apache.org/jira/browse/KAFKA-12698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17330046#comment-17330046 ] priya Vijay commented on KAFKA-12698: - can someone please provide some insight in to this, Is Kafka affected by this CVE-2019-25013? thanks > CVE-2019-25013 high priority vulnerability reported in Kafka > - > > Key: KAFKA-12698 > URL: https://issues.apache.org/jira/browse/KAFKA-12698 > Project: Kafka > Issue Type: Bug >Reporter: priya Vijay >Priority: Major > > On running clair scanner for kafka version 2.6, the following high priority > vulnerability is reported for Kafka. > CVE-2019-25013 [https://nvd.nist.gov/vuln/detail/CVE-2019-25013] > details: The iconv feature in the GNU C Library (aka glibc or libc6) through > 2.32, when processing invalid multi-byte input sequences in the EUC-KR > encoding, may have a buffer over-read -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-12698) CVE-2019-25013 high priority vulnerability reported in Kafka
[ https://issues.apache.org/jira/browse/KAFKA-12698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] priya Vijay updated KAFKA-12698: Summary: CVE-2019-25013 high priority vulnerability reported in Kafka (was: CVE-2019-25013 vulnerability reported in Kafka ) > CVE-2019-25013 high priority vulnerability reported in Kafka > - > > Key: KAFKA-12698 > URL: https://issues.apache.org/jira/browse/KAFKA-12698 > Project: Kafka > Issue Type: Bug >Reporter: priya Vijay >Priority: Major > > On running clair scanner for kafka version 2.6, the following high priority > vulnerability is reported for Kafka. > CVE-2019-25013 [https://nvd.nist.gov/vuln/detail/CVE-2019-25013] > details: The iconv feature in the GNU C Library (aka glibc or libc6) through > 2.32, when processing invalid multi-byte input sequences in the EUC-KR > encoding, may have a buffer over-read -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [kafka] highluck commented on pull request #9640: KAFKA-10283; Consolidate client-level and consumer-level assignment within ClientState
highluck commented on pull request #9640: URL: https://github.com/apache/kafka/pull/9640#issuecomment-825427682 @guozhangwang Yes, I checked, but the logic was not changed except for assignActiveToConsumer, so the test is consistent. There seems to be no problem locally! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
