[jira] [Commented] (KAFKA-13641) Kafka Streams Scala: Add `Option` to `ValueJoiner` parameters
[
https://issues.apache.org/jira/browse/KAFKA-13641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17489966#comment-17489966
]
Matthias J. Sax commented on KAFKA-13641:
-
{quote}Hope I didn't come across as rude or imply that I feel these are big
flaws.
{quote}
Not at all! – Contrary. If there is anything we can improve (not matter how
small), please let us know and help to fix/improve it!
{quote}An example is that for keySerde in Materialized.with, we have an
explicit type called KeySerde which sets the Serde isKey property to true for
things like avro preventing users to have Materialized.with(serde (with is key
= false), serde (with is key = true))
{quote}
That is an interesting example. I am wondering though what the gain would be
(for this example). One could have (eg) and Integer either as key or value, so
just having a IntegerSerde seems ok – if we have a KeyIntererSerde and a
ValueIntegerSerde both would just (de)serialize an integer and thus it seems to
be redundant/artificial to split it up into two types?
The examples about tombstone and Optional for joins is something I agree, too.
When Kafka Streams was introduced we still uses Java7 though, and later it was
hard (for backward compatibility reasons) to change the API... (just want to
give some context). If you have a good idea how we could change the interfaces
without "breaking" everything, it would be great!
> Kafka Streams Scala: Add `Option` to `ValueJoiner` parameters
> -
>
> Key: KAFKA-13641
> URL: https://issues.apache.org/jira/browse/KAFKA-13641
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Reporter: Mohammad Yousuf Minhaj Zia
>Priority: Minor
>
> Since `ValueJoiner` right parameter in `leftJoins`, `outerJoins` can be
> nullable, I am wondering if can wrap them around Scala `Option`.
> However, there is also the concern that the left hand side value can be null
> in the case of tombstone messages, in which the `Option` semantics can be
> misleading. I still feel this could be a useful feature in reducing the
> number of `NullPointerExceptions`.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (KAFKA-13289) Bulk processing correctly ordered input data through a join with kafka-streams results in `Skipping record for expired segment`
[
https://issues.apache.org/jira/browse/KAFKA-13289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17489962#comment-17489962
]
Matthias J. Sax commented on KAFKA-13289:
-
Great news! Thanks for sharing. Happy you got your app stabilized and running!
– On to new adventures!
> Bulk processing correctly ordered input data through a join with
> kafka-streams results in `Skipping record for expired segment`
> ---
>
> Key: KAFKA-13289
> URL: https://issues.apache.org/jira/browse/KAFKA-13289
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.8.0
>Reporter: Matthew Sheppard
>Priority: Minor
>
> When pushing bulk data through a kafka-steams app, I see it log the following
> message many times...
> {noformat}
> WARN
> org.apache.kafka.streams.state.internals.AbstractRocksDBSegmentedBytesStore -
> Skipping record for expired segment.
> {noformat}
> ...and data which I expect to have been joined through a leftJoin step
> appears to be lost.
> I've seen this in practice either when my application has been shut down for
> a while and then is brought back up, or when I've used something like the
> [app-reset-rool](https://docs.confluent.io/platform/current/streams/developer-guide/app-reset-tool.html)
> in an attempt to have the application reprocess past data.
> I was able to reproduce this behaviour in isolation by generating 1000
> messages to two topics spaced an hour apart (with the original timestamps in
> order), then having kafka streams select a key for them and try to leftJoin
> the two rekeyed streams.
> Self contained source code for that reproduction is available at
> https://github.com/mattsheppard/ins14809/blob/main/src/test/java/ins14809/Ins14809Test.java
> The actual kafka-streams topology in there looks like this.
> {code:java}
> final StreamsBuilder builder = new StreamsBuilder();
> final KStream leftStream =
> builder.stream(leftTopic);
> final KStream rightStream =
> builder.stream(rightTopic);
> final KStream rekeyedLeftStream = leftStream
> .selectKey((k, v) -> v.substring(0, v.indexOf(":")));
> final KStream rekeyedRightStream = rightStream
> .selectKey((k, v) -> v.substring(0, v.indexOf(":")));
> JoinWindows joinWindow = JoinWindows.of(Duration.ofSeconds(5));
> final KStream joined = rekeyedLeftStream.leftJoin(
> rekeyedRightStream,
> (left, right) -> left + "/" + right,
> joinWindow
> );
> {code}
> ...and the eventual output I produce looks like this...
> {code}
> ...
> 523 [523,left/null]
> 524 [524,left/null, 524,left/524,right]
> 525 [525,left/525,right]
> 526 [526,left/null]
> 527 [527,left/null]
> 528 [528,left/528,right]
> 529 [529,left/null]
> 530 [530,left/null]
> 531 [531,left/null, 531,left/531,right]
> 532 [532,left/null]
> 533 [533,left/null]
> 534 [534,left/null, 534,left/534,right]
> 535 [535,left/null]
> 536 [536,left/null]
> 537 [537,left/null, 537,left/537,right]
> 538 [538,left/null]
> 539 [539,left/null]
> 540 [540,left/null]
> 541 [541,left/null]
> 542 [542,left/null]
> 543 [543,left/null]
> ...
> {code}
> ...where as, given the input data, I expect to see every row end with the two
> values joined, rather than the right value being null.
> Note that I understand it's expected that we initially get the left/null
> values for many values since that's the expected semantics of kafka-streams
> left join, at least until
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Streams+Join+Semantics#KafkaStreamsJoinSemantics-ImprovedLeft/OuterStream-StreamJoin(v3.1.xandnewer)spurious
> I've noticed that if I set a very large grace value on the join window the
> problem is solved, but since the input I provide is not out of order I did
> not expect to need to do that, and I'm weary of the resource requirements
> doing so in practice on an application with a lot of volume.
> My suspicion is that something is happening such that when one partition is
> processed it causes the stream time to be pushed forward to the newest
> message in that partition, meaning when the next partition is then examined
> it is found to contain many records which are 'too old' compared to the
> stream time.
> I ran across this discussion thread which seems to cover the same issue
> http://mail-archives.apache.org/mod_mbox/kafka-users/202002.mbox/%3cCAB0tB9p_vijMS18jWXBqp7TQozL__ANoo3=h57q6z3y4hzt...@mail.gmail.com%3e
> and had a request from [~cadonna] for a reproduction case, so I'm hoping my
> example above might make the issue easier to tackle!
--
Thi
[jira] [Comment Edited] (KAFKA-13655) Cannot edit clients page
[ https://issues.apache.org/jira/browse/KAFKA-13655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17489958#comment-17489958 ] Matthias J. Sax edited comment on KAFKA-13655 at 2/10/22, 5:05 AM: --- Did you create an account (note the Jira and wiki are two independent account). If you have an account, please share you account id so we can grant writes to edit the wiki. Closing this ticket as it's not a bug. was (Author: mjsax): Did you create an account (note the Jira and wiki are two independent account). If you have an account, please share you account id so we can grant writes to edit the wiki. > Cannot edit clients page > > > Key: KAFKA-13655 > URL: https://issues.apache.org/jira/browse/KAFKA-13655 > Project: Kafka > Issue Type: Bug > Components: documentation >Reporter: Mario Mastrodicasa >Priority: Major > > Dear administrator, > I want to add references to a new .NET Kafka Client, but I'm not able to find > a button, or link, to edit the page. The page reports an unrestricted access, > but I can't edit it. > The page I want to edit is > [https://cwiki.apache.org/confluence/display/KAFKA/Clients] and the client is > under GitHub at this link [https://github.com/masesgroup/KafkaBridge] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (KAFKA-13655) Cannot edit clients page
[ https://issues.apache.org/jira/browse/KAFKA-13655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-13655. - Resolution: Not A Bug > Cannot edit clients page > > > Key: KAFKA-13655 > URL: https://issues.apache.org/jira/browse/KAFKA-13655 > Project: Kafka > Issue Type: Bug > Components: documentation >Reporter: Mario Mastrodicasa >Priority: Major > > Dear administrator, > I want to add references to a new .NET Kafka Client, but I'm not able to find > a button, or link, to edit the page. The page reports an unrestricted access, > but I can't edit it. > The page I want to edit is > [https://cwiki.apache.org/confluence/display/KAFKA/Clients] and the client is > under GitHub at this link [https://github.com/masesgroup/KafkaBridge] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (KAFKA-13655) Cannot edit clients page
[ https://issues.apache.org/jira/browse/KAFKA-13655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17489958#comment-17489958 ] Matthias J. Sax commented on KAFKA-13655: - Did you create an account (note the Jira and wiki are two independent account). If you have an account, please share you account id so we can grant writes to edit the wiki. > Cannot edit clients page > > > Key: KAFKA-13655 > URL: https://issues.apache.org/jira/browse/KAFKA-13655 > Project: Kafka > Issue Type: Bug > Components: documentation >Reporter: Mario Mastrodicasa >Priority: Major > > Dear administrator, > I want to add references to a new .NET Kafka Client, but I'm not able to find > a button, or link, to edit the page. The page reports an unrestricted access, > but I can't edit it. > The page I want to edit is > [https://cwiki.apache.org/confluence/display/KAFKA/Clients] and the client is > under GitHub at this link [https://github.com/masesgroup/KafkaBridge] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [kafka] dengziming commented on a change in pull request #11603: MINOR: MetadataShell should handle ProducerIdsRecord
dengziming commented on a change in pull request #11603:
URL: https://github.com/apache/kafka/pull/11603#discussion_r803264649
##
File path: shell/src/main/java/org/apache/kafka/shell/MetadataNodeManager.java
##
@@ -318,6 +320,15 @@ private void handleCommitImpl(MetadataRecordType type,
ApiMessage message)
node.create(record.key()).setContents(record.value() + "");
break;
}
+case PRODUCER_IDS_RECORD: {
+ProducerIdsRecord record = (ProducerIdsRecord) message;
+DirectoryNode producerIdNode = data.root.mkdirs("producerIds");
+producerIdNode.create("broker").setContents(record.brokerId()
+ "");
Review comment:
Thank you for these suggestions, I think assignedBrokerId and
assignedBrokerEpoch are enough since we don't use JSON format when parsing
other metadata records.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] hachikuji commented on pull request #11520: KAFKA-13468: Consumers may hang because IOException in Log# does not trigger KafkaStorageException
hachikuji commented on pull request #11520: URL: https://github.com/apache/kafka/pull/11520#issuecomment-1034437206 @functioner Good find. It does indeed look like `IoException` raised from `LogManager.getOrCreateLog` is not caught anywhere. In addition to catching, we probably need to add the dir to `LogDirFailureChannel` as you suggested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] artemlivshits commented on a change in pull request #11688: KAFKA-13435; Static membership protocol should let the leader skip assignment (KIP-814)
artemlivshits commented on a change in pull request #11688:
URL: https://github.com/apache/kafka/pull/11688#discussion_r803251907
##
File path: core/src/main/scala/kafka/coordinator/group/GroupCoordinator.scala
##
@@ -1306,23 +1326,46 @@ class GroupCoordinator(val brokerId: Int,
protocolType = group.protocolType,
protocolName = group.protocolName,
leaderId = currentLeader,
+skipAssignment = false,
error = error
))
+} else if (supportSkippingAssignment) {
+ // Starting from version 9 of the JoinGroup API, static members
are able to
+ // skip running the assignor based on the `SkipAssignment`
field. We leverage
+ // this to tell the leader that it is the leader of the group
but by skipping
+ // running the assignor while the group is in stable state.
Review comment:
Can you mention somewhere in the comments that this approach still
doesn't fully handle a condition if metadata has changed while the leader was
down?
##
File path:
connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/WorkerCoordinator.java
##
@@ -211,7 +211,13 @@ protected void onJoinComplete(int generation, String
memberId, String protocol,
}
@Override
-protected Map performAssignment(String leaderId,
String protocol, List allMemberMetadata) {
+protected Map onLeaderElected(String leaderId,
+ String protocol,
+
List allMemberMetadata,
+ boolean skipAssignment) {
+if (skipAssignment)
+throw new IllegalStateException("Can't skip assignment because
Connect does not support static membership.");
Review comment:
Would it be safer to just log an error and ignore? That would make a
difference if we say have a bug and return skipAssignment=true by mistake.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] hachikuji edited a comment on pull request #11558: KAFKA-13323 Fixed variable name in KafkaConsumer
hachikuji edited a comment on pull request #11558: URL: https://github.com/apache/kafka/pull/11558#issuecomment-1034428663 @vijaykriishna Thanks, the variable name fix LGTM. It does look a little strange to modify the one html snippet as part of this patch. Would it make sense to split that out into a separate PR? Maybe it makes sense to add that change to https://github.com/apache/kafka/pull/11542 instead. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] dengziming commented on a change in pull request #11655: KAFKA-13316; Enable KRaft mode in CreateTopics tests
dengziming commented on a change in pull request #11655:
URL: https://github.com/apache/kafka/pull/11655#discussion_r803251469
##
File path:
core/src/test/scala/unit/kafka/server/CreateTopicsRequestWithPolicyTest.scala
##
@@ -55,11 +63,9 @@ class CreateTopicsRequestWithPolicyTest extends
AbstractCreateTopicsRequestTest
assignment = Map(0 -> List(1, 0), 1 -> List(0, 1))
}
- @Test
- def testErrorCreateTopicsRequests(): Unit = {
-val existingTopic = "existing-topic"
-createTopic(existingTopic, 1, 1)
Review comment:
Thank you for pointing out this, firstly I think we are testing creating
a topic that violates the policy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] hachikuji commented on pull request #11558: KAFKA-13323 Fixed variable name in KafkaConsumer
hachikuji commented on pull request #11558: URL: https://github.com/apache/kafka/pull/11558#issuecomment-1034428663 @vijaykriishna Thanks, the variable name fix LGTM. It does look a little strange to modify the one html snippet as part of this patch. Would it make sense to split that out into a separate PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] dengziming commented on a change in pull request #11667: MINOR; Enable Kraft in ApiVersionTest
dengziming commented on a change in pull request #11667:
URL: https://github.com/apache/kafka/pull/11667#discussion_r803248389
##
File path:
core/src/test/scala/unit/kafka/server/AbstractApiVersionsRequestTest.scala
##
@@ -17,35 +17,45 @@
package kafka.server
import java.util.Properties
-
import kafka.test.ClusterInstance
+import org.apache.kafka.clients.NodeApiVersions
import org.apache.kafka.common.message.ApiMessageType.ListenerType
import org.apache.kafka.common.message.ApiVersionsResponseData.ApiVersion
+import org.apache.kafka.common.message.{ApiMessageType,
ApiVersionsResponseData}
import org.apache.kafka.common.network.ListenerName
import org.apache.kafka.common.protocol.ApiKeys
+import org.apache.kafka.common.record.RecordVersion
import org.apache.kafka.common.requests.{ApiVersionsRequest,
ApiVersionsResponse, RequestUtils}
import org.apache.kafka.common.utils.Utils
import org.junit.jupiter.api.Assertions._
import org.junit.jupiter.api.Tag
+import scala.compat.java8.OptionConverters._
import scala.jdk.CollectionConverters._
@Tag("integration")
abstract class AbstractApiVersionsRequestTest(cluster: ClusterInstance) {
def sendApiVersionsRequest(request: ApiVersionsRequest, listenerName:
ListenerName): ApiVersionsResponse = {
-IntegrationTestUtils.connectAndReceive[ApiVersionsResponse](request,
cluster.brokerSocketServers().asScala.head, listenerName)
+val socket = if
(cluster.controlPlaneListenerName().asScala.contains(listenerName) ||
Review comment:
Yes, you are right, I forget about these 2 things.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] hachikuji commented on a change in pull request #11603: MINOR: MetadataShell should handle ProducerIdsRecord
hachikuji commented on a change in pull request #11603:
URL: https://github.com/apache/kafka/pull/11603#discussion_r803244663
##
File path: shell/src/main/java/org/apache/kafka/shell/MetadataNodeManager.java
##
@@ -318,6 +320,15 @@ private void handleCommitImpl(MetadataRecordType type,
ApiMessage message)
node.create(record.key()).setContents(record.value() + "");
break;
}
+case PRODUCER_IDS_RECORD: {
+ProducerIdsRecord record = (ProducerIdsRecord) message;
+DirectoryNode producerIdNode = data.root.mkdirs("producerIds");
+producerIdNode.create("broker").setContents(record.brokerId()
+ "");
Review comment:
Maybe `assignedBrokerId`? As in, this was the brokerId that the block
was assigned to. Also, perhaps we may as well add `assignedBrokerEpoch` as
well? I guess we could even do it like
`/lastProducerBlock/assignedBroker/{id,epoch}`, but maybe that's overkill.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #11534: KAFKA-12939: After migrating processors, search the codebase for missed migrations
vvcephei commented on pull request #11534: URL: https://github.com/apache/kafka/pull/11534#issuecomment-1034421613 Hey @jeqo , it looks like some of those test failures are related to this PR. I took a look at one of them, and it looks like it's due to us trying to cast the processor context, which in that test is just a mock. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] hachikuji commented on a change in pull request #11603: MINOR: MetadataShell should handle ProducerIdsRecord
hachikuji commented on a change in pull request #11603:
URL: https://github.com/apache/kafka/pull/11603#discussion_r803244663
##
File path: shell/src/main/java/org/apache/kafka/shell/MetadataNodeManager.java
##
@@ -318,6 +320,15 @@ private void handleCommitImpl(MetadataRecordType type,
ApiMessage message)
node.create(record.key()).setContents(record.value() + "");
break;
}
+case PRODUCER_IDS_RECORD: {
+ProducerIdsRecord record = (ProducerIdsRecord) message;
+DirectoryNode producerIdNode = data.root.mkdirs("producerIds");
+producerIdNode.create("broker").setContents(record.brokerId()
+ "");
Review comment:
Maybe `assignedBrokerId`? As in, this was the brokerId that the block
was assigned to. Also, perhaps we may as well add `assignedBrokerEpoch` as
well? I guess we could even do it like
`/lastProducerBlock/assignedBroker/{id,epoch}`
##
File path: shell/src/main/java/org/apache/kafka/shell/MetadataNodeManager.java
##
@@ -318,6 +320,15 @@ private void handleCommitImpl(MetadataRecordType type,
ApiMessage message)
node.create(record.key()).setContents(record.value() + "");
break;
}
+case PRODUCER_IDS_RECORD: {
+ProducerIdsRecord record = (ProducerIdsRecord) message;
+DirectoryNode producerIdNode = data.root.mkdirs("producerIds");
Review comment:
How about `lastProducerIdBlock`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (KAFKA-13660) Replace log4j with reload4j
[ https://issues.apache.org/jira/browse/KAFKA-13660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17489905#comment-17489905 ] Dongjin Lee commented on KAFKA-13660: - Hi [~FireBurn], Thanks for your interest in this issue. I think reload4j is a promising project but, it seems not proven yet. Also, the log4j issue is already under progress with KAFKA-9366. Plus, these kinds of issues need a process named Kafka Improvement Proposal. Please have a look at [this page|https://cwiki.apache.org/confluence/display/kafka/kafka+improvement+proposals]. > Replace log4j with reload4j > --- > > Key: KAFKA-13660 > URL: https://issues.apache.org/jira/browse/KAFKA-13660 > Project: Kafka > Issue Type: Bug > Components: logging >Affects Versions: 2.4.0, 3.0.0 >Reporter: Mike Lothian >Priority: Major > > Kafka is using a known vulnerable version of log4j, the reload4j project was > created by the code's original authors to address those issues. It is > designed as a drop in replacement without any api changes > > https://reload4j.qos.ch/ > > I've raised a merge request, replacing log4j with reload4j, slf4j-log4j12 > with slf4j-reload4j and bumping the slf4j version > > This is my first time contributing to the Kafka project and I'm not too > familiar with the process, I'll go back and amend my PR with this issue number -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [kafka] mjsax commented on a change in pull request #11584: MINOR: improve logging
mjsax commented on a change in pull request #11584:
URL: https://github.com/apache/kafka/pull/11584#discussion_r803222877
##
File path:
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerCoordinator.java
##
@@ -1516,4 +1527,20 @@ RebalanceProtocol getProtocol() {
boolean poll(Timer timer) {
return poll(timer, true);
}
+
+
+
+final static class TopicPartitionComparator implements
Comparator, Serializable {
Review comment:
Ack. Just pushed an update with the refactoring, adding a `Utils` class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] hachikuji commented on a change in pull request #11655: KAFKA-13316; Enable KRaft mode in CreateTopics tests
hachikuji commented on a change in pull request #11655:
URL: https://github.com/apache/kafka/pull/11655#discussion_r802933225
##
File path:
core/src/test/scala/unit/kafka/server/CreateTopicsRequestWithPolicyTest.scala
##
@@ -55,11 +63,9 @@ class CreateTopicsRequestWithPolicyTest extends
AbstractCreateTopicsRequestTest
assignment = Map(0 -> List(1, 0), 1 -> List(0, 1))
}
- @Test
- def testErrorCreateTopicsRequests(): Unit = {
-val existingTopic = "existing-topic"
-createTopic(existingTopic, 1, 1)
Review comment:
Do we need to? Seems like the main thing we're testing here is the case
when the topic already exists. Would it be just as good to create with the
admin client and then verify the `TOPIC_ALREADY_EXISTS` error code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] hachikuji commented on a change in pull request #11688: KAFKA-13435; Static membership protocol should let the leader skip assignment (KIP-814)
hachikuji commented on a change in pull request #11688:
URL: https://github.com/apache/kafka/pull/11688#discussion_r803165736
##
File path: core/src/test/scala/integration/kafka/api/PlaintextConsumerTest.scala
##
@@ -1791,4 +1793,34 @@ class PlaintextConsumerTest extends BaseConsumerTest {
assertTrue(records2.count() == 1 &&
records2.records(tp).asScala.head.offset == 1,
"Expected consumer2 to consume one message from offset 1, which is the
committed offset of consumer1")
}
+
+ @Test
+ def testStaticConsumerDetectsNewPartitionCreatedAfterRestart(): Unit = {
+val foo = "foo"
+val foo0 = new TopicPartition(foo, 0)
+val foo1 = new TopicPartition(foo, 1)
+
+val admin = createAdminClient()
+admin.createTopics(Seq(new NewTopic(foo, 1,
1.asInstanceOf[Short])).asJava).all.get
Review comment:
nit: I think `1.toShort` works?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] hachikuji commented on a change in pull request #11688: KAFKA-13435; Static membership protocol should let the leader skip assignment (KIP-814)
hachikuji commented on a change in pull request #11688:
URL: https://github.com/apache/kafka/pull/11688#discussion_r803159171
##
File path:
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractCoordinator.java
##
@@ -699,11 +702,15 @@ public void handle(JoinGroupResponse joinResponse,
RequestFuture fut
return sendSyncGroupRequest(requestBuilder);
}
-private RequestFuture onJoinLeader(JoinGroupResponse
joinResponse) {
+private RequestFuture onLeaderElected(JoinGroupResponse
joinResponse) {
try {
// perform the leader synchronization and send back the assignment
for the group
-Map groupAssignment =
performAssignment(joinResponse.data().leader(),
joinResponse.data().protocolName(),
-joinResponse.data().members());
+Map groupAssignment = onLeaderElected(
+joinResponse.data().leader(),
+joinResponse.data().protocolName(),
+joinResponse.data().members(),
+joinResponse.data().skipAssignment()
+);
Review comment:
Do you think it is worthwhile validating that `groupAssignment` is empty
when `skipAssignment` is set?
##
File path:
clients/src/test/java/org/apache/kafka/clients/consumer/internals/ConsumerCoordinatorTest.java
##
@@ -1691,6 +1717,64 @@ public void testMetadataChangeTriggersRebalance() {
assertTrue(coordinator.rejoinNeededOrPending());
}
+@Test
+public void testStaticLeaderRejoinsGroupAndCanTriggersRebalance() {
+// ensure metadata is up-to-date for leader
+subscriptions.subscribe(singleton(topic1), rebalanceListener);
+client.updateMetadata(metadataResponse);
+
+client.prepareResponse(groupCoordinatorResponse(node, Errors.NONE));
+coordinator.ensureCoordinatorReady(time.timer(Long.MAX_VALUE));
+
+// the leader is responsible for picking up metadata changes and
forcing a group rebalance.
+// note that `MockPartitionAssignor.prepare` is not called therefore
calling `MockPartitionAssignor.assign`
+// will throw a IllegalStateException. this indirectly verifies that
`assign` is correctly skipped.
+Map> memberSubscriptions =
singletonMap(consumerId, singletonList(topic1));
+client.prepareResponse(joinGroupLeaderResponse(1, consumerId,
memberSubscriptions, true, Errors.NONE));
+client.prepareResponse(syncGroupResponse(singletonList(t1p),
Errors.NONE));
+
+coordinator.poll(time.timer(Long.MAX_VALUE));
+
+assertFalse(coordinator.rejoinNeededOrPending());
+
+// a new partition is added to the topic
+
metadata.updateWithCurrentRequestVersion(RequestTestUtils.metadataUpdateWith(1,
singletonMap(topic1, 2)), false, time.milliseconds());
+coordinator.maybeUpdateSubscriptionMetadata();
+
+// we should detect the change and ask for reassignment
+assertTrue(coordinator.rejoinNeededOrPending());
+}
+
+@Test
+public void
testStaticLeaderRejoinsGroupAndCanDetectMetadataChangesForOtherMembers() {
+// ensure metadata is up-to-date for leader
+subscriptions.subscribe(singleton(topic1), rebalanceListener);
+client.updateMetadata(metadataResponse);
+
+client.prepareResponse(groupCoordinatorResponse(node, Errors.NONE));
+coordinator.ensureCoordinatorReady(time.timer(Long.MAX_VALUE));
+
+// the leader is responsible for picking up metadata changes and
forcing a group rebalance.
+// note that `MockPartitionAssignor.prepare` is not called therefore
calling `MockPartitionAssignor.assign`
+// will throw a IllegalStateException. this indirectly verifies that
`assign` is correctly skipped.
+Map> memberSubscriptions = new HashMap<>();
+memberSubscriptions.put(consumerId, singletonList(topic1));
+memberSubscriptions.put(consumerId2, singletonList(topic2));
+client.prepareResponse(joinGroupLeaderResponse(1, consumerId,
memberSubscriptions, true, Errors.NONE));
+client.prepareResponse(syncGroupResponse(singletonList(t1p),
Errors.NONE));
+
+coordinator.poll(time.timer(Long.MAX_VALUE));
+
+assertFalse(coordinator.rejoinNeededOrPending());
+
Review comment:
Could we add an assertion for `SubscriptionState.metadataTopics`?
##
File path: clients/src/main/resources/common/message/JoinGroupResponse.json
##
@@ -49,6 +51,8 @@
"about": "The group protocol selected by the coordinator." },
{ "name": "Leader", "type": "string", "versions": "0+",
"about": "The leader of the group." },
+{ "name": "SkipAssignment", "type": "bool", "versions": "9+", "default":
"false",
+ "about": "True is the leader must skip running the assignment." },
Review comment:
nit: is -> if
##
File path: core/src/test/scala/integration/kafka/api/PlaintextConsumerTest
[GitHub] [kafka] hachikuji commented on a change in pull request #11667: MINOR; Enable Kraft in ApiVersionTest
hachikuji commented on a change in pull request #11667:
URL: https://github.com/apache/kafka/pull/11667#discussion_r803154384
##
File path:
core/src/test/scala/unit/kafka/server/AbstractApiVersionsRequestTest.scala
##
@@ -17,35 +17,45 @@
package kafka.server
import java.util.Properties
-
import kafka.test.ClusterInstance
+import org.apache.kafka.clients.NodeApiVersions
import org.apache.kafka.common.message.ApiMessageType.ListenerType
import org.apache.kafka.common.message.ApiVersionsResponseData.ApiVersion
+import org.apache.kafka.common.message.{ApiMessageType,
ApiVersionsResponseData}
import org.apache.kafka.common.network.ListenerName
import org.apache.kafka.common.protocol.ApiKeys
+import org.apache.kafka.common.record.RecordVersion
import org.apache.kafka.common.requests.{ApiVersionsRequest,
ApiVersionsResponse, RequestUtils}
import org.apache.kafka.common.utils.Utils
import org.junit.jupiter.api.Assertions._
import org.junit.jupiter.api.Tag
+import scala.compat.java8.OptionConverters._
import scala.jdk.CollectionConverters._
@Tag("integration")
abstract class AbstractApiVersionsRequestTest(cluster: ClusterInstance) {
def sendApiVersionsRequest(request: ApiVersionsRequest, listenerName:
ListenerName): ApiVersionsResponse = {
-IntegrationTestUtils.connectAndReceive[ApiVersionsResponse](request,
cluster.brokerSocketServers().asScala.head, listenerName)
+val socket = if
(cluster.controlPlaneListenerName().asScala.contains(listenerName) ||
Review comment:
I think there might still be some confusion about the control plane
listener. It is not expected to be the listener that the controller listens on
for zk clusters. Instead, it is the listener that the controller uses to
connect to other brokers. If we are testing `ApiVersions` on the control plane
listener, then we can choose any broker. We do not need to find the active
controller. So I think this logic should just be doing this:
```scala
val socket = if
(cluster.controllerListenerName().asScala.contains(listenerName)) {
cluster.controllerSocketServers().asScala.head
} else {
cluster.brokerSocketServers().asScala.head
}
```
Then just to avoid confusion, maybe we can call it `CONTROL_PLANE` in
`brokerPropertyOverrides` below. Would that work?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] hachikuji commented on a change in pull request #11667: MINOR; Enable Kraft in ApiVersionTest
hachikuji commented on a change in pull request #11667:
URL: https://github.com/apache/kafka/pull/11667#discussion_r803154384
##
File path:
core/src/test/scala/unit/kafka/server/AbstractApiVersionsRequestTest.scala
##
@@ -17,35 +17,45 @@
package kafka.server
import java.util.Properties
-
import kafka.test.ClusterInstance
+import org.apache.kafka.clients.NodeApiVersions
import org.apache.kafka.common.message.ApiMessageType.ListenerType
import org.apache.kafka.common.message.ApiVersionsResponseData.ApiVersion
+import org.apache.kafka.common.message.{ApiMessageType,
ApiVersionsResponseData}
import org.apache.kafka.common.network.ListenerName
import org.apache.kafka.common.protocol.ApiKeys
+import org.apache.kafka.common.record.RecordVersion
import org.apache.kafka.common.requests.{ApiVersionsRequest,
ApiVersionsResponse, RequestUtils}
import org.apache.kafka.common.utils.Utils
import org.junit.jupiter.api.Assertions._
import org.junit.jupiter.api.Tag
+import scala.compat.java8.OptionConverters._
import scala.jdk.CollectionConverters._
@Tag("integration")
abstract class AbstractApiVersionsRequestTest(cluster: ClusterInstance) {
def sendApiVersionsRequest(request: ApiVersionsRequest, listenerName:
ListenerName): ApiVersionsResponse = {
-IntegrationTestUtils.connectAndReceive[ApiVersionsResponse](request,
cluster.brokerSocketServers().asScala.head, listenerName)
+val socket = if
(cluster.controlPlaneListenerName().asScala.contains(listenerName) ||
Review comment:
I think there might still be some confusion about the control plane
listener. It is not expected to be the listener that the controller listens on
for zk clusters. Instead, it is the listener that the controller uses to
connect to other brokers. If we are test ApiVersions on the control plane
listener, then we can choose any broker. We do not need to find the active
controller. So I think this logic should just be doing this:
```scala
val socket = if
(cluster.controllerListenerName().asScala.contains(listenerName)) {
cluster.controllerSocketServers().asScala.head
} else {
cluster.brokerSocketServers().asScala.head
}
```
Then just to avoid confusion, maybe we can call it `CONTROL_PLANE` in
`brokerPropertyOverrides` below. Would that work?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] hachikuji opened a new pull request #11745: KAFKA-13661; Consistent permissions for CreatePartitions API
hachikuji opened a new pull request #11745: URL: https://github.com/apache/kafka/pull/11745 In #11649, we fixed one permission inconsistency between kraft and zk authorization for the `CreatePartitions` request. Previously kraft was requireing `CREATE` permission on the `Topic` resource when it should have required `ALTER`. A second inconsistency is that kraft was also allowing `CREATE` on the `Cluster` resource, which is not supported in zk clusters and was not documented in KIP-195: https://cwiki.apache.org/confluence/display/KAFKA/KIP-195%3A+AdminClient.createPartitions. This patch fixes this inconsistency and adds additional test coverage for both cases. ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KAFKA-13661) KRaft uses the wrong permission for adding topic partitions
[ https://issues.apache.org/jira/browse/KAFKA-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17489845#comment-17489845 ] Jason Gustafson commented on KAFKA-13661: - Another minor difference in behavior is that KRaft allows CREATE on the Cluster resource. This is not allowed in zk clusters and was not documented by KIP-195: https://cwiki.apache.org/confluence/display/KAFKA/KIP-195%3A+AdminClient.createPartitions. Probably not a big deal to allow this, but I'm inclined nevertheless to drop it for consistency. > KRaft uses the wrong permission for adding topic partitions > --- > > Key: KAFKA-13661 > URL: https://issues.apache.org/jira/browse/KAFKA-13661 > Project: Kafka > Issue Type: Bug >Affects Versions: 3.1.0, 3.0.0 >Reporter: Jason Gustafson >Assignee: Jason Gustafson >Priority: Major > > [~cmccabe] caught this as part of KAFKA-13646. KRaft currently checks CREATE > on the topic resource. It should be ALTER. This will be fixed in trunk as > part of KAFKA-13646, but it would be good to fix for 3.0 and 3.1 as well. > Note this does not affect zookeeper-based clusters. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [kafka] vvcephei commented on a change in pull request #11534: KAFKA-12939: After migrating processors, search the codebase for missed migrations
vvcephei commented on a change in pull request #11534:
URL: https://github.com/apache/kafka/pull/11534#discussion_r803128144
##
File path:
streams/src/main/java/org/apache/kafka/streams/processor/internals/RecordDeserializer.java
##
@@ -70,7 +70,10 @@
} catch (final Exception deserializationException) {
final
DeserializationExceptionHandler.DeserializationHandlerResponse response;
try {
-response =
deserializationExceptionHandler.handle(processorContext, rawRecord,
deserializationException);
+response = deserializationExceptionHandler.handle(
+(InternalProcessorContext) processorContext,
Review comment:
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] prince-mahajan commented on a change in pull request #11742: KAFKA-13636: Fix for the group coordinator issue where the offsets are deleted for unstable groups
prince-mahajan commented on a change in pull request #11742:
URL: https://github.com/apache/kafka/pull/11742#discussion_r803115724
##
File path:
core/src/test/scala/unit/kafka/coordinator/group/GroupMetadataTest.scala
##
@@ -259,6 +259,47 @@ class GroupMetadataTest {
assertFalse(group.supportsProtocols(protocolType, Set("range", "foo")))
}
+ @Test
+ def testOffsetRemovalDuringTransitionFromEmptyToNonEmpty(): Unit = {
+val topic = "foo"
+val partition = new TopicPartition(topic, 0)
+val time = new MockTime()
Review comment:
makes sense. thanks for catching that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] prince-mahajan commented on a change in pull request #11742: KAFKA-13636: Fix for the group coordinator issue where the offsets are deleted for unstable groups
prince-mahajan commented on a change in pull request #11742:
URL: https://github.com/apache/kafka/pull/11742#discussion_r803107096
##
File path: core/src/main/scala/kafka/coordinator/group/GroupMetadata.scala
##
@@ -763,7 +763,7 @@ private[group] class GroupMetadata(val groupId: String,
initialState: GroupState
}
val expiredOffsets: Map[TopicPartition, OffsetAndMetadata] = protocolType
match {
- case Some(_) if is(Empty) =>
+ case Some(_) if is(Empty) || !is(Stable)=>
Review comment:
Yes, I was saying that we can expire offsets with the larger expiration
time (per KIP-211): if currentStateTimestamp has elapsed. But, you are right,
it is better to not expire at all when in this state. I assume the group will
eventually transition to either Stable or Empty state and we can expire then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] jeqo commented on a change in pull request #11534: KAFKA-12939: After migrating processors, search the codebase for missed migrations
jeqo commented on a change in pull request #11534:
URL: https://github.com/apache/kafka/pull/11534#discussion_r803098389
##
File path:
streams/src/main/java/org/apache/kafka/streams/processor/internals/RecordDeserializer.java
##
@@ -70,7 +70,10 @@
} catch (final Exception deserializationException) {
final
DeserializationExceptionHandler.DeserializationHandlerResponse response;
try {
-response =
deserializationExceptionHandler.handle(processorContext, rawRecord,
deserializationException);
+response = deserializationExceptionHandler.handle(
+(InternalProcessorContext) processorContext,
Review comment:
https://issues.apache.org/jira/browse/KAFKA-13662
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (KAFKA-13662) Migrate DeserializationExceptionHandler to latest ProcessorContext API
Jorge Esteban Quilcate Otoya created KAFKA-13662: Summary: Migrate DeserializationExceptionHandler to latest ProcessorContext API Key: KAFKA-13662 URL: https://issues.apache.org/jira/browse/KAFKA-13662 Project: Kafka Issue Type: Task Reporter: Jorge Esteban Quilcate Otoya DeserializationExceptionHandler depends on old ProcessorContext API. This API has been deprecated and migrating to the latest one requires a public API that hasn't been considered as part of KIP-478. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [kafka] jeffreyolchovy commented on pull request #719: KAFKA-3049: VerifiableProperties does not respect 'default' properties of underlying java.util.Properties instance
jeffreyolchovy commented on pull request #719: URL: https://github.com/apache/kafka/pull/719#issuecomment-1034179990 @vvcephei The way I read the situation was that the ball was always in the court of the Kafka maintainers, and it sounded like they did not want to go down the approach of merging this in. The PR was code complete. Even though this is over six years old, I would be willing to update it and get it back into a merge-able state -- but again, the direction and decision needs to come from you all. Best, Jeff -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #11534: KAFKA-12939: After migrating processors, search the codebase for missed migrations
vvcephei commented on pull request #11534: URL: https://github.com/apache/kafka/pull/11534#issuecomment-1034166065 I've just kicked off a fresh CIT run (https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-11534/4/) to be sure this is still mergeable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei commented on a change in pull request #11534: KAFKA-12939: After migrating processors, search the codebase for missed migrations
vvcephei commented on a change in pull request #11534:
URL: https://github.com/apache/kafka/pull/11534#discussion_r803052837
##
File path:
streams/src/main/java/org/apache/kafka/streams/processor/internals/RecordDeserializer.java
##
@@ -70,7 +70,10 @@
} catch (final Exception deserializationException) {
final
DeserializationExceptionHandler.DeserializationHandlerResponse response;
try {
-response =
deserializationExceptionHandler.handle(processorContext, rawRecord,
deserializationException);
+response = deserializationExceptionHandler.handle(
+(InternalProcessorContext) processorContext,
Review comment:
Oy. Do you mind filing a `needs-kip` Jira to migrate
DeserializationExceptionHandler to the new API?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Resolved] (KAFKA-3790) Default options when removing ACLs do not comply with documentation
[
https://issues.apache.org/jira/browse/KAFKA-3790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sébastien Launay resolved KAFKA-3790.
-
Resolution: Won't Fix
> Default options when removing ACLs do not comply with documentation
> ---
>
> Key: KAFKA-3790
> URL: https://issues.apache.org/jira/browse/KAFKA-3790
> Project: Kafka
> Issue Type: Bug
> Components: documentation, security
>Affects Versions: 0.9.0.1, 0.10.0.0
>Reporter: Sébastien Launay
>Priority: Minor
>
> When removing ACLs without providing options like principal, host or
> operation, we got a prompt for removing all the matching ACLs but when
> executing the command none get removed.
> The following commands can be used to reproduce the inconsistency:
> {noformat}
> $ ./bin/kafka-acls.sh --authorizer-properties
> zookeeper.connect=localhost:2181 -list -topic test
> Current ACLs for resource `Topic:test`:
> $ ./bin/kafka-acls.sh --authorizer-properties
> zookeeper.connect=localhost:2181 --add --allow-principal User:Alice
> --operation Write --topic test --allow-host 1.2.3.4
> Adding ACLs for resource `Topic:test`:
> User:Alice has Allow permission for operations: Write from hosts:
> 1.2.3.4
> Current ACLs for resource `Topic:test`:
> User:Alice has Allow permission for operations: Write from hosts:
> 1.2.3.4
> $ ./bin/kafka-acls.sh --authorizer-properties
> zookeeper.connect=localhost:2181 --remove --allow-principal User:Alice
> --topic test
> Are you sure you want to remove ACLs:
> User:Alice has Allow permission for operations: All from hosts: *
> from resource `Topic:test`? (y/n)
> y
> Current ACLs for resource `Topic:test`:
> User:Alice has Allow permission for operations: Write from hosts:
> 1.2.3.4
> {noformat}
> *The Current ACLs for resource {{Topic:test}} is expected to be empty after
> the last command.*
> Only a specific ACL (when all options mentioned above are provided) or else
> all the ACLs for a given resource (none of the options mentioned above are
> provided) can get removed as shown by the following code snippets:
> {noformat}
> // AclCommand.scala
> ...
> private def removeAcl(opts: AclCommandOptions) {
> withAuthorizer(opts) { authorizer =>
> val resourceToAcl = getResourceToAcls(opts)
> for ((resource, acls) <- resourceToAcl) {
> if (acls.isEmpty) {
> if (confirmAction(opts, s"Are you sure you want to delete all ACLs
> for resource `${resource}`? (y/n)"))
> authorizer.removeAcls(resource)
> } else {
> if (confirmAction(opts, s"Are you sure you want to remove ACLs:
> $Newline ${acls.map("\t" + _).mkString(Newline)} $Newline from resource
> `${resource}`? (y/n)"))
> authorizer.removeAcls(acls, resource)
> }
> }
> listAcl(opts)
> }
> }
> ...
> // SimpleAclAuthorizer.scala
> ...
> override def removeAcls(aclsTobeRemoved: Set[Acl], resource: Resource):
> Boolean = {
> inWriteLock(lock) {
>updateResourceAcls(resource) { currentAcls =>
> currentAcls -- aclsTobeRemoved
>}
> }
>}
> {noformat}
> A workaround consists of listing the ACL in order to know which exact one to
> remove which make the automation of ACL management trickier.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (KAFKA-3790) Default options when removing ACLs do not comply with documentation
[
https://issues.apache.org/jira/browse/KAFKA-3790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17489798#comment-17489798
]
Sébastien Launay commented on KAFKA-3790:
-
It's an oldie and since then we have access to the Admin API so I don't think
it makes sense anymore to modify such long running behaviour of {{kafka-acls}}.
> Default options when removing ACLs do not comply with documentation
> ---
>
> Key: KAFKA-3790
> URL: https://issues.apache.org/jira/browse/KAFKA-3790
> Project: Kafka
> Issue Type: Bug
> Components: documentation, security
>Affects Versions: 0.9.0.1, 0.10.0.0
>Reporter: Sébastien Launay
>Priority: Minor
>
> When removing ACLs without providing options like principal, host or
> operation, we got a prompt for removing all the matching ACLs but when
> executing the command none get removed.
> The following commands can be used to reproduce the inconsistency:
> {noformat}
> $ ./bin/kafka-acls.sh --authorizer-properties
> zookeeper.connect=localhost:2181 -list -topic test
> Current ACLs for resource `Topic:test`:
> $ ./bin/kafka-acls.sh --authorizer-properties
> zookeeper.connect=localhost:2181 --add --allow-principal User:Alice
> --operation Write --topic test --allow-host 1.2.3.4
> Adding ACLs for resource `Topic:test`:
> User:Alice has Allow permission for operations: Write from hosts:
> 1.2.3.4
> Current ACLs for resource `Topic:test`:
> User:Alice has Allow permission for operations: Write from hosts:
> 1.2.3.4
> $ ./bin/kafka-acls.sh --authorizer-properties
> zookeeper.connect=localhost:2181 --remove --allow-principal User:Alice
> --topic test
> Are you sure you want to remove ACLs:
> User:Alice has Allow permission for operations: All from hosts: *
> from resource `Topic:test`? (y/n)
> y
> Current ACLs for resource `Topic:test`:
> User:Alice has Allow permission for operations: Write from hosts:
> 1.2.3.4
> {noformat}
> *The Current ACLs for resource {{Topic:test}} is expected to be empty after
> the last command.*
> Only a specific ACL (when all options mentioned above are provided) or else
> all the ACLs for a given resource (none of the options mentioned above are
> provided) can get removed as shown by the following code snippets:
> {noformat}
> // AclCommand.scala
> ...
> private def removeAcl(opts: AclCommandOptions) {
> withAuthorizer(opts) { authorizer =>
> val resourceToAcl = getResourceToAcls(opts)
> for ((resource, acls) <- resourceToAcl) {
> if (acls.isEmpty) {
> if (confirmAction(opts, s"Are you sure you want to delete all ACLs
> for resource `${resource}`? (y/n)"))
> authorizer.removeAcls(resource)
> } else {
> if (confirmAction(opts, s"Are you sure you want to remove ACLs:
> $Newline ${acls.map("\t" + _).mkString(Newline)} $Newline from resource
> `${resource}`? (y/n)"))
> authorizer.removeAcls(acls, resource)
> }
> }
> listAcl(opts)
> }
> }
> ...
> // SimpleAclAuthorizer.scala
> ...
> override def removeAcls(aclsTobeRemoved: Set[Acl], resource: Resource):
> Boolean = {
> inWriteLock(lock) {
>updateResourceAcls(resource) { currentAcls =>
> currentAcls -- aclsTobeRemoved
>}
> }
>}
> {noformat}
> A workaround consists of listing the ACL in order to know which exact one to
> remove which make the automation of ACL management trickier.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [kafka] slaunay edited a comment on pull request #1468: KAFKA-3790: Allow for removal of non specific ACLs
slaunay edited a comment on pull request #1468: URL: https://github.com/apache/kafka/pull/1468#issuecomment-1034159058 It's an oldie and since then we have access to the Admin API so I don't think it makes sense anymore to modify such long running behaviour of `kafka-acls`. I will close the JIRA ticket as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KAFKA-12256) auto commit causes delays due to retriable UNKNOWN_TOPIC_OR_PARTITION
[ https://issues.apache.org/jira/browse/KAFKA-12256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17489797#comment-17489797 ] Ryan Leslie commented on KAFKA-12256: - [~RivenSun] [~guozhang] Hey guys, I see that [https://github.com/apache/kafka/pull/11340] was just recently merged for KAFKA-13310. Thank you [~RivenSun] for your work on this! It looks like this ticket should be the same issue. Please let me know if you guys agree and we can close it out. I also wanted to see how you guys felt about potentially backporting the fix to earlier branches like 2.8, or maybe 3.0 if 2.8 is considered dead already. It looks like the 3.0.1 will happen soon at least. > auto commit causes delays due to retriable UNKNOWN_TOPIC_OR_PARTITION > - > > Key: KAFKA-12256 > URL: https://issues.apache.org/jira/browse/KAFKA-12256 > Project: Kafka > Issue Type: Bug > Components: consumer >Affects Versions: 2.0.0 >Reporter: Ryan Leslie >Priority: Minor > > In KAFKA-6829 a change was made to the consumer to internally retry commits > upon receiving UNKNOWN_TOPIC_OR_PARTITION. > Though this helped mitigate issues around stale broker metadata, there were > some valid concerns around the negative effects for routine topic deletion: > https://github.com/apache/kafka/pull/4948 > In particular, if a commit is issued for a deleted topic, retries can block > the consumer for up to max.poll.interval.ms. This is tunable of course, but > any amount of stalling in a consumer can lead to unnecessary lag. > One of the assumptions while permitting the change was that in practice it > should be rare for commits to occur for deleted topics, since that would > imply messages were being read or published at the time of deletion. It's > fair to expect users to not delete topics that are actively published to. But > this assumption is false in cases where auto commit is enabled. > With the current implementation of auto commit, the consumer will regularly > issue commits for all topics being fetched from, regardless of whether or not > messages were actually received. The fetch positions are simply flushed, even > when they are 0. This is simple and generally efficient, though it does mean > commits are often redundant. Besides the auto commit interval, commits are > also issued at the time of rebalance, which is often precisely at the time > topics are deleted. > This means that in practice commits for deleted topics are not really rare. > This is particularly an issue when the consumer is subscribed to a multitude > of topics using a wildcard. For example, a consumer might subscribe to a > particular "flavor" of topic with the aim of auditing all such data, and > these topics might dynamically come and go. The consumer's metadata and > rebalance mechanisms are meant to handle this gracefully, but the end result > is that such groups are often blocked in a commit for several seconds or > minutes (the default is 5 minutes) whenever a delete occurs. This can > sometimes result in significant lag. > Besides having users abandon auto commit in the face of topic deletes, there > are probably multiple ways to deal with this, including reconsidering if > commits still truly need to be retried here, or if this behavior should be > more configurable; e.g. having a separate commit timeout or policy. In some > cases the loss of a commit and subsequent message duplication is still > preferred to processing delays. And having an artificially low > max.poll.interval.ms or rebalance.timeout.ms comes with its own set of > concerns. > In the very least the current behavior and pitfalls around delete with active > consumers should be documented. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [kafka] slaunay commented on pull request #1468: KAFKA-3790: Allow for removal of non specific ACLs
slaunay commented on pull request #1468: URL: https://github.com/apache/kafka/pull/1468#issuecomment-1034159058 It's an oldie and since then we have access to Admin API so I don't think it makes sense anymore to modify such long running behaviour of `kafka-acls`. I will close the JIRA ticket as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KAFKA-9366) Upgrade log4j to log4j2
[ https://issues.apache.org/jira/browse/KAFKA-9366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17489791#comment-17489791 ] Akansh Shandilya commented on KAFKA-9366: - Voting has been done by multiple users, and more are the requests for upgrading log4j. Can we set or request ETA for log4j upgrade to log4j2. Any other challenge in doing so. [https://logging.apache.org/log4j/1.2/download.html] Log4j 1.x was End-Of-Llife on August 5, 2015. Kafka and Log4j, both are connected to Apache. As a strong community we need to think :: Does Apache-Kafka require more than 6 years of time to upgrade a log4j library, which was declared End-of-Life by Apache-log4j in 2015. > Upgrade log4j to log4j2 > --- > > Key: KAFKA-9366 > URL: https://issues.apache.org/jira/browse/KAFKA-9366 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 2.2.0, 2.1.1, 2.3.0, 2.4.0 >Reporter: leibo >Assignee: Dongjin Lee >Priority: Critical > Labels: needs-kip > Fix For: 3.2.0 > > > h2. CVE-2019-17571 Detail > Included in Log4j 1.2 is a SocketServer class that is vulnerable to > deserialization of untrusted data which can be exploited to remotely execute > arbitrary code when combined with a deserialization gadget when listening to > untrusted network traffic for log data. This affects Log4j versions up to 1.2 > up to 1.2.17. > > [https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-17571] > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (KAFKA-12495) Unbalanced connectors/tasks distribution will happen in Connect's incremental cooperative assignor
[
https://issues.apache.org/jira/browse/KAFKA-12495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17489790#comment-17489790
]
Chris Egerton commented on KAFKA-12495:
---
Hi [~showuon]! I'm taking time off right now but should be able to try to get
up-to-speed on the state of incremental rebalancing logic in Connect and take a
look at your PR and the issue description here sometime next week.
I'll note that the assertion that "This issue corresponds to a corner case that
does not seem to appear in practice often." seems to be empirically incorrect
given the level of attention that this ticket has received, and given some
recent conversations on the dev mailing list around timely review of PRs from
non-committer contributors, would love to see this get the attention it
deserves.
We should all obviously try to maintain a high level of quality and efficacy in
the Connect framework and the Kafka code base in general, but at this point it
seems like we've let perfect become the enemy of good and have allowed this
issue to remain unaddressed for far too long (nearly a year!) despite showing
promise as an improvement to Connect. There's only so much I can do as a
contributor, but I'd be happy to take the first step and give this a look of my
own to try to help move things along.
> Unbalanced connectors/tasks distribution will happen in Connect's incremental
> cooperative assignor
> --
>
> Key: KAFKA-12495
> URL: https://issues.apache.org/jira/browse/KAFKA-12495
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Reporter: Luke Chen
>Assignee: Luke Chen
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: image-2021-03-18-15-04-57-854.png,
> image-2021-03-18-15-05-52-557.png, image-2021-03-18-15-07-27-103.png
>
>
> In Kafka Connect, we implement incremental cooperative rebalance algorithm
> based on KIP-415
> ([https://cwiki.apache.org/confluence/display/KAFKA/KIP-415%3A+Incremental+Cooperative+Rebalancing+in+Kafka+Connect)|https://cwiki.apache.org/confluence/display/KAFKA/KIP-415%3A+Incremental+Cooperative+Rebalancing+in+Kafka+Connect].
> However, we have a bad assumption in the algorithm implementation, which is:
> after revoking rebalance completed, the member(worker) count will be the same
> as the previous round of reblance.
>
> Let's take a look at the example in the KIP-415:
> !image-2021-03-18-15-07-27-103.png|width=441,height=556!
> It works well for most cases. But what if W4 added after 1st rebalance
> completed and before 2nd rebalance started? Let's see what will happened?
> Let's see this example: (we'll use 10 tasks here):
>
> {code:java}
> Initial group and assignment: W1([AC0, AT1, AT2, AT3, AT4, AT5, BC0, BT1,
> BT2, BT4, BT4, BT5])
> Config topic contains: AC0, AT1, AT2, AT3, AT4, AT5, BC0, BT1, BT2, BT4, BT4,
> BT5
> W1 is current leader
> W2 joins with assignment: []
> Rebalance is triggered
> W3 joins while rebalance is still active with assignment: []
> W1 joins with assignment: [AC0, AT1, AT2, AT3, AT4, AT5, BC0, BT1, BT2, BT4,
> BT4, BT5]
> W1 becomes leader
> W1 computes and sends assignments:
> W1(delay: 0, assigned: [AC0, AT1, AT2, AT3], revoked: [AT4, AT5, BC0, BT1,
> BT2, BT4, BT4, BT5])
> W2(delay: 0, assigned: [], revoked: [])
> W3(delay: 0, assigned: [], revoked: [])
> W1 stops revoked resources
> W1 rejoins with assignment: [AC0, AT1, AT2, AT3]
> Rebalance is triggered
> W2 joins with assignment: []
> W3 joins with assignment: []
> // one more member joined
> W4 joins with assignment: []
> W1 becomes leader
> W1 computes and sends assignments:
> // We assigned all the previous revoked Connectors/Tasks to the new member,
> but we didn't revoke any more C/T in this round, which cause unbalanced
> distribution
> W1(delay: 0, assigned: [AC0, AT1, AT2, AT3], revoked: [])
> W2(delay: 0, assigned: [AT4, AT5, BC0], revoked: [])
> W2(delay: 0, assigned: [BT1, BT2, BT4], revoked: [])
> W2(delay: 0, assigned: [BT4, BT5], revoked: [])
> {code}
> Because we didn't allow to do consecutive revoke in two consecutive
> rebalances (under the same leader), we will have this uneven distribution
> under this situation. We should allow consecutive rebalance to have another
> round of revocation to revoke the C/T to the other members in this case.
> expected:
> {code:java}
> Initial group and assignment: W1([AC0, AT1, AT2, AT3, AT4, AT5, BC0, BT1,
> BT2, BT4, BT4, BT5])
> Config topic contains: AC0, AT1, AT2, AT3, AT4, AT5, BC0, BT1, BT2, BT4, BT4,
> BT5
> W1 is current leader
> W2 joins with assignment: []
> Rebalance is triggered
> W3 joins while rebalance is still active with assignment: []
> W1 joins with assignment: [AC0, AT1, AT2, AT3, AT4, AT5, BC0, BT1
[jira] [Assigned] (KAFKA-12495) Unbalanced connectors/tasks distribution will happen in Connect's incremental cooperative assignor
[
https://issues.apache.org/jira/browse/KAFKA-12495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Egerton reassigned KAFKA-12495:
-
Assignee: Chris Egerton (was: Luke Chen)
> Unbalanced connectors/tasks distribution will happen in Connect's incremental
> cooperative assignor
> --
>
> Key: KAFKA-12495
> URL: https://issues.apache.org/jira/browse/KAFKA-12495
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Reporter: Luke Chen
>Assignee: Chris Egerton
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: image-2021-03-18-15-04-57-854.png,
> image-2021-03-18-15-05-52-557.png, image-2021-03-18-15-07-27-103.png
>
>
> In Kafka Connect, we implement incremental cooperative rebalance algorithm
> based on KIP-415
> ([https://cwiki.apache.org/confluence/display/KAFKA/KIP-415%3A+Incremental+Cooperative+Rebalancing+in+Kafka+Connect)|https://cwiki.apache.org/confluence/display/KAFKA/KIP-415%3A+Incremental+Cooperative+Rebalancing+in+Kafka+Connect].
> However, we have a bad assumption in the algorithm implementation, which is:
> after revoking rebalance completed, the member(worker) count will be the same
> as the previous round of reblance.
>
> Let's take a look at the example in the KIP-415:
> !image-2021-03-18-15-07-27-103.png|width=441,height=556!
> It works well for most cases. But what if W4 added after 1st rebalance
> completed and before 2nd rebalance started? Let's see what will happened?
> Let's see this example: (we'll use 10 tasks here):
>
> {code:java}
> Initial group and assignment: W1([AC0, AT1, AT2, AT3, AT4, AT5, BC0, BT1,
> BT2, BT4, BT4, BT5])
> Config topic contains: AC0, AT1, AT2, AT3, AT4, AT5, BC0, BT1, BT2, BT4, BT4,
> BT5
> W1 is current leader
> W2 joins with assignment: []
> Rebalance is triggered
> W3 joins while rebalance is still active with assignment: []
> W1 joins with assignment: [AC0, AT1, AT2, AT3, AT4, AT5, BC0, BT1, BT2, BT4,
> BT4, BT5]
> W1 becomes leader
> W1 computes and sends assignments:
> W1(delay: 0, assigned: [AC0, AT1, AT2, AT3], revoked: [AT4, AT5, BC0, BT1,
> BT2, BT4, BT4, BT5])
> W2(delay: 0, assigned: [], revoked: [])
> W3(delay: 0, assigned: [], revoked: [])
> W1 stops revoked resources
> W1 rejoins with assignment: [AC0, AT1, AT2, AT3]
> Rebalance is triggered
> W2 joins with assignment: []
> W3 joins with assignment: []
> // one more member joined
> W4 joins with assignment: []
> W1 becomes leader
> W1 computes and sends assignments:
> // We assigned all the previous revoked Connectors/Tasks to the new member,
> but we didn't revoke any more C/T in this round, which cause unbalanced
> distribution
> W1(delay: 0, assigned: [AC0, AT1, AT2, AT3], revoked: [])
> W2(delay: 0, assigned: [AT4, AT5, BC0], revoked: [])
> W2(delay: 0, assigned: [BT1, BT2, BT4], revoked: [])
> W2(delay: 0, assigned: [BT4, BT5], revoked: [])
> {code}
> Because we didn't allow to do consecutive revoke in two consecutive
> rebalances (under the same leader), we will have this uneven distribution
> under this situation. We should allow consecutive rebalance to have another
> round of revocation to revoke the C/T to the other members in this case.
> expected:
> {code:java}
> Initial group and assignment: W1([AC0, AT1, AT2, AT3, AT4, AT5, BC0, BT1,
> BT2, BT4, BT4, BT5])
> Config topic contains: AC0, AT1, AT2, AT3, AT4, AT5, BC0, BT1, BT2, BT4, BT4,
> BT5
> W1 is current leader
> W2 joins with assignment: []
> Rebalance is triggered
> W3 joins while rebalance is still active with assignment: []
> W1 joins with assignment: [AC0, AT1, AT2, AT3, AT4, AT5, BC0, BT1, BT2, BT4,
> BT4, BT5]
> W1 becomes leader
> W1 computes and sends assignments:
> W1(delay: 0, assigned: [AC0, AT1, AT2, AT3], revoked: [AT4, AT5, BC0, BT1,
> BT2, BT4, BT4, BT5])
> W2(delay: 0, assigned: [], revoked: [])
> W3(delay: 0, assigned: [], revoked: [])
> W1 stops revoked resources
> W1 rejoins with assignment: [AC0, AT1, AT2, AT3]
> Rebalance is triggered
> W2 joins with assignment: []
> W3 joins with assignment: []
> // one more member joined
> W4 joins with assignment: []
> W1 becomes leader
> W1 computes and sends assignments:
> // We assigned all the previous revoked Connectors/Tasks to the new member,
> **and also revoke some C/T**
> W1(delay: 0, assigned: [AC0, AT1, AT2], revoked: [AT3])
> W2(delay: 0, assigned: [AT4, AT5, BC0], revoked: [])
> W3(delay: 0, assigned: [BT1, BT2, BT4], revoked: [])
> W4(delay: 0, assigned: [BT4, BT5], revoked: [])
> // another round of rebalance to assign the new revoked C/T to the other
> members
> W1 rejoins with assignment: [AC0, AT1, AT2]
> Rebalance is triggered
> W2 joins with assignment: [AT4, AT5, BC0]
> W3 joins with assignment: [BT1, BT2, BT4]
> W4 joins
[jira] [Assigned] (KAFKA-12495) Unbalanced connectors/tasks distribution will happen in Connect's incremental cooperative assignor
[
https://issues.apache.org/jira/browse/KAFKA-12495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Egerton reassigned KAFKA-12495:
-
Assignee: Luke Chen (was: Chris Egerton)
> Unbalanced connectors/tasks distribution will happen in Connect's incremental
> cooperative assignor
> --
>
> Key: KAFKA-12495
> URL: https://issues.apache.org/jira/browse/KAFKA-12495
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Reporter: Luke Chen
>Assignee: Luke Chen
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: image-2021-03-18-15-04-57-854.png,
> image-2021-03-18-15-05-52-557.png, image-2021-03-18-15-07-27-103.png
>
>
> In Kafka Connect, we implement incremental cooperative rebalance algorithm
> based on KIP-415
> ([https://cwiki.apache.org/confluence/display/KAFKA/KIP-415%3A+Incremental+Cooperative+Rebalancing+in+Kafka+Connect)|https://cwiki.apache.org/confluence/display/KAFKA/KIP-415%3A+Incremental+Cooperative+Rebalancing+in+Kafka+Connect].
> However, we have a bad assumption in the algorithm implementation, which is:
> after revoking rebalance completed, the member(worker) count will be the same
> as the previous round of reblance.
>
> Let's take a look at the example in the KIP-415:
> !image-2021-03-18-15-07-27-103.png|width=441,height=556!
> It works well for most cases. But what if W4 added after 1st rebalance
> completed and before 2nd rebalance started? Let's see what will happened?
> Let's see this example: (we'll use 10 tasks here):
>
> {code:java}
> Initial group and assignment: W1([AC0, AT1, AT2, AT3, AT4, AT5, BC0, BT1,
> BT2, BT4, BT4, BT5])
> Config topic contains: AC0, AT1, AT2, AT3, AT4, AT5, BC0, BT1, BT2, BT4, BT4,
> BT5
> W1 is current leader
> W2 joins with assignment: []
> Rebalance is triggered
> W3 joins while rebalance is still active with assignment: []
> W1 joins with assignment: [AC0, AT1, AT2, AT3, AT4, AT5, BC0, BT1, BT2, BT4,
> BT4, BT5]
> W1 becomes leader
> W1 computes and sends assignments:
> W1(delay: 0, assigned: [AC0, AT1, AT2, AT3], revoked: [AT4, AT5, BC0, BT1,
> BT2, BT4, BT4, BT5])
> W2(delay: 0, assigned: [], revoked: [])
> W3(delay: 0, assigned: [], revoked: [])
> W1 stops revoked resources
> W1 rejoins with assignment: [AC0, AT1, AT2, AT3]
> Rebalance is triggered
> W2 joins with assignment: []
> W3 joins with assignment: []
> // one more member joined
> W4 joins with assignment: []
> W1 becomes leader
> W1 computes and sends assignments:
> // We assigned all the previous revoked Connectors/Tasks to the new member,
> but we didn't revoke any more C/T in this round, which cause unbalanced
> distribution
> W1(delay: 0, assigned: [AC0, AT1, AT2, AT3], revoked: [])
> W2(delay: 0, assigned: [AT4, AT5, BC0], revoked: [])
> W2(delay: 0, assigned: [BT1, BT2, BT4], revoked: [])
> W2(delay: 0, assigned: [BT4, BT5], revoked: [])
> {code}
> Because we didn't allow to do consecutive revoke in two consecutive
> rebalances (under the same leader), we will have this uneven distribution
> under this situation. We should allow consecutive rebalance to have another
> round of revocation to revoke the C/T to the other members in this case.
> expected:
> {code:java}
> Initial group and assignment: W1([AC0, AT1, AT2, AT3, AT4, AT5, BC0, BT1,
> BT2, BT4, BT4, BT5])
> Config topic contains: AC0, AT1, AT2, AT3, AT4, AT5, BC0, BT1, BT2, BT4, BT4,
> BT5
> W1 is current leader
> W2 joins with assignment: []
> Rebalance is triggered
> W3 joins while rebalance is still active with assignment: []
> W1 joins with assignment: [AC0, AT1, AT2, AT3, AT4, AT5, BC0, BT1, BT2, BT4,
> BT4, BT5]
> W1 becomes leader
> W1 computes and sends assignments:
> W1(delay: 0, assigned: [AC0, AT1, AT2, AT3], revoked: [AT4, AT5, BC0, BT1,
> BT2, BT4, BT4, BT5])
> W2(delay: 0, assigned: [], revoked: [])
> W3(delay: 0, assigned: [], revoked: [])
> W1 stops revoked resources
> W1 rejoins with assignment: [AC0, AT1, AT2, AT3]
> Rebalance is triggered
> W2 joins with assignment: []
> W3 joins with assignment: []
> // one more member joined
> W4 joins with assignment: []
> W1 becomes leader
> W1 computes and sends assignments:
> // We assigned all the previous revoked Connectors/Tasks to the new member,
> **and also revoke some C/T**
> W1(delay: 0, assigned: [AC0, AT1, AT2], revoked: [AT3])
> W2(delay: 0, assigned: [AT4, AT5, BC0], revoked: [])
> W3(delay: 0, assigned: [BT1, BT2, BT4], revoked: [])
> W4(delay: 0, assigned: [BT4, BT5], revoked: [])
> // another round of rebalance to assign the new revoked C/T to the other
> members
> W1 rejoins with assignment: [AC0, AT1, AT2]
> Rebalance is triggered
> W2 joins with assignment: [AT4, AT5, BC0]
> W3 joins with assignment: [BT1, BT2, BT4]
> W4 joins with
[GitHub] [kafka] lmr3796 commented on a change in pull request #11744: MINOR: Fix JavaDoc of OffsetIndex#append
lmr3796 commented on a change in pull request #11744: URL: https://github.com/apache/kafka/pull/11744#discussion_r803031914 ## File path: core/src/main/scala/kafka/log/OffsetIndex.scala ## @@ -136,7 +136,7 @@ class OffsetIndex(_file: File, baseOffset: Long, maxIndexSize: Int = -1, writabl /** * Append an entry for the given offset/location pair to the index. This entry must have a larger offset than all subsequent entries. - * @throws IndexOffsetOverflowException if the offset causes index offset to overflow + * @throws InvalidOffsetException if the offset causes index offset to overflow Review comment: Thanks for taking time to review @mimaison! I've added the exception kept the original one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei closed pull request #560: KAFKA-2423: Introduce Scalastyle
vvcephei closed pull request #560: URL: https://github.com/apache/kafka/pull/560 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #560: KAFKA-2423: Introduce Scalastyle
vvcephei commented on pull request #560: URL: https://github.com/apache/kafka/pull/560#issuecomment-1034122311 Hi @granthenke , It seems like this PR stalled. I'll close it out for now, but if you or anyone else want to resume this work, please feel free to re-open it (or start a new one)! Thanks, John -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei closed pull request #200: KAFKA-2512: Add version check to broker and clients.
vvcephei closed pull request #200: URL: https://github.com/apache/kafka/pull/200 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #200: KAFKA-2512: Add version check to broker and clients.
vvcephei commented on pull request #200: URL: https://github.com/apache/kafka/pull/200#issuecomment-1034122122 Hi @becketqin , It seems like this PR stalled. I'll close it out for now, but if you or anyone else want to resume this work, please feel free to re-open it (or start a new one)! Thanks, John -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei closed pull request #62: add support libvirt as provider. KAFKA-2183
vvcephei closed pull request #62: URL: https://github.com/apache/kafka/pull/62 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #62: add support libvirt as provider. KAFKA-2183
vvcephei commented on pull request #62: URL: https://github.com/apache/kafka/pull/62#issuecomment-1034121935 Hi @pronix , It seems like this PR stalled. I'll close it out for now, but if you or anyone else want to resume this work, please feel free to re-open it (or start a new one)! Thanks, John -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei closed pull request #719: KAFKA-3049: VerifiableProperties does not respect 'default' properties of underlying java.util.Properties instance
vvcephei closed pull request #719: URL: https://github.com/apache/kafka/pull/719 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #719: KAFKA-3049: VerifiableProperties does not respect 'default' properties of underlying java.util.Properties instance
vvcephei commented on pull request #719: URL: https://github.com/apache/kafka/pull/719#issuecomment-1034121675 Hi @jeffreyolchovy , It seems like this PR stalled. I'll close it out for now, but if you or anyone else want to resume this work, please feel free to re-open it (or start a new one)! Thanks, John -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #735: KAFKA-3065: Remove unused topic partitions from RecordAccumulator
vvcephei commented on pull request #735: URL: https://github.com/apache/kafka/pull/735#issuecomment-1034121499 Hi @rajinisivaram , It seems like this PR stalled. I'll close it out for now, but if you or anyone else want to resume this work, please feel free to re-open it (or start a new one)! Thanks, John -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei closed pull request #735: KAFKA-3065: Remove unused topic partitions from RecordAccumulator
vvcephei closed pull request #735: URL: https://github.com/apache/kafka/pull/735 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei closed pull request #757: KAFKA-3082: Make LogManager.InitialTaskDelayMs configurable
vvcephei closed pull request #757: URL: https://github.com/apache/kafka/pull/757 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #757: KAFKA-3082: Make LogManager.InitialTaskDelayMs configurable
vvcephei commented on pull request #757: URL: https://github.com/apache/kafka/pull/757#issuecomment-1034121306 Ho @j-nowak , It seems like this PR stalled. I'll close it out for now, but if you or anyone else want to resume this work, please feel free to re-open it (or start a new one)! Thanks, John -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #824: KAFKA-3161: Fixed ProducerConfig/ConsumerConfig so that defaults are used in java.util.Properties
vvcephei commented on pull request #824: URL: https://github.com/apache/kafka/pull/824#issuecomment-1034121012 Hi @crhyne , It seems like this PR stalled. I'll close it out for now, but if you or anyone else want to resume this work, please feel free to re-open it (or start a new one)! Thanks, John -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei closed pull request #824: KAFKA-3161: Fixed ProducerConfig/ConsumerConfig so that defaults are used in java.util.Properties
vvcephei closed pull request #824: URL: https://github.com/apache/kafka/pull/824 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #880: KAFKA-3190 Producer should not fire callback in Send() method
vvcephei commented on pull request #880: URL: https://github.com/apache/kafka/pull/880#issuecomment-1034120832 Hi @becketqin , It seems like this PR stalled. I'll close it out for now, but if you or anyone else want to resume this work, please feel free to re-open it (or start a new one)! Thanks, John -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei closed pull request #880: KAFKA-3190 Producer should not fire callback in Send() method
vvcephei closed pull request #880: URL: https://github.com/apache/kafka/pull/880 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei closed pull request #907: KAFKA-3234; Clarify minISR in documentation and auto-generate topic configuration docs
vvcephei closed pull request #907: URL: https://github.com/apache/kafka/pull/907 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #907: KAFKA-3234; Clarify minISR in documentation and auto-generate topic configuration docs
vvcephei commented on pull request #907: URL: https://github.com/apache/kafka/pull/907#issuecomment-1034120532 Hi @jjkoshy , It seems like this PR stalled. I'll close it out for now, but if you or anyone else want to resume this work, please feel free to re-open it (or start a new one)! Thanks, John -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei closed pull request #1035: KAFKA-3359 Parallel log-recovery of un-flushed segments on startup
vvcephei closed pull request #1035: URL: https://github.com/apache/kafka/pull/1035 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei closed pull request #983: KAFKA-3300: Avoid over allocating disk space and memory for index files.
vvcephei closed pull request #983: URL: https://github.com/apache/kafka/pull/983 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #983: KAFKA-3300: Avoid over allocating disk space and memory for index files.
vvcephei commented on pull request #983: URL: https://github.com/apache/kafka/pull/983#issuecomment-1034120088 Hi @becketqin , It seems like this PR stalled. I'll close it out for now, but if you or anyone else want to resume this work, please feel free to re-open it (or start a new one)! Thanks, John -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #1035: KAFKA-3359 Parallel log-recovery of un-flushed segments on startup
vvcephei commented on pull request #1035: URL: https://github.com/apache/kafka/pull/1035#issuecomment-1034119827 Hello, It seems like this PR stalled. I'll close it out for now, but if you or anyone else want to resume this work, please feel free to re-open it (or start a new one)! Thanks, John -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #1078: Fixes for Windows #154
vvcephei commented on pull request #1078: URL: https://github.com/apache/kafka/pull/1078#issuecomment-1034119371 Hi @JeffersJi , It seems like this PR stalled. I'll close it out for now, but if you or anyone else want to resume this work, please feel free to re-open it (or start a new one)! Thanks, John -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei closed pull request #1078: Fixes for Windows #154
vvcephei closed pull request #1078: URL: https://github.com/apache/kafka/pull/1078 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #1111: KAFKA-3428 Remove metadata sync bottleneck from mirrormaker's producer
vvcephei commented on pull request #: URL: https://github.com/apache/kafka/pull/#issuecomment-1034119203 Hi @maysamyabandeh , It seems like this PR stalled. I'll close it out for now, but if you or anyone else want to resume this work, please feel free to re-open it (or start a new one)! Thanks, John -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei closed pull request #1135: [KAFKA-3458] Selector should throw InterruptException when interrupted.
vvcephei closed pull request #1135: URL: https://github.com/apache/kafka/pull/1135 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #1147: [KAFKA-3472] Allow MirrorMaker to copy selected partitions and choose target topic name
vvcephei commented on pull request #1147: URL: https://github.com/apache/kafka/pull/1147#issuecomment-1034118707 Hi @ooasis , It seems like this PR stalled. I'll close it out for now, but if you or anyone else want to resume this work, please feel free to re-open it (or start a new one)! Thanks, John -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei closed pull request #1111: KAFKA-3428 Remove metadata sync bottleneck from mirrormaker's producer
vvcephei closed pull request #: URL: https://github.com/apache/kafka/pull/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #1135: [KAFKA-3458] Selector should throw InterruptException when interrupted.
vvcephei commented on pull request #1135: URL: https://github.com/apache/kafka/pull/1135#issuecomment-1034118967 Hi @sruehl , It seems like this PR stalled. I'll close it out for now, but if you or anyone else want to resume this work, please feel free to re-open it (or start a new one)! Thanks, John -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei closed pull request #1147: [KAFKA-3472] Allow MirrorMaker to copy selected partitions and choose target topic name
vvcephei closed pull request #1147: URL: https://github.com/apache/kafka/pull/1147 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] lmr3796 commented on a change in pull request #11744: MINOR: Fix JavaDoc of OffsetIndex#append
lmr3796 commented on a change in pull request #11744: URL: https://github.com/apache/kafka/pull/11744#discussion_r803018010 ## File path: core/src/main/scala/kafka/log/OffsetIndex.scala ## @@ -136,7 +136,7 @@ class OffsetIndex(_file: File, baseOffset: Long, maxIndexSize: Int = -1, writabl /** * Append an entry for the given offset/location pair to the index. This entry must have a larger offset than all subsequent entries. - * @throws IndexOffsetOverflowException if the offset causes index offset to overflow + * @throws InvalidOffsetException if the offset causes index offset to overflow Review comment: Ah @mimaison you're right. Let me try to change it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei closed pull request #1150: KAFKA-3474: add metrics to track replica fetcher timeouts
vvcephei closed pull request #1150: URL: https://github.com/apache/kafka/pull/1150 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #1150: KAFKA-3474: add metrics to track replica fetcher timeouts
vvcephei commented on pull request #1150: URL: https://github.com/apache/kafka/pull/1150#issuecomment-1034118461 Hi @junrao , It seems like this PR stalled. I'll close it out for now, but if you or anyone else want to resume this work, please feel free to re-open it (or start a new one)! Thanks, John -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei closed pull request #1244: MINOR: Docs for ACLs over SSL auth and KAFKA_OPTS
vvcephei closed pull request #1244: URL: https://github.com/apache/kafka/pull/1244 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #1244: MINOR: Docs for ACLs over SSL auth and KAFKA_OPTS
vvcephei commented on pull request #1244: URL: https://github.com/apache/kafka/pull/1244#issuecomment-1034118238 Hi @QwertyManiac , It seems like this PR stalled. I'll close it out for now, but if you or anyone else want to resume this work, please feel free to re-open it (or start a new one)! Thanks, John -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei closed pull request #1269: KAFKA-3622: Use descriptive error message if port number is missing from url
vvcephei closed pull request #1269: URL: https://github.com/apache/kafka/pull/1269 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #1269: KAFKA-3622: Use descriptive error message if port number is missing from url
vvcephei commented on pull request #1269: URL: https://github.com/apache/kafka/pull/1269#issuecomment-1034118007 Hi @peterableda , It seems like this PR stalled. I'll close it out for now, but if you or anyone else want to resume this work, please feel free to re-open it (or start a new one)! Thanks, John -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #1415: KAFKA-3737: Change log level for error during produce request
vvcephei commented on pull request #1415: URL: https://github.com/apache/kafka/pull/1415#issuecomment-1034117748 Hi @fhussonnois , It seems like this PR stalled. I'll close it out for now, but if you or anyone else want to resume this work, please feel free to re-open it (or start a new one)! Thanks, John -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei closed pull request #1415: KAFKA-3737: Change log level for error during produce request
vvcephei closed pull request #1415: URL: https://github.com/apache/kafka/pull/1415 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #1468: KAFKA-3790: Allow for removal of non specific ACLs
vvcephei commented on pull request #1468: URL: https://github.com/apache/kafka/pull/1468#issuecomment-1034117452 Hi @slaunay , It seems like this PR stalled. I'll close it out for now, but if you or anyone else want to resume this work, please feel free to re-open it (or start a new one)! Thanks, John -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vvcephei closed pull request #1468: KAFKA-3790: Allow for removal of non specific ACLs
vvcephei closed pull request #1468: URL: https://github.com/apache/kafka/pull/1468 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] lmr3796 closed pull request #11744: MINOR: Fix JavaDoc of OffsetIndex#append
lmr3796 closed pull request #11744: URL: https://github.com/apache/kafka/pull/11744 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] mimaison commented on pull request #11731: KAFKA-13293: Reloading SSL Engine Factory
mimaison commented on pull request #11731: URL: https://github.com/apache/kafka/pull/11731#issuecomment-1034113905 Thanks @teabot for the contribution! This PR adds new configurations and these are considered public API. So in order to accept this change we need a [Kafka Improvement Proposal](https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals). Let me know if you have any questions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] mimaison commented on a change in pull request #11744: MINOR: Fix JavaDoc of OffsetIndex#append
mimaison commented on a change in pull request #11744: URL: https://github.com/apache/kafka/pull/11744#discussion_r803007182 ## File path: core/src/main/scala/kafka/log/OffsetIndex.scala ## @@ -136,7 +136,7 @@ class OffsetIndex(_file: File, baseOffset: Long, maxIndexSize: Int = -1, writabl /** * Append an entry for the given offset/location pair to the index. This entry must have a larger offset than all subsequent entries. - * @throws IndexOffsetOverflowException if the offset causes index offset to overflow + * @throws InvalidOffsetException if the offset causes index offset to overflow Review comment: This method does indeed throw `IndexOffsetOverflowException` (from `relativeOffset()`) for the reason listed. I'm assuming you got confused because it also throws `InvalidOffsetException`. It's ok to add another `@throws` tag if you want but we don't want to remove the existing one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] mimaison commented on a change in pull request #11673: KAFKA-13577: Replace easymock with mockito in kafka:core - part 2
mimaison commented on a change in pull request #11673:
URL: https://github.com/apache/kafka/pull/11673#discussion_r802994307
##
File path: core/src/test/scala/unit/kafka/server/ReplicaManagerQuotasTest.scala
##
@@ -157,10 +158,9 @@ class ReplicaManagerQuotasTest {
def shouldIncludeThrottledReplicasForConsumerFetch(): Unit = {
setUpMocks(fetchInfo)
-val quota = mockQuota(100)
-expect(quota.isQuotaExceeded).andReturn(true).once()
-expect(quota.isQuotaExceeded).andReturn(true).once()
-replay(quota)
+val quota = mockQuota()
+when(quota.isQuotaExceeded).thenReturn(true)
+//expect(quota.isQuotaExceeded).andReturn(true).once()
Review comment:
Oops! I forgot to remove it when rebasing. I've fixed it now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (KAFKA-13422) Even if the correct username and password are configured, when ClientBroker or KafkaClient tries to establish a SASL connection to ServerBroker, an exception is thrown
[
https://issues.apache.org/jira/browse/KAFKA-13422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17489744#comment-17489744
]
Guozhang Wang commented on KAFKA-13422:
---
I'm unfortunately less familiar with security.auth module, [~rsivaram] [~ijuma]
could you please chime in with your thoughts?
> Even if the correct username and password are configured, when ClientBroker
> or KafkaClient tries to establish a SASL connection to ServerBroker, an
> exception is thrown: (Authentication failed: Invalid username or password)
> --
>
> Key: KAFKA-13422
> URL: https://issues.apache.org/jira/browse/KAFKA-13422
> Project: Kafka
> Issue Type: Bug
> Components: clients, core
>Affects Versions: 2.7.1, 3.0.0
>Reporter: RivenSun
>Priority: Major
> Attachments: CustomerAuthCallbackHandler.java,
> LoginContext_login_debug.png, SaslClientCallbackHandler_handle_debug.png
>
>
>
> h1. Foreword:
> When deploying a Kafka cluster with a higher version (2.7.1), I encountered
> an exception of communication identity authentication failure between
> brokers. In the current latest version 3.0.0, this problem can also be
> reproduced.
> h1. Problem recurring:
> h2. 1）broker Version is 3.0.0
> h3. The content of kafka_server_jaas.conf of each broker is exactly the same,
> the content is as follows：
>
>
> {code:java}
> KafkaServer {
> org.apache.kafka.common.security.plain.PlainLoginModule required
> username="admin"
> password="kJTVDziatPgjXG82sFHc4O1EIuewmlvS"
> user_admin="kJTVDziatPgjXG82sFHc4O1EIuewmlvS"
> user_alice="alice";
> org.apache.kafka.common.security.scram.ScramLoginModule required
> username="admin_scram"
> password="admin_scram_password";
>
> };
> {code}
>
>
> h3. broker server.properties：
> One of the broker configuration files is provided, and the content of the
> configuration files of other brokers is only different from the localPublicIp
> of advertised.listeners.
>
> {code:java}
> broker.id=1
> broker.rack=us-east-1a
> advertised.listeners=SASL_PLAINTEXT://localPublicIp:9779,SASL_SSL://localPublicIp:9889,INTERNAL_SSL://:9009,PLAIN_PLUGIN_SSL://localPublicIp:9669
> log.dirs=/asyncmq/kafka/data_1,/asyncmq/kafka/data_2
> zookeeper.connect=***
> listeners=SASL_PLAINTEXT://:9779,SASL_SSL://:9889,INTERNAL_SSL://:9009,PLAIN_PLUGIN_SSL://:9669
> listener.security.protocol.map=INTERNAL_SSL:SASL_SSL,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL,PLAIN_PLUGIN_SSL:SASL_SSL
> listener.name.plain_plugin_ssl.plain.sasl.server.callback.handler.class=org.apache.kafka.common.security.plain.internals.PlainServerCallbackHandler
> #ssl config
> ssl.keystore.password=***
> ssl.key.password=***
> ssl.truststore.password=***
> ssl.keystore.location=***
> ssl.truststore.location=***
> ssl.client.auth=none
> ssl.endpoint.identification.algorithm=
> #broker communicate config
> #security.inter.broker.protocol=SASL_PLAINTEXT
> inter.broker.listener.name=INTERNAL_SSL
> sasl.mechanism.inter.broker.protocol=PLAIN
> #sasl authentication config
> sasl.kerberos.service.name=kafka
> sasl.enabled.mechanisms=PLAIN,SCRAM-SHA-256,SCRAM-SHA-512,GSSAPI
> delegation.token.master.key=***
> delegation.token.expiry.time.ms=8640
> delegation.token.max.lifetime.ms=31536
> {code}
>
>
> Then start all brokers at the same time. Each broker has actually been
> started successfully, but when establishing a connection between the
> controller node and all brokers, the identity authentication has always
> failed. The connection between brokers cannot be established normally,
> causing the entire Kafka cluster to be unable to provide external services.
> h3. The server log keeps printing abnormally like crazy：
> The real ip sensitive information of the broker in the log, I use **
> instead of here
>
> {code:java}
> [2021-10-29 14:16:19,831] INFO [SocketServer listenerType=ZK_BROKER,
> nodeId=3] Started socket server acceptors and processors
> (kafka.network.SocketServer)
> [2021-10-29 14:16:19,836] INFO Kafka version: 3.0.0
> (org.apache.kafka.common.utils.AppInfoParser)
> [2021-10-29 14:16:19,836] INFO Kafka commitId: 8cb0a5e9d3441962
> (org.apache.kafka.common.utils.AppInfoParser)
> [2021-10-29 14:16:19,836] INFO Kafka startTimeMs: 1635516979831
> (org.apache.kafka.common.utils.AppInfoParser)
> [2021-10-29 14:16:19,837] INFO [KafkaServer id=3] started
> (kafka.server.KafkaServer)
> [2021-10-29 14:16:20,249] INFO [SocketServer listenerType=ZK_BROKER,
> nodeId=3] Failed authentication with /** (Authentication failed: Invalid
> username or password) (org.apache.kaf
[jira] [Commented] (KAFKA-13638) Slow KTable update when forwarding multiple values from transformer
[
https://issues.apache.org/jira/browse/KAFKA-13638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17489741#comment-17489741
]
Bruno Cadonna commented on KAFKA-13638:
---
[~Lejon] Could you try to use another state store directory ({{state.dir}}
config). By default that config points to {{/tmp/kafka-streams}}. Maybe the OS
makes something weird with the temporary directory. Just an idea!
> Slow KTable update when forwarding multiple values from transformer
> ---
>
> Key: KAFKA-13638
> URL: https://issues.apache.org/jira/browse/KAFKA-13638
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.1.0, 3.0.0
>Reporter: Ulrik
>Priority: Major
> Attachments: KafkaTest.java
>
>
> I have a topology where I stream messages from an input topic, transform the
> message to multiple messages (via context.forward), and then store those
> messages in a KTable.
> Since upgrading from kafka-streams 2.8.1 to 3.1.0 I have noticed that my
> tests take significantly longer time to run.
>
> I have attached a test class to demonstrate my scenario. When running this
> test with kafka-streams versions 2.8.1 and 3.1.0 I came up with the following
> numbers:
>
> *Version 2.8.1*
> * one input message and one output message: 541 ms
> * 8 input message and 30 output message per input message (240 output
> messages in total): 919 ms
>
> *Version 3.1.0*
> * one input message and one output message: 908 ms
> * 8 input message and 30 output message per input message (240 output
> messages in total): 6 sec 94 ms
>
> Even when the transformer just transforms and forwards one input message to
> one output message, the test takes approx. 400 ms longer to run.
> When transforming 8 input messages to 240 output messages it takes approx 5
> seconds longer.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (KAFKA-13646) Implement KIP-801: KRaft authorizer
[ https://issues.apache.org/jira/browse/KAFKA-13646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-13646. - Fix Version/s: 3.2.0 Resolution: Fixed > Implement KIP-801: KRaft authorizer > --- > > Key: KAFKA-13646 > URL: https://issues.apache.org/jira/browse/KAFKA-13646 > Project: Kafka > Issue Type: Improvement >Reporter: Colin McCabe >Assignee: Colin McCabe >Priority: Major > Labels: kip-500, kip-801 > Fix For: 3.2.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [kafka] hachikuji merged pull request #11649: KAFKA-13646: Implement KIP-801: KRaft authorizer
hachikuji merged pull request #11649: URL: https://github.com/apache/kafka/pull/11649 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] tombentley commented on a change in pull request #11673: KAFKA-13577: Replace easymock with mockito in kafka:core - part 2
tombentley commented on a change in pull request #11673:
URL: https://github.com/apache/kafka/pull/11673#discussion_r802922650
##
File path: core/src/test/scala/unit/kafka/server/ReplicaManagerQuotasTest.scala
##
@@ -157,10 +158,9 @@ class ReplicaManagerQuotasTest {
def shouldIncludeThrottledReplicasForConsumerFetch(): Unit = {
setUpMocks(fetchInfo)
-val quota = mockQuota(100)
-expect(quota.isQuotaExceeded).andReturn(true).once()
-expect(quota.isQuotaExceeded).andReturn(true).once()
-replay(quota)
+val quota = mockQuota()
+when(quota.isQuotaExceeded).thenReturn(true)
+//expect(quota.isQuotaExceeded).andReturn(true).once()
Review comment:
Can we remove this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[ https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17489723#comment-17489723 ] Guram Savinov commented on KAFKA-7500: -- Please update KIP-382 documentation: LegacyReplicationPolicy -> IdentityReplicationPolicy https://issues.apache.org/jira/browse/KAFKA-9726 > MirrorMaker 2.0 (KIP-382) > - > > Key: KAFKA-7500 > URL: https://issues.apache.org/jira/browse/KAFKA-7500 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect, mirrormaker >Affects Versions: 2.4.0 >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Major > Labels: pull-request-available, ready-to-commit > Fix For: 2.4.0 > > Attachments: Active-Active XDCR setup.png > > > Implement a drop-in replacement for MirrorMaker leveraging the Connect > framework. > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0] > [https://github.com/apache/kafka/pull/6295] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (KAFKA-13661) KRaft uses the wrong permission for creating partitions
Jason Gustafson created KAFKA-13661: --- Summary: KRaft uses the wrong permission for creating partitions Key: KAFKA-13661 URL: https://issues.apache.org/jira/browse/KAFKA-13661 Project: Kafka Issue Type: Bug Affects Versions: 3.0.0, 3.1.0 Reporter: Jason Gustafson Assignee: Jason Gustafson [~cmccabe] caught this as part of KAFKA-13646. KRaft currently checks CREATE on the topic resource. It should be ALTER. This will be fixed in trunk as part of KAFKA-13646, but it would be good to fix for 3.0 and 3.1 as well. Note this does not affect zookeeper-based clusters. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (KAFKA-13661) KRaft uses the wrong permission for adding topic partitions
[ https://issues.apache.org/jira/browse/KAFKA-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson updated KAFKA-13661: Summary: KRaft uses the wrong permission for adding topic partitions (was: KRaft uses the wrong permission for creating partitions) > KRaft uses the wrong permission for adding topic partitions > --- > > Key: KAFKA-13661 > URL: https://issues.apache.org/jira/browse/KAFKA-13661 > Project: Kafka > Issue Type: Bug >Affects Versions: 3.1.0, 3.0.0 >Reporter: Jason Gustafson >Assignee: Jason Gustafson >Priority: Major > > [~cmccabe] caught this as part of KAFKA-13646. KRaft currently checks CREATE > on the topic resource. It should be ALTER. This will be fixed in trunk as > part of KAFKA-13646, but it would be good to fix for 3.0 and 3.1 as well. > Note this does not affect zookeeper-based clusters. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (KAFKA-12468) Initial offsets are copied from source to target cluster
[
https://issues.apache.org/jira/browse/KAFKA-12468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17489716#comment-17489716
]
Guram Savinov commented on KAFKA-12468:
---
[~bdeneuter] there is IdentityReplicationPolicy which can be used to preserve
topic names, maybe you don't need to implement your CustomReplicationPolicy.
https://issues.apache.org/jira/browse/KAFKA-9726
> Initial offsets are copied from source to target cluster
>
>
> Key: KAFKA-12468
> URL: https://issues.apache.org/jira/browse/KAFKA-12468
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 2.7.0
>Reporter: Bart De Neuter
>Priority: Major
>
> We have an active-passive setup where the 3 connectors from mirror maker 2
> (heartbeat, checkpoint and source) are running on a dedicated Kafka connect
> cluster on the target cluster.
> Offset syncing is enabled as specified by KIP-545. But when activated, it
> seems the offsets from the source cluster are initially copied to the target
> cluster without translation. This causes a negative lag for all synced
> consumer groups. Only when we reset the offsets for each topic/partition on
> the target cluster and produce a record on the topic/partition in the source,
> the sync starts working correctly.
> I would expect that the consumer groups are synced but that the current
> offsets of the source cluster are not copied to the target cluster.
> This is the configuration we are currently using:
> Heartbeat connector
>
> {code:xml}
> {
> "name": "mm2-mirror-heartbeat",
> "config": {
> "name": "mm2-mirror-heartbeat",
> "connector.class":
> "org.apache.kafka.connect.mirror.MirrorHeartbeatConnector",
> "source.cluster.alias": "eventador",
> "target.cluster.alias": "msk",
> "source.cluster.bootstrap.servers": "",
> "target.cluster.bootstrap.servers": "",
> "topics": ".*",
> "groups": ".*",
> "tasks.max": "1",
> "replication.policy.class": "CustomReplicationPolicy",
> "sync.group.offsets.enabled": "true",
> "sync.group.offsets.interval.seconds": "5",
> "emit.checkpoints.enabled": "true",
> "emit.checkpoints.interval.seconds": "30",
> "emit.heartbeats.interval.seconds": "30",
> "key.converter": "
> org.apache.kafka.connect.converters.ByteArrayConverter",
> "value.converter":
> "org.apache.kafka.connect.converters.ByteArrayConverter"
> }
> }
> {code}
> Checkpoint connector:
> {code:xml}
> {
> "name": "mm2-mirror-checkpoint",
> "config": {
> "name": "mm2-mirror-checkpoint",
> "connector.class":
> "org.apache.kafka.connect.mirror.MirrorCheckpointConnector",
> "source.cluster.alias": "eventador",
> "target.cluster.alias": "msk",
> "source.cluster.bootstrap.servers": "",
> "target.cluster.bootstrap.servers": "",
> "topics": ".*",
> "groups": ".*",
> "tasks.max": "40",
> "replication.policy.class": "CustomReplicationPolicy",
> "sync.group.offsets.enabled": "true",
> "sync.group.offsets.interval.seconds": "5",
> "emit.checkpoints.enabled": "true",
> "emit.checkpoints.interval.seconds": "30",
> "emit.heartbeats.interval.seconds": "30",
> "key.converter": "
> org.apache.kafka.connect.converters.ByteArrayConverter",
> "value.converter":
> "org.apache.kafka.connect.converters.ByteArrayConverter"
> }
> }
> {code}
> Source connector:
> {code:xml}
> {
> "name": "mm2-mirror-source",
> "config": {
> "name": "mm2-mirror-source",
> "connector.class":
> "org.apache.kafka.connect.mirror.MirrorSourceConnector",
> "source.cluster.alias": "eventador",
> "target.cluster.alias": "msk",
> "source.cluster.bootstrap.servers": "",
> "target.cluster.bootstrap.servers": "",
> "topics": ".*",
> "groups": ".*",
> "tasks.max": "40",
> "replication.policy.class": "CustomReplicationPolicy",
> "sync.group.offsets.enabled": "true",
> "sync.group.offsets.interval.seconds": "5",
> "emit.checkpoints.enabled": "true",
> "emit.checkpoints.interval.seconds": "30",
> "emit.heartbeats.interval.seconds": "30",
> "key.converter": "
> org.apache.kafka.connect.converters.ByteArrayConverter",
> "value.converter":
> "org.apache.kafka.connect.converters.ByteArrayConverter"
> }
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (KAFKA-13517) Add ConfigurationKeys to ConfigResource class
[
https://issues.apache.org/jira/browse/KAFKA-13517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mickael Maison updated KAFKA-13517:
---
Fix Version/s: (was: 2.8.1)
> Add ConfigurationKeys to ConfigResource class
> -
>
> Key: KAFKA-13517
> URL: https://issues.apache.org/jira/browse/KAFKA-13517
> Project: Kafka
> Issue Type: Improvement
> Components: clients
>Affects Versions: 2.8.1, 3.0.0
>Reporter: Vikas Singh
>Assignee: Vikas Singh
>Priority: Major
>
> A list of {{ConfigResource}} class is passed as argument to
> {{AdminClient::describeConfigs}} api to indicate configuration of the
> entities to fetch. The {{ConfigResource}} class is made up of two fields,
> name and type of entity. Kafka returns *all* configurations for the entities
> provided to the admin client api.
> This admin api in turn uses {{DescribeConfigsRequest}} kafka api to get the
> configuration for the entities in question. In addition to name and type of
> entity whose configuration to get, Kafka {{DescribeConfigsResource}}
> structure also lets users provide {{ConfigurationKeys}} list, which allows
> users to fetch only the configurations that are needed.
> However, this field isn't exposed in the {{ConfigResource}} class that is
> used by AdminClient, so users of AdminClient have no way to ask for specific
> configuration. The API always returns *all* configurations. Then the user of
> the {{AdminClient::describeConfigs}} go over the returned list and filter out
> the config keys that they are interested in.
> This results in boilerplate code for all users of
> {{AdminClient::describeConfigs}} api, in addition to being wasteful use of
> resource. It becomes painful in large cluster case where to fetch one
> configuration of all topics, we need to fetch all configuration of all
> topics, which can be huge in size.
> Creating this Jira to add same field (i.e. {{{}ConfigurationKeys{}}}) to the
> {{ConfigResource}} structure to bring it to parity to
> {{DescribeConfigsResource}} Kafka API structure. There should be no backward
> compatibility issue as the field will be optional and will behave same way if
> it is not specified (i.e. by passing null to backend kafka api)
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (KAFKA-13242) KRaft Controller doesn't handle UpdateFeaturesRequest
[ https://issues.apache.org/jira/browse/KAFKA-13242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mickael Maison updated KAFKA-13242: --- Fix Version/s: (was: 3.0.1) > KRaft Controller doesn't handle UpdateFeaturesRequest > - > > Key: KAFKA-13242 > URL: https://issues.apache.org/jira/browse/KAFKA-13242 > Project: Kafka > Issue Type: Sub-task >Reporter: dengziming >Assignee: dengziming >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (KAFKA-13228) ApiVersionRequest are not correctly handled in kraft mode
[ https://issues.apache.org/jira/browse/KAFKA-13228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mickael Maison updated KAFKA-13228: --- Fix Version/s: (was: 3.0.1) > ApiVersionRequest are not correctly handled in kraft mode > - > > Key: KAFKA-13228 > URL: https://issues.apache.org/jira/browse/KAFKA-13228 > Project: Kafka > Issue Type: Bug >Reporter: dengziming >Assignee: dengziming >Priority: Major > > I'am trying to describe quorum in kraft mode but got > `org.apache.kafka.common.errors.UnsupportedVersionException: The broker does > not support DESCRIBE_QUORUM`. > This happens because we only concerns `ApiKeys.zkBrokerApis()` when we call > `NodeApiVersions.create()` -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (KAFKA-13188) Release the memory back into MemoryPool
[ https://issues.apache.org/jira/browse/KAFKA-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mickael Maison updated KAFKA-13188: --- Fix Version/s: (was: 3.0.1) > Release the memory back into MemoryPool > --- > > Key: KAFKA-13188 > URL: https://issues.apache.org/jira/browse/KAFKA-13188 > Project: Kafka > Issue Type: Improvement >Reporter: Lucas Wang >Assignee: Alok Nikhil >Priority: Major > > Tushar made a [hotfix change|https://github.com/linkedin/kafka/pull/186] to > the linkedin/kafka repo hosting apache kafka 2.4. > The change is about releasing memory back to the MemoryPool for the kafka > consumer, and his benchmark showed significant improvement in terms of the > memory graduating from Young Gen and promoted to Old Gen. > Given the benefit, the change should also be added trunk. -- This message was sent by Atlassian Jira (v8.20.1#820001)
