[jira] [Created] (KAFKA-17094) Make it possible to list registered KRaft nodes in order to know which nodes should be unregistered
Jakub Scholz created KAFKA-17094:
Summary: Make it possible to list registered KRaft nodes in order
to know which nodes should be unregistered
Key: KAFKA-17094
URL: https://issues.apache.org/jira/browse/KAFKA-17094
Project: Kafka
Issue Type: Bug
Affects Versions: 3.7.1
Reporter: Jakub Scholz
Kafka seems to require nodes that are removed from the cluster to be
unregistered using the Kafka Admin API. If they are unregistred, that you might
run into problems later. For example, after upgrade when you try to bump the
KRaft metadata version, you might get an error like this:
{code:java}
g.apache.kafka.common.errors.InvalidUpdateVersionException: Invalid update
version 19 for feature metadata.version. Broker 3002 only supports versions
1-14 {code}
In this case, 3002 is an old node that was removed before the upgrade and
doesn't support the KRaft metadata version 19 and blocks the metadata update.
However, it seems to be impossible to list the registered nodes in order to
unregister them:
* The describe cluster metadata request in the Admin API seems to return only
the IDs of running brokers
* The describe metadata quorum command seems to list the removed nodes in the
list of observers. But it does so only until the controller nodes are restarted.
If Kafka expects the inactive nodes to be registered, it should provide a list
of the registered nodes so that it can be checked what nodes to unregister.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-16606) JBOD support in KRaft does not seem to be gated by the metadata version
[
https://issues.apache.org/jira/browse/KAFKA-16606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakub Scholz resolved KAFKA-16606.
--
Resolution: Not A Problem
> JBOD support in KRaft does not seem to be gated by the metadata version
> ---
>
> Key: KAFKA-16606
> URL: https://issues.apache.org/jira/browse/KAFKA-16606
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 3.7.0
>Reporter: Jakub Scholz
>Priority: Major
>
> JBOD support in KRaft should be supported since Kafka 3.7.0. The Kafka
> [source
> code|https://github.com/apache/kafka/blob/1b301b30207ed8fca9f0aea5cf940b0353a1abca/server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java#L194-L195]
> suggests that it is supported with the metadata version {{{}3.7-IV2{}}}.
> However, it seems to be possible to run KRaft cluster with JBOD even with
> older metadata versions such as {{{}3.6{}}}. For example, I have a cluster
> using the {{3.6}} metadata version:
> {code:java}
> bin/kafka-features.sh --bootstrap-server localhost:9092 describe
> Feature: metadata.version SupportedMinVersion: 3.0-IV1
> SupportedMaxVersion: 3.7-IV4 FinalizedVersionLevel: 3.6-IV2 Epoch: 1375
> {code}
> Yet a KRaft cluster with JBOD seems to run fine:
> {code:java}
> bin/kafka-log-dirs.sh --bootstrap-server localhost:9092 --describe
> Querying brokers for log directories information
> Received log directory information from brokers 2000,3000,1000
>
[jira] [Created] (KAFKA-16606) JBOD support in KRaft does not seem to be gated by the metadata version
Jakub Scholz created KAFKA-16606:
Summary: JBOD support in KRaft does not seem to be gated by the
metadata version
Key: KAFKA-16606
URL: https://issues.apache.org/jira/browse/KAFKA-16606
Project: Kafka
Issue Type: Bug
Affects Versions: 3.7.0
Reporter: Jakub Scholz
JBOD support in KRaft should be supported since Kafka 3.7.0. The Kafka [source
code|https://github.com/apache/kafka/blob/1b301b30207ed8fca9f0aea5cf940b0353a1abca/server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java#L194-L195]
suggests that it is supported with the metadata version {{{}3.7-IV2{}}}.
However, it seems to be possible to run KRaft cluster with JBOD even with older
metadata versions such as {{{}3.6{}}}. For example, I have a cluster using the
{{3.6}} metadata version:
{code:java}
bin/kafka-features.sh --bootstrap-server localhost:9092 describe
Feature: metadata.version SupportedMinVersion: 3.0-IV1
SupportedMaxVersion: 3.7-IV4 FinalizedVersionLevel: 3.6-IV2 Epoch: 1375
{code}
Yet a KRaft cluster with JBOD seems to run fine:
{code:java}
bin/kafka-log-dirs.sh --bootstrap-server localhost:9092 --describe
Querying brokers for log directories information
Received log directory information from brokers 2000,3000,1000
[jira] [Created] (KAFKA-16131) Repeated UnsupportedVersionException logged when running Kafka 3.7.0-RC2 KRaft cluster with metadata version 3.6
Jakub Scholz created KAFKA-16131:
Summary: Repeated UnsupportedVersionException logged when running
Kafka 3.7.0-RC2 KRaft cluster with metadata version 3.6
Key: KAFKA-16131
URL: https://issues.apache.org/jira/browse/KAFKA-16131
Project: Kafka
Issue Type: Bug
Reporter: Jakub Scholz
Fix For: 3.7.0
When running Kafka 3.7.0-RC2 as a KRaft cluster with metadata version set to
3.6-IV2 metadata version, it throws repeated errors like this in the controller
logs:
{quote}2024-01-13 16:58:01,197 INFO [QuorumController id=0]
assignReplicasToDirs: event failed with UnsupportedVersionException in 15
microseconds. (org.apache.kafka.controller.QuorumController)
[quorum-controller-0-event-handler]
2024-01-13 16:58:01,197 ERROR [ControllerApis nodeId=0] Unexpected error
handling request RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS, apiVersion=0,
clientId=1000, correlationId=14, headerVersion=2) –
AssignReplicasToDirsRequestData(brokerId=1000, brokerEpoch=5,
directories=[DirectoryData(id=w_uxN7pwQ6eXSMrOKceYIQ,
topics=[TopicData(topicId=bvAKLSwmR7iJoKv2yZgygQ,
partitions=[PartitionData(partitionIndex=2), PartitionData(partitionIndex=1)]),
TopicData(topicId=uNe7f5VrQgO0zST6yH1jDQ,
partitions=[PartitionData(partitionIndex=0)])])]) with context
RequestContext(header=RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
apiVersion=0, clientId=1000, correlationId=14, headerVersion=2),
connectionId='172.16.14.219:9090-172.16.14.217:53590-7',
clientAddress=/[172.16.14.217|http://172.16.14.217/],
principal=User:CN=my-cluster-kafka,O=io.strimzi,
listenerName=ListenerName(CONTROLPLANE-9090), securityProtocol=SSL,
clientInformation=ClientInformation(softwareName=apache-kafka-java,
softwareVersion=3.7.0), fromPrivilegedListener=false,
principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@71004ad2])
(kafka.server.ControllerApis) [quorum-controller-0-event-handler]
java.util.concurrent.CompletionException:
org.apache.kafka.common.errors.UnsupportedVersionException: Directory
assignment is not supported yet.
at
java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
at
java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
at
java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at
java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
at
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:880)
at
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:871)
at
org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
at
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
at
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
at
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.apache.kafka.common.errors.UnsupportedVersionException:
Directory assignment is not supported yet.
{quote}
With the metadata version set to 3.6-IV2, it makes sense that the request is
not supported. But the request should in such case not be sent at all.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-16075) TLS configuration not validated in KRaft controller-only nodes
Jakub Scholz created KAFKA-16075:
Summary: TLS configuration not validated in KRaft controller-only
nodes
Key: KAFKA-16075
URL: https://issues.apache.org/jira/browse/KAFKA-16075
Project: Kafka
Issue Type: Bug
Components: kraft
Affects Versions: 3.6.1
Reporter: Jakub Scholz
When the Kafka broker node (either a broker in ZooKeeper based cluster or node
with a broker role in a KRaft cluster) has an incorrect TLS configuration such
as unsupported TLS cipher suite, it seems to throw a {{ConfigException}} and
shutdown:
{code:java}
2024-01-02 13:50:24,895 ERROR Exiting Kafka due to fatal exception during
startup. (kafka.Kafka$) [main]
org.apache.kafka.common.config.ConfigException: Invalid value
java.lang.IllegalArgumentException: Unsupported CipherSuite:
TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305 for configuration A client SSLEngine
created with the provided settings can't connect to a server SSLEngine created
with those settings.
at
org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:102)
at
org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:73)
at
org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:192)
at
org.apache.kafka.common.network.ChannelBuilders.serverChannelBuilder(ChannelBuilders.java:107)
at kafka.network.Processor.(SocketServer.scala:973)
at kafka.network.Acceptor.newProcessor(SocketServer.scala:879)
at
kafka.network.Acceptor.$anonfun$addProcessors$1(SocketServer.scala:849)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:190)
at kafka.network.Acceptor.addProcessors(SocketServer.scala:848)
at kafka.network.DataPlaneAcceptor.configure(SocketServer.scala:523)
at
kafka.network.SocketServer.createDataPlaneAcceptorAndProcessors(SocketServer.scala:251)
at kafka.network.SocketServer.$anonfun$new$31(SocketServer.scala:175)
at
kafka.network.SocketServer.$anonfun$new$31$adapted(SocketServer.scala:175)
at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:576)
at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:574)
at scala.collection.AbstractIterable.foreach(Iterable.scala:933)
at kafka.network.SocketServer.(SocketServer.scala:175)
at kafka.server.BrokerServer.startup(BrokerServer.scala:242)
at
kafka.server.KafkaRaftServer.$anonfun$startup$2(KafkaRaftServer.scala:96)
at
kafka.server.KafkaRaftServer.$anonfun$startup$2$adapted(KafkaRaftServer.scala:96)
at scala.Option.foreach(Option.scala:437)
at kafka.server.KafkaRaftServer.startup(KafkaRaftServer.scala:96)
at kafka.Kafka$.main(Kafka.scala:113)
at kafka.Kafka.main(Kafka.scala) {code}
But in a KRaft controller-only nodes, such validation does not seem to happen
and the broker keeps running and looping with this warning:
{code:java}
2024-01-02 13:53:10,186 WARN [RaftManager id=1] Error connecting to node
my-cluster-controllers-0.my-cluster-kafka-brokers.myproject.svc.cluster.local:9090
(id: 0 rack: null) (org.apache.kafka.clients.NetworkClient)
[kafka-1-raft-outbound-request-thread]
java.io.IOException: Channel could not be created for socket
java.nio.channels.SocketChannel[closed]
at
org.apache.kafka.common.network.Selector.buildAndAttachKafkaChannel(Selector.java:348)
at
org.apache.kafka.common.network.Selector.registerChannel(Selector.java:329)
at org.apache.kafka.common.network.Selector.connect(Selector.java:256)
at
org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:1032)
at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:301)
at
org.apache.kafka.server.util.InterBrokerSendThread.sendRequests(InterBrokerSendThread.java:145)
at
org.apache.kafka.server.util.InterBrokerSendThread.pollOnce(InterBrokerSendThread.java:108)
at
org.apache.kafka.server.util.InterBrokerSendThread.doWork(InterBrokerSendThread.java:136)
at
org.apache.kafka.server.util.ShutdownableThread.run(ShutdownableThread.java:130)
Caused by: org.apache.kafka.common.KafkaException:
java.lang.IllegalArgumentException: Unsupported CipherSuite:
TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
at
org.apache.kafka.common.network.SslChannelBuilder.buildChannel(SslChannelBuilder.java:111)
at
org.apache.kafka.common.network.Selector.buildAndAttachKafkaChannel(Selector.java:338)
... 8 more
Caused by: java.lang.IllegalArgumentException: Unsupported CipherSuite:
TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
at
java.base/sun.security.ssl.CipherSuite.validValuesOf(CipherSuite.java:978)
at
java.base/sun.security.ssl.SSLEngineImpl.setEnabledCipherSuites(SSLEngineImpl.java:864)
at
[jira] [Created] (KAFKA-15949) Improve the KRaft metadata version related messages
Jakub Scholz created KAFKA-15949:
Summary: Improve the KRaft metadata version related messages
Key: KAFKA-15949
URL: https://issues.apache.org/jira/browse/KAFKA-15949
Project: Kafka
Issue Type: Improvement
Affects Versions: 3.6.0
Reporter: Jakub Scholz
Various error messages related to KRaft seem to use very different style and
formatting. Just for example in the {{StorageTool}} Scala class, there are two
different examples:
* {{Must specify a valid KRaft metadata version of at least 3.0.}}
** Refers to "metadata version"
** Refers to the version as 3.0 (although strictly speaking 3.0-IV0 is not
valid for KRaft)
* {{SCRAM is only supported in metadataVersion IBP_3_5_IV2 or later.}}
** Talks about "metadataVersion"
** Refers to "IBP_3_5_IV2" instead of "3.5" or "3.5-IV2"
Other pieces of Kafka code seem to also talk about "metadata.version" for
example.
For users, it would be nice if the style and formats used were the same
everywhere. Would it be worth unifying messages like this? If yes, what would
be the preferred style to use?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-15464) Allow dynamic reloading of certificates with different DN / SANs
Jakub Scholz created KAFKA-15464: Summary: Allow dynamic reloading of certificates with different DN / SANs Key: KAFKA-15464 URL: https://issues.apache.org/jira/browse/KAFKA-15464 Project: Kafka Issue Type: Improvement Reporter: Jakub Scholz Kafka currently doesn't allow dynamic reloading of keystores when the new key has a different DN or removes some of the SANs. While it might help to prevent users from breaking their cluster, in some cases it would be great to be able to bypass this validation when desired. More details are in the [KIP-978: Allow dynamic reloading of certificates with different DN / SANs|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263429128] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-14941) Document which configuration options are applicable only to processes with broker role or controller role
Jakub Scholz created KAFKA-14941:
Summary: Document which configuration options are applicable only
to processes with broker role or controller role
Key: KAFKA-14941
URL: https://issues.apache.org/jira/browse/KAFKA-14941
Project: Kafka
Issue Type: Improvement
Reporter: Jakub Scholz
When running in KRaft mode, some of the configuration options are applicable
only to nodes with the broker process role and some are applicable only to the
nodes with the controller process roles. It would be great if this information
was part of the documentation (e.g. in the [Broker
Configs|https://kafka.apache.org/documentation/#brokerconfigs] table on the
website), but if it was also part of the config classes so that it can be used
in situations when the configuration is dynamically configured to for example
filter the options applicable to different nodes. This would allow having
configuration files with only the actually used configuration options and for
example, help to reduce unnecessary restarts when rolling out new
configurations etc.
For some options, it seems clear and the Kafka node would refuse to start if
they are set - for example the configurations of the non-controler-listeners in
controller-only nodes. For others, it seems a bit less clear (Does
{{compression.type}} option apply to controller-only nodes? Or the
configurations for the offset topic? etc.).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-14357) Make it possible to batch describe requests in the Kafka Admin API
Jakub Scholz created KAFKA-14357:
Summary: Make it possible to batch describe requests in the Kafka
Admin API
Key: KAFKA-14357
URL: https://issues.apache.org/jira/browse/KAFKA-14357
Project: Kafka
Issue Type: Improvement
Reporter: Jakub Scholz
The Admin API has several methods to describe different objects such as ACLs,
Quotas or SCRAM-SHA users. But these API seem to be usable only in one for the
two modes:
* Query or one users ACLs / Quotas / SCRAM-SHA credentials
* Query all existing ACLs / Quotas / SCRAM-SHA credentials
But there seems to be no way how to batch the describe requests for multiple
users. E.g. {_}describe ACLs of users Joe, John and Mike{_}. It would be nice
to have such option as it might make it easier for applications using the Admin
API to make less different API calls.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-14356) Make it possible to detect changes to SCRAM-SHA credentials using the Admin API
Jakub Scholz created KAFKA-14356: Summary: Make it possible to detect changes to SCRAM-SHA credentials using the Admin API Key: KAFKA-14356 URL: https://issues.apache.org/jira/browse/KAFKA-14356 Project: Kafka Issue Type: Improvement Reporter: Jakub Scholz When using the Kafka Admin API to manage SCRAM-SHA credentials, the API seems to offer only three options: * Find out if given user has any credentials * Set SCRAM-SHA credentials * Delete SCRAM-SHA credentials There is now way how to find out what the current credentials are. That makes sense as that can lead to the credentials being leaked which would be a security issue. However, there is also no way how to find out if the credentials changed since last time. So if you have an external tool which is managing the SCRAM-SHA credentials based on some desired state in a controller loop (such as for example a Kubernetes Operator would do), there is no way to know whether you need to update the password in Kafka or not. And as a result, you always have to update the credentials. It would be great to have some mechanism to detect if the credentials changes since last time. E.g.: * Timestamp of the last change * Some random hash assigned during each change of the credentials which can be compared before updating the credentials * Or possibly some offset of the KRaft metadata log where the credentials are stored. An application managing the passwords, would get the indicator as a response to the the call updating the password and can store it. And in the next loop, it could describe the credentials which would return the latest indicator, compare it with what it stored and if they would be equal, it would know that it does not need to update the credentials. If providing such indicator as part of the describe request would not be considered secure, maybe at least there can be some kind of conditional update call. Where the tool managing the passwords would get the change indicator as response to the update call. And will pass it in the subsequent update calls and the broker will evaluate it server-side if it changed or not and if it should be updated or ignored. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-13937) StandardAuthorizer throws "ID 5t1jQ3zWSfeVLMYkN3uong not found in aclsById" exceptions into broker logs
Jakub Scholz created KAFKA-13937:
Summary: StandardAuthorizer throws "ID 5t1jQ3zWSfeVLMYkN3uong not
found in aclsById" exceptions into broker logs
Key: KAFKA-13937
URL: https://issues.apache.org/jira/browse/KAFKA-13937
Project: Kafka
Issue Type: Bug
Affects Versions: 3.2.0
Reporter: Jakub Scholz
I'm trying to use the new {{StandardAuthorizer}} in a Kafka cluster running in
KRaft mode. When managing the ACLs using the Admin API, the authorizer seems to
throw a lot of runtime exceptions in the log. For example ...
When creating an ACL rule, it seems to create it just fine. But it throws the
following exception:
{code:java}
2022-05-25 11:09:18,074 ERROR [StandardAuthorizer 0] addAcl error
(org.apache.kafka.metadata.authorizer.StandardAuthorizerData) [EventHandler]
java.lang.RuntimeException: An ACL with ID 5t1jQ3zWSfeVLMYkN3uong already
exists.
at
org.apache.kafka.metadata.authorizer.StandardAuthorizerData.addAcl(StandardAuthorizerData.java:169)
at
org.apache.kafka.metadata.authorizer.StandardAuthorizer.addAcl(StandardAuthorizer.java:83)
at
kafka.server.metadata.BrokerMetadataPublisher.$anonfun$publish$19(BrokerMetadataPublisher.scala:234)
at
java.base/java.util.LinkedHashMap$LinkedEntrySet.forEach(LinkedHashMap.java:671)
at
kafka.server.metadata.BrokerMetadataPublisher.$anonfun$publish$18(BrokerMetadataPublisher.scala:232)
at
kafka.server.metadata.BrokerMetadataPublisher.$anonfun$publish$18$adapted(BrokerMetadataPublisher.scala:221)
at scala.Option.foreach(Option.scala:437)
at
kafka.server.metadata.BrokerMetadataPublisher.publish(BrokerMetadataPublisher.scala:221)
at
kafka.server.metadata.BrokerMetadataListener.kafka$server$metadata$BrokerMetadataListener$$publish(BrokerMetadataListener.scala:258)
at
kafka.server.metadata.BrokerMetadataListener$HandleCommitsEvent.$anonfun$run$2(BrokerMetadataListener.scala:119)
at
kafka.server.metadata.BrokerMetadataListener$HandleCommitsEvent.$anonfun$run$2$adapted(BrokerMetadataListener.scala:119)
at scala.Option.foreach(Option.scala:437)
at
kafka.server.metadata.BrokerMetadataListener$HandleCommitsEvent.run(BrokerMetadataListener.scala:119)
at
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:121)
at
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:200)
at
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:173)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-05-25 11:09:18,076 ERROR [BrokerMetadataPublisher id=0] Error publishing
broker metadata at OffsetAndEpoch(offset=3, epoch=1)
(kafka.server.metadata.BrokerMetadataPublisher) [EventHandler]
java.lang.RuntimeException: An ACL with ID 5t1jQ3zWSfeVLMYkN3uong already
exists.
at
org.apache.kafka.metadata.authorizer.StandardAuthorizerData.addAcl(StandardAuthorizerData.java:169)
at
org.apache.kafka.metadata.authorizer.StandardAuthorizer.addAcl(StandardAuthorizer.java:83)
at
kafka.server.metadata.BrokerMetadataPublisher.$anonfun$publish$19(BrokerMetadataPublisher.scala:234)
at
java.base/java.util.LinkedHashMap$LinkedEntrySet.forEach(LinkedHashMap.java:671)
at
kafka.server.metadata.BrokerMetadataPublisher.$anonfun$publish$18(BrokerMetadataPublisher.scala:232)
at
kafka.server.metadata.BrokerMetadataPublisher.$anonfun$publish$18$adapted(BrokerMetadataPublisher.scala:221)
at scala.Option.foreach(Option.scala:437)
at
kafka.server.metadata.BrokerMetadataPublisher.publish(BrokerMetadataPublisher.scala:221)
at
kafka.server.metadata.BrokerMetadataListener.kafka$server$metadata$BrokerMetadataListener$$publish(BrokerMetadataListener.scala:258)
at
kafka.server.metadata.BrokerMetadataListener$HandleCommitsEvent.$anonfun$run$2(BrokerMetadataListener.scala:119)
at
kafka.server.metadata.BrokerMetadataListener$HandleCommitsEvent.$anonfun$run$2$adapted(BrokerMetadataListener.scala:119)
at scala.Option.foreach(Option.scala:437)
at
kafka.server.metadata.BrokerMetadataListener$HandleCommitsEvent.run(BrokerMetadataListener.scala:119)
at
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:121)
at
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:200)
at
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:173)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-05-25 11:09:18,077 ERROR [BrokerMetadataListener id=0] Unexpected error
handling HandleCommitsEvent (kafka.server.metadata.BrokerMetadataListener)
[EventHandler]
java.lang.RuntimeException: An ACL with ID 5t1jQ3zWSfeVLMYkN3uong already
exists.
at
org.apache.kafka.metadata.authorizer.StandardAuthorizerData.addAcl(StandardAuthorizerData.java:169)
at
[jira] [Created] (KAFKA-13839) Example connectors Maven artifact should use provided dependencies
Jakub Scholz created KAFKA-13839:
Summary: Example connectors Maven artifact should use provided
dependencies
Key: KAFKA-13839
URL: https://issues.apache.org/jira/browse/KAFKA-13839
Project: Kafka
Issue Type: Bug
Reporter: Jakub Scholz
The {{connect-file}} artifact which contains the sample
{{FileStreamSourceConnector}} and {{FileStreamSourceConnector}} connectors has
currently 2 Maven dependencies:
* {{connect-api}}
* {{slf4j}}
Both are marked as runtime dependencies. So when the connectors are pulled from
Maven repository to be added to the plugin path, they pull with them also these
dependencies and other transitive dependencies (such as {{{}kafka-clients{}}}).
This seems unnecessary since all these dependencies are already in the
classpath of Kafka itself and do not need to be there again in the plugin path.
They should be configured as {{provided}} so that they are not downloaded with
the connectors.
This is now more relevant after
https://issues.apache.org/jira/browse/KAFKA-13748, when the connectors are not
anymore on the main classpath and downloading them from Maven to the plugin
path would be more common.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Created] (KAFKA-9121) Mirror Maker 2.0 doesn't handle the topic names in consumer checkpoints properly when topic name contain separator
Jakub Scholz created KAFKA-9121:
---
Summary: Mirror Maker 2.0 doesn't handle the topic names in
consumer checkpoints properly when topic name contain separator
Key: KAFKA-9121
URL: https://issues.apache.org/jira/browse/KAFKA-9121
Project: Kafka
Issue Type: Bug
Components: mirrormaker
Affects Versions: 2.4.0
Reporter: Jakub Scholz
I was trying the Kafka Mirror Maker 2.0 and run into the following situation:
1) I have 2 Kafka clusters with topic {{kafka-test-apps}} topic
2) I configured Mirror Maker with {{replication.policy.separator=-}} and with
mirroring between cluster {{a}} and {{b}}.
3) When running Mirror Maker the mirroring of topics works fine. But when I use
the {{RemoteClusterUtils}} to recover the offsets, the names of the topics for
which the offsets are found are {{a-kafka-test-apps}} and {{apps}}. While the
expected topic names would be {{a-kafka-test-apps}} and {{kafka-test-apps}}.
I tried to find the issue, but didn't found it so far. But it doesn't seem to
be in {{RemoteClusterUtils}} because the topic names seem to be wrong already
in {{checkpoints.internal}} topic. So it is probably already processed in the
wrong way in the source cluster.
When I use {{.}} as the separator, it seems to work fine for me. It looks like
the problem is only when the topci names contain already the separator in the
original topic name. But using the right separator might not be a solution for
this, because you migth have topics with different characters and always have
this problem.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-8053) kafka-topics.sh gives confusing error message when the topic doesn't exist
Jakub Scholz created KAFKA-8053:
---
Summary: kafka-topics.sh gives confusing error message when the
topic doesn't exist
Key: KAFKA-8053
URL: https://issues.apache.org/jira/browse/KAFKA-8053
Project: Kafka
Issue Type: Bug
Reporter: Jakub Scholz
The kafka-topics.sh utility gives a confusing message when the topic it is
called with doesn't exist or when no topics exist at all:
{code}
bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic xxx
Error while executing topic command : Topics in [] does not exist
[2019-03-06 13:26:33,982] ERROR java.lang.IllegalArgumentException: Topics in
[] does not exist
at
kafka.admin.TopicCommand$.kafka$admin$TopicCommand$$ensureTopicExists(TopicCommand.scala:416)
at
kafka.admin.TopicCommand$ZookeeperTopicService.describeTopic(TopicCommand.scala:332)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:66)
at kafka.admin.TopicCommand.main(TopicCommand.scala)
(kafka.admin.TopicCommand$)
{code}
It tries to list the topics, but because list of topics is always empty, it
always prints just `[]`. The error message should be more useful and instead
list the topic passed by the user as the parameter or not try to list anything
at all.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KAFKA-2544) Replication tools wiki page needs to be updated
[ https://issues.apache.org/jira/browse/KAFKA-2544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakub Scholz resolved KAFKA-2544. - Resolution: Fixed Assignee: Jakub Scholz This issue has been resolved. > Replication tools wiki page needs to be updated > --- > > Key: KAFKA-2544 > URL: https://issues.apache.org/jira/browse/KAFKA-2544 > Project: Kafka > Issue Type: Improvement > Components: website >Affects Versions: 0.8.2.1 >Reporter: Stevo Slavic >Assignee: Jakub Scholz >Priority: Minor > Labels: documentation, newbie > > https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools is > outdated, mentions tools which have been heavily refactored or replaced by > other tools, e.g. add partition tool, list/create topics tools, etc. > Please have the replication tools wiki page updated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-6008) Kafka Connect: Unsanitized workerID causes exception during startup
Jakub Scholz created KAFKA-6008:
---
Summary: Kafka Connect: Unsanitized workerID causes exception
during startup
Key: KAFKA-6008
URL: https://issues.apache.org/jira/browse/KAFKA-6008
Project: Kafka
Issue Type: Bug
Components: KafkaConnect
Affects Versions: 1.0.0
Environment: MacOS, Java 1.8.0_77-b03
Reporter: Jakub Scholz
Fix For: 1.0.0
When KAfka Connect starts, it seems to use unsanitized workerId for creating
Metrics. As a result it throws following exception:
{code}
[2017-10-04 13:16:08,886] WARN Error registering AppInfo mbean
(org.apache.kafka.common.utils.AppInfoParser:66)
javax.management.MalformedObjectNameException: Invalid character ':' in value
part of property
at javax.management.ObjectName.construct(ObjectName.java:618)
at javax.management.ObjectName.(ObjectName.java:1382)
at
org.apache.kafka.common.utils.AppInfoParser.registerAppInfo(AppInfoParser.java:60)
at
org.apache.kafka.connect.runtime.ConnectMetrics.(ConnectMetrics.java:77)
at org.apache.kafka.connect.runtime.Worker.(Worker.java:88)
at
org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:81)
{code}
It looks like in my case the generated workerId is :. The
workerId should be sanitized before creating the metric.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (KAFKA-5969) bin/kafka-preferred-replica-election.sh gives misleading error when invalid JSON file is passed as parameter
Jakub Scholz created KAFKA-5969:
---
Summary: bin/kafka-preferred-replica-election.sh gives misleading
error when invalid JSON file is passed as parameter
Key: KAFKA-5969
URL: https://issues.apache.org/jira/browse/KAFKA-5969
Project: Kafka
Issue Type: Bug
Components: tools
Reporter: Jakub Scholz
When invalid JSON file is passed to the bin/kafka-preferred-replica-election.sh
/ PreferredReplicaLeaderElectionCommand tool it gives a misleading error:
{code}
kafka.admin.AdminOperationException: Preferred replica election data is empty
at
kafka.admin.PreferredReplicaLeaderElectionCommand$.parsePreferredReplicaElectionData(PreferredReplicaLeaderElectionCommand.scala:97)
at
kafka.admin.PreferredReplicaLeaderElectionCommand$.main(PreferredReplicaLeaderElectionCommand.scala:66)
at
kafka.admin.PreferredReplicaLeaderElectionCommand.main(PreferredReplicaLeaderElectionCommand.scala)
{code}
It suggests that the data is empty instead of invalid. This can confuse people.
The exception text should be fixed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (KAFKA-5940) kafka-delete-records.sh doesn't give any feedback when the JSON offset configuration file is invalid
Jakub Scholz created KAFKA-5940:
---
Summary: kafka-delete-records.sh doesn't give any feedback when
the JSON offset configuration file is invalid
Key: KAFKA-5940
URL: https://issues.apache.org/jira/browse/KAFKA-5940
Project: Kafka
Issue Type: Improvement
Components: tools
Reporter: Jakub Scholz
Assignee: Jakub Scholz
When deleting records using {{bin/kafka-delete-records.sh}}, the user has to
pass a JSON file with the list of topics/partitions and the offset to which the
records should be deleted. However, currently when such file is invalid the
utility doesn't print any visible error:
{code}
$ bin/kafka-delete-records.sh --bootstrap-server localhost:9092
--offset-json-file offset.json
Executing records delete operation
Records delete operation completed:
$
{code}
Instead, I would suggest that it throws an exception to make it clear that the
problem is the invalid JSON file:
{code}
$ bin/kafka-delete-records.sh --bootstrap-server localhost:9092
--offset-json-file offset.json
Exception in thread "main" kafka.common.AdminCommandFailedException: Offset
json file doesn't contain valid JSON data.
at
kafka.admin.DeleteRecordsCommand$.parseOffsetJsonStringWithoutDedup(DeleteRecordsCommand.scala:54)
at
kafka.admin.DeleteRecordsCommand$.execute(DeleteRecordsCommand.scala:62)
at kafka.admin.DeleteRecordsCommand$.main(DeleteRecordsCommand.scala:37)
at kafka.admin.DeleteRecordsCommand.main(DeleteRecordsCommand.scala)
$
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (KAFKA-5933) Move timeout and validate_only protocol fields into CommonFields class
Jakub Scholz created KAFKA-5933: --- Summary: Move timeout and validate_only protocol fields into CommonFields class Key: KAFKA-5933 URL: https://issues.apache.org/jira/browse/KAFKA-5933 Project: Kafka Issue Type: Bug Reporter: Jakub Scholz Most of the fields which are shared by multiple protocol messages (requests ad responses) are in the CommonFields class. However there are still some fields used multiple times which are not there yet: * timeout * validate_only It would be good to move also these two fields into the CommonFields class so that they can be easily shared by different messages. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-5918) Fix minor typos and errors in the Kafka Streams turotial
Jakub Scholz created KAFKA-5918:
---
Summary: Fix minor typos and errors in the Kafka Streams turotial
Key: KAFKA-5918
URL: https://issues.apache.org/jira/browse/KAFKA-5918
Project: Kafka
Issue Type: Bug
Components: documentation
Reporter: Jakub Scholz
Priority: Minor
Fix For: 1.0.0
I found several minor issues with the Kafka Streams tutorial:
* Some typos
** "As shown above, it illustrate that the constructed ..." instead of "As
shown above, it illustrate_s_ that the constructed ..."
** "same as Pipe.java below" instead of "same as Pipe.java _above_"
** Wrong class name in the {{LineSplit}} example
* Incorrect imports for the code examples
** Missing {{import org.apache.kafka.streams.kstream.KStream;}} in
{{LineSplit}} and {{WordCount}} example
* Unnecessary (and potentially confusing) split by whitespaces in the
{{WorkCount}} class (the split into words happened already in {{LineSplit}})
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
