[jira] [Created] (KAFKA-8695) Metrics UnderReplicated and UnderMinSir are diverging when configuration is inconsistent
Alexandre Dupriez created KAFKA-8695: Summary: Metrics UnderReplicated and UnderMinSir are diverging when configuration is inconsistent Key: KAFKA-8695 URL: https://issues.apache.org/jira/browse/KAFKA-8695 Project: Kafka Issue Type: Bug Components: core Affects Versions: 2.3.0, 2.1.1, 2.2.0, 2.1.0 Reporter: Alexandre Dupriez As of now, Kafka allows the replication factor of a topic and "min.insync.replicas" to be set such that "min.insync.replicas" > the topic's replication factor. As a consequences, the JMX beans {code:java} kafka.cluster:type=Partition,name=UnderReplicated{code} and {code:java} kafka.cluster:type=Partition,name=UnderMinIsr{code} can report diverging views on the replication for a topic. The former can report no under replicated partition, while the second will report under in-sync replicas. Even worse, consumption of topics which exhibit this behaviour seems to fail, the Kafka broker throwing a NotEnoughReplicasException. {code:java} [2019-07-22 10:44:29,913] ERROR [ReplicaManager broker=0] Error processing append operation on partition __consumer_offsets-0 (kafka.server.ReplicaManager) org.apache.kafka.common.errors.NotEnoughReplicasException: The size of the current ISR Set(0) is insufficient to satisfy the min.isr requirement of 2 for partition __consumer_offsets-0{code} In order to avoid this scenario, one possibility would be to check the values of "min.insync.replicas" and "default.replication.factor" when the broker starts, and "min.insync.replicas" and the replication factor given to a topic at creation time, and refuses to create the topic if those are inconsistently set. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (KAFKA-8752) Ensure plugin classes are instantiable when discovering plugins
Alexandre Dupriez created KAFKA-8752: Summary: Ensure plugin classes are instantiable when discovering plugins Key: KAFKA-8752 URL: https://issues.apache.org/jira/browse/KAFKA-8752 Project: Kafka Issue Type: Bug Components: core Reporter: Alexandre Dupriez Assignee: Alexandre Dupriez While running integration tests from the IntelliJ IDE, it appears plugins fail to load in {{DelegatingClassLoader.scanUrlsAndAddPlugins}}. The reason was, in this case, that the class {{org.apache.kafka.connect.connector.ConnectorReconfigurationTest$TestConnector}} could not be instantiated - which it does not intend to be. The problem does not when running integration tests with Gradle as the runtime closure is different from IntelliJ - which includes test sources from modules depended on on the classpath. While debugging this minor inconvenience, I could see that {{DelegatingClassLoader}} performs a sanity check on the plugin class to instantiate - as of now, it verifies the class is concrete. A quick fix for the problem highlighted above could to add an extra condition on the Java modifiers of the class to ensure it will be instantiable. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (KAFKA-8695) Metrics UnderReplicated and UnderMinIsr are diverging when configuration is inconsistent
[ https://issues.apache.org/jira/browse/KAFKA-8695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexandre Dupriez resolved KAFKA-8695. -- Resolution: Duplicate De-duplicating in favor of [KAFKA-4680|http://issues.apache.org/jira/browse/KAFKA-4680]. > Metrics UnderReplicated and UnderMinIsr are diverging when configuration is > inconsistent > > > Key: KAFKA-8695 > URL: https://issues.apache.org/jira/browse/KAFKA-8695 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 2.1.0, 2.2.0, 2.1.1, 2.3.0 >Reporter: Alexandre Dupriez >Assignee: Alexandre Dupriez >Priority: Minor > > As of now, Kafka allows the replication factor of a topic and > "min.insync.replicas" to be set such that "min.insync.replicas" > the topic's > replication factor. > As a consequence, the JMX beans > {code:java} > kafka.cluster:type=Partition,name=UnderReplicated{code} > and > {code:java} > kafka.cluster:type=Partition,name=UnderMinIsr{code} > can report diverging views on the replication for a topic. The former can > report no under replicated partition, while the second will report under > in-sync replicas. > Even worse, consumption of topics which exhibit this behaviour seems to fail, > the Kafka broker throwing a NotEnoughReplicasException. > {code:java} > [2019-07-22 10:44:29,913] ERROR [ReplicaManager broker=0] Error processing > append operation on partition __consumer_offsets-0 > (kafka.server.ReplicaManager) > org.apache.kafka.common.errors.NotEnoughReplicasException: The size of the > current ISR Set(0) is insufficient to satisfy the min.isr requirement of 2 > for partition __consumer_offsets-0 {code} > In order to avoid this scenario, one possibility would be to check the values > of "min.insync.replicas" and "default.replication.factor" when the broker > starts, and "min.insync.replicas" and the replication factor given to a topic > at creation time, and refuses to create the topic if those are inconsistently > set. > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (KAFKA-8815) Kafka broker blocked on I/O primitive
[ https://issues.apache.org/jira/browse/KAFKA-8815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexandre Dupriez resolved KAFKA-8815. -- Resolution: Not A Problem System failure. Not related to Kafka. > Kafka broker blocked on I/O primitive > - > > Key: KAFKA-8815 > URL: https://issues.apache.org/jira/browse/KAFKA-8815 > Project: Kafka > Issue Type: Bug > Components: core, log >Affects Versions: 1.1.1 >Reporter: Alexandre Dupriez >Priority: Major > > This JIRA is for tracking a problem we run into on a production cluster. > *Scenario* > Cluster of 15 brokers and an average ingress throughput of ~4 MB/s and egress > of ~4 MB/s per broker. > Brokers are running on OpenJDK 8. They are configured with a heap size of 1 > GB. > There is around ~1,000 partition replicas per broker. Load is evenly > balanced. Each broker instance is under fair CPU load, but not overloaded > (50-60%). G1 is used for garbage collection and doesn't exhibit any pressure, > with mostly short young GC observed and an heap-after-GC usage of 70%. > Replication factor is 3. > *Symptom* > One broker on the cluster suddenly became "unresponsive". Other brokers, > Zookeeper and producers/consumers requests were failing with timeouts. The > Kafka process, however, was still alive and doing some background work > (truncating logs and rolling segments) This lasted for hours. At some point, > several thread dumps were taken at few minutes interval. Most of the threads > were "blocked". Deadlock was ruled out. The most suspicious stack is the > following > {code:java} > Thread 7801: (state = BLOCKED) > - sun.nio.ch.FileChannelImpl.write(java.nio.ByteBuffer) @bci=25, line=202 > (Compiled frame) > - > org.apache.kafka.common.record.MemoryRecords.writeFullyTo(java.nio.channels.GatheringByteChannel) > @bci=24, line=93 (Compiled frame) > - > org.apache.kafka.common.record.FileRecords.append(org.apache.kafka.common.record.MemoryRecords) > @bci=5, line=152 (Compiled frame) > - kafka.log.LogSegment.append(long, long, long, long, > org.apache.kafka.common.record.MemoryRecords) @bci=82, line=136 (Compiled > frame) > - kafka.log.Log.$anonfun$append$2(kafka.log.Log, > org.apache.kafka.common.record.MemoryRecords, boolean, boolean, int, > java.lang.Object) @bci=1080, line=757 (Compiled frame) > - kafka.log.Log$$Lambda$614.apply() @bci=24 (Compiled frame) > - kafka.log.Log.maybeHandleIOException(scala.Function0, scala.Function0) > @bci=1, line=1696 (Compiled frame) > - kafka.log.Log.append(org.apache.kafka.common.record.MemoryRecords, > boolean, boolean, int) @bci=29, line=642 (Compiled frame) > - kafka.log.Log.appendAsLeader(org.apache.kafka.common.record.MemoryRecords, > int, boolean) @bci=5, line=612 (Compiled frame) > - > kafka.cluster.Partition.$anonfun$appendRecordsToLeader$1(kafka.cluster.Partition, > org.apache.kafka.common.record.MemoryRecords, boolean, int) @bci=148, > line=609 (Compiled frame) > - kafka.cluster.Partition$$Lambda$837.apply() @bci=16 (Compiled frame) > - kafka.utils.CoreUtils$.inLock(java.util.concurrent.locks.Lock, > scala.Function0) @bci=7, line=250 (Compiled frame) > - > kafka.utils.CoreUtils$.inReadLock(java.util.concurrent.locks.ReadWriteLock, > scala.Function0) @bci=8, line=256 (Compiled frame) > - > kafka.cluster.Partition.appendRecordsToLeader(org.apache.kafka.common.record.MemoryRecords, > boolean, int) @bci=16, line=597 (Compiled frame) > - > kafka.server.ReplicaManager.$anonfun$appendToLocalLog$2(kafka.server.ReplicaManager, > boolean, boolean, short, scala.Tuple2) @bci=295, line=739 (Compiled frame) > - kafka.server.ReplicaManager$$Lambda$836.apply(java.lang.Object) @bci=20 > (Compiled frame) > - scala.collection.TraversableLike.$anonfun$map$1(scala.Function1, > scala.collection.mutable.Builder, java.lang.Object) @bci=3, line=234 > (Compiled frame) > - scala.collection.TraversableLike$$Lambda$14.apply(java.lang.Object) @bci=9 > (Compiled frame) > - scala.collection.mutable.HashMap.$anonfun$foreach$1(scala.Function1, > scala.collection.mutable.DefaultEntry) @bci=16, line=138 (Compiled frame) > - scala.collection.mutable.HashMap$$Lambda$31.apply(java.lang.Object) @bci=8 > (Compiled frame) > - scala.collection.mutable.HashTable.foreachEntry(scala.Function1) @bci=39, > line=236 (Compiled frame) > - > scala.collection.mutable.HashTable.foreachEntry$(scala.collection.mutable.HashTable, > scala.Function1) @bci=2, line=229 (Compiled frame) > - scala.collection.mutable.HashMap.foreachEntry(scala.Function1) @bci=2, > line=40 (Compiled frame) > - scala.collection.mutable.HashMap.foreach(scala.Function1) @bci=7, line=138 > (Compiled frame) > - scala.collection.TraversableLike.map(scala.Function1, > scala.collection.generic.CanBuildFrom) @bci=14, line=234 (Compiled fra
[jira] [Resolved] (KAFKA-8916) Unreliable kafka-reassign-partitions.sh affecting performance
[ https://issues.apache.org/jira/browse/KAFKA-8916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexandre Dupriez resolved KAFKA-8916. -- Resolution: Invalid Closing this as there is no bug or development required. Please kindly reach out to the user mailing list for this type of question: us...@kafka.apache.org. More information to engage the Kafka community is available on https://kafka.apache.org/contact.html. > Unreliable kafka-reassign-partitions.sh affecting performance > - > > Key: KAFKA-8916 > URL: https://issues.apache.org/jira/browse/KAFKA-8916 > Project: Kafka > Issue Type: Task > Components: admin, config >Affects Versions: 2.1.1 > Environment: CentOS 7 >Reporter: VinayKumar >Priority: Major > Labels: performance > > Currently I have 3 node kafka cluster, and I want to add 2 more nodes to make > it 5 node cluster. > *After adding the nodes to cluster, I need all the topic partitions to be > evenly distributed across all the 5 nodes. > **In the past, when I ran kafka-reassign-partitions.sh & > kafka-preferred-replica-election.sh, it ran for very long time, hung & made > the cluster unstable. So I'm afraid to use this method. > Can you please suggest the best & foolproof way to assign partitions among > all the cluster nodes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-6959) Any impact we foresee if we upgrade Linux version or move to VM instead of physical Linux server
[ https://issues.apache.org/jira/browse/KAFKA-6959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexandre Dupriez resolved KAFKA-6959. -- Resolution: Fixed > Any impact we foresee if we upgrade Linux version or move to VM instead of > physical Linux server > > > Key: KAFKA-6959 > URL: https://issues.apache.org/jira/browse/KAFKA-6959 > Project: Kafka > Issue Type: Task > Components: admin >Affects Versions: 0.11.0.2 > Environment: Prod >Reporter: Gene Yi >Priority: Trivial > Labels: patch, performance, security > > As we know that the recent issue on the Liunx Meltdown and Spectre. all the > Linux servers need to deploy the patch and the OS version at least to be 6.9. > we want to know the impact to Kafka, is there any side effect if we directly > upgrade the OS to 7.0, also is there any limitation if we deploy Kafka to VM > instead of the physical servers? > currently the Kafka version we used is 0.11.0.2 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9554) Define the SPI for Tiered Storage framework
Alexandre Dupriez created KAFKA-9554: Summary: Define the SPI for Tiered Storage framework Key: KAFKA-9554 URL: https://issues.apache.org/jira/browse/KAFKA-9554 Project: Kafka Issue Type: Sub-task Components: clients, core Reporter: Alexandre Dupriez Assignee: Alexandre Dupriez The goal of this task is to define the SPI (service provider interfaces) which will be used by vendors to implement plug-ins to communicate with specific storage system. Done means: * Package with interfaces and key objects available and published for review. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9555) Topic-based implementation for the RLMM
Alexandre Dupriez created KAFKA-9555: Summary: Topic-based implementation for the RLMM Key: KAFKA-9555 URL: https://issues.apache.org/jira/browse/KAFKA-9555 Project: Kafka Issue Type: Sub-task Components: core Reporter: Alexandre Dupriez -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9564) Integration Tests for Tiered Storage
Alexandre Dupriez created KAFKA-9564: Summary: Integration Tests for Tiered Storage Key: KAFKA-9564 URL: https://issues.apache.org/jira/browse/KAFKA-9564 Project: Kafka Issue Type: Sub-task Reporter: Alexandre Dupriez Assignee: Alexandre Dupriez -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9565) Implementation of Tiered Storage SPI to integrate with S3
Alexandre Dupriez created KAFKA-9565: Summary: Implementation of Tiered Storage SPI to integrate with S3 Key: KAFKA-9565 URL: https://issues.apache.org/jira/browse/KAFKA-9565 Project: Kafka Issue Type: Sub-task Reporter: Alexandre Dupriez -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-7958) Transactions are broken with kubernetes hosted brokers
[ https://issues.apache.org/jira/browse/KAFKA-7958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexandre Dupriez resolved KAFKA-7958. -- Fix Version/s: 2.1.1 Resolution: Fixed > Transactions are broken with kubernetes hosted brokers > -- > > Key: KAFKA-7958 > URL: https://issues.apache.org/jira/browse/KAFKA-7958 > Project: Kafka > Issue Type: Bug >Affects Versions: 2.1.0 > Environment: cp-kakfka 2.1.1-1, kafka-streams 2.1.1 >Reporter: Thomas Dickinson >Priority: Major > Fix For: 2.1.1 > > > After a rolling re-start in a kubernetes-like environment, brokers may change > IP address. From our logs it seems that the transaction manager in the > brokers never re-resolves the DNS name of other brokers, keeping stale pod > IPs. Thus transactions stop working. > ??[2019-02-20 02:20:20,085] WARN [TransactionCoordinator id=1001] Connection > to node 0 > (khaki-joey-kafka-0.khaki-joey-kafka-headless.hyperspace-dev/[10.233.124.181:9092|http://10.233.124.181:9092/]) > could not be established. Broker may not be available. > (org.apache.kafka.clients.NetworkClient)?? > ??[2019-02-20 02:20:57,205] WARN [TransactionCoordinator id=1001] Connection > to node 1 > (khaki-joey-kafka-1.khaki-joey-kafka-headless.hyperspace-dev/[10.233.122.67:9092|http://10.233.122.67:9092/]) > could not be established. Broker may not be available. > (org.apache.kafka.clients.NetworkClient)?? > This is from the log from broker 1001 which was restarted first, followed by > 1 and then 0. The log entries are from the day after the rolling restart. > I note a similar issue was fixed for clients 2.1.1 > https://issues.apache.org/jira/browse/KAFKA-7755. We are using streams lib > 2.1.1 > *Update* We are now testing with Kafka 2.1.1 (docker cp-kafka 5.1.2-1) and it > looks like this may be resolved. Would be good to get confirmation though. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-9549) Local storage implementations for RSM which can be used in tests
[ https://issues.apache.org/jira/browse/KAFKA-9549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexandre Dupriez resolved KAFKA-9549. -- Resolution: Fixed > Local storage implementations for RSM which can be used in tests > > > Key: KAFKA-9549 > URL: https://issues.apache.org/jira/browse/KAFKA-9549 > Project: Kafka > Issue Type: Sub-task > Components: system tests >Reporter: Satish Duggana >Assignee: Alexandre Dupriez >Priority: Major > > The goal of this task is to implement a straightforward file-system based > implementation of the {{RemoteStorageManager}} defined as part of the SPI for > Tiered Storage. > It is intended to be used in single-host integration tests where the remote > storage is or can be exercised. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-14190) Corruption of Topic IDs with pre-2.8.0 admin clients
Alexandre Dupriez created KAFKA-14190: - Summary: Corruption of Topic IDs with pre-2.8.0 admin clients Key: KAFKA-14190 URL: https://issues.apache.org/jira/browse/KAFKA-14190 Project: Kafka Issue Type: Bug Components: admin, core, zkclient Affects Versions: 3.2.1, 3.1.1, 3.2.0, 3.0.1, 3.0.0, 2.8.1, 3.1.0 Reporter: Alexandre Dupriez h4. Scope The problem reported below has been verified to occur with Zookeeper controllers. It has not been attempted with Kraft controllers, although it is unlikely to be reproduced in Kraft mode given the nature of the issue and clients involved. h4. Problem Description There is a loss of topic IDs when an AdminClient of version < 2.8.0 is used to increase the number of partitions of a topic for a cluster with version >= 2.8.0. This results in the controller re-creating topic IDs upon restart, eventually conflicting with the topic ID of broker’s partition.metadata files in the partition directories of the impacted topic, leading to an availability loss of the partitions which do not accept leadership / follower-ship when the topic ID indicated by a LeaderAndIsr request differ from their own locally cached ID. One mitigation post-corruption is to substitute the stale topic ID in the partition.metadata files with the new topic ID referenced by the controller, or alternatively, delete the partition.metadata file altogether. h4. Steps to reproduce 1. Set-up and launch a two-nodes Kafka cluster in Zookeeper mode. 2. Create a topic e.g. via {{kafka-topics.sh}} {noformat} ./bin/kafka-topics.sh --bootstrap-server :9092 --create --topic myTopic --partitions 2 --replication-factor 2{noformat} 3. Capture the topic ID using a 2.8.0+ client. {noformat} ./kafka/bin/kafka-topics.sh --bootstrap-server :9092 --topic myTopic --describe Topic: myTopic TopicId: jKTRaM_TSNqocJeQI2aYOQ PartitionCount: 2 ReplicationFactor: 2 Configs: segment.bytes=1073741824 Topic: myTopic Partition: 0 Leader: 0 Replicas: 1,0 Isr: 0,1 Topic: myTopic Partition: 1 Leader: 0 Replicas: 0,1 Isr: 0,1{noformat} 4. Restart one of the broker. This will make each broker create the {{partition.metadata}} files in the partition directories since it will already have loaded the {{Log}} instance in memory. 5. Using a pre-2.8.0 client library, run the following command. {noformat} ./kafka/bin/kafka-topics.sh --zookeeper :2181 --alter --topic myTopic --partitions 3{noformat} 6. Using a 2.8.0+ client library, describe the topic via Zookeeper and notice the absence of topic ID from the output, where it is otherwise expected. {noformat} ./kafka/bin/kafka-topics.sh —zookeeper :2181 —describe —topic myTopic Topic: myTopic PartitionCount: 3 ReplicationFactor: 2 Configs: Topic: myTopic Partition: 0 Leader: 1 Replicas: 1,0 Isr: 0,1 Topic: myTopic Partition: 1 Leader: 0 Replicas: 0,1 Isr: 0,1 Topic: myTopic Partition: 2 Leader: 1 Replicas: 1,0 Isr: 1,0{noformat} 7. Using a 2.8.0+ client library, describe the topic via a broker endpoint and notice the topic ID changed. {noformat} ./kafka/bin/kafka-topics.sh —bootstrap-server :9093 —describe —topic myTopic Topic: myTopic TopicId: nI-JQtPwQwGiylMfm8k13w PartitionCount: 3 ReplicationFactor: 2 Configs: segment.bytes=1073741824 Topic: myTopic Partition: 0 Leader: 1 Replicas: 1,0 Isr: 1,0 Topic: myTopic Partition: 1 Leader: 1 Replicas: 0,1 Isr: 1,0 Topic: myTopic Partition: 2 Leader: 1 Replicas: 1,0 Isr: 1,0{noformat} 8. Restart the controller. 9. Check the state-change.log file on the controller broker. The following type of logs will appear. {noformat} [2022-08-25 17:44:05,308] ERROR [Broker id=0] Topic Id in memory: jKTRaM_TSNqocJeQI2aYOQ does not match the topic Id for partition myTopic-0 provided in the request: nI-JQtPwQwGiylMfm8k13w. (state.change.logger){noformat} 10. Restart the other broker. 11. Describe the topic via the broker endpoint or Zookeeper with a 2.8.0+ client library {noformat} ./kafka/bin/kafka-topics.sh --zookeeper :2181 --describe --topic myTopic Topic: myTopic TopicId: nI-JQtPwQwGiylMfm8k13w PartitionCount: 3 ReplicationFactor: 2 Configs: Topic: myTopic Partition: 0 Leader: 0 Replicas: 1,0 Isr: 0 Topic: myTopic Partition: 1 Leader: 0 Replicas: 0,1 Isr: 0 Topic: myTopic Partition: 2 Leader: 0 Replicas: 1,0 Isr: 0{noformat} Notice the abnormal state the topic is in: ISR is reduced to one single broker which is claimed to be the leader by the controller (here, broker 0). The controller believes 0 is the leader because it does not handle the error response from peer brokers when sending the requests for them to become a leader or follower of a partition. 12. Verify produce is unavailable. {noformat} ./kafka/bin/kafka-console-producer.sh —bootstrap-server:9092 —topic myTopic [2022-08-25 17:52:59,962] ERROR Error when sending message to topic myTopic
[jira] [Resolved] (KAFKA-8752) Ensure plugin classes are instantiable when discovering plugins
[ https://issues.apache.org/jira/browse/KAFKA-8752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexandre Dupriez resolved KAFKA-8752. -- Resolution: Not A Problem > Ensure plugin classes are instantiable when discovering plugins > --- > > Key: KAFKA-8752 > URL: https://issues.apache.org/jira/browse/KAFKA-8752 > Project: Kafka > Issue Type: Bug > Components: core >Reporter: Alexandre Dupriez >Assignee: Alexandre Dupriez >Priority: Minor > Attachments: stacktrace.log > > > While running integration tests from the IntelliJ IDE, it appears plugins > fail to load in {{DelegatingClassLoader.scanUrlsAndAddPlugins}}. The reason > was, in this case, that the class > {{org.apache.kafka.connect.connector.ConnectorReconfigurationTest$TestConnector}} > could not be instantiated - which it does not intend to be. > The problem does not appear when running integration tests with Gradle as the > runtime closure is different from IntelliJ - which includes test sources from > module dependencies on the classpath. > While debugging this minor inconvenience, I could see that > {{DelegatingClassLoader}} performs a sanity check on the plugin class to > instantiate - as of now, it verifies the class is concrete. A quick fix for > the problem highlighted above could to add an extra condition on the Java > modifiers of the class to ensure it will be instantiable. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15486) Include NIO exceptions as I/O exceptions to be part of disk failure handling
Alexandre Dupriez created KAFKA-15486: - Summary: Include NIO exceptions as I/O exceptions to be part of disk failure handling Key: KAFKA-15486 URL: https://issues.apache.org/jira/browse/KAFKA-15486 Project: Kafka Issue Type: Improvement Components: core, jbod Reporter: Alexandre Dupriez Currently, Apache Kafka offers the ability to detect and capture I/O errors when accessing the file system via the standard {{IOException}} from the JDK. There are cases however, where I/O errors are only reported via exceptions such as {{{}BufferOverflowException{}}}, without associated {{IOException}} on the produce or read path, so that the data volume is not detected as unhealthy and not included in the list of offline directories. Specifically, we faced the following scenario on a broker: * The data volume hosting a log directory became saturated. * As expected, {{IOException}} were generated on the read/write path. * The log directory was set as offline and since it was the only log directory configured on the broker, Kafka automatically shut down. * Additional space was added to the data volume. * Kafka was then restarted. * No more {{IOException}} occurred, however {{BufferOverflowException}} *[*]* were raised while trying to delete log segments in oder to honour the retention settings of a topic. The log directory was not moved to offline and the exceptions kept re-occurring indefinitely. The retention settings were therefore not applied in this case. The mitigation consisted in restarting Kafka. It may be worth considering adding {{BufferOverflowException}} and {{BufferUnderflowException}} (and any other related exception from the JDK NIO library which surfaces an I/O error) to the current {{IOException}} as a proxy of storage I/O failure, although there may be known unintended consequences in doing so which is the reason they were not added already, or, it may be too marginal of an impact to modify the main I/O failure handing path to risk exposing it to such unknown unintended consequences. *[*]* {code:java} java.nio.BufferOverflowException at java.base/java.nio.Buffer.nextPutIndex(Buffer.java:674) at java.base/java.nio.DirectByteBuffer.putLong(DirectByteBuffer.java:882) at kafka.log.TimeIndex.$anonfun$maybeAppend$1(TimeIndex.scala:134) at kafka.log.TimeIndex.maybeAppend(TimeIndex.scala:114) at kafka.log.LogSegment.onBecomeInactiveSegment(LogSegment.scala:506) at kafka.log.Log.$anonfun$roll$8(Log.scala:2066) at kafka.log.Log.$anonfun$roll$8$adapted(Log.scala:2066) at scala.Option.foreach(Option.scala:437) at kafka.log.Log.$anonfun$roll$2(Log.scala:2066) at kafka.log.Log.roll(Log.scala:2482) at kafka.log.Log.maybeRoll(Log.scala:2017) at kafka.log.Log.append(Log.scala:1292) at kafka.log.Log.appendAsFollower(Log.scala:1155) at kafka.cluster.Partition.doAppendRecordsToFollowerOrFutureReplica(Partition.scala:1023) at kafka.cluster.Partition.appendRecordsToFollowerOrFutureReplica(Partition.scala:1030) at kafka.server.ReplicaFetcherThread.processPartitionData(ReplicaFetcherThread.scala:178) at kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$7(AbstractFetcherThread.scala:356) at scala.Option.foreach(Option.scala:437) at kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$6(AbstractFetcherThread.scala:345) at kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$6$adapted(AbstractFetcherThread.scala:344) at kafka.utils.Implicits$MapExtensionMethods$.$anonfun$forKeyValue$1(Implicits.scala:62) at scala.collection.convert.JavaCollectionWrappers$JMapWrapperLike.foreachEntry(JavaCollectionWrappers.scala:359) at scala.collection.convert.JavaCollectionWrappers$JMapWrapperLike.foreachEntry$(JavaCollectionWrappers.scala:355) at scala.collection.convert.JavaCollectionWrappers$AbstractJMapWrapper.foreachEntry(JavaCollectionWrappers.scala:309) at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:344) at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3(AbstractFetcherThread.scala:141) at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3$adapted(AbstractFetcherThread.scala:140) at scala.Option.foreach(Option.scala:437) at kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:140) at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:123) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96) {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15490) Invalid path provided to the log failure channel upon I/O error when writing broker metadata checkpoint
Alexandre Dupriez created KAFKA-15490: - Summary: Invalid path provided to the log failure channel upon I/O error when writing broker metadata checkpoint Key: KAFKA-15490 URL: https://issues.apache.org/jira/browse/KAFKA-15490 Project: Kafka Issue Type: Bug Components: core Reporter: Alexandre Dupriez There is a small bug/typo in the handling of I/O error when writing broker metadata checkpoint in {{{}KafkaServer{}}}. The path provided to the log dir failure channel is the full path of the checkpoint file whereas only the log directory is expected ([source|https://github.com/apache/kafka/blob/3.4/core/src/main/scala/kafka/server/KafkaServer.scala#L958C8-L961C8]). {code:java} case e: IOException => val dirPath = checkpoint.file.getAbsolutePath logDirFailureChannel.maybeAddOfflineLogDir(dirPath, s"Error while writing meta.properties to $dirPath", e){code} As a result, after an {{IOException}} is captured and enqueued in the log dir failure channel {{}} {code:java} {code} {{[2023-09-22 17:07:32,052] ERROR Error while writing meta.properties to /meta.properties (kafka.server.LogDirFailureChannel) java.io.IOException}} The log dir failure handler cannot lookup the log directory: {{}} {code:java} {code} {{[2023-09-22 17:07:32,053] ERROR [LogDirFailureHandler]: Error due to (kafka.server.ReplicaManager$LogDirFailureHandler) org.apache.kafka.common.errors.LogDirNotFoundException: Log dir /meta.properties is not found in the config.}} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15678) [Tiered Storage] Stall remote reads with long-spanning transactions
Alexandre Dupriez created KAFKA-15678: - Summary: [Tiered Storage] Stall remote reads with long-spanning transactions Key: KAFKA-15678 URL: https://issues.apache.org/jira/browse/KAFKA-15678 Project: Kafka Issue Type: Bug Components: Tiered-Storage Affects Versions: 3.6.0 Reporter: Alexandre Dupriez Hi team, I am facing an issue on the remote data path for uncommitted reads. As mentioned in [the original PR|https://github.com/apache/kafka/pull/13535#discussion_r1166887367], if a transaction spans over a long sequence of segments, the time taken to retrieve the producer snapshots from the remote storage can, in the worst case, become redhibitory and block the reads if it consistently exceed the deadline of fetch requests ({{{}fetch.max.wait.ms{}}}). Essentially, the method used to compute the uncommitted records to return have an asymptotic complexity proportional to the number of segments in the log. That is not a problem with local storage since the constant factor to traverse the files is small enough, but that is not the case with a remote storage which exhibits higher read latency. An aggravating factor was the lock contention in the remote index cache which was then mitigated by KAFKA-15084. But unfortunately, despite the improvements observed without said contention, the algorithmic complexity of the current method used to compute uncommitted records can always defeat any optimisation made on the remote read path. Maybe we could start thinking (if not already) about a different construct which would reduce that complexity to O(1) - i.e. to make the computation independent from the number of segments and irrespective of the transaction spans. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-14777) Add support of topic id for OffsetCommitRequests in CommitRequestManager
Alexandre Dupriez created KAFKA-14777: - Summary: Add support of topic id for OffsetCommitRequests in CommitRequestManager Key: KAFKA-14777 URL: https://issues.apache.org/jira/browse/KAFKA-14777 Project: Kafka Issue Type: Sub-task Reporter: Alexandre Dupriez Topic IDs have been introduced to the {{OffsetCommitRequest}} in KAFKA-14690. The consumer coordinator now generates these requests with topic ids when all topics present in the request have a resolved id. This change was not added to the commit request manager to limit the scope of [PR-13240|https://github.com/apache/kafka/pull/13240]. The purpose of this PR is to extend the support of topic ids to this new component. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-14779) Add ACL Authorizer integration test for authorized OffsetCommits with an unknown topic
Alexandre Dupriez created KAFKA-14779: - Summary: Add ACL Authorizer integration test for authorized OffsetCommits with an unknown topic Key: KAFKA-14779 URL: https://issues.apache.org/jira/browse/KAFKA-14779 Project: Kafka Issue Type: Sub-task Reporter: Alexandre Dupriez Discovered as part of [PR-13240|https://github.com/apache/kafka/pull/13240),], it seems the use case where a group and topic have the necessary ACLs to allow for offsets for that topic and consumer group to be committed, but the topic is unknown by the broker (either by name or id), is not covered. This purpose of this ticket is to add this coverage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-14780) Make RefreshingHttpsJwksTest#testSecondaryRefreshAfterElapsedDelay deterministic
Alexandre Dupriez created KAFKA-14780: - Summary: Make RefreshingHttpsJwksTest#testSecondaryRefreshAfterElapsedDelay deterministic Key: KAFKA-14780 URL: https://issues.apache.org/jira/browse/KAFKA-14780 Project: Kafka Issue Type: Test Reporter: Alexandre Dupriez The test `RefreshingHttpsJwksTest#testSecondaryRefreshAfterElapsedDelay` relies on the actual system clock which makes it frequently fail on my poor intellij setup. The `RefreshingHttpsJwks` component creates and uses a scheduled executor service. We could expose the scheduling mechanism to be able to mock its behaviour. One way to do could be to use the `KafkaScheduler` which has a `MockScheduler` implementation which relies on `MockTime` instead of the real time clock. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-14793) Propagate topic ids to the group coordinator
Alexandre Dupriez created KAFKA-14793: - Summary: Propagate topic ids to the group coordinator Key: KAFKA-14793 URL: https://issues.apache.org/jira/browse/KAFKA-14793 Project: Kafka Issue Type: Sub-task Reporter: Alexandre Dupriez KAFKA-14690 introduces topic ids in the OffsetCommit API in the request layer. Propagation of topic ids within the group coordinator has been left out of scope. Whether topic ids are re-mapped internally in the group coordinator or the group coordinator starts to rely on {{{}TopicIdPartition{}}}. Note that with KAFKA-14690, the offset commit response data built by the coordinator includes topic names only, and topic ids need to be injected afterwards outside of the coordinator before serializing the response. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-14806) Add connection timeout in PlaintextSender used by SelectorTests
Alexandre Dupriez created KAFKA-14806: - Summary: Add connection timeout in PlaintextSender used by SelectorTests Key: KAFKA-14806 URL: https://issues.apache.org/jira/browse/KAFKA-14806 Project: Kafka Issue Type: Test Reporter: Alexandre Dupriez Tests in `SelectorTest` can fail due to spurious connection timeouts. One example can be found in [this build|https://github.com/apache/kafka/pull/13378/checks?check_run_id=11970595528] where the client connection the `PlaintextSender` tried to open could not be established before the test timed out. It may be worth enforcing connection timeout and retries if this can add to the selector tests resiliency. Note that `PlaintextSender` is only used by the `SelectorTest` so the scope of the change would remain local. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-14845) Broker ZNode creation can fail due to a session ID unknown to the broker
Alexandre Dupriez created KAFKA-14845: - Summary: Broker ZNode creation can fail due to a session ID unknown to the broker Key: KAFKA-14845 URL: https://issues.apache.org/jira/browse/KAFKA-14845 Project: Kafka Issue Type: Bug Reporter: Alexandre Dupriez Attachments: broker-registration.drawio Our production environment faced a use case where registration of a broker failed due to the presence of a "conflicting" broker znode in Zookeeper. This case is not without familiarity to that fixed by KAFKA-6584 and induced by the Zookeeper bug (or feature) tracked in ZOOKEEPER-2985 opened as of today. A network partition disturbed communication channels between the Kafka and Zookeeper clusters for about 20% of the brokers in the cluster. One of this broker was not able to re-register with Zookeeper and was excluded from the cluster until it was restarted. Broker logs show the failed registration due to a "conflicting" znode write which in this case is not covered by KAFKA-6584. The broker did not restart and was not unhealthy. In the following logs, the broker IP is 1.2.3.4. The sequence of logs on the broker is as follows. First, a connection is established with the Zookeeper node 3. {code:java} [2023-03-05 16:01:55,342] INFO Socket connection established, initiating session, client: /1.2.3.4:40200, server: zk.3/5.6.7.8:2182 (org.apache.zookeeper.ClientCnxn) [2023-03-05 16:01:55,342] INFO channel is connected: [id: 0x2b45ae40, L:/1.1.3.4:40200 - R:zk.3/5.6.7.8:2182] (org.apache.zookeeper.ClientCnxnSocketNetty){code} An existing Zookeeper session was expired, and upon reconnection, the Zookeeper state change handler was invoked. The creation of the ephemeral znode /brokers/ids/18 started on the controller thread. {code:java} [2023-03-05 16:01:55,345] INFO Creating /brokers/ids/18 (is it secure? false) (kafka.zk.KafkaZkClient){code} The client "session" timed out after 6 seconds. Note the session is 0x0 and the absence of "{_}Session establishment complete{_}" log: the broker appears to have never received or processed the response from the Zookeeper node. {code:java} [2023-03-05 16:02:01,343] INFO Client session timed out, have not heard from server in 6000ms for sessionid 0x0, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn) [2023-03-05 16:02:01,343] INFO channel is disconnected: [id: 0x2b45ae40, L:/1.2.3.4:40200 ! R:zk.3/5.6.7.8:2182] (org.apache.zookeeper.ClientCnxnSocketNetty){code} Pending requests were aborted with a {{CONNECTIONLOSS}} error and the client started waiting on a new connection notification. {code:java} [2023-03-05 16:02:01,343] INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient){code} A new connection was created with the Zookeeper node 1. Note that a valid (new) session ({{{}0x1006c6e0b830001{}}}) was reported by Kafka this time. {code:java} [2023-03-05 16:02:02,037] INFO Socket connection established, initiating session, client: /1.2.3.4:58080, server: zk.1/9.10.11.12:2182 (org.apache.zookeeper.ClientCnxn) [2023-03-05 16:02:02,037] INFO channel is connected: [id: 0x68fba106, L:/1.2.3.4:58080 - R:zk.1/9.10.11.12:2182] (org.apache.zookeeper.ClientCnxnSocketNetty) [2023-03-05 16:02:03,054] INFO Session establishment complete on server zk.1/9.10.11.12:2182, sessionid = 0x1006c6e0b830001, negotiated timeout = 18000 (org.apache.zookeeper.ClientCnxn){code} The Kafka ZK client is notified of the connection. {code:java} [2023-03-05 16:02:03,054] INFO [ZooKeeperClient Kafka server] Connected. (kafka.zookeeper.ZooKeeperClient){code} The broker sends the request to create the znode {{/brokers/ids/18}} which already exists. The error path implemented for KAFKA-6584 is then followed. However, in this case, the session owning the ephemeral node {{0x30043230ac1}} ({{{}216172783240153793{}}}) is different from the last active Zookeeper session which the broker has recorded. And it is also different from the current session {{0x1006c6e0b830001}} ({{{}72176813933264897{}}}), hence the recreation of the broker znode is not attempted. {code:java} [2023-03-05 16:02:04,466] ERROR Error while creating ephemeral at /brokers/ids/18, node already exists and owner '216172783240153793' does not match current session '72176813933264897' (kafka.zk.KafkaZkClient$CheckedEphemeral) org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists at org.apache.zookeeper.KeeperException.create(KeeperException.java:126) at kafka.zk.KafkaZkClient$CheckedEphemeral.getAfterNodeExists(KafkaZkClient.scala:1821) at kafka.zk.KafkaZkClient$CheckedEphemeral.create(KafkaZkClient.scala:1759) at kafka.zk.KafkaZkClient.checkedEphemeralCreate(KafkaZkClient.scala:1726) at kafka.zk.Kaf
[jira] [Created] (KAFKA-14852) Propagate Topic Ids to the Group Coordinator for Offset Fetch
Alexandre Dupriez created KAFKA-14852: - Summary: Propagate Topic Ids to the Group Coordinator for Offset Fetch Key: KAFKA-14852 URL: https://issues.apache.org/jira/browse/KAFKA-14852 Project: Kafka Issue Type: Sub-task Reporter: Alexandre Dupriez Fix For: 3.5.0 This task is the sibling of KAFKA-14793 which propagates topic ids in the group coordinator on the offset commit (write) path. The purpose of this JIRA is to change the interfaces of the group coordinator and group coordinator adapter to propagate topic ids in a similar way. KAFKA-14691 will add the topic ids to the OffsetFetch API itself so that topic ids are propagated from clients to the coordinator on the offset fetch path. Changes to the persisted data model (group metadata and keys) are out of scope. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15038) Use topic id/name mapping from the Metadata cache in RLM
Alexandre Dupriez created KAFKA-15038: - Summary: Use topic id/name mapping from the Metadata cache in RLM Key: KAFKA-15038 URL: https://issues.apache.org/jira/browse/KAFKA-15038 Project: Kafka Issue Type: Sub-task Reporter: Alexandre Dupriez Assignee: Alexandre Dupriez Currently, the {{RemoteLogManager}} maintains its own cache of topic name to topic id [[1]|https://github.com/apache/kafka/blob/trunk/core/src/main/java/kafka/log/remote/RemoteLogManager.java#L138] using the information provided during leadership changes, and removing the mapping upon receiving the notification of partition stopped. It should be possible to re-use the mapping in a broker's metadata cache, removing the need for the RLM to build and update a local cache thereby duplicating the information in the metadata cache. It also allows to preserve a single source of authority regarding the association between topic names and ids. [1] https://github.com/apache/kafka/blob/trunk/core/src/main/java/kafka/log/remote/RemoteLogManager.java#L138 -- This message was sent by Atlassian Jira (v8.20.10#820010)