date:20211012

[jira] [Created] (KAFKA-13372) failed authentication due to: SSL handshake failed

2021-10-12 Thread Maria Isabel Florez Rodriguez (Jira)

Maria Isabel Florez Rodriguez created KAFKA-13372:
-

 Summary: failed authentication due to: SSL handshake failed
 Key: KAFKA-13372
 URL: https://issues.apache.org/jira/browse/KAFKA-13372
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 2.2.2
Reporter: Maria Isabel Florez Rodriguez


Hi everyone,
 
I have the next issue about authentication SCRAM + SSL. I’m using the CLI and 
this is the version of my client (./kafka_2.13-2.8.1/bin/kafka-topics.sh). In 
this example I will talk about list topics, but another operations (consumer, 
producer) failed too.
 
 
First, let me describe the current scenario:
 
 * I have 5 Kafka servers with 
 * kafka-broker-0.mydomain.com
 * kafka-broker-1.mydomain.com
 * kafka-broker-2.mydomain.com
 * kafka-broker-3.mydomain.com
 * kafka-broker-4.mydomain.com

 
 * I have a DNS principal configured with Round Robin to IPs broker:
 * kafka-broker-princial.mydomain.com (Round Robin)

 
 I have configured for each broker the next listeners (I'm using 3 ports):
{quote}advertised.listeners=SASL_SSL://kafka-broker-0.mydomain.com:9094,SASL_PLAINTEXT://kafka-broker-0.mydomain.com:9093,PLAINTEXT://kafka-broker-0.mydomain.com:9092{quote}
 * 9092 for PLAINTEXT
 * 9093 for SASL_PLAINTEXT
 * 9094 for SASL_SSL

 
My Kafka broker servers have the next config server.properties:
{quote}advertised.listeners=SASL_SSL://kafka-broker-X.mydomain.com:9094,SASL_PLAINTEXT://kafka-broker-X.mydomain.com:9093,PLAINTEXT://kafka-broker-X.mydomain.com:9092
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
auto.create.topics.enable=false
auto.leader.rebalance.enable=true
background.threads=10
broker.id=X
broker.rack=us-east-1c
compression.type=producer
connections.max.idle.ms=270
controlled.shutdown.enable=true
delete.topic.enable=true
host.name=localhost
leader.imbalance.check.interval.seconds=300
leader.imbalance.per.broker.percentage=10
listeners=SASL_SSL://0.0.0.0:9094,SASL_PLAINTEXT://0.0.0.0:9093,PLAINTEXT://0.0.0.0:9092
log.cleaner.enable=true
log.dirs=/var/lib/kafka/log/data1,/var/lib/kafka/log/data2,/var/lib/kafka/log/data3
log.retention.check.interval.ms=30
log.retention.hours=336
log.segment.bytes=1073741824
message.max.bytes=112
min.insync.replicas=2
num.io.threads=8
num.network.threads=3
num.partitions=3
num.recovery.threads.per.data.dir=1
num.replica.fetchers=1
offset.metadata.max.bytes=4096
offsets.commit.timeout.ms=5000
offsets.retention.minutes=129600
offsets.topic.num.partitions=50
offsets.topic.replication.factor=3
port=9092
queued.max.requests=500
replica.fetch.min.bytes=1
replica.fetch.wait.max.ms=500
sasl.enabled.mechanisms=SCRAM-SHA-256,GSSAPI
sasl.kerberos.service.name=x
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-256
security.inter.broker.protocol=SASL_SSL
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
socket.send.buffer.bytes=102400
ssl.client.auth=required
{{ssl.endpoint.identification.algorithm=""}}
ssl.enabled.protocols=TLSv1.2
ssl.key.password=
ssl.keystore.location=/etc/ssl/default_keystore.jks
ssl.keystore.password=
ssl.truststore.location=/usr/lib/jvm/java-11-adoptopenjdk-hotspot/lib/security/cacerts
ssl.truststore.password= 
ssl.truststore.type=JKS
super.users=User:x
zookeeper.connect=kafka-zk-X.mydomain.com:2181,kafka-zk-X.mydomain.com:2181,kafka-zk-X.mydomain.com:2181,kafka-zk-X.mydomain.com
 :2181,kafka-zk-X.mydomain.com:218/my-environment
zookeeper.connection.timeout.ms=6000
zookeeper.sasl.client=false{quote}
 
 
I was trying the next things:
 
 * (/)*PLAINTEXT:* I can consume directly to broker to broker with port *9092* 
(Using IP or dns broker) 
 * (/)*PLAINTEXT:* I also can consume directly to DNS principal configured with 
Round Robin  with port *9092* (Using DNS principal)
 * (/)*SASL_SSL:* I can consume directly to broker to broker with port *9094* 
(Using only dns broker due it needs to validate the certificate)
 * (x)*SASL_SSL:* I cannot consume directly to DNS principal configured with 
Round Robin with port *9094*

The issue is: * *(x)SASL_SSL(x):* I cannot consume directly to DNS principal 
configured with Round Robin with port *9094*. Only I have the issue with I try 
to connect directly to DNS principal. My certificates contains permissions with 
all my subdomains under the domain. 

 * I have the next _file.config_ when that I use when I try to connect to  DNS 
principal. (Is the same file that I used for consume directly to broker to 
broker with port 9094)

{quote}# Required connection configs for Kafka producer, consumer, and 
admin{quote}
{quote}ssl.keystore.location=/My/Path/default_keystore.jks
ssl.keystore.password=x
ssl.truststore.location=/My/Path/cacerts
ssl.truststore.password= x
ssl.truststore.type=JKS
ssl.enabled.protocols=TLSv1.2
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-256

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #520

2021-10-12 Thread Apache Jenkins Server

See 


Changes:


--
[...truncated 4076 lines...]
[2021-10-13T00:25:48.751Z] 
[2021-10-13T00:25:48.751Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 8.0.
[2021-10-13T00:25:48.751Z] 
[2021-10-13T00:25:48.751Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2021-10-13T00:25:48.751Z] 
[2021-10-13T00:25:48.751Z] See 
https://docs.gradle.org/7.2/userguide/command_line_interface.html#sec:command_line_warnings
[2021-10-13T00:25:48.751Z] 
[2021-10-13T00:25:48.751Z] Execution optimizations have been disabled for 2 
invalid unit(s) of work during this build to ensure correctness.
[2021-10-13T00:25:48.751Z] Please consult deprecation warnings for more details.
[2021-10-13T00:25:48.751Z] 
[2021-10-13T00:25:48.751Z] BUILD FAILED in 6m 7s
[2021-10-13T00:25:48.751Z] 226 actionable tasks: 184 executed, 42 up-to-date
[2021-10-13T00:25:48.751Z] 
[2021-10-13T00:25:48.751Z] See the profiling report at: 
file:///home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/build/reports/profile/profile-2021-10-13-00-19-42.html
[2021-10-13T00:25:48.751Z] A fine-grained performance profile is available: use 
the --scan option.
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
Failed in branch JDK 17 and Scala 2.13
[2021-10-13T00:26:00.284Z] 
[2021-10-13T00:26:00.284Z] > Task :streams:compileTestJava
[2021-10-13T00:26:00.284Z] 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/src/test/java/org/apache/kafka/streams/kstream/TimeWindowsTest.java:65:
 warning: [deprecation] grace(java.time.Duration) in 
org.apache.kafka.streams.kstream.TimeWindows has been deprecated
[2021-10-13T00:26:00.284Z] assertThrows(IllegalStateException.class, () 
-> TimeWindows.ofSizeAndGrace(ofMillis(10), ofMillis(10)).grace(ofMillis(10)));
[2021-10-13T00:26:00.284Z]  
 ^
[2021-10-13T00:26:00.284Z] 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/src/test/java/org/apache/kafka/streams/kstream/TimeWindowsTest.java:66:
 warning: [deprecation] grace(java.time.Duration) in 
org.apache.kafka.streams.kstream.TimeWindows has been deprecated
[2021-10-13T00:26:00.284Z] assertThrows(IllegalStateException.class, () 
-> TimeWindows.ofSizeWithNoGrace(ofMillis(10)).grace(ofMillis(10)));
[2021-10-13T00:26:00.284Z]  
  ^
[2021-10-13T00:26:04.002Z] 1 error
[2021-10-13T00:26:04.002Z] 6 warnings
[2021-10-13T00:26:04.002Z] 
[2021-10-13T00:26:04.002Z] > Task :streams:compileTestJava FAILED
[2021-10-13T00:26:04.002Z] 
[2021-10-13T00:26:04.002Z] FAILURE: Build failed with an exception.
[2021-10-13T00:26:04.002Z] 
[2021-10-13T00:26:04.002Z] * What went wrong:
[2021-10-13T00:26:04.002Z] Execution failed for task ':streams:compileTestJava'.
[2021-10-13T00:26:04.002Z] > Compilation failed; see the compiler error output 
for details.
[2021-10-13T00:26:04.002Z] 
[2021-10-13T00:26:04.002Z] * Try:
[2021-10-13T00:26:04.002Z] Run with --stacktrace option to get the stack trace. 
Run with --info or --debug option to get more log output. Run with --scan to 
get full insights.
[2021-10-13T00:26:04.002Z] 
[2021-10-13T00:26:04.002Z] * Get more help at https://help.gradle.org
[2021-10-13T00:26:04.002Z] 
[2021-10-13T00:26:04.002Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 8.0.
[2021-10-13T00:26:04.002Z] 
[2021-10-13T00:26:04.002Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2021-10-13T00:26:04.002Z] 
[2021-10-13T00:26:04.002Z] See 
https://docs.gradle.org/7.2/userguide/command_line_interface.html#sec:command_line_warnings
[2021-10-13T00:26:04.002Z] 
[2021-10-13T00:26:04.002Z] Execution optimizations have been disabled for 2 
invalid unit(s) of work during this build to ensure correctness.
[2021-10-13T00:26:04.002Z] Please consult deprecation warnings for more details.
[2021-10-13T00:26:04.002Z] 
[2021-10-13T00:26:04.002Z] BUILD FAILED in 6m 28s
[2021-10-13T00:26:04.002Z] 226 actionable tasks: 184 executed, 42 up-to-date
[2021-10-13T00:26:05.217Z] 
[2021-10-13T00:26:05.217Z] See the profiling report at: 
file:///home/jenkins/workspace/Kafka_kafka_trunk/build/reports/profile/profile-2021-10-13-00-19-39.html
[2021-10-13T00:26:05.217Z] A fine-grained performance profile is available: use 
the --scan option.
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #519

2021-10-12 Thread Apache Jenkins Server

See 


Changes:


--
[...truncated 4067 lines...]
[2021-10-13T00:12:13.852Z] > Compilation failed; see the compiler error output 
for details.
[2021-10-13T00:12:13.852Z] 
[2021-10-13T00:12:13.852Z] * Try:
[2021-10-13T00:12:13.852Z] Run with --stacktrace option to get the stack trace. 
Run with --info or --debug option to get more log output. Run with --scan to 
get full insights.
[2021-10-13T00:12:13.852Z] 
[2021-10-13T00:12:13.852Z] * Get more help at https://help.gradle.org
[2021-10-13T00:12:13.852Z] 
[2021-10-13T00:12:13.852Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 8.0.
[2021-10-13T00:12:13.852Z] 
[2021-10-13T00:12:13.852Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2021-10-13T00:12:13.852Z] 
[2021-10-13T00:12:13.852Z] See 
https://docs.gradle.org/7.2/userguide/command_line_interface.html#sec:command_line_warnings
[2021-10-13T00:12:13.852Z] 
[2021-10-13T00:12:13.852Z] Execution optimizations have been disabled for 2 
invalid unit(s) of work during this build to ensure correctness.
[2021-10-13T00:12:13.852Z] Please consult deprecation warnings for more details.
[2021-10-13T00:12:13.852Z] 
[2021-10-13T00:12:13.852Z] BUILD FAILED in 6m 26s
[2021-10-13T00:12:13.852Z] 226 actionable tasks: 184 executed, 42 up-to-date
[2021-10-13T00:12:13.852Z] 
[2021-10-13T00:12:13.852Z] See the profiling report at: 
file:///home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/build/reports/profile/profile-2021-10-13-00-05-49.html
[2021-10-13T00:12:13.853Z] A fine-grained performance profile is available: use 
the --scan option.
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
Failed in branch ARM
[2021-10-13T00:12:20.518Z] [Warn] 
/home/jenkins/workspace/Kafka_kafka_trunk/core/src/test/scala/unit/kafka/log/OffsetIndexTest.scala:52:
 @nowarn annotation does not suppress any warnings
[2021-10-13T00:12:20.518Z] one warning found
[2021-10-13T00:12:20.518Z] 
[2021-10-13T00:12:20.518Z] > Task :core:testClasses
[2021-10-13T00:12:20.518Z] > Task :core:checkstyleMain
[2021-10-13T00:12:20.518Z] > Task :storage:compileTestJava
[2021-10-13T00:12:20.518Z] > Task :storage:testClasses
[2021-10-13T00:12:20.518Z] > Task :core:checkstyleTest
[2021-10-13T00:12:22.290Z] > Task :storage:checkstyleTest
[2021-10-13T00:12:23.238Z] > Task :jmh-benchmarks:compileJava
[2021-10-13T00:12:23.238Z] > Task :jmh-benchmarks:classes
[2021-10-13T00:12:23.238Z] > Task :jmh-benchmarks:compileTestJava NO-SOURCE
[2021-10-13T00:12:24.185Z] > Task :jmh-benchmarks:checkstyleMain
[2021-10-13T00:12:24.185Z] > Task :jmh-benchmarks:testClasses UP-TO-DATE
[2021-10-13T00:12:24.185Z] > Task :jmh-benchmarks:checkstyleTest NO-SOURCE
[2021-10-13T00:12:24.185Z] > Task :connect:runtime:compileTestJava
[2021-10-13T00:12:24.185Z] > Task :connect:runtime:testClasses
[2021-10-13T00:12:25.156Z] > Task :connect:mirror:compileTestJava
[2021-10-13T00:12:25.156Z] > Task :connect:mirror:testClasses
[2021-10-13T00:12:26.116Z] > Task :connect:mirror:checkstyleTest
[2021-10-13T00:12:28.941Z] [Warn] 
/home/jenkins/workspace/Kafka_kafka_trunk/core/src/test/scala/unit/kafka/log/OffsetIndexTest.scala:52:
 @nowarn annotation does not suppress any warnings
[2021-10-13T00:12:28.941Z] one warning found
[2021-10-13T00:12:33.193Z] > Task :connect:runtime:checkstyleTest
[2021-10-13T00:12:33.598Z] Unexpected javac output: warning: [options] 
bootstrap class path not set in conjunction with -source 8
[2021-10-13T00:12:33.598Z] 1 warning.
[2021-10-13T00:12:37.208Z] 
[2021-10-13T00:12:37.208Z] > Task :core:testClasses
[2021-10-13T00:12:38.159Z] > Task :core:checkstyleMain
[2021-10-13T00:12:39.107Z] > Task :core:checkstyleTest
[2021-10-13T00:12:40.055Z] > Task :storage:compileTestJava
[2021-10-13T00:12:40.056Z] > Task :storage:testClasses
[2021-10-13T00:12:41.827Z] > Task :storage:checkstyleTest
[2021-10-13T00:12:42.774Z] > Task :jmh-benchmarks:compileJava
[2021-10-13T00:12:42.774Z] > Task :jmh-benchmarks:classes
[2021-10-13T00:12:42.774Z] > Task :jmh-benchmarks:compileTestJava NO-SOURCE
[2021-10-13T00:12:43.721Z] > Task :jmh-benchmarks:checkstyleMain
[2021-10-13T00:12:43.721Z] > Task :jmh-benchmarks:testClasses UP-TO-DATE
[2021-10-13T00:12:43.721Z] > Task :jmh-benchmarks:checkstyleTest NO-SOURCE
[2021-10-13T00:12:46.381Z] > Task :connect:runtime:compileTestJava
[2021-10-13T00:12:46.381Z] > Task :connect:runtime:testClasses
[2021-10-13T00:12:48.154Z] > Task :connect:mirror:compileTestJava
[2021-10-13T00:12:48.154Z] > Task :connect:mirror:testClasses
[2021-10-13T00:12:49.101Z] > Task :connect:mirror:checkstyleTest
[2021-10-13T00:12:49.687Z] > Task :streams:compileTestJava

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #518

2021-10-12 Thread Apache Jenkins Server

See 


Changes:


--
[...truncated 4076 lines...]
[2021-10-12T23:59:47.979Z] > Task :connect:file:classes
[2021-10-12T23:59:47.979Z] > Task :storage:compileJava
[2021-10-12T23:59:47.979Z] > Task :storage:classes
[2021-10-12T23:59:49.542Z] > Task :connect:json:compileJava
[2021-10-12T23:59:49.542Z] > Task :connect:json:classes
[2021-10-12T23:59:49.542Z] > Task :raft:compileJava
[2021-10-12T23:59:49.542Z] > Task :connect:transforms:compileJava
[2021-10-12T23:59:49.542Z] > Task :raft:classes
[2021-10-12T23:59:50.857Z] > Task :trogdor:compileJava
[2021-10-12T23:59:50.857Z] > Task :trogdor:classes
[2021-10-12T23:59:50.857Z] > Task :connect:transforms:classes
[2021-10-12T23:59:50.857Z] > Task :tools:checkstyleMain
[2021-10-12T23:59:50.857Z] > Task :connect:api:checkstyleMain
[2021-10-12T23:59:52.419Z] > Task :storage:checkstyleMain
[2021-10-12T23:59:52.419Z] > Task :connect:basic-auth-extension:checkstyleMain
[2021-10-12T23:59:52.419Z] > Task :connect:file:checkstyleMain
[2021-10-12T23:59:52.419Z] > Task :metadata:compileJava
[2021-10-12T23:59:52.419Z] > Task :core:compileJava NO-SOURCE
[2021-10-12T23:59:52.419Z] > Task :connect:mirror-client:checkstyleMain
[2021-10-12T23:59:52.419Z] > Task :metadata:classes
[2021-10-12T23:59:52.419Z] > Task :connect:json:checkstyleMain
[2021-10-12T23:59:53.987Z] > Task :raft:checkstyleMain
[2021-10-12T23:59:53.987Z] > Task :connect:runtime:compileJava
[2021-10-12T23:59:53.987Z] > Task :storage:api:checkstyleMain
[2021-10-12T23:59:53.987Z] > Task :connect:runtime:classes
[2021-10-12T23:59:53.987Z] > Task :connect:transforms:checkstyleMain
[2021-10-12T23:59:53.987Z] > Task :trogdor:checkstyleMain
[2021-10-12T23:59:55.809Z] > Task :connect:mirror:compileJava
[2021-10-12T23:59:55.809Z] > Task :connect:mirror:classes
[2021-10-12T23:59:55.809Z] > Task :log4j-appender:spotbugsMain
[2021-10-12T23:59:55.809Z] > Task :connect:mirror:checkstyleMain
[2021-10-12T23:59:58.733Z] > Task :streams:compileJava
[2021-10-12T23:59:58.733Z] > Task :server-common:spotbugsMain
[2021-10-12T23:59:58.733Z] > Task :streams:streams-scala:compileJava NO-SOURCE
[2021-10-12T23:59:58.733Z] > Task :streams:examples:compileJava
[2021-10-13T00:00:00.295Z] > Task :metadata:checkstyleMain
[2021-10-13T00:00:00.295Z] > Task :streams:classes
[2021-10-13T00:00:00.295Z] > Task :streams:examples:classes
[2021-10-13T00:00:00.295Z] > Task :streams:test-utils:compileJava
[2021-10-13T00:00:00.295Z] > Task :streams:test-utils:classes
[2021-10-13T00:00:00.295Z] > Task :streams:examples:checkstyleMain
[2021-10-13T00:00:01.658Z] > Task :streams:test-utils:checkstyleMain
[2021-10-13T00:00:01.658Z] > Task :connect:runtime:checkstyleMain
[2021-10-13T00:00:17.254Z] > Task :clients:compileTestJava
[2021-10-13T00:00:21.700Z] > Task :streams:checkstyleMain
[2021-10-13T00:00:23.261Z] > Task :core:compileScala
[2021-10-13T00:00:23.261Z] > Task :raft:spotbugsMain
[2021-10-13T00:00:26.179Z] > Task :storage:spotbugsMain
[2021-10-13T00:00:29.192Z] > Task :streams:streams-scala:compileScala
[2021-10-13T00:00:30.754Z] > Task :metadata:spotbugsMain
[2021-10-13T00:00:32.575Z] > Task :tools:spotbugsMain
[2021-10-13T00:00:35.487Z] 
[2021-10-13T00:00:35.487Z] > Task :streams:streams-scala:compileScala
[2021-10-13T00:00:35.487Z] [Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/streams-scala/src/main/scala/org/apache/kafka/streams/scala/kstream/KStream.scala:818:
 Usage of named or default arguments transformed this annotation
[2021-10-13T00:00:35.487Z] constructor call into a block. The corresponding 
AnnotationInfo
[2021-10-13T00:00:35.487Z] will contain references to local values and default 
getters instead
[2021-10-13T00:00:35.487Z] of the actual argument trees
[2021-10-13T00:00:35.487Z] [Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/streams-scala/src/main/scala/org/apache/kafka/streams/scala/kstream/KStream.scala:858:
 Usage of named or default arguments transformed this annotation
[2021-10-13T00:00:35.487Z] constructor call into a block. The corresponding 
AnnotationInfo
[2021-10-13T00:00:35.487Z] will contain references to local values and default 
getters instead
[2021-10-13T00:00:35.487Z] of the actual argument trees
[2021-10-13T00:00:35.487Z] 
[2021-10-13T00:00:35.487Z] > Task :clients:testClasses
[2021-10-13T00:00:35.487Z] > Task :connect:basic-auth-extension:spotbugsMain
[2021-10-13T00:00:37.048Z] > Task :log4j-appender:compileTestJava
[2021-10-13T00:00:37.048Z] > Task :server-common:compileTestJava
[2021-10-13T00:00:38.561Z] > Task :tools:compileTestJava
[2021-10-13T00:00:38.561Z] > Task :connect:file:spotbugsMain
[2021-10-13T00:00:40.123Z] > Task :trogdor:compileTestJava
[2021-10-13T00:00:41.682Z] > Task :connect:api:spotbugsMain
[2021-10-13T00:00:41.682Z] > Task :connect:basic-auth-extension:compileTestJava
[2021-10-13T00:00:41.682Z] > Task :connect:api:compileTestJava

Re: [kafka-clients] [VOTE] 2.7.2 RC0

2021-10-12 Thread Israel Ekpo

Mickael,

For patch or bug fix releases like this one, should we exclude the Javadocs
and site docs if they have not changed?

https://github.com/apache/kafka-site

The site docs were last changed about 6 months ago and it appears it may
not have changed or needs validation



On Tue, Oct 12, 2021 at 2:17 PM Mickael Maison  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the first candidate for release of Apache Kafka 2.7.2.
>
> Apache Kafka 2.7.2 is a bugfix release and 26 issues, as well as
> CVE-2021-38153, have been fixed since 2.7.1.
>
> Release notes for the 2.7.2 release:
> https://home.apache.org/~mimaison/kafka-2.7.2-rc0/RELEASE_NOTES.html
>
> *** Please download, test and vote by Friday, October 15, 5pm CET
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~mimaison/kafka-2.7.2-rc0/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~mimaison/kafka-2.7.2-rc0/javadoc/
>
> * Tag to be voted upon (off 2.7 branch) is the 2.7.2 tag:
> https://github.com/apache/kafka/releases/tag/2.7.2-rc0
>
> * Documentation:
> https://kafka.apache.org/27/documentation.html
>
> * Protocol:
> https://kafka.apache.org/27/protocol.html
>
> * Successful Jenkins builds for the 2.7 branch:
> I'll share a link once the build completes
>
>
> /**
>
> Thanks,
> Mickael
>
> --
> You received this message because you are subscribed to the Google Groups
> "kafka-clients" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kafka-clients+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kafka-clients/CA%2BOCqnY9TikXuCjEyr%2BA2bSjG_Zkd-zFvx9_1Bx%3DiOwpWWN1Sg%40mail.gmail.com
> .
>

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #517

2021-10-12 Thread Apache Jenkins Server

See 


Changes:


--
[...truncated 494616 lines...]
[2021-10-12T22:47:41.182Z] 
[2021-10-12T22:47:41.182Z] PlaintextConsumerTest > 
testMultiConsumerDefaultAssignor() STARTED
[2021-10-12T22:47:51.397Z] 
[2021-10-12T22:47:51.397Z] PlaintextConsumerTest > 
testMultiConsumerDefaultAssignor() PASSED
[2021-10-12T22:47:51.397Z] 
[2021-10-12T22:47:51.397Z] PlaintextConsumerTest > testInterceptors() STARTED
[2021-10-12T22:47:56.131Z] 
[2021-10-12T22:47:56.131Z] > Task :streams:integrationTest
[2021-10-12T22:47:56.131Z] 
[2021-10-12T22:47:56.131Z] 
org.apache.kafka.streams.integration.StateRestorationIntegrationTest > 
shouldRestoreNullRecord PASSED
[2021-10-12T22:48:03.360Z] 
[2021-10-12T22:48:03.360Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorLargeNumConsumers STARTED
[2021-10-12T22:48:15.463Z] 
[2021-10-12T22:48:15.463Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorLargeNumConsumers PASSED
[2021-10-12T22:48:15.463Z] 
[2021-10-12T22:48:15.463Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorLargePartitionCount STARTED
[2021-10-12T22:48:15.463Z] 
[2021-10-12T22:48:15.463Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorLargePartitionCount PASSED
[2021-10-12T22:48:15.463Z] 
[2021-10-12T22:48:15.463Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyThreadsPerClient STARTED
[2021-10-12T22:48:16.403Z] 
[2021-10-12T22:48:16.403Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyThreadsPerClient PASSED
[2021-10-12T22:48:16.403Z] 
[2021-10-12T22:48:16.403Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testStickyTaskAssignorManyStandbys STARTED
[2021-10-12T22:48:21.197Z] 
[2021-10-12T22:48:21.197Z] > Task :core:integrationTest
[2021-10-12T22:48:21.197Z] 
[2021-10-12T22:48:21.197Z] PlaintextConsumerTest > testInterceptors() PASSED
[2021-10-12T22:48:21.197Z] 
[2021-10-12T22:48:21.197Z] PlaintextConsumerTest > 
testConsumingWithEmptyGroupId() STARTED
[2021-10-12T22:48:21.197Z] 
[2021-10-12T22:48:21.197Z] PlaintextConsumerTest > 
testConsumingWithEmptyGroupId() PASSED
[2021-10-12T22:48:21.197Z] 
[2021-10-12T22:48:21.197Z] PlaintextConsumerTest > testPatternUnsubscription() 
STARTED
[2021-10-12T22:48:21.197Z] 
[2021-10-12T22:48:21.197Z] PlaintextConsumerTest > testPatternUnsubscription() 
PASSED
[2021-10-12T22:48:21.197Z] 
[2021-10-12T22:48:21.197Z] PlaintextConsumerTest > testGroupConsumption() 
STARTED
[2021-10-12T22:48:21.197Z] 
[2021-10-12T22:48:21.197Z] PlaintextConsumerTest > testGroupConsumption() PASSED
[2021-10-12T22:48:21.197Z] 
[2021-10-12T22:48:21.197Z] PlaintextConsumerTest > testPartitionsFor() STARTED
[2021-10-12T22:48:21.197Z] 
[2021-10-12T22:48:21.197Z] PlaintextConsumerTest > testPartitionsFor() PASSED
[2021-10-12T22:48:21.197Z] 
[2021-10-12T22:48:21.197Z] PlaintextConsumerTest > 
testMultiConsumerDefaultAssignorAndVerifyAssignment() STARTED
[2021-10-12T22:48:22.137Z] 
[2021-10-12T22:48:22.137Z] PlaintextConsumerTest > 
testMultiConsumerDefaultAssignorAndVerifyAssignment() PASSED
[2021-10-12T22:48:22.137Z] 
[2021-10-12T22:48:22.137Z] PlaintextConsumerTest > testAutoCommitOnRebalance() 
STARTED
[2021-10-12T22:48:26.808Z] 
[2021-10-12T22:48:26.808Z] PlaintextConsumerTest > testAutoCommitOnRebalance() 
PASSED
[2021-10-12T22:48:26.808Z] 
[2021-10-12T22:48:26.808Z] PlaintextConsumerTest > 
testInterceptorsWithWrongKeyValue() STARTED
[2021-10-12T22:48:30.524Z] 
[2021-10-12T22:48:30.524Z] PlaintextConsumerTest > 
testInterceptorsWithWrongKeyValue() PASSED
[2021-10-12T22:48:30.524Z] 
[2021-10-12T22:48:30.524Z] PlaintextConsumerTest > 
testPerPartitionLeadWithMaxPollRecords() STARTED
[2021-10-12T22:48:36.315Z] 
[2021-10-12T22:48:36.315Z] PlaintextConsumerTest > 
testPerPartitionLeadWithMaxPollRecords() PASSED
[2021-10-12T22:48:36.315Z] 
[2021-10-12T22:48:36.315Z] PlaintextConsumerTest > testHeaders() STARTED
[2021-10-12T22:48:41.172Z] 
[2021-10-12T22:48:41.172Z] PlaintextConsumerTest > testHeaders() PASSED
[2021-10-12T22:48:41.172Z] 
[2021-10-12T22:48:41.172Z] PlaintextConsumerTest > 
testMaxPollIntervalMsDelayInAssignment() STARTED
[2021-10-12T22:48:45.976Z] 
[2021-10-12T22:48:45.976Z] > Task :streams:integrationTest
[2021-10-12T22:48:45.976Z] 
[2021-10-12T22:48:45.976Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testStickyTaskAssignorManyStandbys PASSED
[2021-10-12T22:48:45.976Z] 
[2021-10-12T22:48:45.976Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testStickyTaskAssignorManyThreadsPerClient STARTED
[2021-10-12T22:48:45.976Z]

Re: [DISCUSS] KIP-778 KRaft Upgrades

2021-10-12 Thread David Arthur

Jun and Colin, thanks very much for the comments. See below

10. Colin, I agree that ApiVersionsRequest|Response is the most
straightforward approach here

11. This brings up an interesting point. Once a UpdateFeature request is
finished processing, a subsequent "describe features" (ApiVersionsRequest)
made by the admin client will go to a random broker, and so might not
reflect the feature update. We could add blocking behavior to the admin
client so it would polls ApiVersions for some time until the expected
version is reflected on that broker. However, this does not mean _every_
broker has finished upgrading/downgrading -- just the one handling that
request. Maybe we could have the admin client poll all the brokers until
the expected version is seen.

If at a later time a broker comes online that needs to process an
upgrade/downgrade, I don't think there will be a problem since the broker
will be fenced until it catches up on latest metadata (which will include
the feature level).

12. Yes, we will need changes to the admin client for "updateFeatures".
I'll update the KIP to reflect this.

13. I'll expand the paragraph on the initial "metadata.version" into its
own section and add some detail.

14/15. As mentioned, I think we can avoid this and rely on
brokers/controllers generating their own snapshots. We will probably want
some kind of manual recovery mode where we can force load a snapshot, but
that is out of scope here (I think..)

16. Automatic upgrades should be feasible, but I think we will want to
start with manual upgrades (while we work out the design and fix bugs).
Following the design detailed in this KIP, we could have a controller
component that (as you suggest) automatically finalizes the feature to the
max of all broker supported versions. I can include a section on this or we
could defer to a future KIP. WDYT?

-David

On Tue, Oct 12, 2021 at 1:57 PM Colin McCabe  wrote:

> On Thu, Oct 7, 2021, at 17:19, Jun Rao wrote:
> > Hi, David,
> >
> > Thanks for the KIP. A few comments below.
> >
> > 10. It would be useful to describe how the controller node determines the
> > RPC version used to communicate to other controller nodes. There seems to
> > be a bootstrap problem. A controller node can't read the log and
> > therefore the feature level until a quorum leader is elected. But leader
> > election requires an RPC.
> >
>
> Hi Jun,
>
> I agree that we need to figure this out. I think it would be best to
> simply use ApiVersionsRequest and ApiVersionsResponse. That way each
> controller can use the appropriate RPC versions when communicating with
> each other controller. This would allow us to upgrade them one by one.
>
> > 11. For downgrades, it would be useful to describe how to determine the
> > downgrade process (generating new snapshot, propagating the snapshot,
> etc)
> > has completed. We could block the UpdateFeature request until the process
> > is completed. However, since the process could take time, the request
> could
> > time out. Another way is through DescribeFeature and the server only
> > reports downgraded versions after the process is completed.
>
> Hmm.. I think we need to avoid blocking, since we don't know how long it
> will take for all nodes to act on the downgrade request. After all, some
> nodes may be down.
>
> But I agree we should have some way of knowing when the upgrade is done.
> DescribeClusterResponse seems like the natural place to put information
> about each node's feature level. While we're at it, we should also add a
> boolean to indicate whether the given node is fenced. (This will always be
> false for ZK mode, of course...)
>
> >
> > 12. Since we are changing UpdateFeaturesRequest, do we need to change the
> > AdminClient api for updateFeatures too?
> >
> > 13. For the paragraph starting with "In the absence of an operator
> defined
> > value for metadata.version", in KIP-584, we described how to finalize
> > features with New cluster bootstrap. In that case, it's inconvenient for
> > the users to have to run an admin tool to finalize the version for each
> > feature. Instead, the system detects that the /features path is missing
> in
> > ZK and thus automatically finalizes every feature with the latest
> supported
> > version. Could we do something similar in the KRaft mode?
> >
>
> Yes, I think we need to have a section describing how this ties into
> creating new clusters. The simplest thing is probably to have the
> controller notice that there are no FeatureRecords at all, and then create
> a record for the latest metadata.version. This is analogous to how we
> assume the latest IBP if no IBP is configured.
>
> There is also the question of how to create a cluster that starts up with
> something other than the latest metadata.version. We could have a config
> for that, like initial.metadata.version, or pass a flag to the
> controllers... alternately, we could pass a flag to "kafka-storage.sh
> format".
>
> > 14. After the quorum leader

Re: [DISCUSS] KIP-778 KRaft Upgrades

2021-10-12 Thread Colin McCabe

On Wed, Sep 29, 2021, at 10:34, David Arthur wrote:
> Hey all,
>
> I'd like to start a discussion around upgrades for KRaft. This design does
> not cover any of the ZooKeeper to KRaft migration (which will be covered in
> a future KIP).
>
> The proposal includes the addition of a "metadata.version" feature flag and
> deprecates the IBP. A procedure for downgrades is also included in the KIP.
>
> https://cwiki.apache.org/confluence/x/WAxACw
>
> Thanks!
> David

Hi David,

Thanks for the KIP! It's good to see us finding a use for the feature flag 
mechanism.

Since this is the first use of the mechanism, I think we should take some time 
to fix some issues with the original KIP, before the design gets "set in 
stone," so to speak.

1. It is not necessary to have two version numbers for each feature flag. I 
believe the rationale for adding this is that a few features would find it 
useful. However, those features can just define two flags (x_min and x_max). So 
I think each feature should have only one feature level number rather than a 
min and a max. This will simplify a lot of things (including your KIP...)

2. kafka-features.sh should use the more modern Argparse4J format rather than 
the old school "everything is a separate flag" format. This would allow us to 
have subcommands. For example, we could have an "upgrade" subcommand, a "nodes" 
subcommand, etc.

Check out kafka-storage.sh for an example of how this can work. There are three 
subcommands (info, format, random-uuid) and they each have a separate set of 
flags that they take. For example, "kafka-storage.sh format" takes an 
--ignore-formatted flag, but this is not valid for "kafka-storage.sh info".

best,
Colin

[jira] [Resolved] (KAFKA-13319) Do not send AddOffsetsToTxn/TxnOffsetCommit if offsets map is empty

2021-10-12 Thread Guozhang Wang (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-13319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-13319.
---
Fix Version/s: 3.1.0
 Assignee: Guozhang Wang  (was: Ryan)
   Resolution: Fixed

> Do not send AddOffsetsToTxn/TxnOffsetCommit if offsets map is empty
> ---
>
> Key: KAFKA-13319
> URL: https://issues.apache.org/jira/browse/KAFKA-13319
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jason Gustafson
>Assignee: Guozhang Wang
>Priority: Major
>  Labels: newbie
> Fix For: 3.1.0
>
>
> If a user calls `Producer.sendOffsetsToTransaction` with an empty map of 
> offsets, we can shortcut return and skip the logic to add the offsets topic 
> to the transaction. The main benefit is avoiding the unnecessary accumulation 
> of markers in __consumer_offsets.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-752: Support --bootstrap-server in ReplicaVerificationTool

2021-10-12 Thread Dongjin Lee

Bumping up the voting thread for the 3.1.0 KIP freeze deadline.

As of present:

Binding: +2 (Gwen, Randall)
Non-binding: +1 (Luke)

Please note that this tool is one of the only two that are omitted from the
'--bootstrap-server' standard parameter. The other one is StreamsResetter,
which uses '--bootstrap-servers' yet.

Thanks,
Dongjin.

On Mon, Jun 21, 2021 at 8:05 PM Dongjin Lee  wrote:

> Hi Guozhang,
>
> Thanks for the comment. I updated the KIP applying your comment.
>
> @All
>
> Any other thoughts?
>
> Regards,
> Dongjin
>
> On Tue, Jun 15, 2021 at 1:40 AM Guozhang Wang  wrote:
>
>> If we believe this tool does not work anymore and there's other ways to
>> achieve the intended function, then we should remove it in the next
>> release; otherwise, I think this KIP still is worthy. In any ways, we
>> should not left a cmd tool not maintained but not removed either.
>>
>> Guozhang
>>
>> On Thu, Jun 10, 2021 at 10:05 PM Dongjin Lee  wrote:
>>
>> > Hi Ismael,
>> >
>> > > I am not convinced this tool is actually useful, I haven't seen anyone
>> > using it in years.
>> >
>> > Sure, you may right indeed. The `ReplicaVerificationTool` may not be so
>> > useful.[^0] However, I hope to propose another perspective.
>> >
>> > As long as this tool is provided with a launcher script in a
>> distribution,
>> > its command-line parameters look so weird to the users since it breaks
>> > consistency. It is even worse with KIP-499
>> > <
>> >
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=123899170
>> > >[^1],
>> > which tries to unify the command line parameters and deprecate old ones
>> -
>> > even the tools without launcher script (e.g., VerifiableLog4jAppender)
>> now
>> > uses `--bootstrap-server` parameter. This situation is rather odd, isn't
>> > it?
>> >
>> > This improvement may not have a great value, but it may reduce
>> awkwardness
>> > from the user's viewpoint.
>> >
>> > Best,
>> > Dongjin
>> >
>> > [^0]: With my personal experience, I used it to validate the replication
>> > when working with a client so sensitive to replication missing, like a
>> > Semiconductor manufacturing company.
>> > [^1]: Somewhat strange, two omitted tools from KIP-499
>> > <
>> >
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=123899170
>> > >
>> > all have their own launcher script.
>> >
>> > On Thu, Jun 10, 2021 at 2:02 PM Ismael Juma  wrote:
>> >
>> > > KAFKA-12600 was a general change, not related to this tool
>> specifically.
>> > I
>> > > am not convinced this tool is actually useful, I haven't seen anyone
>> > using
>> > > it in years.
>> > >
>> > > Ismael
>> > >
>> > > On Wed, Jun 9, 2021 at 9:51 PM Dongjin Lee 
>> wrote:
>> > >
>> > > > Hi Ismael,
>> > > >
>> > > > Before I submit this KIP, I reviewed some history. When KIP-499
>> > > > <
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-499+-+Unify+connection+name+flag+for+command+line+tool
>> > > > >
>> > > > tried to resolve the inconsistencies between the command line tools,
>> > two
>> > > > tools were omitted, probably by mistake.
>> > > >
>> > > > - KAFKA-12878: Support --bootstrap-server
>> > kafka-streams-application-reset
>> > > > 
>> > > > - KAFKA-12899: Support --bootstrap-server in ReplicaVerificationTool
>> > > >  (this one)
>> > > >
>> > > > And it seems like this tool is still working. The last update was
>> > > > KAFKA-12600  by
>> > you,
>> > > > which will also be included in this 3.0.0 release. It is why I
>> > determined
>> > > > that this tool is worth updating.
>> > > >
>> > > > Thanks,
>> > > > Dongjin
>> > > >
>> > > > On Thu, Jun 10, 2021 at 1:26 PM Ismael Juma 
>> wrote:
>> > > >
>> > > > > Hi Dongjin,
>> > > > >
>> > > > > Does this tool still work? I recall that there were some doubts
>> about
>> > > it
>> > > > > and that's why it wasn't updated previously.
>> > > > >
>> > > > > Ismael
>> > > > >
>> > > > > On Sat, Jun 5, 2021 at 2:38 PM Dongjin Lee 
>> > wrote:
>> > > > >
>> > > > > > Hi all,
>> > > > > >
>> > > > > > I'd like to call for a vote on KIP-752: Support
>> --bootstrap-server
>> > in
>> > > > > > ReplicaVerificationTool:
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-752%3A+Support+--bootstrap-server+in+ReplicaVerificationTool
>> > > > > >
>> > > > > > Best,
>> > > > > > Dongjin
>> > > > > >
>> > > > > > --
>> > > > > > *Dongjin Lee*
>> > > > > >
>> > > > > > *A hitchhiker in the mathematical world.*
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > *github:  github.com/dongjinleekr
>> > > > > > keybase:
>> > > > > https://keybase.io/dongjinleekr
>> > > > > > linkedin:
>> > > > > kr.linkedin.com/in/dongjinleekr
>> > > > > >

Re: [VOTE] KIP-719: Add Log4J2 Appender

2021-10-12 Thread Dongjin Lee

Bumping up the voting thread for the 3.1.0 KIP freeze deadline.

As of present:

Binding: +1 (Konstantine)
Non-binding: +1 (Boojapho)

Again, without this feature, we can't entirely remove the log4j dependency
from the classpath - and it can cause a logging problem after the log4j2
upgrade provided by KIP-653. I found this problem while testing the preview
build.

I also updated the proposal document discussing the details of the problem
and why the other approaches were all rejected.

Thanks,
Dongjin.

On Mon, Jun 14, 2021 at 3:54 PM Dongjin Lee  wrote:

> > VerifiableLog4jAppender is used for system tests, it's not a user facing
> tool. We don't need it to support log4j 2.
>
> I mean, as long as there is VerifiableLog4jAppender which uses
> log4j-appender, we can't entirely remove log4j 1.x artifact from the
> classpath, regardless of if it is a user-facing tool or not.
>
> As I wrote above, I found that some users with the preview build reported
> that if log4j 1.x and 2.x artifacts co-exist in the classpath, sometimes
> the slg4j can't find the appropriate binding, resulting in logging failing.
>
> Providing a log4j2 appender is not only for providing an updated one for
> the log4j-appender users, but for removing a potential problem in the
> classpath also.
>
> Regards,
> Dongjin
>
> On Mon, Jun 14, 2021 at 12:17 AM Ismael Juma  wrote:
>
>> VerifiableLog4jAppender is used for system tests, it's not a user facing
>> tool. We don't need it to support log4j 2.
>>
>> Ismael
>>
>> On Sun, Jun 13, 2021 at 8:12 AM Dongjin Lee  wrote:
>>
>> > Hi Ismael,
>> >
>> > > Can't we work with the log4j community to support the alternative
>> format?
>> >
>> > It seems not.
>> >
>> > 1. Changing the format of current log4j2's Kafka Appender implementation
>> > means causing inconvenience to the existing users by changing API
>> > semantics. (In my opinion, the log4j community did not take the
>> > compatibility with the log4j-appender into account when they developed
>> this
>> > module.)
>> >
>> > 2. Providing a new implementation (which is compatible with
>> log4j-appender)
>> > alongside with the existing one means there will be two APIs with
>> similar
>> > goals and functionality under the same artifact coordinate of log4j.
>> This
>> > confuses the users.
>> >
>> > Either of the approaches above is not feasible to the log4j community.
>> >
>> > Moreover, this approach does not resolve the classpath problem; To
>> entirely
>> > remove the log4j 1.x artifact from the classpath, the log4j2 appender
>> > should be released first, since VerifiableLog4jAppender uses it. This
>> means
>> > that the release of Kafka 3.0 depends upon when the log4j community
>> > releases the appender - it is also not feasible for us.
>> >
>> > Thanks,
>> > Dongjin
>> >
>> > On Sun, Jun 13, 2021 at 12:20 AM Ismael Juma  wrote:
>> >
>> > > Can't we work with the log4j community to support the alternative
>> format?
>> > >
>> > > Ismael
>> > >
>> > > On Fri, Jun 11, 2021, 10:54 PM Dongjin Lee 
>> wrote:
>> > >
>> > > > Hi all,
>> > > >
>> > > > @Ismael
>> > > >
>> > > > As I stated in the KIP (see subsection 2 of 'Motivation'), the log4j
>> > > > community's implementation can't be an alternative for the existing
>> > > > 'log4j-appender' users since their message format is different,
>> > breaking
>> > > > the related, already running pipelines.
>> > > >
>> > > > Add to this, the log4j appender can be best maintained when log4j2
>> and
>> > > > Kafka versions are updated together. (see subsection 1 of
>> > 'Motivation'.)
>> > > >
>> > > > @Israel
>> > > >
>> > > > Then, you mean instead of creating a new artifact
>> ('log4j2-appender'),
>> > > just
>> > > > substituting the traditional artifact ('log4j-appender') with a new
>> > > > implementation would be better. Do I understand correctly?
>> > > >
>> > > > After all, one main reason I hurried this proposal is that for the
>> > > > VerifiableLog4jAppender tool, we can't entirely remove log4j 1.x
>> > artifact
>> > > > from the classpath - making classpath logic more complex. (see here
>> > > > <
>> > > >
>> > >
>> >
>> https://github.com/apache/kafka/pull/7898/commits/f56f491e68ef2a976c0e3331a48dd881b74a06b3
>> > > > >
>> > > > for KIP-653
>> > > > <
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-653%3A+Upgrade+log4j+to+log4j2
>> > > > >
>> > > > and here
>> > > > <
>> > > >
>> > >
>> >
>> https://github.com/apache/kafka/pull/10244/commits/b66fce3d04e005b1eaeae006d78bd8e698f417c6
>> > > > >
>> > > > for KIP-719
>> > > > <
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-719%3A+Add+Log4J2+Appender
>> > > > >
>> > > > .)
>> > > >
>> > > > @Boojapho, @Konstantine
>> > > >
>> > > > Thanks for the voting. Currently:
>> > > >
>> > > > - binding: +1 (Konstantine)
>> > > > - non-binding: +1 (Boojapho)
>> > > >
>> > > > Regards,
>> > > > Dongjin
>> > > >
>> > > > On Sat, Jun 12, 2021 at 10:29 AM Israel Ekpo 
>> > > wrote:
>> > > >
>> > >

Build failed in Jenkins: Kafka » kafka-2.6-jdk8 #137

2021-10-12 Thread Apache Jenkins Server

See 


Changes:


--
[...truncated 3.18 MB...]

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest >

[jira] [Created] (KAFKA-13371) Consider consolidating Joined / StreamJoined / TableJoined

2021-10-12 Thread Guozhang Wang (Jira)

Guozhang Wang created KAFKA-13371:
-

 Summary: Consider consolidating Joined / StreamJoined / TableJoined
 Key: KAFKA-13371
 URL: https://issues.apache.org/jira/browse/KAFKA-13371
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Guozhang Wang


This is an idea while reviewing KAFKA-13261 (adding TabledJoined). We have now 
three control objects: Joined, StreamJoined, TableJoined. All of them extends 
NamedOperations and hence has the `name` field inherited which would be used 
for the processor node's name and potentially store names. In addition to that

* Joined: used in stream-table joins. Contains key and two value serdes used 
for serializing the bytes for repartitioning (however since today we only 
repartition one side if needed, the other value serde is never used). 
* StreamJoined: used in stream-stream joins. It includes the serdes, AND also 
the store suppliers and other control variables on the store names.
* TableJoined: used in table-table foreign key joins. It does not include any 
serdes but includes the partitioner information.

The main difference between these different constructs are:

* KTables themselves have embedded a materialized mechanism via 
`valueGetterSupplier` whenever they are created, either from source, or from 
aggregate / join operators, so they do not need extra materialization 
indicators when participated in a follow-up join --- i.e. they either are 
already materialized from the operators that generate them, or they will 
"grandfather" back to the upstream KTable on the fly with a logical view when 
that view is being fetched via the `ValueGetterSupplier`. On the other hand, 
KStreams do not have materialization mechanism inherently and hence operators 
that do need to materialize the streams then need to provide such methods.
* Table-table foreign-key join has a special needs for partitioners.

[~vvcephei] has a good proposal for 
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Streams+DSL+Grammar and 
as part of that proposal we could consider adding partitioner for source 
streams / tables and inherit throughout the topology pipeline. Following that 
idea, we can consider consolidating the above "Joined" objects by isolating the 
materialization / partitioner variables. More specifically, here's a concrete 
proposal:

1) `StreamsBuilder.table/stream` would pass in an optional partitioner.
2) And similarly all operators that changes the key would allow an optional 
partitioner:
2.a) `KStream.repartition/groupBy` and `KTable.groupBy` would allow an optional 
partitioner in `Repartitioned`, as piggy-backed we would also deprecate 
`Grouped` with `Repartitioned` since the latter would subsume the former.
2.b) `KStream.map/flatMap/selectKey` stays as is, and similar to serdes, these 
operators would stop the inheritance of partitioners of the upstream entities.
3) `Repartition` would also add the key/value serdes used for serializing for 
the repartition topics.
4) `KStream.join(KTable)` and `KStream.join(KStream)` would pass in an optional 
`Repartitioned` in addition to `Joined` which can be used to encode the 
partitioner info.
5) Foreign-key `KTable.join(KTable)` would pass in an optional `Repartitioned` 
which can be used to encode the partitioner info.
7) As a result of all above points, we can then reduce `StreamJoined` / 
`TableJoined` / `Joined` since all their enwrapped control objects are not 
separated in `Repartitioned` and `Materialized`: note that for `StreamJoined`, 
the store suppliers / names / configs would now be wrapped in two Materialized 
objects which would still not be exposed for IQ.







--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[VOTE] 2.7.2 RC0

2021-10-12 Thread Mickael Maison

Hello Kafka users, developers and client-developers,

This is the first candidate for release of Apache Kafka 2.7.2.

Apache Kafka 2.7.2 is a bugfix release and 26 issues, as well as
CVE-2021-38153, have been fixed since 2.7.1.

Release notes for the 2.7.2 release:
https://home.apache.org/~mimaison/kafka-2.7.2-rc0/RELEASE_NOTES.html

*** Please download, test and vote by Friday, October 15, 5pm CET

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~mimaison/kafka-2.7.2-rc0/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~mimaison/kafka-2.7.2-rc0/javadoc/

* Tag to be voted upon (off 2.7 branch) is the 2.7.2 tag:
https://github.com/apache/kafka/releases/tag/2.7.2-rc0

* Documentation:
https://kafka.apache.org/27/documentation.html

* Protocol:
https://kafka.apache.org/27/protocol.html

* Successful Jenkins builds for the 2.7 branch:
I'll share a link once the build completes


/**

Thanks,
Mickael

[jira] [Created] (KAFKA-13370) Offset commit failure percentage incorrect (regression)

2021-10-12 Thread Vincent Giroux (Jira)

Vincent Giroux created KAFKA-13370:
--

 Summary: Offset commit failure percentage incorrect (regression)
 Key: KAFKA-13370
 URL: https://issues.apache.org/jira/browse/KAFKA-13370
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect, metrics
Affects Versions: 2.8.0
 Environment: Confluent Platform Helm Chart (v6.2.0)
Reporter: Vincent Giroux
 Fix For: 2.8.0


There seems to have been a regression in the way the *offset-commit-** metrics 
are calculated for *source* Kafka Connect connectors since version 2.8.0.

Before this version, any timeout or interruption while trying to commit offsets 
for source connectors (e.g. MM2 MirrorSourceConnector) would get correctly 
flagged as an offset commit failure (i.e the *offset-commit-failure-percentage* 
metric ** would be non-zero). Since version 2.8.0, these errors are considered 
as successes.

After digging through the code, the commit where this bug was introduced 
appears to be this one : 
[https://github.com/apache/kafka/commit/047ad654da7903f3903760b0e6a6a58648ca7715]

I believe removing the boolean *success* argument in the *recordCommit* method 
of the *WorkerTask* class (argument deemed redundant because of the presence of 
the Throwable *error* argument) and only considering the presence of a non-null 
error to determine if a commit is a success or failure might be a mistake. This 
is because in the *commitOffsets* method of the *WorkerSourceTask* class, there 
are multiple cases where an exception object is either not available or is not 
passed to the *recordCommitFailure* method, e.g. :
 * *TImeout #1* : 
[https://github.com/apache/kafka/blob/2.8/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSourceTask.java#L519]
 
 * *Timeout #2* : 
[https://github.com/apache/kafka/blob/2.8/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSourceTask.java#L584]
 
 * *Interruption* : 
[https://github.com/apache/kafka/blob/2.8/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSourceTask.java#L529]
 
 * *Unserializable offset* : 
[https://github.com/apache/kafka/blob/2.8/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSourceTask.java#L562]
 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-778 KRaft Upgrades

2021-10-12 Thread Colin McCabe

On Thu, Oct 7, 2021, at 17:19, Jun Rao wrote:
> Hi, David,
>
> Thanks for the KIP. A few comments below.
>
> 10. It would be useful to describe how the controller node determines the
> RPC version used to communicate to other controller nodes. There seems to
> be a bootstrap problem. A controller node can't read the log and
> therefore the feature level until a quorum leader is elected. But leader
> election requires an RPC.
>

Hi Jun,

I agree that we need to figure this out. I think it would be best to simply use 
ApiVersionsRequest and ApiVersionsResponse. That way each controller can use 
the appropriate RPC versions when communicating with each other controller. 
This would allow us to upgrade them one by one.

> 11. For downgrades, it would be useful to describe how to determine the
> downgrade process (generating new snapshot, propagating the snapshot, etc)
> has completed. We could block the UpdateFeature request until the process
> is completed. However, since the process could take time, the request could
> time out. Another way is through DescribeFeature and the server only
> reports downgraded versions after the process is completed.

Hmm.. I think we need to avoid blocking, since we don't know how long it will 
take for all nodes to act on the downgrade request. After all, some nodes may 
be down.

But I agree we should have some way of knowing when the upgrade is done. 
DescribeClusterResponse seems like the natural place to put information about 
each node's feature level. While we're at it, we should also add a boolean to 
indicate whether the given node is fenced. (This will always be false for ZK 
mode, of course...)

>
> 12. Since we are changing UpdateFeaturesRequest, do we need to change the
> AdminClient api for updateFeatures too?
>
> 13. For the paragraph starting with "In the absence of an operator defined
> value for metadata.version", in KIP-584, we described how to finalize
> features with New cluster bootstrap. In that case, it's inconvenient for
> the users to have to run an admin tool to finalize the version for each
> feature. Instead, the system detects that the /features path is missing in
> ZK and thus automatically finalizes every feature with the latest supported
> version. Could we do something similar in the KRaft mode?
>

Yes, I think we need to have a section describing how this ties into creating 
new clusters. The simplest thing is probably to have the controller notice that 
there are no FeatureRecords at all, and then create a record for the latest 
metadata.version. This is analogous to how we assume the latest IBP if no IBP 
is configured.

There is also the question of how to create a cluster that starts up with 
something other than the latest metadata.version. We could have a config for 
that, like initial.metadata.version, or pass a flag to the controllers... 
alternately, we could pass a flag to "kafka-storage.sh format".

> 14. After the quorum leader generates a new snapshot, how do we force other
> nodes to pick up the new snapshot?
>
> 15. I agree with Jose that it will be useful to describe when generating a
> new snapshot is needed. To me, it seems the new snapshot is only needed
> when incompatible changes are made.
>

I think it would be good to always generate a snapshot right before the 
upgrade. Then, if the upgrade goes wrong, we have a metadata state we could 
revert back to, albeit with some effort and potential data loss. But, I agree 
that the rationale for this should be spelled out in the KIP.

I also think that the brokers should generate their own snapshots rather than 
fetching from the controller, both in the upgrade and downgrade case. Jose 
mentioned this earlier and I agree.

best,
Colin

> 7. Jose, what control records were you referring?
>
> Thanks,
>
> Jun
>
>
> On Tue, Oct 5, 2021 at 8:53 AM David Arthur  wrote:
>
>> Jose, thanks for the thorough review and comments!
>>
>> I am out of the office until next week, so I probably won't be able to
>> update the KIP until then. Here are some replies to your questions:
>>
>> 1. Generate snapshot on upgrade
>> > > Metadata snapshot is generated and sent to the other nodes
>> > Why does the Active Controller need to generate a new snapshot and
>> > force a snapshot fetch from the replicas (inactive controller and
>> > brokers) on an upgrade? Isn't writing the FeatureLevelRecord good
>> > enough to communicate the upgrade to the replicas?
>>
>>
>> You're right, we don't necessarily need to _transmit_ a snapshot, since
>> each node can generate its own equivalent snapshot
>>
>> 2. Generate snapshot on downgrade
>> > > Metadata snapshot is generated and sent to the other inactive
>> > controllers and to brokers (this snapshot may be lossy!)
>> > Why do we need to send this downgraded snapshot to the brokers? The
>> > replicas have seen the FeatureLevelRecord and noticed the downgrade.
>> > Can we have the replicas each independently generate a downgraded
>> > snapshot at the offset for

Build failed in Jenkins: Kafka » kafka-2.6-jdk8 #136

2021-10-12 Thread Apache Jenkins Server

See 


Changes:


--
[...truncated 6.35 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureInternalTopicNamesIfWrittenInto[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureInternalTopicNamesIfWrittenInto[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest >

Re: [DISCUSS] KIP-778 KRaft Upgrades

2021-10-12 Thread Jun Rao

Hi, David,

One more comment.

16. The main reason why KIP-584 requires finalizing a feature manually is
that in the ZK world, the controller doesn't know all brokers in a cluster.
A broker temporarily down is not registered in ZK. in the KRaft world, the
controller keeps track of all brokers, including those that are temporarily
down. This makes it possible for the controller to automatically finalize a
feature---it's safe to do so when all brokers support that feature. This
will make the upgrade process much simpler since no manual command is
required to turn on a new feature. Have we considered this?

Thanks,

Jun

On Thu, Oct 7, 2021 at 5:19 PM Jun Rao  wrote:

> Hi, David,
>
> Thanks for the KIP. A few comments below.
>
> 10. It would be useful to describe how the controller node determines the
> RPC version used to communicate to other controller nodes. There seems to
> be a bootstrap problem. A controller node can't read the log and
> therefore the feature level until a quorum leader is elected. But leader
> election requires an RPC.
>
> 11. For downgrades, it would be useful to describe how to determine the
> downgrade process (generating new snapshot, propagating the snapshot, etc)
> has completed. We could block the UpdateFeature request until the process
> is completed. However, since the process could take time, the request could
> time out. Another way is through DescribeFeature and the server only
> reports downgraded versions after the process is completed.
>
> 12. Since we are changing UpdateFeaturesRequest, do we need to change the
> AdminClient api for updateFeatures too?
>
> 13. For the paragraph starting with "In the absence of an operator
> defined value for metadata.version", in KIP-584, we described how to
> finalize features with New cluster bootstrap. In that case, it's
> inconvenient for the users to have to run an admin tool to finalize the
> version for each feature. Instead, the system detects that the /features
> path is missing in ZK and thus automatically finalizes every feature with
> the latest supported version. Could we do something similar in the KRaft
> mode?
>
> 14. After the quorum leader generates a new snapshot, how do we force
> other nodes to pick up the new snapshot?
>
> 15. I agree with Jose that it will be useful to describe when generating a
> new snapshot is needed. To me, it seems the new snapshot is only needed
> when incompatible changes are made.
>
> 7. Jose, what control records were you referring?
>
> Thanks,
>
> Jun
>
>
> On Tue, Oct 5, 2021 at 8:53 AM David Arthur 
> wrote:
>
>> Jose, thanks for the thorough review and comments!
>>
>> I am out of the office until next week, so I probably won't be able to
>> update the KIP until then. Here are some replies to your questions:
>>
>> 1. Generate snapshot on upgrade
>> > > Metadata snapshot is generated and sent to the other nodes
>> > Why does the Active Controller need to generate a new snapshot and
>> > force a snapshot fetch from the replicas (inactive controller and
>> > brokers) on an upgrade? Isn't writing the FeatureLevelRecord good
>> > enough to communicate the upgrade to the replicas?
>>
>>
>> You're right, we don't necessarily need to _transmit_ a snapshot, since
>> each node can generate its own equivalent snapshot
>>
>> 2. Generate snapshot on downgrade
>> > > Metadata snapshot is generated and sent to the other inactive
>> > controllers and to brokers (this snapshot may be lossy!)
>> > Why do we need to send this downgraded snapshot to the brokers? The
>> > replicas have seen the FeatureLevelRecord and noticed the downgrade.
>> > Can we have the replicas each independently generate a downgraded
>> > snapshot at the offset for the downgraded FeatureLevelRecord? I assume
>> > that the active controller will guarantee that all records after the
>> > FatureLevelRecord use the downgraded version. If so, it would be good
>> > to mention that explicitly.
>>
>>
>> Similar to above, yes a broker that detects a downgrade via
>> FeatureLevelRecord could generate its own downgrade snapshot and reload
>> its
>> state from that. This does get a little fuzzy when we consider cases where
>> brokers are on different software versions and could be generating a
>> downgrade snapshot for version X, but using different versions of the
>> code.
>> It might be safer to let the controller generate the snapshot so each
>> broker (regardless of software version) gets the same records. However,
>> for
>> upgrades (or downgrades) we expect the whole cluster to be running the
>> same
>> software version before triggering the metadata.version change, so perhaps
>> this isn't a likely scenario. Thoughts?
>>
>>
>> 3. Max metadata version
>> > >For the first release that supports metadata.version, we can simply
>> > initialize metadata.version with the current (and only) version. For
>> future
>> > releases, we will need a mechanism to bootstrap a particular version.
>> This
>> > could be done using the meta.properties

Build failed in Jenkins: Kafka » kafka-2.7-jdk8 #180

2021-10-12 Thread Apache Jenkins Server

See 


Changes:

[github] MINOR: Update LICENSE-binary (#11391)


--
[...truncated 3.45 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotRequireParameters[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotRequireParameters[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldCloseProcessor[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldCloseProcessor[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFeedStoreFromGlobalKTable[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFeedStoreFromGlobalKTable[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCleanUpPersistentStateStoresOnClose[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCleanUpPersistentStateStoresOnClose[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowPatternNotValidForTopicNameException[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowPatternNotValidForTopicNameException[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldEnqueueLaterOutputsAfterEarlierOnes[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldEnqueueLaterOutputsAfterEarlierOnes[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializersDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializersDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfEvenTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfEvenTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowNoSuchElementExceptionForUnusedOutputTopicWithDynamicRouting[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowNoSuchElementExceptionForUnusedOutputTopicWithDynamicRouting[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldInitProcessor[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldInitProcessor[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureGlobalTopicNameIfWrittenInto[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureGlobalTopicNameIfWrittenInto[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest >

[VOTE] 2.6.3 RC0

2021-10-12 Thread Mickael Maison

Hello Kafka users, developers and client-developers,

This is the first candidate for release of Apache Kafka 2.6.3.

Apache Kafka 2.6.3 is a bugfix release and 11 issues, as well as
CVE-2021-38153, have been fixed since 2.6.2.

Release notes for the 2.6.3 release:
https://home.apache.org/~mimaison/kafka-2.6.3-rc0/RELEASE_NOTES.html

*** Please download, test and vote by Friday, October 15, 5pm CET

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~mimaison/kafka-2.6.3-rc0/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~mimaison/kafka-2.6.3-rc0/javadoc/

* Tag to be voted upon (off 2.6 branch) is the 2.6.3 tag:
https://github.com/apache/kafka/releases/tag/2.6.3-rc0

* Documentation:
https://kafka.apache.org/26/documentation.html

* Protocol:
https://kafka.apache.org/26/protocol.html

* Successful Jenkins builds for the 2.6 branch:
I'll share a link once the build completes

/**

Thanks,
Mickael

Build failed in Jenkins: Kafka » kafka-2.6-jdk8 #135

2021-10-12 Thread Apache Jenkins Server

See 


Changes:

[github] MINOR: Update LICENSE-binary (#11389)


--
[...truncated 3.18 MB...]
org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest >

[jira] [Created] (KAFKA-13369) Follower fetch protocol enhancements for tiered storage.

2021-10-12 Thread Satish Duggana (Jira)

Satish Duggana created KAFKA-13369:
--

 Summary: Follower fetch protocol enhancements for tiered storage.
 Key: KAFKA-13369
 URL: https://issues.apache.org/jira/browse/KAFKA-13369
 Project: Kafka
  Issue Type: Sub-task
Reporter: Satish Duggana
Assignee: Satish Duggana






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Requesting Project Access

2021-10-12 Thread Bruno Cadonna


Hi,

You should be all set up, now!

Thank you for your interest in Apache Kafka!

Best,
Bruno

On 11.10.21 22:44, Mason Legere wrote:

Hi, I am requesting access in order to make contributions to Kafka. My Jira
and Wiki ID are both "masonlegere"

Best,

[jira] [Created] (KAFKA-13368) Support smart topic polling for consumer with multiple topic subscriptions

2021-10-12 Thread Pedro Cardoso Silva (Jira)

Pedro Cardoso Silva created KAFKA-13368:
---

 Summary: Support smart topic polling for consumer with multiple 
topic subscriptions
 Key: KAFKA-13368
 URL: https://issues.apache.org/jira/browse/KAFKA-13368
 Project: Kafka
  Issue Type: Wish
  Components: consumer
Reporter: Pedro Cardoso Silva


Currently there is no way to control how a Kafka consumer polls messages from a 
list of topics that it has subscribed to. If I understand correctly, the 
current approach is a round-robin polling mechanism across all topics that a 
consumer has subscribed to. 
This works reasonably well when the consumer's offset is aligned with the 
latest message offset of the topics, however if we configured the Kafka 
consumer to consume from the earliest offset where the topics have very 
distinct amounts of messages each, there is no guarantee/control on how to 
selectively read from topics.

Depending on the use-case it may be useful for the Kafka consumer developer to 
override this polling mechanism with a custom solution that makes sense for 
downstream applications.

Suppose you have 2 or more topics, where you want to merge the topics into a 
single topic but due to large differences between the topic's message rates you 
want to control from which topics to poll at a given time. 

As an example consider 2 topics with the following schemas:

{code:java}
Topic1 Schema: {
   timestamp: Long,
   key: String,
   col1: String,
   col2: String
}

Topic2 Schema: { 
   timestamp: Long,
   key: String,
   col3: String,
   col4: String 
}
{code}

Where Topic1 has 1,000,000 events from timestamp 0 to 1,000 (1000 ev/s) & 
topic2 has 50,000 events from timestamp 0 to 50,000 (1 ev/s).

Next we define a Kafka consumer that subscribes to Topic1 & Topic2. In the 
current situation (round robin), assuming a polling batch of 100 messages,  we 
would read 50,000 from each topic which maps to 50 seconds worth of messages on 
Topic1 and 50,000 seconds worth of messages on Topic2. 

If we then try to sort the messages by timestamp we have incorrect results, 
missing 500,000 messages from Topic1 that should be inserted between message 0 
& 1,000 of Topic2.

The workaround solution is either to buffer the messages from Topic2 of have 1 
Kafka consumer per topic which has significant overhead with periodic 
heartbeats, consumer registration in consumer groups, re-balancing, etc... 
For a couple of topics this approach may be OK, but it does not scale for 10's, 
100's or more topics in a subscription.

The ideal solution would be to extend the Kafka consumer API to allow a user to 
define how to selectively poll messages from a subscription.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-13367) Performance Degradation during introducing Network Delay

2021-10-12 Thread Thomas Heinze (Jira)

Thomas Heinze created KAFKA-13367:
-

 Summary: Performance Degradation during introducing Network Delay
 Key: KAFKA-13367
 URL: https://issues.apache.org/jira/browse/KAFKA-13367
 Project: Kafka
  Issue Type: Bug
 Environment: We are running Kafka 2.5 on m4.xlarge VMs on AWS.
Reporter: Thomas Heinze


Hi Kafka community,

 

we are running a few chaos experiments to simulate Kafka's behaviour during 
issues in the data center. To simulate a slow network we run the following 
command on two out of six brokers (the brokers are spread across 3 AZs on AWS, 
we run the command on two brokers in the same AZ):
{code:java}
tc qdisc add dev eth0 root netem delay x ms 
 {code}
 
 At the same time we are running some Kafka producers inserting roughly 4k 
messages per second to a Kafka topic with 10 partitions with 3 replicas and 
using min-isr=2. What we observe is the following:
 * *Introducing a 1000 ms delay*: The producer see significant response time 
delays, the throughput drops to 2k per second
 * *Introducing a 2000 ms delay*: The producer delay increases further, the 
throughput drops to 300 messages per second
 * *Introducing a 5000 ms delay*: The Kafka clusters remove the slow brokers 
from the list of active replicas and the incoming messages for the remaining 
brokers increases. This is the expected behaviour imho.

What parameters would influence this behaviour? How can I make sure Kafka shows 
the behaviour like for 5 seconds even for smaller delays? We would like to make 
sure that we can guarantee around a certain throughput, even if one AZ is very 
slow.

I already tried to set "replica.lag.time.max.ms" to very small values, but I 
only observe that Kafka adds and remove the replicas on the slow nodes 
constantly from the set of ISR.

 

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-13366) JMX metric for leader.imbalance.per.broker.percentage value

2021-10-12 Thread Giovanni Marigi (Jira)

Giovanni Marigi created KAFKA-13366:
---

 Summary: JMX metric for leader.imbalance.per.broker.percentage 
value
 Key: KAFKA-13366
 URL: https://issues.apache.org/jira/browse/KAFKA-13366
 Project: Kafka
  Issue Type: Improvement
  Components: metrics
Affects Versions: 3.0.0
Reporter: Giovanni Marigi


The value of leader.imbalance.per.broker.percentage is calculated by Controller

It could be useful to expose the current value as JMX metric.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-13372) failed authentication due to: SSL handshake failed

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #520

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #519

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #518

Re: [kafka-clients] [VOTE] 2.7.2 RC0

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #517

Re: [DISCUSS] KIP-778 KRaft Upgrades

Re: [DISCUSS] KIP-778 KRaft Upgrades

[jira] [Resolved] (KAFKA-13319) Do not send AddOffsetsToTxn/TxnOffsetCommit if offsets map is empty

Re: [VOTE] KIP-752: Support --bootstrap-server in ReplicaVerificationTool

Re: [VOTE] KIP-719: Add Log4J2 Appender

Build failed in Jenkins: Kafka » kafka-2.6-jdk8 #137

[jira] [Created] (KAFKA-13371) Consider consolidating Joined / StreamJoined / TableJoined

[VOTE] 2.7.2 RC0

[jira] [Created] (KAFKA-13370) Offset commit failure percentage incorrect (regression)

Re: [DISCUSS] KIP-778 KRaft Upgrades

Build failed in Jenkins: Kafka » kafka-2.6-jdk8 #136

Re: [DISCUSS] KIP-778 KRaft Upgrades

Build failed in Jenkins: Kafka » kafka-2.7-jdk8 #180

[VOTE] 2.6.3 RC0

Build failed in Jenkins: Kafka » kafka-2.6-jdk8 #135

[jira] [Created] (KAFKA-13369) Follower fetch protocol enhancements for tiered storage.

Re: Requesting Project Access

[jira] [Created] (KAFKA-13368) Support smart topic polling for consumer with multiple topic subscriptions

[jira] [Created] (KAFKA-13367) Performance Degradation during introducing Network Delay

[jira] [Created] (KAFKA-13366) JMX metric for leader.imbalance.per.broker.percentage value

26 matches

Site Navigation

Mail list logo

Footer information