*> Hi Jakub,> > Thanks for trying the RC. I think what you found is a
blocker bug because it *
*> will generate huge amount of logspam. I guess we didn't find it in
junit
tests *
*> since logspam doesn't fail the automated tests. But certainly it's not
suitable *
*> for production. Did you file a JIRA yet?*
Hi Colin,
I opened https://issues.apache.org/jira/browse/KAFKA-16131.
Thanks & Regards
Jakub
On Mon, Jan 15, 2024 at 8:57 AM Colin McCabe <cmcc...@apache.org> wrote:
Hi Stanislav,
Thanks for making the first RC. The fact that it's titled RC2 is messing
with my mind a bit. I hope this doesn't make people think that we're
farther along than we are, heh.
On Sun, Jan 14, 2024, at 13:54, Jakub Scholz wrote:
*> Nice catch! It does seem like we should have gated this behind the
metadata> version as KIP-858 implies. Is the cluster configured with
multiple log> dirs? What is the impact of the error messages?*
I did not observe any obvious impact. I was able to send and receive
messages as normally. But to be honest, I have no idea what else
this might impact, so I did not try anything special.
I think everyone upgrading an existing KRaft cluster will go through
this
stage (running Kafka 3.7 with an older metadata version for at least a
while). So even if it is just a logged exception without any other
impact I
wonder if it might scare users from upgrading. But I leave it to
others
to
decide if this is a blocker or not.
Hi Jakub,
Thanks for trying the RC. I think what you found is a blocker bug
because
it will generate huge amount of logspam. I guess we didn't find it in
junit
tests since logspam doesn't fail the automated tests. But certainly it's
not suitable for production. Did you file a JIRA yet?
On Sun, Jan 14, 2024 at 10:17 PM Stanislav Kozlovski
<stanis...@confluent.io.invalid> wrote:
Hey Luke,
This is an interesting problem. Given the fact that the KIP for
having a
3.8 release passed, I think it weights the scale towards not calling
this a
blocker and expecting it to be solved in 3.7.1.
It is unfortunate that it would not seem safe to migrate to KRaft in
3.7.0
(given the inability to rollback safely), but if that's true - the
same
case would apply for 3.6.0. So in any case users w\ould be expected
to
use a
patch release for this.
Hi Luke,
Thanks for testing rollback. I think this is a case where the
documentation is wrong. The intention was to for the steps to basically
be:
1. roll all the brokers into zk mode, but with migration enabled
2. take down the kraft quorum
3. rmr /controller, allowing a hybrid broker to take over.
4. roll all the brokers into zk mode without migration enabled (if
desired)
With these steps, there isn't really unavailability since a ZK
controller
can be elected quickly after the kraft quorum is gone.
Further, since we will have a 3.8 release - it is
likely we will ultimately recommend users upgrade from that version
given
its aim is to have strategic KRaft feature parity with ZK.
That being said, I am not 100% on this. Let me know whether you think
this
should block the release, Luke. I am also tagging Colin and David to
weigh
in with their opinions, as they worked on the migration logic.
The rollback docs are new in 3.7 so the fact that they're wrong is a
clear
blocker, I think. But easy to fix, I believe. I will create a PR.
best,
Colin
Hey Kirk and Chris,
Unless I'm missing something - KAFKALESS-16029 is simply a bad log
due
to
improper closing. And the PR description implies this has been
present
since 3.5. While annoying, I don't see a strong reason for this to
block
the release.
Hey Jakub,
Nice catch! It does seem like we should have gated this behind the
metadata
version as KIP-858 implies. Is the cluster configured with multiple
log
dirs? What is the impact of the error messages?
Tagging Igor (the author of the KIP) to weigh in.
Best,
Stanislav
On Sat, Jan 13, 2024 at 7:22 PM Jakub Scholz <ja...@scholz.cz>
wrote:
Hi,
I was trying the RC2 and run into the following issue ... when I
run
3.7.0-RC2 KRaft cluster with metadata version set to 3.6-IV2
metadata
version, I seem to be getting repeated errors like this in the
controller
logs:
2024-01-13 16:58:01,197 INFO [QuorumController id=0]
assignReplicasToDirs:
event failed with UnsupportedVersionException in 15 microseconds.
(org.apache.kafka.controller.QuorumController)
[quorum-controller-0-event-handler]
2024-01-13 16:58:01,197 ERROR [ControllerApis nodeId=0] Unexpected
error
handling request RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
apiVersion=0, clientId=1000, correlationId=14, headerVersion=2) --
AssignReplicasToDirsRequestData(brokerId=1000, brokerEpoch=5,
directories=[DirectoryData(id=w_uxN7pwQ6eXSMrOKceYIQ,
topics=[TopicData(topicId=bvAKLSwmR7iJoKv2yZgygQ,
partitions=[PartitionData(partitionIndex=2),
PartitionData(partitionIndex=1)]),
TopicData(topicId=uNe7f5VrQgO0zST6yH1jDQ,
partitions=[PartitionData(partitionIndex=0)])])]) with context
RequestContext(header=RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
apiVersion=0, clientId=1000, correlationId=14, headerVersion=2),
connectionId='172.16.14.219:9090-172.16.14.217:53590-7',
clientAddress=/
172.16.14.217, principal=User:CN=my-cluster-kafka,O=io.strimzi,
listenerName=ListenerName(CONTROLPLANE-9090), securityProtocol=SSL,
clientInformation=ClientInformation(softwareName=apache-kafka-java,
softwareVersion=3.7.0), fromPrivilegedListener=false,
principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@71004ad2
])
(kafka.server.ControllerApis) [quorum-controller-0-event-handler]
java.util.concurrent.CompletionException:
org.apache.kafka.common.errors.UnsupportedVersionException:
Directory
assignment is not supported yet.
at
java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
at
java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
at
java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at
java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
at
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:880)
at
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:871)
at
org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
at
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
at
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
at
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by:
org.apache.kafka.common.errors.UnsupportedVersionException:
Directory assignment is not supported yet.
Is that expected? I guess with the metadata version set to
3.6-IV2, it
makes sense that the request is not supported. But shouldn't then
the
request not be sent at all by the brokers? (I did not opened a JIRA
for
it,
but I can open one if you agree this is not expected)
Thanks & Regards
Jakub
On Sat, Jan 13, 2024 at 8:03 AM Luke Chen <show...@gmail.com>
wrote:
Hi Stanislav,
I commented in the "Apache Kafka 3.7.0 Release" thread, but maybe
you
missed it.
cross-posting here:
There is a bug KAFKA-16101
<https://issues.apache.org/jira/browse/KAFKA-16101> reporting
that
"Kafka
cluster will be unavailable during KRaft migration rollback".
The impact for this issue is that if brokers try to rollback to
ZK
mode
during KRaft migration process, there will be a period of time
the
cluster
is unavailable.
Since ZK migrating to KRaft feature is a production ready
feature, I
think
this should be addressed soon.
Do you think this is a blocker for v3.7.0?
Thanks.
Luke
On Sat, Jan 13, 2024 at 8:36 AM Chris Egerton <
fearthecel...@gmail.com
wrote:
Thanks, Kirk!
@Stanislav--do you believe that this warrants a new RC?
On Fri, Jan 12, 2024, 19:08 Kirk True <k...@kirktrue.pro>
wrote:
Hi Chris/Stanislav,
I'm working on the 'Unable to find FetchSessionHandler' log
problem
(KAFKA-16029) and have put out a draft PR (
https://github.com/apache/kafka/pull/15186). I will use the
quickstart
approach as a second means to reproduce/verify while I wait
for
the
PR's
Jenkins job to finish.
Thanks,
Kirk
On Fri, Jan 12, 2024, at 11:31 AM, Chris Egerton wrote:
Hi Stanislav,
Thanks for running this release!
To verify, I:
- Built from source using Java 11 with both:
- - the 3.7.0-rc2 tag on GitHub
- - the kafka-3.7.0-src.tgz artifact from
https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
- Checked signatures and checksums
- Ran the quickstart using both:
- - The kafka_2.13-3.7.0.tgz artifact from
https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
with
Java
11
and Scala 13 in KRaft mode
- - Our shiny new broker Docker image,
apache/kafka:3.7.0-rc2
- Ran all unit tests
- Ran all integration tests for Connect and MM2
I found two minor areas for concern:
1. (Possibly a blocker)
When running the quickstart, I noticed this ERROR-level log
message
being
emitted frequently (not not every time) when I killed my
console
consumer
via ctrl-C:
[2024-01-12 11:00:31,088] ERROR [Consumer
clientId=console-consumer,
groupId=console-consumer-74388] Unable to find
FetchSessionHandler
for
node
1. Ignoring fetch response
(org.apache.kafka.clients.consumer.internals.AbstractFetch)
I see that this error message is already reported in
https://issues.apache.org/jira/browse/KAFKA-16029. I
think we
should
prioritize fixing it for this release. I know it's probably
benign
but
it's
really not a good look for us when basic operations log
error
messages,
and
it may give new users some headaches.
2. (Probably not a blocker)
The following unit tests failed the first time around, and
all of
them
passed the second time I ran them:
- (clients)
ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup()
- (clients) SelectorTest.testConnectionsByClientMetric()
- (clients)
Tls13SelectorTest.testConnectionsByClientMetric()
- (connect)
TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound
(I
thought I fixed this one! 🤬🤬)
- (core)
ProducerIdManagerTest.testUnrecoverableErrors(Errors)[2]
Thanks again for your work on this release, and
congratulations
to
Kafka
Streams for having zero flaky unit tests during my
highly-experimental
single laptop run!
Cheers,
Chris
On Thu, Jan 11, 2024 at 1:33 PM Stanislav Kozlovski
<stanis...@confluent.io.invalid> wrote:
Hello Kafka users, developers, and client-developers,
This is the first candidate for release of Apache Kafka
3.7.0.
Note it's named "RC2" because I had a few "failed" RCs
that
I
had
cut/uploaded but ultimately had to scrap prior to
announcing
due
to
new
blockers arriving before I could even announce them.
Further - I haven't yet been able to set up the system
tests
successfully.
And the integration/unit tests do have a few failures
that I
have
to
spend
time triaging. I would appreciate any help in case anyone
notices
any
tests
failing that they're subject matters experts in. Expect
me
to
follow
up in
a day or two with more detailed analysis.
Major changes include:
- Early Access to KIP-848 - the next generation of the
consumer
rebalance
protocol
- KIP-858: Adding JBOD support to KRaft
- KIP-714: Observability into Client metrics via a
standardized
interface
Check more information in the WIP blog post:
https://github.com/apache/kafka-site/pull/578
Release notes for the 3.7.0 release:
https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/RELEASE_NOTES.html
*** Please download, test and vote by Thursday, January
18,
9am
PT
***
Usually these deadlines tend to be 2-3 days, but due to
this
being
the
first RC and the tests not having ran yet, I am giving
it a
bit
more
time.
Kafka's KEYS file containing PGP keys we use to sign the
release:
https://kafka.apache.org/KEYS
* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
* Docker release artifact to be voted upon:
apache/kafka:3.7.0-rc2
* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/
* Javadoc:
https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/javadoc/
* Tag to be voted upon (off 3.7 branch) is the 3.7.0 tag:
https://github.com/apache/kafka/releases/tag/3.7.0-rc2
* Documentation:
https://kafka.apache.org/37/documentation.html
* Protocol:
https://kafka.apache.org/37/protocol.html
* Successful Jenkins builds for the 3.7 branch:
Unit/integration tests:
https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/
There are failing tests here. I have to follow up with
triaging
some
of
the failures and figuring out if they're actual problems
or
simply
flakes.
System tests:
https://jenkins.confluent.io/job/system-test-kafka/job/3.7/
No successful system test runs yet. I am working on
getting
the
job
to
run.
* Successful Docker Image Github Actions Pipeline for 3.7
branch:
Attached are the scan_report and report_jvm output files
from
the
Docker
Build run:
https://github.com/apache/kafka/actions/runs/7486094960/job/20375761673
And the final docker image build job - Docker Build Test
Pipeline:
https://github.com/apache/kafka/actions/runs/7486178277
The image is apache/kafka:3.7.0-rc2 -
https://hub.docker.com/layers/apache/kafka/3.7.0-rc2/images/sha256-5b4707c08170d39549fbb6e2a3dbb83936a50f987c0c097f23cb26b4c210c226?context=explore
/**************************************
Thanks,
Stanislav Kozlovski
--
Best,
Stanislav
