Apologies, I duplicated KAFKA-16157 twice in my previous message. I intended to mention KAFKA-16195 with the PR at https://github.com/apache/kafka/pull/15262 as the second JIRA.
Thanks, Gaurav > On 26 Jan 2024, at 15:34, ka...@gnarula.com wrote: > > Hi Stan, > > I wanted to share some updates about the bugs you shared earlier. > > - KAFKA-14616: I've reviewed and tested the PR from Colin and have observed > the fix works as intended. > - KAFKA-16162: I reviewed Proven's PR and found some gaps in the proposed > fix. I've > therefore raised https://github.com/apache/kafka/pull/15270 following a > discussion with Luke in JIRA. > - KAFKA-16082: I don't think this is marked as a blocker anymore. I'm awaiting > feedback/reviews at https://github.com/apache/kafka/pull/15136 > > In addition to the above, there are 2 JIRAs I'd like to bring everyone's > attention to: > > - KAFKA-16157: This is similar to KAFKA-14616 and is marked as a blocker. > I've raised > https://github.com/apache/kafka/pull/15263 and am awaiting reviews on it. > - KAFKA-16157: I raised this yesterday and have addressed feedback from Luke. > This should > hopefully get merged soon. > > Regards, > Gaurav > > >> On 24 Jan 2024, at 11:51, ka...@gnarula.com wrote: >> >> Hi Stanislav, >> >> Thanks for bringing these JIRAs/PRs up. >> >> I'll be testing the open PRs for KAFKA-14616 and KAFKA-16162 this week and I >> hope to have some feedback >> by Friday. I gather the latter JIRA is marked as a WIP by Proven and he's >> away. I'll try to build on his work in the meantime. >> >> As for KAFKA-16082, we haven't been able to deduce a data loss scenario. >> There's a PR open >> by me for promoting an abandoned future replica with approvals from Omnia >> and Proven, >> so I'd appreciate a committer reviewing it. >> >> Regards, >> Gaurav >> >> On 23 Jan 2024, at 20:17, Stanislav Kozlovski >> <stanis...@confluent.io.INVALID> wrote: >>> >>> Hey all, I figured I'd give an update about what known blockers we have >>> right now: >>> >>> - KAFKA-16101: KRaft migration rollback documentation is incorrect - >>> https://github.com/apache/kafka/pull/15193; This need not block RC >>> creation, but we need the docs updated so that people can test properly >>> - KAFKA-14616: Topic recreation with offline broker causes permanent URPs - >>> https://github.com/apache/kafka/pull/15230 ; I am of the understanding that >>> this is blocking JBOD for 3.7 >>> - KAFKA-16162: New created topics are unavailable after upgrading to 3.7 - >>> a strict blocker with an open PR https://github.com/apache/kafka/pull/15232 >>> - although I understand Proveen is out of office >>> - KAFKA-16082: JBOD: Possible dataloss when moving leader partition - I am >>> hearing mixed opinions on whether this is a blocker ( >>> https://github.com/apache/kafka/pull/15136) >>> >>> Given that there are 3 JBOD blocker bugs, and I am not confident they will >>> all be merged this week - I am on the edge of voting to revert JBOD from >>> this release, or mark it early access. >>> >>> By all accounts, it seems that if we keep with JBOD the release will have >>> to spill into February, which is a month extra from the time-based release >>> plan we had of start of January. >>> >>> Can I ask others for an opinion? >>> >>> Best, >>> Stan >>> >>> On Thu, Jan 18, 2024 at 1:21 PM Luke Chen <show...@gmail.com> wrote: >>> >>>> Hi all, >>>> >>>> I think I've found another blocker issue: KAFKA-16162 >>>> <https://issues.apache.org/jira/browse/KAFKA-16162> . >>>> The impact is after upgrading to 3.7.0, any new created topics/partitions >>>> will be unavailable. >>>> I've put my findings in the JIRA. >>>> >>>> Thanks. >>>> Luke >>>> >>>> On Thu, Jan 18, 2024 at 9:50 AM Matthias J. Sax <mj...@apache.org> wrote: >>>> >>>>> Stan, thanks for driving this all forward! Excellent job. >>>>> >>>>> About >>>>> >>>>>> StreamsStandbyTask - https://issues.apache.org/jira/browse/KAFKA-16141 >>>>>> StreamsUpgradeTest - https://issues.apache.org/jira/browse/KAFKA-16139 >>>>> >>>>> For `StreamsUpgradeTest` it was a test setup issue and should be fixed >>>>> now in trunk and 3.7 (and actually also in 3.6...) >>>>> >>>>> For `StreamsStandbyTask` the failing test exposes a regression bug, so >>>>> it's a blocker. I updated the ticket accordingly. We already have an >>>>> open PR that reverts the code introducing the regression. >>>>> >>>>> >>>>> -Matthias >>>>> >>>>> On 1/17/24 9:44 AM, Proven Provenzano wrote: >>>>>> We have another blocking issue for the RC : >>>>>> https://issues.apache.org/jira/browse/KAFKA-16157. This bug is similar >>>>> to >>>>>> https://issues.apache.org/jira/browse/KAFKA-14616. The new issue >>>> however >>>>>> can lead to the new topic having partitions that a producer cannot >>>> write >>>>> to. >>>>>> >>>>>> --Proven >>>>>> >>>>>> On Tue, Jan 16, 2024 at 12:04 PM Proven Provenzano < >>>>> pprovenz...@confluent.io> >>>>>> wrote: >>>>>> >>>>>>> >>>>>>> I have a PR https://github.com/apache/kafka/pull/15197 for >>>>>>> https://issues.apache.org/jira/browse/KAFKA-16131 that is building >>>> now. >>>>>>> --Proven >>>>>>> >>>>>>> On Mon, Jan 15, 2024 at 5:03 AM Jakub Scholz <ja...@scholz.cz> wrote: >>>>>>> >>>>>>>> *> Hi Jakub,> > Thanks for trying the RC. I think what you found is a >>>>>>>> blocker bug because it * >>>>>>>> *> will generate huge amount of logspam. I guess we didn't find it in >>>>>>>> junit >>>>>>>> tests * >>>>>>>> *> since logspam doesn't fail the automated tests. But certainly it's >>>>> not >>>>>>>> suitable * >>>>>>>> *> for production. Did you file a JIRA yet?* >>>>>>>> >>>>>>>> Hi Colin, >>>>>>>> >>>>>>>> I opened https://issues.apache.org/jira/browse/KAFKA-16131. >>>>>>>> >>>>>>>> Thanks & Regards >>>>>>>> Jakub >>>>>>>> >>>>>>>> On Mon, Jan 15, 2024 at 8:57 AM Colin McCabe <cmcc...@apache.org> >>>>> wrote: >>>>>>>> >>>>>>>>> Hi Stanislav, >>>>>>>>> >>>>>>>>> Thanks for making the first RC. The fact that it's titled RC2 is >>>>> messing >>>>>>>>> with my mind a bit. I hope this doesn't make people think that we're >>>>>>>>> farther along than we are, heh. >>>>>>>>> >>>>>>>>> On Sun, Jan 14, 2024, at 13:54, Jakub Scholz wrote: >>>>>>>>>> *> Nice catch! It does seem like we should have gated this behind >>>> the >>>>>>>>>> metadata> version as KIP-858 implies. Is the cluster configured >>>> with >>>>>>>>>> multiple log> dirs? What is the impact of the error messages?* >>>>>>>>>> >>>>>>>>>> I did not observe any obvious impact. I was able to send and >>>> receive >>>>>>>>>> messages as normally. But to be honest, I have no idea what else >>>>>>>>>> this might impact, so I did not try anything special. >>>>>>>>>> >>>>>>>>>> I think everyone upgrading an existing KRaft cluster will go >>>> through >>>>>>>> this >>>>>>>>>> stage (running Kafka 3.7 with an older metadata version for at >>>> least >>>>> a >>>>>>>>>> while). So even if it is just a logged exception without any other >>>>>>>>> impact I >>>>>>>>>> wonder if it might scare users from upgrading. But I leave it to >>>>>>>> others >>>>>>>>> to >>>>>>>>>> decide if this is a blocker or not. >>>>>>>>>> >>>>>>>>> >>>>>>>>> Hi Jakub, >>>>>>>>> >>>>>>>>> Thanks for trying the RC. I think what you found is a blocker bug >>>>>>>> because >>>>>>>>> it will generate huge amount of logspam. I guess we didn't find it >>>> in >>>>>>>> junit >>>>>>>>> tests since logspam doesn't fail the automated tests. But certainly >>>>> it's >>>>>>>>> not suitable for production. Did you file a JIRA yet? >>>>>>>>> >>>>>>>>>> On Sun, Jan 14, 2024 at 10:17 PM Stanislav Kozlovski >>>>>>>>>> <stanis...@confluent.io.invalid> wrote: >>>>>>>>>> >>>>>>>>>>> Hey Luke, >>>>>>>>>>> >>>>>>>>>>> This is an interesting problem. Given the fact that the KIP for >>>>>>>> having a >>>>>>>>>>> 3.8 release passed, I think it weights the scale towards not >>>> calling >>>>>>>>> this a >>>>>>>>>>> blocker and expecting it to be solved in 3.7.1. >>>>>>>>>>> >>>>>>>>>>> It is unfortunate that it would not seem safe to migrate to KRaft >>>> in >>>>>>>>> 3.7.0 >>>>>>>>>>> (given the inability to rollback safely), but if that's true - the >>>>>>>> same >>>>>>>>>>> case would apply for 3.6.0. So in any case users w\ould be >>>> expected >>>>>>>> to >>>>>>>>> use a >>>>>>>>>>> patch release for this. >>>>>>>>> >>>>>>>>> Hi Luke, >>>>>>>>> >>>>>>>>> Thanks for testing rollback. I think this is a case where the >>>>>>>>> documentation is wrong. The intention was to for the steps to >>>>> basically >>>>>>>> be: >>>>>>>>> >>>>>>>>> 1. roll all the brokers into zk mode, but with migration enabled >>>>>>>>> 2. take down the kraft quorum >>>>>>>>> 3. rmr /controller, allowing a hybrid broker to take over. >>>>>>>>> 4. roll all the brokers into zk mode without migration enabled (if >>>>>>>> desired) >>>>>>>>> >>>>>>>>> With these steps, there isn't really unavailability since a ZK >>>>>>>> controller >>>>>>>>> can be elected quickly after the kraft quorum is gone. >>>>>>>>> >>>>>>>>>>> Further, since we will have a 3.8 release - it is >>>>>>>>>>> likely we will ultimately recommend users upgrade from that >>>> version >>>>>>>>> given >>>>>>>>>>> its aim is to have strategic KRaft feature parity with ZK. >>>>>>>>>>> That being said, I am not 100% on this. Let me know whether you >>>>> think >>>>>>>>> this >>>>>>>>>>> should block the release, Luke. I am also tagging Colin and David >>>> to >>>>>>>>> weigh >>>>>>>>>>> in with their opinions, as they worked on the migration logic. >>>>>>>>> >>>>>>>>> The rollback docs are new in 3.7 so the fact that they're wrong is a >>>>>>>> clear >>>>>>>>> blocker, I think. But easy to fix, I believe. I will create a PR. >>>>>>>>> >>>>>>>>> best, >>>>>>>>> Colin >>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Hey Kirk and Chris, >>>>>>>>>>> >>>>>>>>>>> Unless I'm missing something - KAFKALESS-16029 is simply a bad log >>>>>>>> due >>>>>>>>> to >>>>>>>>>>> improper closing. And the PR description implies this has been >>>>>>>> present >>>>>>>>>>> since 3.5. While annoying, I don't see a strong reason for this to >>>>>>>> block >>>>>>>>>>> the release. >>>>>>>>>>> >>>>>>>>>>> Hey Jakub, >>>>>>>>>>> >>>>>>>>>>> Nice catch! It does seem like we should have gated this behind the >>>>>>>>> metadata >>>>>>>>>>> version as KIP-858 implies. Is the cluster configured with >>>> multiple >>>>>>>> log >>>>>>>>>>> dirs? What is the impact of the error messages? >>>>>>>>>>> >>>>>>>>>>> Tagging Igor (the author of the KIP) to weigh in. >>>>>>>>>>> >>>>>>>>>>> Best, >>>>>>>>>>> Stanislav >>>>>>>>>>> >>>>>>>>>>> On Sat, Jan 13, 2024 at 7:22 PM Jakub Scholz <ja...@scholz.cz> >>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> I was trying the RC2 and run into the following issue ... when I >>>>>>>> run >>>>>>>>>>>> 3.7.0-RC2 KRaft cluster with metadata version set to 3.6-IV2 >>>>>>>> metadata >>>>>>>>>>>> version, I seem to be getting repeated errors like this in the >>>>>>>>> controller >>>>>>>>>>>> logs: >>>>>>>>>>>> >>>>>>>>>>>> 2024-01-13 16:58:01,197 INFO [QuorumController id=0] >>>>>>>>>>> assignReplicasToDirs: >>>>>>>>>>>> event failed with UnsupportedVersionException in 15 microseconds. >>>>>>>>>>>> (org.apache.kafka.controller.QuorumController) >>>>>>>>>>>> [quorum-controller-0-event-handler] >>>>>>>>>>>> 2024-01-13 16:58:01,197 ERROR [ControllerApis nodeId=0] >>>> Unexpected >>>>>>>>> error >>>>>>>>>>>> handling request RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS, >>>>>>>>>>>> apiVersion=0, clientId=1000, correlationId=14, headerVersion=2) >>>> -- >>>>>>>>>>>> AssignReplicasToDirsRequestData(brokerId=1000, brokerEpoch=5, >>>>>>>>>>>> directories=[DirectoryData(id=w_uxN7pwQ6eXSMrOKceYIQ, >>>>>>>>>>>> topics=[TopicData(topicId=bvAKLSwmR7iJoKv2yZgygQ, >>>>>>>>>>>> partitions=[PartitionData(partitionIndex=2), >>>>>>>>>>>> PartitionData(partitionIndex=1)]), >>>>>>>>>>>> TopicData(topicId=uNe7f5VrQgO0zST6yH1jDQ, >>>>>>>>>>>> partitions=[PartitionData(partitionIndex=0)])])]) with context >>>>>>>>>>>> >>>> RequestContext(header=RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS, >>>>>>>>>>>> apiVersion=0, clientId=1000, correlationId=14, headerVersion=2), >>>>>>>>>>>> connectionId='172.16.14.219:9090-172.16.14.217:53590-7', >>>>>>>>> clientAddress=/ >>>>>>>>>>>> 172.16.14.217, principal=User:CN=my-cluster-kafka,O=io.strimzi, >>>>>>>>>>>> listenerName=ListenerName(CONTROLPLANE-9090), >>>> securityProtocol=SSL, >>>>>>>>>>>> >>>> clientInformation=ClientInformation(softwareName=apache-kafka-java, >>>>>>>>>>>> softwareVersion=3.7.0), fromPrivilegedListener=false, >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>> >>>> principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@71004ad2 >>>>>>>>>>>> ]) >>>>>>>>>>>> (kafka.server.ControllerApis) [quorum-controller-0-event-handler] >>>>>>>>>>>> java.util.concurrent.CompletionException: >>>>>>>>>>>> org.apache.kafka.common.errors.UnsupportedVersionException: >>>>>>>> Directory >>>>>>>>>>>> assignment is not supported yet. >>>>>>>>>>>> >>>>>>>>>>>> at >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>> >>>> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332) >>>>>>>>>>>> at >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>> >>>> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347) >>>>>>>>>>>> at >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>> >>>> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636) >>>>>>>>>>>> at >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>> >>>> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) >>>>>>>>>>>> at >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>> >>>> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) >>>>>>>>>>>> at >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>> >>>> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:880) >>>>>>>>>>>> at >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>> >>>> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:871) >>>>>>>>>>>> at >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>> >>>> org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148) >>>>>>>>>>>> at >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>> >>>> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137) >>>>>>>>>>>> at >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>> >>>> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210) >>>>>>>>>>>> at >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>> >>>> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181) >>>>>>>>>>>> at java.base/java.lang.Thread.run(Thread.java:840) >>>>>>>>>>>> >>>>>>>>>>>> Caused by: >>>>>>>> org.apache.kafka.common.errors.UnsupportedVersionException: >>>>>>>>>>>> Directory assignment is not supported yet. >>>>>>>>>>>> >>>>>>>>>>>> Is that expected? I guess with the metadata version set to >>>>>>>> 3.6-IV2, it >>>>>>>>>>>> makes sense that the request is not supported. But shouldn't then >>>>>>>> the >>>>>>>>>>>> request not be sent at all by the brokers? (I did not opened a >>>> JIRA >>>>>>>>> for >>>>>>>>>>> it, >>>>>>>>>>>> but I can open one if you agree this is not expected) >>>>>>>>>>>> >>>>>>>>>>>> Thanks & Regards >>>>>>>>>>>> Jakub >>>>>>>>>>>> >>>>>>>>>>>> On Sat, Jan 13, 2024 at 8:03 AM Luke Chen <show...@gmail.com> >>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Stanislav, >>>>>>>>>>>>> >>>>>>>>>>>>> I commented in the "Apache Kafka 3.7.0 Release" thread, but >>>> maybe >>>>>>>>> you >>>>>>>>>>>>> missed it. >>>>>>>>>>>>> cross-posting here: >>>>>>>>>>>>> >>>>>>>>>>>>> There is a bug KAFKA-16101 >>>>>>>>>>>>> <https://issues.apache.org/jira/browse/KAFKA-16101> reporting >>>>>>>> that >>>>>>>>>>>> "Kafka >>>>>>>>>>>>> cluster will be unavailable during KRaft migration rollback". >>>>>>>>>>>>> The impact for this issue is that if brokers try to rollback to >>>>>>>> ZK >>>>>>>>> mode >>>>>>>>>>>>> during KRaft migration process, there will be a period of time >>>>>>>> the >>>>>>>>>>>> cluster >>>>>>>>>>>>> is unavailable. >>>>>>>>>>>>> Since ZK migrating to KRaft feature is a production ready >>>>>>>> feature, I >>>>>>>>>>>> think >>>>>>>>>>>>> this should be addressed soon. >>>>>>>>>>>>> Do you think this is a blocker for v3.7.0? >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks. >>>>>>>>>>>>> Luke >>>>>>>>>>>>> >>>>>>>>>>>>> On Sat, Jan 13, 2024 at 8:36 AM Chris Egerton < >>>>>>>>> fearthecel...@gmail.com >>>>>>>>>>>> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Kirk! >>>>>>>>>>>>>> >>>>>>>>>>>>>> @Stanislav--do you believe that this warrants a new RC? >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, Jan 12, 2024, 19:08 Kirk True <k...@kirktrue.pro> >>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Chris/Stanislav, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I'm working on the 'Unable to find FetchSessionHandler' log >>>>>>>>> problem >>>>>>>>>>>>>>> (KAFKA-16029) and have put out a draft PR ( >>>>>>>>>>>>>>> https://github.com/apache/kafka/pull/15186). I will use the >>>>>>>>>>>> quickstart >>>>>>>>>>>>>>> approach as a second means to reproduce/verify while I wait >>>>>>>> for >>>>>>>>> the >>>>>>>>>>>>> PR's >>>>>>>>>>>>>>> Jenkins job to finish. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> Kirk >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Fri, Jan 12, 2024, at 11:31 AM, Chris Egerton wrote: >>>>>>>>>>>>>>>> Hi Stanislav, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks for running this release! >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> To verify, I: >>>>>>>>>>>>>>>> - Built from source using Java 11 with both: >>>>>>>>>>>>>>>> - - the 3.7.0-rc2 tag on GitHub >>>>>>>>>>>>>>>> - - the kafka-3.7.0-src.tgz artifact from >>>>>>>>>>>>>>>> >>>>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/ >>>>>>>>>>>>>>>> - Checked signatures and checksums >>>>>>>>>>>>>>>> - Ran the quickstart using both: >>>>>>>>>>>>>>>> - - The kafka_2.13-3.7.0.tgz artifact from >>>>>>>>>>>>>>>> >>>>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/ >>>>>>>>>>> with >>>>>>>>>>>>> Java >>>>>>>>>>>>>>> 11 >>>>>>>>>>>>>>>> and Scala 13 in KRaft mode >>>>>>>>>>>>>>>> - - Our shiny new broker Docker image, >>>>>>>> apache/kafka:3.7.0-rc2 >>>>>>>>>>>>>>>> - Ran all unit tests >>>>>>>>>>>>>>>> - Ran all integration tests for Connect and MM2 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I found two minor areas for concern: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 1. (Possibly a blocker) >>>>>>>>>>>>>>>> When running the quickstart, I noticed this ERROR-level log >>>>>>>>>>> message >>>>>>>>>>>>>> being >>>>>>>>>>>>>>>> emitted frequently (not not every time) when I killed my >>>>>>>>> console >>>>>>>>>>>>>> consumer >>>>>>>>>>>>>>>> via ctrl-C: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [2024-01-12 11:00:31,088] ERROR [Consumer >>>>>>>>>>>>> clientId=console-consumer, >>>>>>>>>>>>>>>> groupId=console-consumer-74388] Unable to find >>>>>>>>>>> FetchSessionHandler >>>>>>>>>>>>> for >>>>>>>>>>>>>>> node >>>>>>>>>>>>>>>> 1. Ignoring fetch response >>>>>>>>>>>>>>>> (org.apache.kafka.clients.consumer.internals.AbstractFetch) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I see that this error message is already reported in >>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/KAFKA-16029. I >>>>>>>> think we >>>>>>>>>>>> should >>>>>>>>>>>>>>>> prioritize fixing it for this release. I know it's probably >>>>>>>>>>> benign >>>>>>>>>>>>> but >>>>>>>>>>>>>>> it's >>>>>>>>>>>>>>>> really not a good look for us when basic operations log >>>>>>>> error >>>>>>>>>>>>> messages, >>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>> it may give new users some headaches. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2. (Probably not a blocker) >>>>>>>>>>>>>>>> The following unit tests failed the first time around, and >>>>>>>>> all of >>>>>>>>>>>>> them >>>>>>>>>>>>>>>> passed the second time I ran them: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> - (clients) >>>>>>>>>>>>>>> >>>>>>>> ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup() >>>>>>>>>>>>>>>> - (clients) SelectorTest.testConnectionsByClientMetric() >>>>>>>>>>>>>>>> - (clients) >>>>>>>> Tls13SelectorTest.testConnectionsByClientMetric() >>>>>>>>>>>>>>>> - (connect) >>>>>>>>>>>>> TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound >>>>>>>>>>>>>> (I >>>>>>>>>>>>>>>> thought I fixed this one! 🤬🤬) >>>>>>>>>>>>>>>> - (core) >>>>>>>>> ProducerIdManagerTest.testUnrecoverableErrors(Errors)[2] >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks again for your work on this release, and >>>>>>>>> congratulations >>>>>>>>>>> to >>>>>>>>>>>>>> Kafka >>>>>>>>>>>>>>>> Streams for having zero flaky unit tests during my >>>>>>>>>>>>> highly-experimental >>>>>>>>>>>>>>>> single laptop run! >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Chris >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Thu, Jan 11, 2024 at 1:33 PM Stanislav Kozlovski >>>>>>>>>>>>>>>> <stanis...@confluent.io.invalid> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hello Kafka users, developers, and client-developers, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> This is the first candidate for release of Apache Kafka >>>>>>>>> 3.7.0. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Note it's named "RC2" because I had a few "failed" RCs >>>>>>>> that >>>>>>>>> I >>>>>>>>>>> had >>>>>>>>>>>>>>>>> cut/uploaded but ultimately had to scrap prior to >>>>>>>> announcing >>>>>>>>>>> due >>>>>>>>>>>> to >>>>>>>>>>>>>> new >>>>>>>>>>>>>>>>> blockers arriving before I could even announce them. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Further - I haven't yet been able to set up the system >>>>>>>> tests >>>>>>>>>>>>>>> successfully. >>>>>>>>>>>>>>>>> And the integration/unit tests do have a few failures >>>>>>>> that I >>>>>>>>>>> have >>>>>>>>>>>>> to >>>>>>>>>>>>>>> spend >>>>>>>>>>>>>>>>> time triaging. I would appreciate any help in case anyone >>>>>>>>>>> notices >>>>>>>>>>>>> any >>>>>>>>>>>>>>> tests >>>>>>>>>>>>>>>>> failing that they're subject matters experts in. Expect >>>>>>>> me >>>>>>>>> to >>>>>>>>>>>>> follow >>>>>>>>>>>>>>> up in >>>>>>>>>>>>>>>>> a day or two with more detailed analysis. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Major changes include: >>>>>>>>>>>>>>>>> - Early Access to KIP-848 - the next generation of the >>>>>>>>> consumer >>>>>>>>>>>>>>> rebalance >>>>>>>>>>>>>>>>> protocol >>>>>>>>>>>>>>>>> - KIP-858: Adding JBOD support to KRaft >>>>>>>>>>>>>>>>> - KIP-714: Observability into Client metrics via a >>>>>>>>> standardized >>>>>>>>>>>>>>> interface >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Check more information in the WIP blog post: >>>>>>>>>>>>>>>>> https://github.com/apache/kafka-site/pull/578 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Release notes for the 3.7.0 release: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>> >>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/RELEASE_NOTES.html >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> *** Please download, test and vote by Thursday, January >>>>>>>> 18, >>>>>>>>> 9am >>>>>>>>>>>> PT >>>>>>>>>>>>>> *** >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Usually these deadlines tend to be 2-3 days, but due to >>>>>>>> this >>>>>>>>>>>> being >>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>> first RC and the tests not having ran yet, I am giving >>>>>>>> it a >>>>>>>>> bit >>>>>>>>>>>>> more >>>>>>>>>>>>>>> time. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Kafka's KEYS file containing PGP keys we use to sign the >>>>>>>>>>> release: >>>>>>>>>>>>>>>>> https://kafka.apache.org/KEYS >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> * Release artifacts to be voted upon (source and binary): >>>>>>>>>>>>>>>>> >>>>>>>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/ >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> * Docker release artifact to be voted upon: >>>>>>>>>>>>>>>>> apache/kafka:3.7.0-rc2 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> * Maven artifacts to be voted upon: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>> >>>> https://repository.apache.org/content/groups/staging/org/apache/kafka/ >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> * Javadoc: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>> >>>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/javadoc/ >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> * Tag to be voted upon (off 3.7 branch) is the 3.7.0 tag: >>>>>>>>>>>>>>>>> https://github.com/apache/kafka/releases/tag/3.7.0-rc2 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> * Documentation: >>>>>>>>>>>>>>>>> https://kafka.apache.org/37/documentation.html >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> * Protocol: >>>>>>>>>>>>>>>>> https://kafka.apache.org/37/protocol.html >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> * Successful Jenkins builds for the 3.7 branch: >>>>>>>>>>>>>>>>> Unit/integration tests: >>>>>>>>>>>>>>>>> >>>>>>>>> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/ >>>>>>>>>>>>>>>>> There are failing tests here. I have to follow up with >>>>>>>>> triaging >>>>>>>>>>>>> some >>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>> the failures and figuring out if they're actual problems >>>>>>>> or >>>>>>>>>>>> simply >>>>>>>>>>>>>>> flakes. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> System tests: >>>>>>>>>>>>>>> https://jenkins.confluent.io/job/system-test-kafka/job/3.7/ >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> No successful system test runs yet. I am working on >>>>>>>> getting >>>>>>>>> the >>>>>>>>>>>> job >>>>>>>>>>>>>> to >>>>>>>>>>>>>>> run. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> * Successful Docker Image Github Actions Pipeline for 3.7 >>>>>>>>>>> branch: >>>>>>>>>>>>>>>>> Attached are the scan_report and report_jvm output files >>>>>>>>> from >>>>>>>>>>> the >>>>>>>>>>>>>>> Docker >>>>>>>>>>>>>>>>> Build run: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>> >>>>> https://github.com/apache/kafka/actions/runs/7486094960/job/20375761673 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> And the final docker image build job - Docker Build Test >>>>>>>>>>>> Pipeline: >>>>>>>>>>>>>>>>> https://github.com/apache/kafka/actions/runs/7486178277 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The image is apache/kafka:3.7.0-rc2 - >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>> >>>> https://hub.docker.com/layers/apache/kafka/3.7.0-rc2/images/sha256-5b4707c08170d39549fbb6e2a3dbb83936a50f987c0c097f23cb26b4c210c226?context=explore >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> /************************************** >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>> Stanislav Kozlovski >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Best, >>>>>>>>>>> Stanislav >>>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >>> >>> -- >>> Best, >>> Stanislav >> >
