First off, I'd suggest reading https://kafka.apache.org/documentation/#upgrade <https://kafka.apache.org/documentation/#upgrade> pretty carefully. It has your exact question more or less defined and outlined. As I mentioned, they are putting a lot of effort into how the world should migrate from 2.x to 3.x in the last few releases.
Without digging into it too much, it looks like you can upgrade to version 3.7.0 (I'd suggest 3.6.2 however or waiting until 3.7.1 is released) from any version of 0.8.x and up without needing to upgrade to the latest of that series. Test to be sure, it was a bit painful to upgrade to the 3.3.x series when we did that a while back but was totally worth it afterwards. But we also switched at the same time to a raft thing as well and I'm unsure how much of it was the zookeeper to raft and how much was simply the broker upgrades. Just read the upgrade guide. On Mon Jun 17, 2024, 12:58 PM GMT, Wojciech Frycz <mailto:wojciech.fr...@grandparade.co.uk> wrote: > Hi, > Thanks for the answer. However I still do not have a clear picture on how we > should go from version 2.2.1 which we are on at the moment. Options > 1. Straight jump from 2.2.1 to 3.6.3 > 2. Jump from 2.2.1 to 2.6.2 and then the second jump from 2.6.2. to 3.6.3 > What’s you recommendation here? > Thanks, [image] > Wojciech Frycz > Group Head of Engineering – Sports, MrGreen, Evoke, Data Migrations > Mobile: +48 790 417 534 > Grand Parade part of William Hill > ul. Kotlarska 11 | 31-539 Krakow > From: Sejal Patel <se...@playerzero.ai> > Date: Monday, 17 June 2024 at 14:16 > To: users@kafka.apache.org <users@kafka.apache.org>, > kafka-clie...@googlegroups.com <kafka-clie...@googlegroups.com>, > d...@kafka.apache.org <d...@kafka.apache.org>, Wojciech Frycz > <wojciech.fr...@grandparade.co.uk> > Cc: Cezary Gajdzinski <cezary.gajdzin...@williamhill.com>, Slawomir Kmiecik > <slawomir.kmie...@grandparade.co.uk>, Ricardo Antunes (WH Online) > <ricardo.antu...@williamhill.com> > Subject: Re: [EXTERNAL] Re: Kafka upgrade recommendation > Hey Wojciech, > It has been a long time since we did the jump but we had waited until 3.3.x > release when we did the jump and there were a lot of manual steps at that > time. We then had another manual step when going to either 3.4 or 3.5 don't > remember which but that one was easier cause we did have the option to turn > off every broker and kraft server at once to do that one safely. I would > suggest going from 2.6.2 to 3.6.2 and then stopping and waiting until 3.7.1 > comes out as there is a pretty serious flaw in 3.7.0. And I would suggest > probably seeing what the official migration guides have to say because my > understanding is that the bulk of the changes since we started the migration > is around making sure it is a smooth and safe upgrade process overall. > The other option that you could always do (our fallback plan) is to spinup a > brand new cluster and to copy all the topics from the old cluster to the new > cluster. Tedious and takes several days depending on how much data you have > but it is safe and works as well. > Sorry I don't know what are all of the options and a lot has changed since we > had to do the hard step of going from 2 to 3. Once on 3.x, the upgrades have > been pretty smooth. > On Mon Jun 17, 2024, 10:06 AM GMT, Wojciech Frycz > <mailto:wojciech.fr...@grandparade.co.uk> wrote: >> Hi Sejal, >> Thanks for reaching out. >> What would be your recommended strategy to go from 2.2.1 into 3.6.2 version? >> Can we securely do it in one jump or we should use some step in between >> (e.g. to jump form 2.2.1 to 2.6.2 and then from 2.6.2 to 3.6.2)? >> Thanks, >> Error! Filename not specified. >> Wojciech Frycz >> Group Head of Engineering – Sports, MrGreen, Evoke, Data Migrations >> Mobile: +48 790 417 534 >> Grand Parade part of William Hill >> ul. Kotlarska 11 | 31-539 Krakow >> From: Sejal Patel <se...@playerzero.ai> >> Date: Saturday, 15 June 2024 at 17:28 >> To: users@kafka.apache.org <users@kafka.apache.org>, >> kafka-clie...@googlegroups.com <kafka-clie...@googlegroups.com>, >> d...@kafka.apache.org <d...@kafka.apache.org> >> Cc: Cezary Gajdzinski <cezary.gajdzin...@williamhill.com>, Wojciech Frycz >> <wojciech.fr...@grandparade.co.uk>, Slawomir Kmiecik >> <slawomir.kmie...@grandparade.co.uk>, Ricardo Antunes (WH Online) >> <ricardo.antu...@williamhill.com> >> Subject: [EXTERNAL] Re: Kafka upgrade recommendation >> I would suggest going from 2.6.2 to 3.6.2 and then stopping and waiting >> until 3.7.1 comes out. There is a pretty serious flaw in 3.7.0 that hit us >> and we ended up having to downgrade to 3.6.2 to recover. The bug is fixed >> but not going to be released until 3.7.1 comes out. The bug prevented >> metadata from being readable and thus anytime a node restarted (including >> the kraft server itself), it could never come back online or be unregistered >> from the cluster either. >> Sejal [Patel] | CTO >> se...@playerzero.ai <mailto:se...@playerzero.ai> | 470.440.1255 >> On Thu Jun 13, 2024, 03:26 PM GMT, Jose Manuel Monroy >> <mailto:jose.mon...@888williamhill.com> wrote: >>> Hi there, >>> We have few clusters in AWS world and being forced to bump cluster version. >>> So, we are considering options to be in safer way possible, mitigating >>> risks, and without outage as possible (our business is 24/7). >>> Upgrade would be from 2.2.1 / 2.6.2 to 3.7.0 (MSK recommended version). >>> Things to consider: >>> - Recommended configurations in place so rolling restart is not impacting >>> producers/consumers >>> - Clients from different technologies (Java official client/Java SpringBoot >>> Spring Cloud Stream/Alpakka Scala/NodeJS) around those versions >>> - 2.2.1 cluster version -> TLS SSL SASL/SCRAM authenticated >>> - 2.2.6 cluster version -> unathenticated >>> - We don’t use Kafka streams >>> Could you please which would be your recommendation in terms of upgrade >>> brokers? Can be done directly or in several steps? We should upgrade >>> clients before than brokers? If so, in which way? >>> Client side, any recommendation of clients? (example: we use NodeJs rdkafka >>> and looking could be limited to brokers 2.3.0, so probably we will need to >>> jump into another client) >>> Thanks >>> Error! Filename not specified. >>> Jose Manuel Vega Monroy >>> Software Engineer Team Leader >>> Direct: +350 >>> Mobile: +34(0) 633710634 >>> WHG (International) Ltd | 6/1 Waterport Place | Gibraltar | >>> >>> Confidentiality notice: This email message and its attachments are for the >>> sole use of the intended recipient(s) and may not be shared with any other >>> party. They may be privileged or otherwise protected from disclosure, >>> contain confidential information of evoke plc or its direct and indirect >>> subsidiaries (together, the “evoke Group”) and regarded as confidential >>> information under any non-disclosure agreement. Any review, use, disclosure >>> or distribution by persons or entities other than the intended recipient(s) >>> is prohibited. Nothing in this message is capable of or intended to create >>> any legally binding obligation. The evoke Group will only ever assume a >>> legally binding obligation where recorded in a written agreement duly >>> executed by the authorized signatories of the relevant evoke Group company. >>> If you are not the intended recipient, please contact the sender by return >>> and destroy all copies of the original message and its attachments. >>> >>> Unless specifically indicated otherwise, the contents of this e-mail are >>> subject to contract; and are not an official statement and does not >>> necessarily represent the views of evoke plc, its subsidiaries, or >>> affiliated companies. Please note that, whilst the evoke Group has taken >>> reasonable precautions to ensure no viruses are present in this email, >>> neither evoke plc, nor its subsidiaries and affiliated companies can accept >>> any responsibility for any viruses contained within this e-mail and it is >>> your responsibility to scan any emails and their attachments. evoke plc, >>> its subsidiaries and affiliated companies may monitor e-mail traffic data >>> and also the content of e-mails for effective operation of the e-mail >>> system, or for security, purposes. For information about how we process >>> data and monitor communications please see our Privacy policy and for terms >>> of use please see our Website Terms and Conditions. >>> >>> evoke plc is a publicly limited company registered in Gibraltar with >>> registered number 90099 and is listed on the premium segment of the London >>> Stock Exchange. Our registered office is at Suite 601/701 Europort, >>> Europort Road, Gibraltar. This e-mail is sent by an evoke plc Group company >>> and may represent one of our three flagship brands, 888, William Hill, and >>> Mr Green. The evoke Group of companies includes, among others, William Hill >>> Limited (registered number 4212563), William Hill Organization Limited >>> (registered number 278208), WHG (International) Limited (registered number >>> 99191) and Mr Green Limited (registered number C43260). Each of William >>> Hill Limited and William Hill Organization Limited is registered in England >>> and Wales and has its registered office at 1 Bedford Avenue, London, WC1B >>> 3AU, UK. WHG (International) Limited is registered in Gibraltar and has its >>> registered office at 6/1 Waterport Place, Gibraltar. Mr Green Limited is >>> registered in Malta and has its registered office at Tagliaferro Business >>> Centre, Level 7, 14 High Street, Sliema SLM 1549, Malta. A full list of the >>> 888 William Hill Group companies and affiliates can be found in the 888 >>> Holdings plc annual report available from www.evokeplc.com. >> >> Confidentiality notice: This email message and its attachments are for the >> sole use of the intended recipient(s) and may not be shared with any other >> party. They may be privileged or otherwise protected from disclosure, >> contain confidential information of evoke plc or its direct and indirect >> subsidiaries (together, the “evoke Group”) and regarded as confidential >> information under any non-disclosure agreement. Any review, use, disclosure >> or distribution by persons or entities other than the intended recipient(s) >> is prohibited. Nothing in this message is capable of or intended to create >> any legally binding obligation. The evoke Group will only ever assume a >> legally binding obligation where recorded in a written agreement duly >> executed by the authorized signatories of the relevant evoke Group company. >> If you are not the intended recipient, please contact the sender by return >> and destroy all copies of the original message and its attachments. >> >> Unless specifically indicated otherwise, the contents of this e-mail are >> subject to contract; and are not an official statement and does not >> necessarily represent the views of evoke plc, its subsidiaries, or >> affiliated companies. Please note that, whilst the evoke Group has taken >> reasonable precautions to ensure no viruses are present in this email, >> neither evoke plc, nor its subsidiaries and affiliated companies can accept >> any responsibility for any viruses contained within this e-mail and it is >> your responsibility to scan any emails and their attachments. evoke plc, its >> subsidiaries and affiliated companies may monitor e-mail traffic data and >> also the content of e-mails for effective operation of the e-mail system, or >> for security, purposes. For information about how we process data and >> monitor communications please see our Privacy policy and for terms of use >> please see our Website Terms and Conditions. >> >> evoke plc is a publicly limited company registered in Gibraltar with >> registered number 90099 and is listed on the premium segment of the London >> Stock Exchange. Our registered office is at Suite 601/701 Europort, Europort >> Road, Gibraltar. This e-mail is sent by an evoke plc Group company and may >> represent one of our three flagship brands, 888, William Hill, and Mr Green. >> The evoke Group of companies includes, among others, William Hill Limited >> (registered number 4212563), William Hill Organization Limited (registered >> number 278208), WHG (International) Limited (registered number 99191) and Mr >> Green Limited (registered number C43260). Each of William Hill Limited and >> William Hill Organization Limited is registered in England and Wales and has >> its registered office at 1 Bedford Avenue, London, WC1B 3AU, UK. WHG >> (International) Limited is registered in Gibraltar and has its registered >> office at 6/1 Waterport Place, Gibraltar. Mr Green Limited is registered in >> Malta and has its registered office at Tagliaferro Business Centre, Level 7, >> 14 High Street, Sliema SLM 1549, Malta. A full list of the 888 William Hill >> Group companies and affiliates can be found in the 888 Holdings plc annual >> report available from www.evokeplc.com. >> Confidentiality: The contents of this e-mail and any attachments transmitted >> with it are intended to be confidential to the intended recipient; and may >> be privileged or otherwise protected from disclosure. If you are not an >> intended recipient of this e-mail, do not duplicate or redistribute it by >> any means. Please delete it and any attachments and notify the sender that >> you have received it in error. This e-mail is sent by a William Hill PLC >> group company. The William Hill group companies include, among others, >> William Hill PLC (registered number 4212563), William Hill Organization >> Limited (registered number 278208), William Hill US HoldCo Inc, WHG >> (International) Limited (registered number 99191) and Mr Green Limited >> (registered number C43260). Each of William Hill PLC and William Hill >> Organization Limited is registered in England and Wales and has its >> registered office at 1 Bedford Avenue, London, WC1B 3AU, UK. William Hill >> U.S. HoldCo, Inc. is registered in Delaware and has its registered office at >> 1007 N. Orange Street, 9 Floor, Wilmington, New Castle County DE 19801 >> Delaware, United States of America. WHG (International) Limited is >> registered in Gibraltar and has its registered office at 6/1 Waterport >> Place, Gibraltar. Mr Green Limited is registered in Malta and has its >> registered office at Tagliaferro Business Centre, Level 7, 14 High Street, >> Sliema SLM 1549, Malta. Unless specifically indicated otherwise, the >> contents of this e-mail are subject to contract; and are not an official >> statement, and do not necessarily represent the views, of William Hill PLC, >> its subsidiaries or affiliated companies. Please note that neither William >> Hill PLC, nor its subsidiaries and affiliated companies can accept any >> responsibility for any viruses contained within this e-mail and it is your >> responsibility to scan any emails and their attachments. William Hill PLC, >> its subsidiaries and affiliated companies may monitor e-mail traffic data >> and also the content of e-mails for effective operation of the e-mail >> system, or for security, purposes.'. If the disclaimer can't be applied, >> attach the message to a new disclaimer message. > > Confidentiality notice: This email message and its attachments are for the > sole use of the intended recipient(s) and may not be shared with any other > party. They may be privileged or otherwise protected from disclosure, contain > confidential information of evoke plc or its direct and indirect subsidiaries > (together, the “evoke Group”) and regarded as confidential information under > any non-disclosure agreement. Any review, use, disclosure or distribution by > persons or entities other than the intended recipient(s) is prohibited. > Nothing in this message is capable of or intended to create any legally > binding obligation. The evoke Group will only ever assume a legally binding > obligation where recorded in a written agreement duly executed by the > authorized signatories of the relevant evoke Group company. If you are not > the intended recipient, please contact the sender by return and destroy all > copies of the original message and its attachments. > > Unless specifically indicated otherwise, the contents of this e-mail are > subject to contract; and are not an official statement and does not > necessarily represent the views of evoke plc, its subsidiaries, or affiliated > companies. Please note that, whilst the evoke Group has taken reasonable > precautions to ensure no viruses are present in this email, neither evoke > plc, nor its subsidiaries and affiliated companies can accept any > responsibility for any viruses contained within this e-mail and it is your > responsibility to scan any emails and their attachments. evoke plc, its > subsidiaries and affiliated companies may monitor e-mail traffic data and > also the content of e-mails for effective operation of the e-mail system, or > for security, purposes. For information about how we process data and monitor > communications please see our Privacy policy and for terms of use please see > our Website Terms and Conditions. > > evoke plc is a publicly limited company registered in Gibraltar with > registered number 90099 and is listed on the premium segment of the London > Stock Exchange. Our registered office is at Suite 601/701 Europort, Europort > Road, Gibraltar. This e-mail is sent by an evoke plc Group company and may > represent one of our three flagship brands, 888, William Hill, and Mr Green. > The evoke Group of companies includes, among others, William Hill Limited > (registered number 4212563), William Hill Organization Limited (registered > number 278208), WHG (International) Limited (registered number 99191) and Mr > Green Limited (registered number C43260). Each of William Hill Limited and > William Hill Organization Limited is registered in England and Wales and has > its registered office at 1 Bedford Avenue, London, WC1B 3AU, UK. WHG > (International) Limited is registered in Gibraltar and has its registered > office at 6/1 Waterport Place, Gibraltar. Mr Green Limited is registered in > Malta and has its registered office at Tagliaferro Business Centre, Level 7, > 14 High Street, Sliema SLM 1549, Malta. A full list of the 888 William Hill > Group companies and affiliates can be found in the 888 Holdings plc annual > report available from www.evokeplc.com. > Confidentiality: The contents of this e-mail and any attachments transmitted > with it are intended to be confidential to the intended recipient; and may be > privileged or otherwise protected from disclosure. If you are not an intended > recipient of this e-mail, do not duplicate or redistribute it by any means. > Please delete it and any attachments and notify the sender that you have > received it in error. This e-mail is sent by a William Hill PLC group > company. The William Hill group companies include, among others, William Hill > PLC (registered number 4212563), William Hill Organization Limited > (registered number 278208), William Hill US HoldCo Inc, WHG (International) > Limited (registered number 99191) and Mr Green Limited (registered number > C43260). Each of William Hill PLC and William Hill Organization Limited is > registered in England and Wales and has its registered office at 1 Bedford > Avenue, London, WC1B 3AU, UK. William Hill U.S. HoldCo, Inc. is registered in > Delaware and has its registered office at 1007 N. Orange Street, 9 Floor, > Wilmington, New Castle County DE 19801 Delaware, United States of America. > WHG (International) Limited is registered in Gibraltar and has its registered > office at 6/1 Waterport Place, Gibraltar. Mr Green Limited is registered in > Malta and has its registered office at Tagliaferro Business Centre, Level 7, > 14 High Street, Sliema SLM 1549, Malta. Unless specifically indicated > otherwise, the contents of this e-mail are subject to contract; and are not > an official statement, and do not necessarily represent the views, of William > Hill PLC, its subsidiaries or affiliated companies. Please note that neither > William Hill PLC, nor its subsidiaries and affiliated companies can accept > any responsibility for any viruses contained within this e-mail and it is > your responsibility to scan any emails and their attachments. William Hill > PLC, its subsidiaries and affiliated companies may monitor e-mail traffic > data and also the content of e-mails for effective operation of the e-mail > system, or for security, purposes.'. If the disclaimer can't be applied, > attach the message to a new disclaimer message.
