Il giorno mar 11 feb 2020 alle ore 17:17 Andor Molnar <[email protected]> ha scritto: > > The most obvious one which crosses my mind is that I previously worked on: > > 1) run old version cluster, > 2) connect to each node and run smoke tests, > 3) restart one node with new code, > 4) goto 2) until all nodes are upgraded > > I think this wouldn’t work in a “unit test”, we probably need a separate > Jenkins job and a nice python script to do this. > > Andor > > > > > > On 2020. Feb 11., at 16:38, Patrick Hunt <[email protected]> wrote: > > > > Anyone have ideas how we could add testing for upgrade? Obviously something > > we're missing, esp given it's import.
I will send an email next days with a proposal. btw my idea is very like Andor's one Once we have an automatic environment we can launch from Jenkins Enrico > > > > Patrick > > > > On Tue, Feb 11, 2020 at 12:40 AM Enrico Olivelli <[email protected]> > > wrote: > > > >> Il giorno mar 11 feb 2020 alle ore 09:12 Szalay-Bekő Máté > >> <[email protected]> ha scritto: > >>> > >>> Hi All, > >>> > >>> about the question from Michael: > >>>> Regarding the fix, can we just make 3.6.0 aware of the old protocol and > >>>> speak old message format when it's talking to old server? > >>> > >>> In this particular case, it might be enough. The protocol change happened > >>> now in the 'initial message' sent by the QuorumCnxManager. Maybe it is > >> not > >>> a problem if the new servers can not initiate channels to the old > >> servers, > >>> maybe it is enough if these channel gets initiated by the old servers > >> only. > >>> I will test it quickly. > >>> > >>> Although I have no idea if any other thing changed in the quorum protocol > >>> between 3.5 and 3.6. In other cases it might not be enough if the new > >>> servers can understand the old messages, as the old servers can break by > >>> not understanding the messages from the new servers. Also, in the code > >>> currently (AFAIK) there is no generic knowledge of protocol versions, the > >>> servers are not storing that which protocol versions they can/should use > >> to > >>> communicate to which particular other servers. Maybe we don't even need > >>> this, but I would feel better if we would have more tests around these > >>> things. > >>> > >>> My suggestion for the long term: > >>> - let's fix this particular issue now with 3.6.0 quickly (I start doing > >>> this today) > >>> - let's do some automation (backed up with jenkins) that will test a > >> whole > >>> combinations of different ZooKeeper upgrade paths by making rolling > >>> upgrades during some light traffic. Let's have a bit better definition > >>> about what we expect (e.g. the quorum is up, but some clients can get > >>> disconnected? What will happen to the ephemeral nodes? Do we want to > >>> gracefully close or transfer the user sessions before stopping the old > >>> server?) and let's see where this broke. Just by checking the code, I > >> don't > >>> think the quorum will always be up (e.g. between older 3.4 versions and > >>> 3.5). > >> > >> > >> I am happy to work on this topic > >> > >>> - we need to update the Wiki about the working rolling upgrade paths and > >>> maybe about workarounds if needed > >>> - we might need to do some fixes (adding backward compatible versions > >>> and/or specific parameters that enforce old protocol temporary during the > >>> rolling upgrade that can be changed later to the new protocol by either > >>> dynamic reconfig or by rolling restart) > >> > >> it would be much better on 3.6 code to have some support for > >> compatibility with 3.5 servers > >> we can't require old code to be forward compatible but we can make new > >> code be compatible to a certain extend with old code. > >> If we can achieve this compatibility goal without a flag is better, > >> users won't have to care about this part and they simply "trust" on us > >> > >> The rollback story is also important, but maybe we are still not ready > >> for it, in case of local changes to store, > >> it is better to have a clear design and plan and work for a new release > >> (3.7?) > >> > >> Enrico > >> > >>> > >>> Depending on your comments, I am happy to create a few Jira tickets > >> around > >>> these topics. > >>> > >>> Kind regards, > >>> Mate > >>> > >>> ps. Enrico, sorry about your RC... I owe you a beer, let me know if you > >> are > >>> near to Budapest ;) > >>> > >>> On Tue, Feb 11, 2020 at 8:43 AM Enrico Olivelli <[email protected]> > >> wrote: > >>> > >>>> Good. > >>>> > >>>> I will cancel the vote for 3.6.0rc2. > >>>> > >>>> I appreciate very much If Mate and his colleagues have time to work on > >> a > >>>> fix. > >>>> Otherwise I will have cycles next week > >>>> > >>>> I would also like to spend my time in setting up a few minimal > >> integration > >>>> tests about the upgrade story > >>>> > >>>> Enrico > >>>> > >>>> Il Mar 11 Feb 2020, 07:30 Michael Han <[email protected]> ha scritto: > >>>> > >>>>> Kudos Enrico, very thorough work as the final gate keeper of the > >> release! > >>>>> > >>>>> Now with this, I'd like to *vote a -1* on the 3.6.0 RC2. > >>>>> > >>>>> I'd recommend we fix this issue for 3.6.0. ZooKeeper is one of the > >> rare > >>>>> piece of software that put so much emphasis on compatibilities thus > >> it > >>>> just > >>>>> works when upgrade / downgrade, which is amazing. One guarantee we > >> always > >>>>> had is during rolling upgrade, the quorum will always be available, > >>>> leading > >>>>> to no service interruption. It would be sad we lose such capability > >> given > >>>>> this is still a tractable problem. > >>>>> > >>>>> Regarding the fix, can we just make 3.6.0 aware of the old protocol > >> and > >>>>> speak old message format when it's talking to old server? Basically, > >> an > >>>>> ugly if else check against the protocol version should work and > >> there is > >>>> no > >>>>> need to have multiple pass on rolling upgrade process. > >>>>> > >>>>> > >>>>> On Mon, Feb 10, 2020 at 10:23 PM Enrico Olivelli < > >> [email protected]> > >>>>> wrote: > >>>>> > >>>>>> I suggest this plan: > >>>>>> - release 3.6.0 now > >>>>>> - improve the migration story, the flow outlined by Mate is > >>>>>> interesting, but it will take time > >>>>>> > >>>>>> 3.6.0rc2 got enough binding votes so I am going to finalize the > >>>>>> release this evening (within 8-10 hours) if no one comes out in the > >>>>>> VOTE thread with a -1 > >>>>>> > >>>>>> Enrico > >>>>>> > >>>>>> Enrico > >>>>>> > >>>>>> Il giorno lun 10 feb 2020 alle ore 19:33 Patrick Hunt > >>>>>> <[email protected]> ha scritto: > >>>>>>> > >>>>>>> On Mon, Feb 10, 2020 at 3:38 AM Andor Molnar <[email protected]> > >>>> wrote: > >>>>>>> > >>>>>>>> Hi, > >>>>>>>> > >>>>>>>> Answers inline. > >>>>>>>> > >>>>>>>> > >>>>>>>>> In my experience when you are close to a release it is > >> better to > >>>> to > >>>>>>>>> make big changes. (I am among the approvers of that patch, > >> so I > >>>> am > >>>>>>>>> responsible for this change) > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> Although this statement is acceptable for me, I don’t feel this > >>>> patch > >>>>>>>> should not have been merged into 3.6.0. Submission has been > >>>> preceded > >>>>>> by a > >>>>>>>> long argument with MAPR folks who originally wanted to be > >> merged > >>>> into > >>>>>> 3.4 > >>>>>>>> branch (considering the pace how ZooKeeper community is moving > >>>>>> forward) and > >>>>>>>> we reached an agreement that release it with 3.6.0. > >>>>>>>> > >>>>>>>> Make a long story short, this patch has been outstanding for > >> ages > >>>>>> without > >>>>>>>> much attention from the community and contributors made a lot > >> of > >>>>>> effort to > >>>>>>>> get it done before the release. > >>>>>>>> > >>>>>>>> > >>>>>>>>> I would like to ear from people that have been in the > >> community > >>>> for > >>>>>>>>> long time, then I am ready to complete the release process > >> for > >>>>>>>>> 3.6.0rc2. > >>>>>>>> > >>>>>>>> > >>>>>>>> Me too. > >>>>>>>> > >>>>>>>> I tend to accept the way rolling restart works now - as you > >>>> described > >>>>>>>> Enrico - and given that situation was pretty much the same > >> between > >>>>> 3.4 > >>>>>> and > >>>>>>>> 3.5, I don’t feel we have to make additional changes. > >>>>>>>> > >>>>>>>> On the other hand, the fix that Mate suggested sounds quite > >> cool, > >>>> I’m > >>>>>> also > >>>>>>>> happy to work on getting it in. > >>>>>>>> > >>>>>>>> Fyi, Release Management page says the following: > >>>>>>>> > >>>>>> > >>>> > >> https://cwiki.apache.org/confluence/display/ZOOKEEPER/ReleaseManagement > >>>>>>>> > >>>>>>>> "major.minor release of ZooKeeper must be backwards compatible > >> with > >>>>> the > >>>>>>>> previous minor release, major.(minor-1)" > >>>>>>>> > >>>>>>>> > >>>>>>> Our users, direct and indirect, value the ability to migrate to > >> newer > >>>>>>> versions - esp as we drop support for older. Frictions such as > >> this > >>>> can > >>>>>> be > >>>>>>> a reason to go elsewhere. I'm "pro" b/w compact - esp given our > >>>>> published > >>>>>>> guidelines. > >>>>>>> > >>>>>>> Patrick > >>>>>>> > >>>>>>> > >>>>>>>> Andor > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>>> On 2020. Feb 10., at 11:32, Enrico Olivelli < > >> [email protected] > >>>>> > >>>>>> wrote: > >>>>>>>>> > >>>>>>>>> Thank you Mate for checking and explaining this story. > >>>>>>>>> > >>>>>>>>> I find it very interesting that the cause is ZOOKEEPER-3188 > >> as: > >>>>>>>>> - it is the last "big patch" committed to 3.6 before > >> starting the > >>>>>>>>> release process > >>>>>>>>> - it is the cause of the failure of the first RC > >>>>>>>>> > >>>>>>>>> In my experience when you are close to a release it is > >> better to > >>>> to > >>>>>>>>> make big changes. (I am among the approvers of that patch, > >> so I > >>>> am > >>>>>>>>> responsible for this change) > >>>>>>>>> > >>>>>>>>> This is a pointer to the change to whom who wants to > >> understand > >>>>>> better > >>>>>>>>> the context > >>>>>>>>> > >>>>>>>> > >>>>>> > >>>>> > >>>> > >> https://github.com/apache/zookeeper/pull/1048/files#diff-7a209d890686bcba351d758b64b22a7dR11 > >>>>>>>>> > >>>>>>>>> IIUC even for the upgrade from 3.4 to 3.5 the story was the > >> same > >>>>> and > >>>>>>>>> if this statement holds then I feel we can continue > >>>>>>>>> with this release. > >>>>>>>>> > >>>>>>>>> - Reverting ZOOKEEPER-3188 is not an option for me, it is too > >>>>>> complex. > >>>>>>>>> - Making 3.5 and 3.6 "compatible" can be very tricky and we > >> do > >>>> not > >>>>>>>>> have tools to certify this compatibility (at least not in the > >>>> short > >>>>>>>>> term) > >>>>>>>>> > >>>>>>>>> I would like to ear from people that have been in the > >> community > >>>> for > >>>>>>>>> long time, then I am ready to complete the release process > >> for > >>>>>>>>> 3.6.0rc2. > >>>>>>>>> > >>>>>>>>> I will update the website and the release notes with a > >> specific > >>>>>>>>> warning about the upgrade, we should also update the Wiki > >>>>>>>>> > >>>>>>>>> Enrico > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> Il giorno lun 10 feb 2020 alle ore 11:17 Szalay-Bekő Máté > >>>>>>>>> <[email protected]> ha scritto: > >>>>>>>>>> > >>>>>>>>>> Hi Enrico! > >>>>>>>>>> > >>>>>>>>>> This is caused by the different PROTOCOL_VERSION in the > >>>>>>>> QuorumCnxManager. > >>>>>>>>>> The Protocol version was changed last time in > >> ZOOKEEPER-2186 > >>>>>> released > >>>>>>>>>> first in 3.4.7 and 3.5.1 to avoid some crashing / fix some > >> bugs. > >>>>>> Later I > >>>>>>>>>> also changed the protocol version when the format of the > >> initial > >>>>>> message > >>>>>>>>>> changed in ZOOKEEPER-3188. So actually the quorum protocol > >> is > >>>> not > >>>>>>>>>> compatible in this case and is the 'expected' behavior if > >> you > >>>>>> upgrade > >>>>>>>> e.g > >>>>>>>>>> from 3.4.6 to 3.4.7, or 3.4.6 to 3.5.5 or e.g from 3.5.6 to > >>>> 3.6.0. > >>>>>>>>>> > >>>>>>>>>> We had some discussion in the PR of ZOOKEEPER-3188 back > >> then and > >>>>>> got to > >>>>>>>> the > >>>>>>>>>> conclusion that it is not that bad, as there will be no data > >>>> loss > >>>>>> as you > >>>>>>>>>> wrote. The tricky thing is that during rolling upgrade we > >> should > >>>>>> ensure > >>>>>>>>>> both backward and forward compatibility to make sure that > >> the > >>>> old > >>>>>> and > >>>>>>>> the > >>>>>>>>>> new part of the quorum can still speak to each other. The > >>>> current > >>>>>>>> solution > >>>>>>>>>> (simply failing if the protocol versions mismatch) is more > >>>> simple > >>>>>> and > >>>>>>>> still > >>>>>>>>>> working just fine: as the servers are restarted one-by-one, > >> the > >>>>>> nodes > >>>>>>>> with > >>>>>>>>>> the old protocol version and the nodes with the new protocol > >>>>> version > >>>>>>>> will > >>>>>>>>>> form two partitions, but any given time only one partition > >> will > >>>>>> have the > >>>>>>>>>> quorum. > >>>>>>>>>> > >>>>>>>>>> Still, thinking it trough, as a side effect in these cases > >> there > >>>>>> will > >>>>>>>> be a > >>>>>>>>>> short time when none of the partitions will have quorums > >> (when > >>>> we > >>>>>> have N > >>>>>>>>>> servers with the old protocol version, N servers with the > >> new > >>>>>> protocol > >>>>>>>>>> version, and there is one server just being restarted). I > >> am not > >>>>>> sure > >>>>>>>> if we > >>>>>>>>>> can accept this. > >>>>>>>>>> > >>>>>>>>>> For ZOOKEEPER-3188 we can add a small patch to make it > >> possible > >>>> to > >>>>>> parse > >>>>>>>>>> the initial message of the old protocol version with the new > >>>> code. > >>>>>> But > >>>>>>>> I am > >>>>>>>>>> not sure if it would be enough (as the old code will not be > >> able > >>>>> to > >>>>>>>> parse > >>>>>>>>>> the new initial message). > >>>>>>>>>> > >>>>>>>>>> One option can be to make a patch also for 3.5 to have a > >> version > >>>>>> which > >>>>>>>>>> supports both protocol versions. (let's say in 3.5.8) Then > >> we > >>>> can > >>>>>> write > >>>>>>>> to > >>>>>>>>>> the release note, that if you need rolling upgrade from any > >>>>> versions > >>>>>>>> since > >>>>>>>>>> 3.4.7, then you have to first upgrade from 3.5.8 before > >>>> upgrading > >>>>> to > >>>>>>>> 3.6.0. > >>>>>>>>>> We can even make the same thing on the 3.4 branch. > >>>>>>>>>> > >>>>>>>>>> But I am also new to the community... It would be great to > >> hear > >>>>> the > >>>>>>>> opinion > >>>>>>>>>> of more experienced people. > >>>>>>>>>> Whatever the decision will be, I am happy to make the > >> changes. > >>>>>>>>>> > >>>>>>>>>> And sorry for breaking the RC (if we decide that this needs > >> to > >>>> be > >>>>>>>>>> changed...). ZOOKEEPER-3188 was a complex patch. > >>>>>>>>>> > >>>>>>>>>> Kind regards, > >>>>>>>>>> Mate > >>>>>>>>>> > >>>>>>>>>> On Mon, Feb 10, 2020 at 9:47 AM Enrico Olivelli < > >>>>>> [email protected]> > >>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> Hi, > >>>>>>>>>>> even if we had enough binding +1 on 3.6.0rc2 before > >> closing the > >>>>>> VOTE > >>>>>>>>>>> of 3.6.0 I wanted to finish my tests and I am coming to an > >>>>> apparent > >>>>>>>>>>> blocker. > >>>>>>>>>>> > >>>>>>>>>>> I am trying to upgrade a 3.5.6 cluster to 3.6.0, but it > >> looks > >>>>> like > >>>>>>>>>>> peers are not able to talk to each other. > >>>>>>>>>>> I have a cluster of 3, server1, server2 and server3. > >>>>>>>>>>> When I upgrade server1 to 3.6.0rc2 I see this kind of > >> errors on > >>>>> 3.5 > >>>>>>>> nodes: > >>>>>>>>>>> > >>>>>>>>>>> 2020-02-10 09:35:07,745 [myid:3] - INFO > >>>>>>>>>>> [localhost/127.0.0.1:3334:QuorumCnxManager$Listener@918] - > >>>>>> Received > >>>>>>>>>>> connection request 127.0.0.1:62591 > >>>>>>>>>>> 2020-02-10 09:35:07,746 [myid:3] - ERROR > >>>>>>>>>>> [localhost/127.0.0.1:3334:QuorumCnxManager@527] - > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>> > >>>>>> > >>>>> > >>>> > >> org.apache.zookeeper.server.quorum.QuorumCnxManager$InitialMessage$InitialMessageException: > >>>>>>>>>>> Got unrecognized protocol version -65535 > >>>>>>>>>>> > >>>>>>>>>>> Once I upgrade all of the peers the system is up and > >> running, > >>>>>> without > >>>>>>>>>>> apparently no data loss. > >>>>>>>>>>> > >>>>>>>>>>> During the upgrade as soon as I upgrade the first node, > >> say, > >>>>>> server1, > >>>>>>>>>>> server1 is not able to accept connections (error "Close of > >>>>> session > >>>>>> 0x0 > >>>>>>>>>>> java.io.IOException: ZooKeeperServer not running") from > >>>> clients, > >>>>>> this > >>>>>>>>>>> is expected, because as far as it cannot talk with the > >> other > >>>>> peers > >>>>>> it > >>>>>>>>>>> is practically partitioned away from the cluster. > >>>>>>>>>>> > >>>>>>>>>>> My questions are: > >>>>>>>>>>> 1) is this expected ? I can't remember protocol changes > >> from > >>>> 3.5 > >>>>> to > >>>>>>>>>>> 3.6, but actually 3.6 diverged from 3.5 branch so long ago, > >>>> and I > >>>>>> was > >>>>>>>>>>> not in the community as dev so I cannot tell > >>>>>>>>>>> 2) is this a viable option for users ? to have some > >> temporary > >>>>>> glitch > >>>>>>>>>>> during the upgrade and hope that the upgrade completes > >> without > >>>>>>>>>>> troubles ? > >>>>>>>>>>> > >>>>>>>>>>> In theory as long as two servers are running the same major > >>>>> version > >>>>>>>>>>> (3.5 or 3.6) we have a quorum and the system is able to > >> make > >>>>>> progress > >>>>>>>>>>> and to server clients. > >>>>>>>>>>> I feel that this is quite dangerous, but I don't have > >> enough > >>>>>> context > >>>>>>>>>>> to understand how this problem is possible and when we > >> decided > >>>> to > >>>>>>>>>>> break compatibility. > >>>>>>>>>>> > >>>>>>>>>>> The other option is that I am wrong in my test and I am > >> messing > >>>>> up > >>>>>> :-) > >>>>>>>>>>> > >>>>>>>>>>> The other upgrade path I would like to see working like a > >> charm > >>>>> is > >>>>>> the > >>>>>>>>>>> upgrade from 3.4 to 3.6, as I see that as soon as we > >> release > >>>> 3.6 > >>>>> we > >>>>>>>>>>> should encourage users to move to 3.6 and not to 3.5. > >>>>>>>>>>> > >>>>>>>>>>> Regards > >>>>>>>>>>> Enrico > >>>>>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>> > >>>>> > >>>> > >> >
