FYI: PR just submitted, see https://github.com/apache/zookeeper/pull/1251 any comments welcomed! :)
Kind regards, Mate On Wed, Feb 12, 2020 at 1:16 PM Andor Molnar <an...@apache.org> wrote: > Hi Michael, > > "if we can get to rc2 without noticing a showstopper…” > > 200% disagree with this. > > The whole point of release voting system is to identify problems no matter > how big they are. The message of finding a showstopper for me is that > people paying attention and accurately testing the release. This is a very > good thing and emphasises how much effort the ZooKeeper community is > putting into every single release. Otherwise we could just set up a Jenkins > job which creates and publishes a new release in every six months and say > good luck with them. > > I admit that currently we don’t have (rolling) upgrade tests, but I feel > demand from the community to fill this gap. > > “rolling upgrades (and mixed ensembles generally) are effectively untested” > > Not true. That’s exactly what we are currently doing (manually for now). > > "there have to be a hundred corner cases beyond the MultiAddress issue” > > Sure thing. True for every new feature in every release. That’s why I’m > happy disabling it by default. People usually don’t pick up releases ending > with .0, production upgrades are expected from .1 or .2 or maybe later > depending on how much risk would like to be taken. > > Andor > > > > > On 2020. Feb 11., at 23:35, Michael K. Edwards <m.k.edwa...@gmail.com> > wrote: > > > > I think it would be prudent to emphasize in the release notes that > rolling > > upgrades (and mixed ensembles generally) are effectively untested. That > > this was, in practice, a non-goal of this release cycle. Because if we > can > > get to rc2 without noticing a showstopper, clearly it's not something > that > > anyone has gotten around to attempting; and there have to be a hundred > > corner cases beyond the MultiAddress issue. > > > > On Tue, Feb 11, 2020 at 12:27 PM Szalay-Bekő Máté < > > szalay.beko.m...@gmail.com> wrote: > > > >> I see the main problem here in the fact that we are missing proper > >> versioning in the leader election / quorum protocols. I tried to simply > >> implement backward compatibility in 3.6, but it didn't solve the > problem. > >> The new code understands the old protocol, but it can not decide when to > >> use the new or the old protocol during connection initiation. So the old > >> servers can not read the new init messages and we still temporarly end > up > >> having two partitions during rolling restart. > >> > >> I already suggested two ways to handle this later, but I think for 3.6.0 > >> now the simplest solution is to disable the new MultiAddress feature and > >> stick to the old protocol version by default. Plus extend the > >> documentation with the note, that enabling the MultiAddress feature is > not > >> possible during a rolling upgrade, but it needs to be done with a > separate > >> rolling restart. With this approach, the rolling restart should "just > work" > >> with the 3.4 / 3.5 configs and we don't require any extra step / > >> configuration from the users, unless they want to use the new feature. I > >> plan to submit a PR with these changes tomorrow to ZOOKEEPER-3720, if > there > >> isn't any different opinion. > >> > >> P.S. For 4.0 we might need to put some extra thinking into backward > >> compatibility / versioning for the quorum and client protocols. > >> > >> > >> On Tue, Feb 11, 2020, 20:44 Michael K. Edwards <m.k.edwa...@gmail.com> > >> wrote: > >> > >>> I hate to say it, but I think 3.6.0 should release as is. It is > >>> impossible > >>> to *reliably* retrofit backwards compatibility / interoperability onto > a > >>> release that was engineered from the beginning without that goal. > Learn > >>> the lesson, set goals differently in the future. > >>> > >>> On Tue, Feb 11, 2020 at 9:41 AM Szalay-Bekő Máté < > >>> szalay.beko.m...@gmail.com> > >>> wrote: > >>> > >>>> FYI: I created these scripts for my local tests: > >>>> https://github.com/symat/zk-rolling-upgrade-test > >>>> > >>>> For the long term I would also add some script that actually monitors > >>> the > >>>> state of the quorum and also runs continuous traffic, not just 1-2 > >>>> smoketests after each restart. But I don't know how important this > would > >>>> be. > >>>> > >>>> On Tue, Feb 11, 2020 at 5:25 PM Enrico Olivelli <eolive...@gmail.com> > >>>> wrote: > >>>> > >>>>> Il giorno mar 11 feb 2020 alle ore 17:17 Andor Molnar > >>>>> <an...@apache.org> ha scritto: > >>>>>> > >>>>>> The most obvious one which crosses my mind is that I previously > >>> worked > >>>>> on: > >>>>>> > >>>>>> 1) run old version cluster, > >>>>>> 2) connect to each node and run smoke tests, > >>>>>> 3) restart one node with new code, > >>>>>> 4) goto 2) until all nodes are upgraded > >>>>>> > >>>>>> I think this wouldn’t work in a “unit test”, we probably need a > >>>> separate > >>>>> Jenkins job and a nice python script to do this. > >>>>>> > >>>>>> Andor > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>> On 2020. Feb 11., at 16:38, Patrick Hunt <ph...@apache.org> > >>> wrote: > >>>>>>> > >>>>>>> Anyone have ideas how we could add testing for upgrade? Obviously > >>>>> something > >>>>>>> we're missing, esp given it's import. > >>>>> > >>>>> I will send an email next days with a proposal. > >>>>> btw my idea is very like Andor's one > >>>>> > >>>>> Once we have an automatic environment we can launch from Jenkins > >>>>> > >>>>> Enrico > >>>>> > >>>>> > >>>>>>> > >>>>>>> Patrick > >>>>>>> > >>>>>>> On Tue, Feb 11, 2020 at 12:40 AM Enrico Olivelli < > >>>> eolive...@gmail.com> > >>>>>>> wrote: > >>>>>>> > >>>>>>>> Il giorno mar 11 feb 2020 alle ore 09:12 Szalay-Bekő Máté > >>>>>>>> <szalay.beko.m...@gmail.com> ha scritto: > >>>>>>>>> > >>>>>>>>> Hi All, > >>>>>>>>> > >>>>>>>>> about the question from Michael: > >>>>>>>>>> Regarding the fix, can we just make 3.6.0 aware of the old > >>>> protocol > >>>>> and > >>>>>>>>>> speak old message format when it's talking to old server? > >>>>>>>>> > >>>>>>>>> In this particular case, it might be enough. The protocol change > >>>>> happened > >>>>>>>>> now in the 'initial message' sent by the QuorumCnxManager. > >>> Maybe it > >>>>> is > >>>>>>>> not > >>>>>>>>> a problem if the new servers can not initiate channels to the > >>> old > >>>>>>>> servers, > >>>>>>>>> maybe it is enough if these channel gets initiated by the old > >>>> servers > >>>>>>>> only. > >>>>>>>>> I will test it quickly. > >>>>>>>>> > >>>>>>>>> Although I have no idea if any other thing changed in the quorum > >>>>> protocol > >>>>>>>>> between 3.5 and 3.6. In other cases it might not be enough if > >>> the > >>>> new > >>>>>>>>> servers can understand the old messages, as the old servers can > >>>>> break by > >>>>>>>>> not understanding the messages from the new servers. Also, in > >>> the > >>>>> code > >>>>>>>>> currently (AFAIK) there is no generic knowledge of protocol > >>>>> versions, the > >>>>>>>>> servers are not storing that which protocol versions they > >>>> can/should > >>>>> use > >>>>>>>> to > >>>>>>>>> communicate to which particular other servers. Maybe we don't > >>> even > >>>>> need > >>>>>>>>> this, but I would feel better if we would have more tests around > >>>>> these > >>>>>>>>> things. > >>>>>>>>> > >>>>>>>>> My suggestion for the long term: > >>>>>>>>> - let's fix this particular issue now with 3.6.0 quickly (I > >>> start > >>>>> doing > >>>>>>>>> this today) > >>>>>>>>> - let's do some automation (backed up with jenkins) that will > >>> test > >>>> a > >>>>>>>> whole > >>>>>>>>> combinations of different ZooKeeper upgrade paths by making > >>> rolling > >>>>>>>>> upgrades during some light traffic. Let's have a bit better > >>>>> definition > >>>>>>>>> about what we expect (e.g. the quorum is up, but some clients > >>> can > >>>> get > >>>>>>>>> disconnected? What will happen to the ephemeral nodes? Do we > >>> want > >>>> to > >>>>>>>>> gracefully close or transfer the user sessions before stopping > >>> the > >>>>> old > >>>>>>>>> server?) and let's see where this broke. Just by checking the > >>>> code, I > >>>>>>>> don't > >>>>>>>>> think the quorum will always be up (e.g. between older 3.4 > >>> versions > >>>>> and > >>>>>>>>> 3.5). > >>>>>>>> > >>>>>>>> > >>>>>>>> I am happy to work on this topic > >>>>>>>> > >>>>>>>>> - we need to update the Wiki about the working rolling upgrade > >>>> paths > >>>>> and > >>>>>>>>> maybe about workarounds if needed > >>>>>>>>> - we might need to do some fixes (adding backward compatible > >>>> versions > >>>>>>>>> and/or specific parameters that enforce old protocol temporary > >>>>> during the > >>>>>>>>> rolling upgrade that can be changed later to the new protocol by > >>>>> either > >>>>>>>>> dynamic reconfig or by rolling restart) > >>>>>>>> > >>>>>>>> it would be much better on 3.6 code to have some support for > >>>>>>>> compatibility with 3.5 servers > >>>>>>>> we can't require old code to be forward compatible but we can > >>> make > >>>> new > >>>>>>>> code be compatible to a certain extend with old code. > >>>>>>>> If we can achieve this compatibility goal without a flag is > >>> better, > >>>>>>>> users won't have to care about this part and they simply "trust" > >>> on > >>>> us > >>>>>>>> > >>>>>>>> The rollback story is also important, but maybe we are still not > >>>> ready > >>>>>>>> for it, in case of local changes to store, > >>>>>>>> it is better to have a clear design and plan and work for a new > >>>>> release > >>>>>>>> (3.7?) > >>>>>>>> > >>>>>>>> Enrico > >>>>>>>> > >>>>>>>>> > >>>>>>>>> Depending on your comments, I am happy to create a few Jira > >>> tickets > >>>>>>>> around > >>>>>>>>> these topics. > >>>>>>>>> > >>>>>>>>> Kind regards, > >>>>>>>>> Mate > >>>>>>>>> > >>>>>>>>> ps. Enrico, sorry about your RC... I owe you a beer, let me > >>> know if > >>>>> you > >>>>>>>> are > >>>>>>>>> near to Budapest ;) > >>>>>>>>> > >>>>>>>>> On Tue, Feb 11, 2020 at 8:43 AM Enrico Olivelli < > >>>> eolive...@gmail.com > >>>>>> > >>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> Good. > >>>>>>>>>> > >>>>>>>>>> I will cancel the vote for 3.6.0rc2. > >>>>>>>>>> > >>>>>>>>>> I appreciate very much If Mate and his colleagues have time to > >>>> work > >>>>> on > >>>>>>>> a > >>>>>>>>>> fix. > >>>>>>>>>> Otherwise I will have cycles next week > >>>>>>>>>> > >>>>>>>>>> I would also like to spend my time in setting up a few minimal > >>>>>>>> integration > >>>>>>>>>> tests about the upgrade story > >>>>>>>>>> > >>>>>>>>>> Enrico > >>>>>>>>>> > >>>>>>>>>> Il Mar 11 Feb 2020, 07:30 Michael Han <h...@apache.org> ha > >>>> scritto: > >>>>>>>>>> > >>>>>>>>>>> Kudos Enrico, very thorough work as the final gate keeper of > >>> the > >>>>>>>> release! > >>>>>>>>>>> > >>>>>>>>>>> Now with this, I'd like to *vote a -1* on the 3.6.0 RC2. > >>>>>>>>>>> > >>>>>>>>>>> I'd recommend we fix this issue for 3.6.0. ZooKeeper is one of > >>>> the > >>>>>>>> rare > >>>>>>>>>>> piece of software that put so much emphasis on compatibilities > >>>> thus > >>>>>>>> it > >>>>>>>>>> just > >>>>>>>>>>> works when upgrade / downgrade, which is amazing. One > >>> guarantee > >>>> we > >>>>>>>> always > >>>>>>>>>>> had is during rolling upgrade, the quorum will always be > >>>> available, > >>>>>>>>>> leading > >>>>>>>>>>> to no service interruption. It would be sad we lose such > >>>> capability > >>>>>>>> given > >>>>>>>>>>> this is still a tractable problem. > >>>>>>>>>>> > >>>>>>>>>>> Regarding the fix, can we just make 3.6.0 aware of the old > >>>> protocol > >>>>>>>> and > >>>>>>>>>>> speak old message format when it's talking to old server? > >>>>> Basically, > >>>>>>>> an > >>>>>>>>>>> ugly if else check against the protocol version should work > >>> and > >>>>>>>> there is > >>>>>>>>>> no > >>>>>>>>>>> need to have multiple pass on rolling upgrade process. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> On Mon, Feb 10, 2020 at 10:23 PM Enrico Olivelli < > >>>>>>>> eolive...@gmail.com> > >>>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> I suggest this plan: > >>>>>>>>>>>> - release 3.6.0 now > >>>>>>>>>>>> - improve the migration story, the flow outlined by Mate is > >>>>>>>>>>>> interesting, but it will take time > >>>>>>>>>>>> > >>>>>>>>>>>> 3.6.0rc2 got enough binding votes so I am going to finalize > >>> the > >>>>>>>>>>>> release this evening (within 8-10 hours) if no one comes out > >>> in > >>>>> the > >>>>>>>>>>>> VOTE thread with a -1 > >>>>>>>>>>>> > >>>>>>>>>>>> Enrico > >>>>>>>>>>>> > >>>>>>>>>>>> Enrico > >>>>>>>>>>>> > >>>>>>>>>>>> Il giorno lun 10 feb 2020 alle ore 19:33 Patrick Hunt > >>>>>>>>>>>> <ph...@apache.org> ha scritto: > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Mon, Feb 10, 2020 at 3:38 AM Andor Molnar < > >>> an...@apache.org > >>>>> > >>>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> Hi, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Answers inline. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> In my experience when you are close to a release it is > >>>>>>>> better to > >>>>>>>>>> to > >>>>>>>>>>>>>>> make big changes. (I am among the approvers of that patch, > >>>>>>>> so I > >>>>>>>>>> am > >>>>>>>>>>>>>>> responsible for this change) > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Although this statement is acceptable for me, I don’t feel > >>>> this > >>>>>>>>>> patch > >>>>>>>>>>>>>> should not have been merged into 3.6.0. Submission has been > >>>>>>>>>> preceded > >>>>>>>>>>>> by a > >>>>>>>>>>>>>> long argument with MAPR folks who originally wanted to be > >>>>>>>> merged > >>>>>>>>>> into > >>>>>>>>>>>> 3.4 > >>>>>>>>>>>>>> branch (considering the pace how ZooKeeper community is > >>> moving > >>>>>>>>>>>> forward) and > >>>>>>>>>>>>>> we reached an agreement that release it with 3.6.0. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Make a long story short, this patch has been outstanding > >>> for > >>>>>>>> ages > >>>>>>>>>>>> without > >>>>>>>>>>>>>> much attention from the community and contributors made a > >>> lot > >>>>>>>> of > >>>>>>>>>>>> effort to > >>>>>>>>>>>>>> get it done before the release. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I would like to ear from people that have been in the > >>>>>>>> community > >>>>>>>>>> for > >>>>>>>>>>>>>>> long time, then I am ready to complete the release process > >>>>>>>> for > >>>>>>>>>>>>>>> 3.6.0rc2. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Me too. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I tend to accept the way rolling restart works now - as you > >>>>>>>>>> described > >>>>>>>>>>>>>> Enrico - and given that situation was pretty much the same > >>>>>>>> between > >>>>>>>>>>> 3.4 > >>>>>>>>>>>> and > >>>>>>>>>>>>>> 3.5, I don’t feel we have to make additional changes. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On the other hand, the fix that Mate suggested sounds quite > >>>>>>>> cool, > >>>>>>>>>> I’m > >>>>>>>>>>>> also > >>>>>>>>>>>>>> happy to work on getting it in. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Fyi, Release Management page says the following: > >>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>> > >>> > https://cwiki.apache.org/confluence/display/ZOOKEEPER/ReleaseManagement > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> "major.minor release of ZooKeeper must be backwards > >>> compatible > >>>>>>>> with > >>>>>>>>>>> the > >>>>>>>>>>>>>> previous minor release, major.(minor-1)" > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> Our users, direct and indirect, value the ability to > >>> migrate to > >>>>>>>> newer > >>>>>>>>>>>>> versions - esp as we drop support for older. Frictions such > >>> as > >>>>>>>> this > >>>>>>>>>> can > >>>>>>>>>>>> be > >>>>>>>>>>>>> a reason to go elsewhere. I'm "pro" b/w compact - esp given > >>> our > >>>>>>>>>>> published > >>>>>>>>>>>>> guidelines. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Patrick > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>>> Andor > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On 2020. Feb 10., at 11:32, Enrico Olivelli < > >>>>>>>> eolive...@gmail.com > >>>>>>>>>>> > >>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Thank you Mate for checking and explaining this story. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I find it very interesting that the cause is > >>> ZOOKEEPER-3188 > >>>>>>>> as: > >>>>>>>>>>>>>>> - it is the last "big patch" committed to 3.6 before > >>>>>>>> starting the > >>>>>>>>>>>>>>> release process > >>>>>>>>>>>>>>> - it is the cause of the failure of the first RC > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> In my experience when you are close to a release it is > >>>>>>>> better to > >>>>>>>>>> to > >>>>>>>>>>>>>>> make big changes. (I am among the approvers of that patch, > >>>>>>>> so I > >>>>>>>>>> am > >>>>>>>>>>>>>>> responsible for this change) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> This is a pointer to the change to whom who wants to > >>>>>>>> understand > >>>>>>>>>>>> better > >>>>>>>>>>>>>>> the context > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>> > >>>> > >>> > https://github.com/apache/zookeeper/pull/1048/files#diff-7a209d890686bcba351d758b64b22a7dR11 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> IIUC even for the upgrade from 3.4 to 3.5 the story was > >>> the > >>>>>>>> same > >>>>>>>>>>> and > >>>>>>>>>>>>>>> if this statement holds then I feel we can continue > >>>>>>>>>>>>>>> with this release. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> - Reverting ZOOKEEPER-3188 is not an option for me, it is > >>> too > >>>>>>>>>>>> complex. > >>>>>>>>>>>>>>> - Making 3.5 and 3.6 "compatible" can be very tricky and > >>> we > >>>>>>>> do > >>>>>>>>>> not > >>>>>>>>>>>>>>> have tools to certify this compatibility (at least not in > >>> the > >>>>>>>>>> short > >>>>>>>>>>>>>>> term) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I would like to ear from people that have been in the > >>>>>>>> community > >>>>>>>>>> for > >>>>>>>>>>>>>>> long time, then I am ready to complete the release process > >>>>>>>> for > >>>>>>>>>>>>>>> 3.6.0rc2. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I will update the website and the release notes with a > >>>>>>>> specific > >>>>>>>>>>>>>>> warning about the upgrade, we should also update the Wiki > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Enrico > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Il giorno lun 10 feb 2020 alle ore 11:17 Szalay-Bekő Máté > >>>>>>>>>>>>>>> <szalay.beko.m...@gmail.com> ha scritto: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Hi Enrico! > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> This is caused by the different PROTOCOL_VERSION in the > >>>>>>>>>>>>>> QuorumCnxManager. > >>>>>>>>>>>>>>>> The Protocol version was changed last time in > >>>>>>>> ZOOKEEPER-2186 > >>>>>>>>>>>> released > >>>>>>>>>>>>>>>> first in 3.4.7 and 3.5.1 to avoid some crashing / fix > >>> some > >>>>>>>> bugs. > >>>>>>>>>>>> Later I > >>>>>>>>>>>>>>>> also changed the protocol version when the format of the > >>>>>>>> initial > >>>>>>>>>>>> message > >>>>>>>>>>>>>>>> changed in ZOOKEEPER-3188. So actually the quorum > >>> protocol > >>>>>>>> is > >>>>>>>>>> not > >>>>>>>>>>>>>>>> compatible in this case and is the 'expected' behavior if > >>>>>>>> you > >>>>>>>>>>>> upgrade > >>>>>>>>>>>>>> e.g > >>>>>>>>>>>>>>>> from 3.4.6 to 3.4.7, or 3.4.6 to 3.5.5 or e.g from 3.5.6 > >>> to > >>>>>>>>>> 3.6.0. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> We had some discussion in the PR of ZOOKEEPER-3188 back > >>>>>>>> then and > >>>>>>>>>>>> got to > >>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>> conclusion that it is not that bad, as there will be no > >>> data > >>>>>>>>>> loss > >>>>>>>>>>>> as you > >>>>>>>>>>>>>>>> wrote. The tricky thing is that during rolling upgrade we > >>>>>>>> should > >>>>>>>>>>>> ensure > >>>>>>>>>>>>>>>> both backward and forward compatibility to make sure that > >>>>>>>> the > >>>>>>>>>> old > >>>>>>>>>>>> and > >>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>> new part of the quorum can still speak to each other. The > >>>>>>>>>> current > >>>>>>>>>>>>>> solution > >>>>>>>>>>>>>>>> (simply failing if the protocol versions mismatch) is > >>> more > >>>>>>>>>> simple > >>>>>>>>>>>> and > >>>>>>>>>>>>>> still > >>>>>>>>>>>>>>>> working just fine: as the servers are restarted > >>> one-by-one, > >>>>>>>> the > >>>>>>>>>>>> nodes > >>>>>>>>>>>>>> with > >>>>>>>>>>>>>>>> the old protocol version and the nodes with the new > >>> protocol > >>>>>>>>>>> version > >>>>>>>>>>>>>> will > >>>>>>>>>>>>>>>> form two partitions, but any given time only one > >>> partition > >>>>>>>> will > >>>>>>>>>>>> have the > >>>>>>>>>>>>>>>> quorum. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Still, thinking it trough, as a side effect in these > >>> cases > >>>>>>>> there > >>>>>>>>>>>> will > >>>>>>>>>>>>>> be a > >>>>>>>>>>>>>>>> short time when none of the partitions will have quorums > >>>>>>>> (when > >>>>>>>>>> we > >>>>>>>>>>>> have N > >>>>>>>>>>>>>>>> servers with the old protocol version, N servers with the > >>>>>>>> new > >>>>>>>>>>>> protocol > >>>>>>>>>>>>>>>> version, and there is one server just being restarted). I > >>>>>>>> am not > >>>>>>>>>>>> sure > >>>>>>>>>>>>>> if we > >>>>>>>>>>>>>>>> can accept this. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> For ZOOKEEPER-3188 we can add a small patch to make it > >>>>>>>> possible > >>>>>>>>>> to > >>>>>>>>>>>> parse > >>>>>>>>>>>>>>>> the initial message of the old protocol version with the > >>> new > >>>>>>>>>> code. > >>>>>>>>>>>> But > >>>>>>>>>>>>>> I am > >>>>>>>>>>>>>>>> not sure if it would be enough (as the old code will not > >>> be > >>>>>>>> able > >>>>>>>>>>> to > >>>>>>>>>>>>>> parse > >>>>>>>>>>>>>>>> the new initial message). > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> One option can be to make a patch also for 3.5 to have a > >>>>>>>> version > >>>>>>>>>>>> which > >>>>>>>>>>>>>>>> supports both protocol versions. (let's say in 3.5.8) > >>> Then > >>>>>>>> we > >>>>>>>>>> can > >>>>>>>>>>>> write > >>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>> the release note, that if you need rolling upgrade from > >>> any > >>>>>>>>>>> versions > >>>>>>>>>>>>>> since > >>>>>>>>>>>>>>>> 3.4.7, then you have to first upgrade from 3.5.8 before > >>>>>>>>>> upgrading > >>>>>>>>>>> to > >>>>>>>>>>>>>> 3.6.0. > >>>>>>>>>>>>>>>> We can even make the same thing on the 3.4 branch. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> But I am also new to the community... It would be great > >>> to > >>>>>>>> hear > >>>>>>>>>>> the > >>>>>>>>>>>>>> opinion > >>>>>>>>>>>>>>>> of more experienced people. > >>>>>>>>>>>>>>>> Whatever the decision will be, I am happy to make the > >>>>>>>> changes. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> And sorry for breaking the RC (if we decide that this > >>> needs > >>>>>>>> to > >>>>>>>>>> be > >>>>>>>>>>>>>>>> changed...). ZOOKEEPER-3188 was a complex patch. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Kind regards, > >>>>>>>>>>>>>>>> Mate > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On Mon, Feb 10, 2020 at 9:47 AM Enrico Olivelli < > >>>>>>>>>>>> eolive...@gmail.com> > >>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Hi, > >>>>>>>>>>>>>>>>> even if we had enough binding +1 on 3.6.0rc2 before > >>>>>>>> closing the > >>>>>>>>>>>> VOTE > >>>>>>>>>>>>>>>>> of 3.6.0 I wanted to finish my tests and I am coming to > >>> an > >>>>>>>>>>> apparent > >>>>>>>>>>>>>>>>> blocker. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I am trying to upgrade a 3.5.6 cluster to 3.6.0, but it > >>>>>>>> looks > >>>>>>>>>>> like > >>>>>>>>>>>>>>>>> peers are not able to talk to each other. > >>>>>>>>>>>>>>>>> I have a cluster of 3, server1, server2 and server3. > >>>>>>>>>>>>>>>>> When I upgrade server1 to 3.6.0rc2 I see this kind of > >>>>>>>> errors on > >>>>>>>>>>> 3.5 > >>>>>>>>>>>>>> nodes: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> 2020-02-10 09:35:07,745 [myid:3] - INFO > >>>>>>>>>>>>>>>>> [localhost/127.0.0.1:3334:QuorumCnxManager$Listener@918] > >>> - > >>>>>>>>>>>> Received > >>>>>>>>>>>>>>>>> connection request 127.0.0.1:62591 > >>>>>>>>>>>>>>>>> 2020-02-10 09:35:07,746 [myid:3] - ERROR > >>>>>>>>>>>>>>>>> [localhost/127.0.0.1:3334:QuorumCnxManager@527] - > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>> > >>>> > >>> > org.apache.zookeeper.server.quorum.QuorumCnxManager$InitialMessage$InitialMessageException: > >>>>>>>>>>>>>>>>> Got unrecognized protocol version -65535 > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Once I upgrade all of the peers the system is up and > >>>>>>>> running, > >>>>>>>>>>>> without > >>>>>>>>>>>>>>>>> apparently no data loss. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> During the upgrade as soon as I upgrade the first node, > >>>>>>>> say, > >>>>>>>>>>>> server1, > >>>>>>>>>>>>>>>>> server1 is not able to accept connections (error "Close > >>> of > >>>>>>>>>>> session > >>>>>>>>>>>> 0x0 > >>>>>>>>>>>>>>>>> java.io.IOException: ZooKeeperServer not running") from > >>>>>>>>>> clients, > >>>>>>>>>>>> this > >>>>>>>>>>>>>>>>> is expected, because as far as it cannot talk with the > >>>>>>>> other > >>>>>>>>>>> peers > >>>>>>>>>>>> it > >>>>>>>>>>>>>>>>> is practically partitioned away from the cluster. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> My questions are: > >>>>>>>>>>>>>>>>> 1) is this expected ? I can't remember protocol changes > >>>>>>>> from > >>>>>>>>>> 3.5 > >>>>>>>>>>> to > >>>>>>>>>>>>>>>>> 3.6, but actually 3.6 diverged from 3.5 branch so long > >>> ago, > >>>>>>>>>> and I > >>>>>>>>>>>> was > >>>>>>>>>>>>>>>>> not in the community as dev so I cannot tell > >>>>>>>>>>>>>>>>> 2) is this a viable option for users ? to have some > >>>>>>>> temporary > >>>>>>>>>>>> glitch > >>>>>>>>>>>>>>>>> during the upgrade and hope that the upgrade completes > >>>>>>>> without > >>>>>>>>>>>>>>>>> troubles ? > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> In theory as long as two servers are running the same > >>> major > >>>>>>>>>>> version > >>>>>>>>>>>>>>>>> (3.5 or 3.6) we have a quorum and the system is able to > >>>>>>>> make > >>>>>>>>>>>> progress > >>>>>>>>>>>>>>>>> and to server clients. > >>>>>>>>>>>>>>>>> I feel that this is quite dangerous, but I don't have > >>>>>>>> enough > >>>>>>>>>>>> context > >>>>>>>>>>>>>>>>> to understand how this problem is possible and when we > >>>>>>>> decided > >>>>>>>>>> to > >>>>>>>>>>>>>>>>> break compatibility. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> The other option is that I am wrong in my test and I am > >>>>>>>> messing > >>>>>>>>>>> up > >>>>>>>>>>>> :-) > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> The other upgrade path I would like to see working like > >>> a > >>>>>>>> charm > >>>>>>>>>>> is > >>>>>>>>>>>> the > >>>>>>>>>>>>>>>>> upgrade from 3.4 to 3.6, as I see that as soon as we > >>>>>>>> release > >>>>>>>>>> 3.6 > >>>>>>>>>>> we > >>>>>>>>>>>>>>>>> should encourage users to move to 3.6 and not to 3.5. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Regards > >>>>>>>>>>>>>>>>> Enrico > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >>> > >
