This would make the work I’ve done with TTLs and Containers more manageable. I can get rid of the ugly ephemeralOwner hacks.
-Jordan > On May 25, 2016, at 10:35 AM, Mohammad arshad <[email protected]> > wrote: > > Hi All, > Finally, decided to use upgrade tool for backward compatibility. Here is, > what is done overall > > 1) Modified StatPersisted to hold any number of new fields. Now it will be > easy to add new fields without breaking the backward compatibility > > Reference from zookeeper.jute: > class StatPersisted { > ---- > long pzxid; // last modified children > vector<org.apache.zookeeper.data.Property> extAttrs; // ZooKeeper data > structure extended attributes > } > class Property { > ustring key; > ustring value; > } > 2) Modified TxnHeader to pass the extra fields, > class TxnHeader { > ----- > int type; > vector<org.apache.zookeeper.data.Property> extAttrs; // extra > attributes > } > > 3) Created StatPersistedV0 and TxnHeaderV0 which have only old fields. These > are used to upgrade the zookeeper data. > > As zookeeper is not migrating to protocol buffers, at least in near future, > can we use this approach to make the zookeeper database easy to extend? > If yes, I can work on it and contribute it. > > > Best Regards > Mohammad Arshad > HUAWEI TECHNOLOGIES CO.LTD. > Huawei Tecnologies India Pvt. Ltd. > Near EPIP Industrial Area, Kundalahalli Village > Whitefield, Bangalore-560066 > www.huawei.com > ----------------------------------------------------------------------------------------------------------------- > This e-mail and its attachments contain confidential information from HUAWEI, > which > is intended only for the person or entity whose address is listed above. Any > use of the > information contained herein in any way (including, but not limited to, total > or partial > disclosure, reproduction, or dissemination) by persons other than the > intended > recipient(s) is prohibited. If you receive this e-mail in error, please > notify the sender by > phone or email immediately and delete it! > > > -----Original Message----- > From: Mohammad arshad > Sent: 24 May 2016 11:12 > To: DevZooKeeper > Subject: RE: How to extend ZooKeeper data structure without breaking the > backward compatibility > > Thanks Patrick and Falvio > > I also was more inclined towards the option 3 but it does not solve the > problem. > It works for bellow scenarios > i) all new server with fresh data > ii) all new servers but with old data > > As Patrick already pointed out, mixed servers have some unavoidable problems > (AFAIK). > Option 3 fails for bellow mix servers mode scenarios > > i) Suppose existing quorum of old servers is running, another server (new > server) joins the quorum, while receiving the snapshot from the old server > failure occurs. > This is because leader does not send FileHeader. So there is nothing which > can used to decide which version to de serialize. > Also cannot change the leader code to send the FileHeader while sending the > snapshot, as this will cascade the problem. > ii) In the rolling upgrade scenario, suppose new server becomes the leader, > while sending the data to older follower, follower will fail as it will not > know how to de-serialize the new data > > ------------------------------------------------------------ > FYI: > Here is what I was doing to implement option 3 > a) Increment the dbId in the code > b) Decide whether database is old or not based on the dbId from the > FileHeader De-serialize old data model StatPersistedV0 copy data from old > data model StatPersistedV0 to new data model StatPersisted but off course, > the new field cuser will have null. > > stat = new StatPersisted(); > if (dbIdFromFileHeader < currentDbId) > { > StatPersistedV0 old=new StatPersistedV0(); > old.deserialize(archive, "statpersisted"); > copyFromOldToNewVersion(old, stat); > }else > { > stat.deserialize(archive, "statpersisted"); } > ----------------------------------------------------------- > > is there anything which can be done to make the Option 3 work for mix mode > server cases? > > > Best Regards > Mohammad Arshad > HUAWEI TECHNOLOGIES CO.LTD. > Huawei Tecnologies India Pvt. Ltd. > Near EPIP Industrial Area, Kundalahalli Village Whitefield, Bangalore-560066 > www.huawei.com > ----------------------------------------------------------------------------------------------------------------- > This e-mail and its attachments contain confidential information from HUAWEI, > which is intended only for the person or entity whose address is listed > above. Any use of the information contained herein in any way (including, but > not limited to, total or partial disclosure, reproduction, or dissemination) > by persons other than the intended > recipient(s) is prohibited. If you receive this e-mail in error, please > notify the sender by phone or email immediately and delete it! > > > -----Original Message----- > From: Patrick Hunt [mailto:[email protected]] > Sent: 13 May 2016 03:58 > To: DevZooKeeper > Subject: Re: How to extend ZooKeeper data structure without breaking the > backward compatibility > > I thought we added the version to the file header for exactly this case. > Allowing the server to differentiate in case we need to upgrade the on-disk > format. Option 3 seems the best to me -- "one time conversions" typically > have bad corner cases. > > You'll also want to think through the upgrade implications re having mixed > server versions in the ensemble (some old and some new). Might be worth > thinking about downgrade for that matter... > > Patrick > > On Wed, May 11, 2016 at 2:13 PM, Flavio Junqueira <[email protected]> wrote: > >> We currently put a version in the directory name and afaict the >> version isn't used anywhere else. We also have a version in FileHeader. >> >> The options I see are: >> >> - We write a tool to convert the data from one format to another. In >> the case of upgrade, we run the tool before starting the upgraded >> server. If we do it, then we will be simply moving all the data from >> the old version to the new version. >> - We can do the same thing as in the first bullet without the tool in >> the case the server detects that the current data is in the old >> format. In this case, the serves converts it all before it fully >> starts. This procedure will of course introduce some delay when starting an >> upgraded server. >> - Another way is to mix files with different versions and then let the >> server figure out how to process the data with a given version. >> >> Option 3 is a bit messier because we will be mixing files of different >> versions, but I'd be interested in hearing opinions. Does it make >> sense at all? >> >> -Flavio >> >> >>> On 11 May 2016, at 18:06, Flavio P JUNQUEIRA <[email protected]> wrote: >>> >>> We need a file format version in this case. Let me have look and >>> get >> back >>> to you. >>> >>> -Flavio >>> On 11 May 2016 15:49, "Mohammad arshad" <[email protected]> >> wrote: >>> >>> Loading snapshot file into memory(deserialization) does not involve >>> any protocol version. >>> are you suggesting version and dbId from FileHeader? >>> Which field (version, dbId) would be more appropriate to decide >>> which version ,StatPersisted or StatPersistedV0, to be deserialized. >>> I think it should be the dbId. >>> >>> Best Regards >>> Mohammad Arshad >>> HUAWEI TECHNOLOGIES CO.LTD. >>> Huawei Tecnologies India Pvt. Ltd. >>> Near EPIP Industrial Area, Kundalahalli Village Whitefield, >>> Bangalore-560066 www.huawei.com >>> >> ---------------------------------------------------------------------- >> ------------------------------------------- >>> This e-mail and its attachments contain confidential information >>> from HUAWEI, which is intended only for the person or entity whose >>> address is listed above. >>> Any use of the >>> information contained herein in any way (including, but not limited >>> to, total or partial disclosure, reproduction, or dissemination) by >>> persons other than the intended >>> recipient(s) is prohibited. If you receive this e-mail in error, >>> please notify the sender by phone or email immediately and delete >>> it! >>> >>> >>> -----Original Message----- >>> From: Flavio Junqueira [mailto:[email protected]] >>> Sent: 11 May 2016 18:18 >>> To: [email protected] >>> Cc: [email protected] >>> Subject: Re: How to extend ZooKeeper data structure without breaking >>> the backward compatibility >>> >>> In this case, I guess we have to check the protocol version and >>> depending on the version deserialize differently. >>> >>> -Flavio >>> >>>> On 11 May 2016, at 13:21, Mohammad arshad >>>> <[email protected]> >>> wrote: >>>> >>>> Thanks Flavio, >>>> >>>> I think CreateTxn kind of solution cannot be applied in this >>>> scenario I can case of transaction the records are separable but in >>>> case of snapshot records are continuous In this scenario, cuser >>>> de-serialization eats into the next record which causes bigger >>>> problem >>>> >>>> Here is exception I got after adding cuser and running the >>>> ZooKeeper on old data >>>> java.io.IOException: Unreasonable length = 796553071 >>>> at >>> >> org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java >> :127) >>>> at >>> org.apache.jute.BinaryInputArchive.readString(BinaryInputArchive.jav >>> a:81) >>>> at >>> >> org.apache.zookeeper.data.StatPersisted.deserialize(StatPersisted.java >> :143) >>>> at >>> org.apache.zookeeper.server.DataNode.deserialize(DataNode.java:173) >>>> at >>> org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.jav >>> a:99) >>>> at >>>> org.apache.zookeeper.server.DataTree.deserialize(DataTree.java:1278 >>>> ) >>>> >>>> >>>> Best Regards >>>> Mohammad Arshad >>>> HUAWEI TECHNOLOGIES CO.LTD. >>>> Huawei Tecnologies India Pvt. Ltd. >>>> Near EPIP Industrial Area, Kundalahalli Village Whitefield, >>>> Bangalore-560066 www.huawei.com >>>> ------------------------------------------------------------------- >>>> --- >>>> ------------------------------------------- >>>> This e-mail and its attachments contain confidential information >>>> from HUAWEI, which is intended only for the person or entity whose >>>> address is listed above. Any use of the information contained >>>> herein in any way (including, but not limited to, total or partial >>>> disclosure, reproduction, or dissemination) by persons other than >>>> the intended >>>> recipient(s) is prohibited. If you receive this e-mail in error, >>>> please notify the sender by phone or email immediately and delete it! >>>> >>>> -----Original Message----- >>>> From: Flavio Junqueira [mailto:[email protected]] >>>> Sent: 11 May 2016 16:02 >>>> To: [email protected] >>>> Cc: Zookeeper >>>> Subject: Re: How to extend ZooKeeper data structure without >>>> breaking the backward compatibility >>>> >>>> In the past, we simply created two versions of the data structure: >>>> >>>> class CreateTxnV0 { >>>> ustring path; >>>> buffer data; >>>> vector<org.apache.zookeeper.data.ACL> acl; >>>> boolean ephemeral; >>>> } >>>> class CreateTxn { >>>> ustring path; >>>> buffer data; >>>> vector<org.apache.zookeeper.data.ACL> acl; >>>> boolean ephemeral; >>>> int parentCVersion; >>>> } >>>> >>>> and deal with it in the code. It is not ideal and the serialization >>> framework is actually a pretty old one. >>>> >>>> -Flavio >>>> >>>>> On 11 May 2016, at 11:14, Mohammad arshad >>>>> <[email protected]> >>> wrote: >>>>> >>>>> Hello Everyone >>>>> >>>>> is there any way to extend ZooKeeper data structure without >>>>> breaking >> the >>> backward compatibility. >>>>> Suppose I want to add cuser field in StatPersisted class >>>>> StatPersisted { >>>>> ........ >>>>> ustring cuser; //user who create the node >>>>> } >>>>> This is fine with fresh ZooKeeper server installation, where >>>>> ZooKeeper data is created freshly But this causes problem while >>> de-serializing when server is new but points to old ZooKeeper data. >>>>> Here by ZooKeeper data I mean snapshot and transaction log >>>>> >>>>> In protocol buffers a field can be optional but in jute optional >>>>> field >>> is not supported. >>>>> is there any way to overcome this jute limitation? any work around? >>>>> >>>>> Best Regards >>>>> Mohammad Arshad >>>>> HUAWEI TECHNOLOGIES CO.LTD. >>>>> Huawei Tecnologies India Pvt. Ltd. >>>>> Near EPIP Industrial Area, Kundalahalli Village Whitefield, >>>>> Bangalore-560066 www.huawei.com<http://www.huawei.com/> >>>>> ------------------------------------------------------------------ >>>>> --- >>>>> - >>>>> ------------------------------------------- >>>>> This e-mail and its attachments contain confidential information >>>>> from HUAWEI, which is intended only for the person or entity whose >>>>> address is listed above. Any use of the information contained >>>>> herein in any way (including, but not limited to, total or partial >>>>> disclosure, reproduction, or dissemination) by persons other than >>>>> the intended >>>>> recipient(s) is prohibited. If you receive this e-mail in error, >>>>> please notify the sender by phone or email immediately and delete it! >>>>> >>>> >> >>
