I thought we added the version to the file header for exactly this case. Allowing the server to differentiate in case we need to upgrade the on-disk format. Option 3 seems the best to me -- "one time conversions" typically have bad corner cases.
You'll also want to think through the upgrade implications re having mixed server versions in the ensemble (some old and some new). Might be worth thinking about downgrade for that matter... Patrick On Wed, May 11, 2016 at 2:13 PM, Flavio Junqueira <f...@apache.org> wrote: > We currently put a version in the directory name and afaict the version > isn't used anywhere else. We also have a version in FileHeader. > > The options I see are: > > - We write a tool to convert the data from one format to another. In the > case of upgrade, we run the tool before starting the upgraded server. If we > do it, then we will be simply moving all the data from the old version to > the new version. > - We can do the same thing as in the first bullet without the tool in the > case the server detects that the current data is in the old format. In this > case, the serves converts it all before it fully starts. This procedure > will of course introduce some delay when starting an upgraded server. > - Another way is to mix files with different versions and then let the > server figure out how to process the data with a given version. > > Option 3 is a bit messier because we will be mixing files of different > versions, but I'd be interested in hearing opinions. Does it make sense at > all? > > -Flavio > > > > On 11 May 2016, at 18:06, Flavio P JUNQUEIRA <f...@apache.org> wrote: > > > > We need a file format version in this case. Let me have look and get > back > > to you. > > > > -Flavio > > On 11 May 2016 15:49, "Mohammad arshad" <mohammad.ars...@huawei.com> > wrote: > > > > Loading snapshot file into memory(deserialization) does not involve any > > protocol version. > > are you suggesting version and dbId from FileHeader? > > Which field (version, dbId) would be more appropriate to decide which > > version ,StatPersisted or StatPersistedV0, to be deserialized. I think it > > should be the dbId. > > > > Best Regards > > Mohammad Arshad > > HUAWEI TECHNOLOGIES CO.LTD. > > Huawei Tecnologies India Pvt. Ltd. > > Near EPIP Industrial Area, Kundalahalli Village > > Whitefield, Bangalore-560066 > > www.huawei.com > > > ----------------------------------------------------------------------------------------------------------------- > > This e-mail and its attachments contain confidential information from > > HUAWEI, which > > is intended only for the person or entity whose address is listed above. > > Any use of the > > information contained herein in any way (including, but not limited to, > > total or partial > > disclosure, reproduction, or dissemination) by persons other than the > > intended > > recipient(s) is prohibited. If you receive this e-mail in error, please > > notify the sender by > > phone or email immediately and delete it! > > > > > > -----Original Message----- > > From: Flavio Junqueira [mailto:f...@apache.org] > > Sent: 11 May 2016 18:18 > > To: u...@zookeeper.apache.org > > Cc: dev@zookeeper.apache.org > > Subject: Re: How to extend ZooKeeper data structure without breaking the > > backward compatibility > > > > In this case, I guess we have to check the protocol version and depending > > on the version deserialize differently. > > > > -Flavio > > > >> On 11 May 2016, at 13:21, Mohammad arshad <mohammad.ars...@huawei.com> > > wrote: > >> > >> Thanks Flavio, > >> > >> I think CreateTxn kind of solution cannot be applied in this scenario > >> I can case of transaction the records are separable but in case of > >> snapshot records are continuous In this scenario, cuser > >> de-serialization eats into the next record which causes bigger problem > >> > >> Here is exception I got after adding cuser and running the ZooKeeper > >> on old data > >> java.io.IOException: Unreasonable length = 796553071 > >> at > > > org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:127) > >> at > > org.apache.jute.BinaryInputArchive.readString(BinaryInputArchive.java:81) > >> at > > > org.apache.zookeeper.data.StatPersisted.deserialize(StatPersisted.java:143) > >> at > > org.apache.zookeeper.server.DataNode.deserialize(DataNode.java:173) > >> at > > org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99) > >> at > >> org.apache.zookeeper.server.DataTree.deserialize(DataTree.java:1278) > >> > >> > >> Best Regards > >> Mohammad Arshad > >> HUAWEI TECHNOLOGIES CO.LTD. > >> Huawei Tecnologies India Pvt. Ltd. > >> Near EPIP Industrial Area, Kundalahalli Village Whitefield, > >> Bangalore-560066 www.huawei.com > >> ---------------------------------------------------------------------- > >> ------------------------------------------- > >> This e-mail and its attachments contain confidential information from > >> HUAWEI, which is intended only for the person or entity whose address > >> is listed above. Any use of the information contained herein in any > >> way (including, but not limited to, total or partial disclosure, > >> reproduction, or dissemination) by persons other than the intended > >> recipient(s) is prohibited. If you receive this e-mail in error, > >> please notify the sender by phone or email immediately and delete it! > >> > >> -----Original Message----- > >> From: Flavio Junqueira [mailto:f...@apache.org] > >> Sent: 11 May 2016 16:02 > >> To: u...@zookeeper.apache.org > >> Cc: Zookeeper > >> Subject: Re: How to extend ZooKeeper data structure without breaking > >> the backward compatibility > >> > >> In the past, we simply created two versions of the data structure: > >> > >> class CreateTxnV0 { > >> ustring path; > >> buffer data; > >> vector<org.apache.zookeeper.data.ACL> acl; > >> boolean ephemeral; > >> } > >> class CreateTxn { > >> ustring path; > >> buffer data; > >> vector<org.apache.zookeeper.data.ACL> acl; > >> boolean ephemeral; > >> int parentCVersion; > >> } > >> > >> and deal with it in the code. It is not ideal and the serialization > > framework is actually a pretty old one. > >> > >> -Flavio > >> > >>> On 11 May 2016, at 11:14, Mohammad arshad <mohammad.ars...@huawei.com> > > wrote: > >>> > >>> Hello Everyone > >>> > >>> is there any way to extend ZooKeeper data structure without breaking > the > > backward compatibility. > >>> Suppose I want to add cuser field in StatPersisted class > >>> StatPersisted { > >>> ........ > >>> ustring cuser; //user who create the node > >>> } > >>> This is fine with fresh ZooKeeper server installation, where > >>> ZooKeeper data is created freshly But this causes problem while > > de-serializing when server is new but points to old ZooKeeper data. > >>> Here by ZooKeeper data I mean snapshot and transaction log > >>> > >>> In protocol buffers a field can be optional but in jute optional field > > is not supported. > >>> is there any way to overcome this jute limitation? any work around? > >>> > >>> Best Regards > >>> Mohammad Arshad > >>> HUAWEI TECHNOLOGIES CO.LTD. > >>> Huawei Tecnologies India Pvt. Ltd. > >>> Near EPIP Industrial Area, Kundalahalli Village Whitefield, > >>> Bangalore-560066 www.huawei.com<http://www.huawei.com/> > >>> --------------------------------------------------------------------- > >>> - > >>> ------------------------------------------- > >>> This e-mail and its attachments contain confidential information from > >>> HUAWEI, which is intended only for the person or entity whose address > >>> is listed above. Any use of the information contained herein in any > >>> way (including, but not limited to, total or partial disclosure, > >>> reproduction, or dissemination) by persons other than the intended > >>> recipient(s) is prohibited. If you receive this e-mail in error, > >>> please notify the sender by phone or email immediately and delete it! > >>> > >> > >
