>> I'm fine with suggesting we keep the list short, but isn't saying we keep 2 bits free an implementation concern for protocols?
I'd take Chad's suggestion here as an implementation recommendation, not a requirement. I forgot to mention in my API change email that we also need to change the signatures of readFieldBegin and readFieldEnd. Both of those methods will also need to pass/return type hints. How they send the hints over the wire is up to the protocol implementation. Using the top 2 bits of the type field seems a reasonable suggestion in the TBinaryProtocol world. >> I presume this would mean that the default modifier should be interpreted as "variable", and that protocols can feel free to compress all ints by default. Yes, the default behavior should be totally up to the Protocol implementation. -----Original Message----- From: Bryan Duxbury [mailto:[EMAIL PROTECTED] Sent: Thursday, August 21, 2008 5:07 PM To: [email protected] Subject: Re: Adding a "vint" type to Thrift Comments inline. On Aug 21, 2008, at 4:57 PM, Chad Walters wrote: > Looking at things from the standpoint of efficiency also argues for a > limited set of modifiers, since we need to actually pass this > information on the wire. > > In THRIFT-110, I suggest that we reserve 2 bits at the top of the type > byte for modifiers. This allows for 3 different modifiers for any > given primary type (plus a default). I'm fine with suggesting we keep the list short, but isn't saying we keep 2 bits free an implementation concern for protocols? > For integer types, I suggested that we add "fixed" and > "zipper"/"zigzag". The default assumption would be that a variable > encoding could be used. "fixed" would mean that variable encoding is > unlikely to be helpful, and "zipper"/"zigzag" would suggest a variable > encoding suitable for fields likely to contain a mix of negative and > positive values. I presume this would mean that the default modifier should be interpreted as "variable", and that protocols can feel free to compress all ints by default. > For string types, I suggested that we add "extern". Strings marked > with this modifier could be stored in some separate indexed table and > referenced by index when passed through the protocol. > > Does this sound like a good starting point? Are there other modifiers > we'd like to add? > > Chad > > On 8/21/08 4:41 PM, "Mark Slee" <[EMAIL PROTECTED]> wrote: > > I'd propose starting with a fixed set of type hints that we agree > upon. > We could certainly implement a dynamic any-string goes type of thing. > But here's my concern with that: > > 1/ If we agree upon a fixed set of modifiers, we're more likely to get > them implemented across more languages since deciding on the set will > involve getting some basic agreement amongst the Thrift community. > We're > likely to retain more overall compatibility. > > 2/ If the modifiers are dynamic, it implies that people would make > their own local changes, and write their own custom Protocol > implementations. > We're more likely to duplicate this work or disagree on things. This > will make custom protocol submissions back into the trunk more > difficult, and the core project becomes more likely to lose quality > work. > > So, while a dynamic implementation is arguably more elegant/ flexible, > I think having the community agree upon a reasonable set of type > annotations is most healthy for the coherence of Thrift in the long > run. > > Cheers, > Mark > > -----Original Message----- > From: Bryan Duxbury [mailto:[EMAIL PROTECTED] > Sent: Thursday, August 21, 2008 3:40 PM > To: [email protected] > Subject: Re: Adding a "vint" type to Thrift > > I'm not opposed to using a type modifier approach instead of a > separate type. Will we really be adding "arbitrary" annotations to the > objects, or will they be of a specific nature? That is, will we have > modifiers like "fixed", "variable", "extern", etc. be compiler tokens, > or will you just be allowed to add any string you want as a modifier > before the type? The latter would certainly be more flexible, but > would be more complicated as well. > > -Bryan > > On Aug 21, 2008, at 2:07 PM, David Reiss wrote: > >> First, I don't understand why it would be limited to 64-bit. What if >> I have an integer that I want represented as 32 bits in memory but >> serialized with a variable-length encoding. >> >> Second, I wonder if creating a new type is the best solution. I've >> been thinking for a while that it might be useful to add arbitrary >> annotations to types. We already have a few special cases of this >> (cpp_type in C++, slist in Erlang), and vint would be a third. I >> think it might be time to bite the bullet and implement annotations >> so > >> we can stop adding new syntactic constructs for each minor extension >> to the data model. >> >> --David >> >> Bryan Duxbury wrote: >>> All, >>> >>> We've been following the discussion on THRIFT-110. I think as a >>> first > >>> step we should add a "vint" (variable integer) to Thrift. >>> >>> My general proposal is that we add vint as a new type. Protocols >>> that > >>> don't do variable length integers can treat vints as i64s. Protocols >>> that do implement variable length integers can choose encodings that >>> make sense for them. >>> >>> We'd have to change the compiler and all the generators and >>> libraries > >>> to recognize vint, if for no other reason than to treat it as an >>> i64. >>> However, this would make implementing other protocols that included >>> variable length integers very easy. >>> >>> Are there any objections to this proposal? If not, I'll make an >>> issue, and Rapleaf is willing to commit time to this enhancement. >>> >>> -Bryan >>> > > >
