[ https://issues.apache.org/jira/browse/HBASE-8693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13711775#comment-13711775 ]
Nick Dimiduk commented on HBASE-8693: ------------------------------------- bq. Ok, make sense with this limited scope (no schema) have a fixed list of fields. Right. In this implementation Struct is a simple concatenation of fields. No schema information is written into that concatenation because to do so will mess with sort order. Struct is merely API convenience. Now, the field encodings implemented in OrderedBytes include a header byte which is currently used to identify the type of encoded field that follows. The full space of 256 available bit patterns in that header bit is not consumed by the current implementation. I've been thinking about extending that header byte to include some version bits at the very beginning. That would enable evolution of the individual field encodings (say, if you later want to re-implement blob-mid, for example). This doesn't address the user-level logical structure of a Struct data type, only evolution of the OrderedBytes codec. bq. My main concern is: I start use 96 with this struct encoding... is fixed so I can't add fields.. so I work around it adding a version number in front of the struct and then I do the switch for v1, v2, v3 with all the fixed struct that I know... Prepending a version number to the Struct's members will impact sort order. Struct definition is fixed in that you can't prepend or interpose a new field in the middle of an existing encoded value. You're free to append fields. Appending a field would look like the following: # application defines Struct v0 with members [A,B,C] # application writes lots of data # application changes, Struct v1 becomes [A,B,C,D,E] # application writes lots more data At step 3, the application now needs to become version aware. Because the fields of v0 are a subset of v1, the application can use the definition of struct v1 with the following safe-guards. (1) Any place where v0 was used, it now needs to be sure to check for end-of-buffer and skip over the two new elements. (2) Anywhere v1 is used, mindful of truncated records and be prepared to only receive the v0 fields. Maybe the API defined around Struct can be improved to support these needs? Records of v0 and v1 can be intermixed, ie, as rowkeys in the same table. According to the documented sort semantics, they'll sort "left-to-right and depth-first". Meaning, they'll sort first according to v0 values and then within that group, by v1 values. We leave all of this up to user applications today, so this change management isn't mitigated. Changing a compound rowkey today requires rewriting data (or duplication into a new table). A smarter struct encoding, one that's able to preserve the sorted semantics I've described but that can also track more sophisticated schama change would be very useful indeed -- I don't think it exists. Prepending a version field to a Struct will change the sorting behavior; v0 will sort before v1, &c. IMHO, this is a less flexible migration strategy than the append behavior described above. It's also perfectly valid, and the user of the Struct API is free to do so in their own application. In that case, the application is still version-aware. Instead of being cautious about consuming the potentially truncated records, instead it's executing a scan for each version. bq. as you said, data evolution is out of the scope. so if you consider this patch just as a "smarter" alternative to the Bytes encoding. HBASE-8201 is a smarter alternative to Bytes and this ticket adds some higher-level APIs for manipulating them. In short, yes, schema definition and evolution is out of scope. > Implement extensible type API based on serialization primitives > --------------------------------------------------------------- > > Key: HBASE-8693 > URL: https://issues.apache.org/jira/browse/HBASE-8693 > Project: HBase > Issue Type: Sub-task > Components: Client > Reporter: Nick Dimiduk > Assignee: Nick Dimiduk > Fix For: 0.95.2 > > Attachments: 0001-HBASE-8693-Extensible-data-types-API.patch, > 0001-HBASE-8693-Extensible-data-types-API.patch, > 0001-HBASE-8693-Extensible-data-types-API.patch, > 0002-HBASE-8693-example-Use-DataType-API-to-build-regionN.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira