Douglas McClusky created VCL-787:
------------------------------------
Summary: Backend approach to simplify schema updates
Key: VCL-787
URL: https://issues.apache.org/jira/browse/VCL-787
Project: VCL
Issue Type: Improvement
Components: database
Reporter: Douglas McClusky
Have you considered Avro + a document store? Avro is an Apache project for
seralization / deserialization that includes schema validation and nonbreaking
transformations, automatically. Objects and schemata can be expressed in JSON,
so it would lend itself to a json document store like Apache CouchDB very
simply.
You could have for any given storable object, obj, two databases: objdb and
objschemadb. Store schema definitions for object obj in objschemadb, and store
obj objects in objdb, including a field indicating the schema used for
validation. That way, if the schema changes, obj can be read using its old
schema and written using the new schema. Avro provides support for simple
schema changes, such as adding fields, changing variable names or changing a
datatype to a compatible datatype, like int to long. For more complex schema
changes, the transformations necessary would have to be done in code, but you
would at least know exactly what schemata the data was transforming from and
to.
Avro has libraries for both PHP and Ruby, and the JSON object format would be
well suited to working with data in dojo + javascript. The system
configuration could include a target schemata version, and changes to existing
data could be applied either during the configuration change of the target
version, or lazily on updates to existing data (reading with the previous
schema and then writing to the target schema).
Configuration management could define the target schemata by version or
"latest". Schemata of individual objects could be defined by version or
"latest" or "any" (to allow lazy updates later, for example). Backwards
updates should probably be discouraged or not supported, because making changes
like changing a field back from a long to an int could lose information.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
