Hi Christian, On Sat, Jul 20, 2013 at 12:54 PM, <[email protected]> wrote:
> > thanks a lot for sharing your thoughts! I have worked a little on the > topic during the last days and would like to share the results with you. > Pls accept my apologies for the long text. > No probs > > You will have noticed that the current design of the Gora cassandra module > is not really in conformance with the above points. No joke ;) > Still I think Gora did fantastic work in providing basic concepts and > infrastructure for a big data abstraction layer. Moreover, it should be > generally possible to extend the design in a way that satisfies our needs. > I agree with you very mcuh here. Although you point out some fundamental points, which for some may be blockers for using Gora for your use case, there are other low hanging fruit (in the form of tweaks and improvements)... which I assume you are working on right now. It's just a case of delving down into the module. [snip] > The idea to exploit these concepts in Gora builds on adding an additional > key mapping part to the gora-cassandra-mapping. An avro class might define > a primary key element. The key is represented by another avro class. The > mapping defines which fields of the avro key class hold the data for which > parts of the C* compound primary key. Additionally, the field names of the > avro value class are being appended to the composite column name. Nested > complex types that have formerly been mapped to sub columns are handled > accordingly by appending their name and subKeys as column names. As C* > composite column names can be defined dynamically, also an arbitrary level > of nesting complex types would be possible. > I think this is the first serious suggestion we've had for a rethink on the logic behind the gora-cassandra storage model. I like it, and I would really like to see something (code) to map/tie the above to. It is not trivial but you seem to have a comprehensive (better than me ;)) vision of how it should and will work. > > An open question for me is, if this concept is also viable considering > other data stores and if it would make sense to add something along this > lines to the Gora core. At least for Cassandra it seems to do the trick. > One important aspect of core that we have successfully maintained to-date, is that the object2datstore mappings are backend specific. Consequently (and although admittedly there have been many improvements/changes in Gora over the last few revisions) we really aim to be backwards compatible until we reach 1.0. We are seeing (in the pipeline) a complete overhaul/upgrade of the HBase API within possibly 0.4. If you wish to propose changes to core then please feel free to do so... maybe a well defined and comprehensively document Jira issue is in order here? > > As a POC I am implementing this on top of the current Gora HEAD (just the > cassandra module). Thanks to the good Gora infrastructure, I've finished a > first version of the data creation/update part and can give you an example. On the side, what were the nice parts of the core API which you liked right now? What are the bad? > The corresponding gora-cassandra-mapping is shown below. The result of > this is that all meter reading objects of a single year will go into one > partition (row) and can be retrieved by date ranges. Additionally the CF > holds aggregations for months and weeks. Additionally, the mapping > specifies data replication for the keyspace. > > There is no doubt, this certainly makes sense for you use case. It is very very powerful indeed. N.B. I reviewed and acknowledged the avro, xml and Java code you posted, thanks for being so verbose. I'm sure others have thoughts. I for one would like to come back to it once (as below) I can see some code... if at all possible. > ... > I'm trying to finish the Gora C* store this month. I'm happy to share the > patch with you if you are interested. > > It would be really great to have your insight and motivation as part of Gora. Your code would be very welcome here and thanks for dropping in on the dev lists. Thank you Lewis
