On Mar 20, 2009, at 4:43 PM, Jacob Gabrielson wrote:
> > Does anyone have any suggestions for managing both horizontal and > vertical partitioning within the same application? In particular I'd > prefer that most of my code be oblivious to all this and only use a > single session object. To make it a little more concrete, imagine I > have four tables in my application: > > VertA - always use engine db1 > VertB - always use engine db2, > HorizC - sharded across db3, db4 > HorizD - sharded across db3, db4 > > The goal would be for most of my code to not really care about any of > this and just use a single Session object (gotten from somewhere) to > handle all of them, e.g., > > def foo(): > sess = Session() # from somewhere, likely scoped_session > myA = SomeVertAClass(...) > myB = SomeVertBClass(...) > sess.save(myA) > sess.save(myB) > myC = sess.query(SomeHorizCClass).filter(SomeHorizCClass.frotz == > mumble).one() > # ...etc... > # someone else commits the session > > Based on some research I think this might be possible by pretending > that even the vertically-partitioned tables are "shards", but have our > sharding code know that it should only return those shards when the > class involved really belongs there. But I'm not sure that specific > idea would be a good one, or if in general this whole thing is a bad > idea? > > I googled around on this but didn't see anything obvious, apologies in > advance if I missed something obvious... when using ShardedSession, all determination of binds is based on functions which you provide to it. You can write whatever rules you'd like for these functions. However, you should never expect horizontal sharding to be transparent to your application. your usage of this session, at least among the data which is horizontally partitioned, will require many explicit and inefficient steps if you need to construct any kind of aggregation across multiple shards. even the vertically partitioned aspects can cause headaches as you no longer can join between tables (unless they can access each other via schemas or database links). Unless you're planning on storing tens of billions of rows, I'd be hesitatant to build around a horizontally sharded approach. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~----------~----~----~----~------~----~------~--~---
