I think the abstraction is good, but I would rather wait until we have a chance to add redis support.
On 09/10/2010 12:08 PM, Vishvananda Ishaya wrote: > I don't understand the comment that this doesn't play nicely with > non-relational data stores. There is a very clear abstraction layer > db/api.py that would allow a different backend to be plugged in > regardless of whether sqlalchemy ever supports it. I don't think it > would be too difficult to add db/redis/api.py to this system. > > The one possible gotcha is that there isn't a clear definition of the > properties that each object needs to have anywhere outside of > sqlalchemy/models.py and the relations that are needed. This > ultimately should be fixed with either documentation or some kind of > middle tier classes that define the needed properties. > > Delaying until after austin would be a bit troublesome for us since > we have to move of of redis. That means a pretty strongly diverged > branch and any features that anso is working on will probably have to > be delayed as well. > > Vish > > On Sep 10, 2010, at 9:56 AM, Jay Pipes wrote: > >> Hi Vish, >> >> Such a large patch has taken me quite some time to digest. There >> is a larger discussion on large patches without any specifications, >> but I'll leave that for a later time! :) >> >> I am torn on this one, mostly because I spent a bunch of time >> attempting to do the datastore refactoring myself (as did Justin >> Santa Barbara), and thus I know the dragons that live in this layer >> of the code :) >> >> One of the things that both Justin and I had tried was to keep an >> abstraction layer that would allow both NoSQL as well as SQL data >> stores to be used. Unfortunately, it seems that this patch >> removes the ability to use ReDIS, among other NoSQL stores. I >> think this is a mistake, and although I like much of the code in >> this patch, I was hoping that SQLAlchemy could be hidden behind an >> abstraction layer that would play nicely with the non-relational >> data stores. >> >> As this patch stands, we take a 180 degree turn away from NoSQL >> data stores and back into the relatively comfortable norms of the >> SQL databases. While there's nothing particularly wrong with SQL >> databases (as you know, I'm a fan of many of them ;) ), I think >> that keeping non-relational data store capabilities is pretty >> critical. >> >> After an email discussion with SQLAlchemy's Michael Bayer about >> SQLAlchemy's future with NoSQL data stores. Although there is an >> issue in the SQLAlchemy trac system about this (see here: >> http://www.sqlalchemy.org/trac/ticket/1518) the likelihood of this >> module seeing the light of day is unlikely in the next year or >> two. >> >> So...what to do? There are at least four options I can see: >> >> 1) Go forward with this patch and add NoSQL stores back at some >> later time by ourselves 2) Go forward with this patch and wait >> until SQLAlchemy properly supports key value stores 3) Delay this >> patch until after the Austin release and have a larger discussion >> about it here and at the next summit 4) Go back to the drawing >> board and try again with a less ambitious set of patches that >> incrementally changes the way the data stores work. >> >> I'm personally on the fence. I'd prefer to at least delay the >> patch until after Austin, but I understand there are now at least 4 >> branches that depend on this one, which makes things, well, a bit >> difficult. >> >> -jay >> >> On Tue, Aug 31, 2010 at 8:46 PM, Vishvananda Ishaya >> <[email protected]> wrote: >>> I've proposed a merge of the orm refactor branch that a large >>> part of the nasa/anso team has been working on. I'm hoping >>> everyone can pick it apart and we end up with a really clean >>> system that everyone likes. I've copied the description of the >>> change and issues below. If the mailing list debates get too >>> complicated, we should just organize a time to discuss it in >>> IRC. >>> >>> Proposing merge to get feedback on orm refactoring. I am very >>> interested in feedback to all of these changes. >>> >>> This is a huge set of changes, that touches almost all of the >>> files. I'm sure I have broken quite a bit, but better to take the >>> plunge now than to postpone this until later. The idea is to >>> allow for pluggable backends throughout the code. >>> >>> Brief Overview For compute/volume/network, there are multiple >>> classes service - responsible for rpc this currently uses the >>> existing cast and call in rpc.py and a little bit of magic to >>> call public methods on the manager class. each service also >>> reports its state into the database every 10 seconds manager - >>> responsible for managing respective object classes all the >>> business logic for the classes go here db (db_driver) - >>> responsible for abstracting database access driver >>> (domain_driver) - responsible for executing actual shell commands >>> and implementation >>> >>> Compute hasn't been fully cleaned up, but to get an idea of how >>> it works, take a look at volume and network >>> >>> Known issues/Things to be done: >>> >>> * nova-api accesses db objects directly It seems cleaner to have >>> only the managers dealing with their respective objects. This >>> would mean code for 'run_instances' would move into the manager >>> class and it would do the initial setup and cast out to the >>> remote service >>> >>> * db code uses flat methods to define its interface In my mind >>> this is a little prettier as an abstract base class, but driver >>> loading code can load a module or a class. It works, so I'm not >>> sure it needs to be changed but feel free to debate it. >>> >>> * Service classes have no code in them Not sure if this is a >>> problem for people, but the magic of calling the manager's >>> methods is done in the base class. We could remove the magic from >>> the base class and explicitly wrap methods that we want to make >>> available via rpc if this seems nasty. >>> >>> * AuthManager Projects/Users/Roles are not integrated into this >>> system. In order for everything to live happily in the backend, >>> we need some type of adaptor for LDAP >>> >>> * Context is not passed properly across rabbit Context should >>> probably be changed to a simple dictionary so that it can be >>> passed properly through the queue >>> >>> * No authorization checks on access to objects We need to decide >>> on which layer auth checks should happen. >>> >>> * Some of the methods in ComputeManager need to be moved into >>> other layers/managers * Compute driver layer should be abstracted >>> more cleanly * Flat networking is untested and may need to be >>> reworked * Some of the api commands are not working yet * Nova >>> Swift Authentication needs to be refactored(Todd is working on >>> this) >>> >>> _______________________________________________ Mailing list: >>> https://launchpad.net/~nova Post to : >>> [email protected] Unsubscribe : >>> https://launchpad.net/~nova More help : >>> https://help.launchpad.net/ListHelp >>> >>> > > > _______________________________________________ Mailing list: > https://launchpad.net/~nova Post to : [email protected] > Unsubscribe : https://launchpad.net/~nova More help : > https://help.launchpad.net/ListHelp
signature.asc
Description: OpenPGP digital signature
_______________________________________________ Mailing list: https://launchpad.net/~nova Post to : [email protected] Unsubscribe : https://launchpad.net/~nova More help : https://help.launchpad.net/ListHelp
