I'm gong to guess redis' lack of HA. Did I guess right? On 09/10/2010 12:19 PM, Jay Pipes wrote: > On Fri, Sep 10, 2010 at 1:08 PM, Vishvananda Ishaya > <[email protected]> wrote: >> I don't understand the comment that this doesn't play nicely with >> non-relational data stores. There is a very clear abstraction layer >> db/api.py that would allow a different backend to be plugged in regardless >> of whether sqlalchemy ever supports it. I don't think it would be too >> difficult to add db/redis/api.py to this system. >> >> The one possible gotcha is that there isn't a clear definition of the >> properties that each object needs to have anywhere outside of >> sqlalchemy/models.py and the relations that are needed. This ultimately >> should be fixed with either documentation or some kind of middle tier >> classes that define the needed properties. >> >> Delaying until after austin would be a bit troublesome for us since we have >> to move of of redis. That means a pretty strongly diverged branch and any >> features that anso is working on will probably have to be delayed as well. > > Maybe I'm out of some loop, but this is the first I'm hearing of any > need for you guys to move off of Redis. Can I ask why this move is > necessary? Lack of resources who know how to manage ReDIS stores, or > something else? Just curious. > > Also, as far as features that Anso is working on, where can I see > these features? We use the blueprints system on Launchpad to manage > precisely these kind of things. Having a chunk of coders working on > their own ideas and features with no insight into what those things > are spells duplicated effort to me (as this patch duplicated much > effort put in by a few folks on the datastore stuff..) It would be > helpful to know what everyone is working on by looking at the singular > list of blueprints and using the Launchpad milestone/release system to > its full effect. > > -jay > >> Vish >> >> On Sep 10, 2010, at 9:56 AM, Jay Pipes wrote: >> >>> Hi Vish, >>> >>> Such a large patch has taken me quite some time to digest. There is a >>> larger discussion on large patches without any specifications, but >>> I'll leave that for a later time! :) >>> >>> I am torn on this one, mostly because I spent a bunch of time >>> attempting to do the datastore refactoring myself (as did Justin Santa >>> Barbara), and thus I know the dragons that live in this layer of the >>> code :) >>> >>> One of the things that both Justin and I had tried was to keep an >>> abstraction layer that would allow both NoSQL as well as SQL data >>> stores to be used. Unfortunately, it seems that this patch removes >>> the ability to use ReDIS, among other NoSQL stores. I think this is a >>> mistake, and although I like much of the code in this patch, I was >>> hoping that SQLAlchemy could be hidden behind an abstraction layer >>> that would play nicely with the non-relational data stores. >>> >>> As this patch stands, we take a 180 degree turn away from NoSQL data >>> stores and back into the relatively comfortable norms of the SQL >>> databases. While there's nothing particularly wrong with SQL >>> databases (as you know, I'm a fan of many of them ;) ), I think that >>> keeping non-relational data store capabilities is pretty critical. >>> >>> After an email discussion with SQLAlchemy's Michael Bayer about >>> SQLAlchemy's future with NoSQL data stores. Although there is an >>> issue in the SQLAlchemy trac system about this (see here: >>> http://www.sqlalchemy.org/trac/ticket/1518) the likelihood of this >>> module seeing the light of day is unlikely in the next year or two. >>> >>> So...what to do? There are at least four options I can see: >>> >>> 1) Go forward with this patch and add NoSQL stores back at some later >>> time by ourselves >>> 2) Go forward with this patch and wait until SQLAlchemy properly >>> supports key value stores >>> 3) Delay this patch until after the Austin release and have a larger >>> discussion about it here and at the next summit >>> 4) Go back to the drawing board and try again with a less ambitious >>> set of patches that incrementally changes the way the data stores >>> work. >>> >>> I'm personally on the fence. I'd prefer to at least delay the patch >>> until after Austin, but I understand there are now at least 4 branches >>> that depend on this one, which makes things, well, a bit difficult. >>> >>> -jay >>> >>> On Tue, Aug 31, 2010 at 8:46 PM, Vishvananda Ishaya >>> <[email protected]> wrote: >>>> I've proposed a merge of the orm refactor branch that a large part of the >>>> nasa/anso team has been working on. I'm hoping everyone can pick it apart >>>> and we end up with a really clean system that everyone likes. I've copied >>>> the description of the change and issues below. If the mailing list >>>> debates >>>> get too complicated, we should just organize a time to discuss it in IRC. >>>> >>>> Proposing merge to get feedback on orm refactoring. I am very interested in >>>> feedback to all of these changes. >>>> >>>> This is a huge set of changes, that touches almost all of the files. I'm >>>> sure I have broken quite a bit, but better to take the plunge now than to >>>> postpone this until later. The idea is to allow for pluggable backends >>>> throughout the code. >>>> >>>> Brief Overview >>>> For compute/volume/network, there are multiple classes >>>> service - responsible for rpc >>>> this currently uses the existing cast and call in rpc.py and a little bit >>>> of magic >>>> to call public methods on the manager class. >>>> each service also reports its state into the database every 10 seconds >>>> manager - responsible for managing respective object classes >>>> all the business logic for the classes go here >>>> db (db_driver) - responsible for abstracting database access >>>> driver (domain_driver) - responsible for executing actual shell commands >>>> and >>>> implementation >>>> >>>> Compute hasn't been fully cleaned up, but to get an idea of how it works, >>>> take a look >>>> at volume and network >>>> >>>> Known issues/Things to be done: >>>> >>>> * nova-api accesses db objects directly >>>> It seems cleaner to have only the managers dealing with their respective >>>> objects. This would >>>> mean code for 'run_instances' would move into the manager class and it >>>> would do the initial >>>> setup and cast out to the remote service >>>> >>>> * db code uses flat methods to define its interface >>>> In my mind this is a little prettier as an abstract base class, but >>>> driver >>>> loading code >>>> can load a module or a class. It works, so I'm not sure it needs to be >>>> changed but feel >>>> free to debate it. >>>> >>>> * Service classes have no code in them >>>> Not sure if this is a problem for people, but the magic of calling the >>>> manager's methods is >>>> done in the base class. We could remove the magic from the base class and >>>> explicitly >>>> wrap methods that we want to make available via rpc if this seems nasty. >>>> >>>> * AuthManager Projects/Users/Roles are not integrated into this system. >>>> In order for everything to live happily in the backend, we need some type >>>> of adaptor for LDAP >>>> >>>> * Context is not passed properly across rabbit >>>> Context should probably be changed to a simple dictionary so that it can >>>> be >>>> passed properly through the queue >>>> >>>> * No authorization checks on access to objects >>>> We need to decide on which layer auth checks should happen. >>>> >>>> * Some of the methods in ComputeManager need to be moved into other >>>> layers/managers >>>> * Compute driver layer should be abstracted more cleanly >>>> * Flat networking is untested and may need to be reworked >>>> * Some of the api commands are not working yet >>>> * Nova Swift Authentication needs to be refactored(Todd is working on this) >>>> >>>> _______________________________________________ >>>> Mailing list: https://launchpad.net/~nova >>>> Post to : [email protected] >>>> Unsubscribe : https://launchpad.net/~nova >>>> More help : https://help.launchpad.net/ListHelp >>>> >>>> >> >> > > _______________________________________________ > Mailing list: https://launchpad.net/~nova > Post to : [email protected] > Unsubscribe : https://launchpad.net/~nova > More help : https://help.launchpad.net/ListHelp
signature.asc
Description: OpenPGP digital signature
_______________________________________________ Mailing list: https://launchpad.net/~nova Post to : [email protected] Unsubscribe : https://launchpad.net/~nova More help : https://help.launchpad.net/ListHelp
