Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-20 Thread John Bresnahan
Mark, good thoughts (as usual) On 08/19/2013 09:15 PM, Mark Washenberger wrote: > The goal isn't really to replace sqlalchemy completely. Perhaps my problem is that I am not exactly sure what the goals are. Cleanup (BL mixed in the BL seems wrong)? HA or performance (are people hitting limits th

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-20 Thread Mark Washenberger
On Tue, Aug 20, 2013 at 3:20 AM, Flavio Percoco wrote: > On 20/08/13 00:15 -0700, Mark Washenberger wrote: > >> >>2) I highly caution folks who think a No-SQL store is a good >> storage >>solution for any of the data currently used by Nova, Glance >> (registry), >>Cinder (

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-20 Thread Johannes Erdfelt
On Tue, Aug 20, 2013, Flavio Percoco wrote: > There are a couple of things that would worry me about an hypothetic > support for NoSQL but I guess one that I'd consider very critical is > migrations. Some could argue asking whether we'd really need them or > not - when talking about NoSQL databas

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-20 Thread Flavio Percoco
On 20/08/13 00:15 -0700, Mark Washenberger wrote: 2) I highly caution folks who think a No-SQL store is a good storage solution for any of the data currently used by Nova, Glance (registry), Cinder (registry), Ceilometer, and Quantum. All of the data stored and manipu

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-20 Thread Mark Washenberger
So much great stuff to respond to in this snip and response! On Mon, Aug 19, 2013 at 2:17 AM, Flavio Percoco wrote: > On 19/08/13 02:51 -0400, Jay Pipes wrote: > >> On 08/19/2013 12:56 AM, Joshua Harlow wrote: >> >>> Another good article from an ex-coworker that keeps on making more and >>> mor

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-19 Thread Ben Nemec
On 08/19/13 20:34, Joshua Harlow wrote: > Just a related question, > > Oslo 'incubator' db code I think depends on eventlet. This means any code > that uses the oslo.db code could/would(?) be dependent on eventlet. > > Will there be some refactoring there to not require it (useful for pro

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-19 Thread Joshua Harlow
OpenStack Development Mailing List mailto:openstack-dev@lists.openstack.org>> Subject: Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code. Mark, But for a variety of reasons, I do not consider the general thrust of "use oslo db code" to be approved. Instead, l

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-19 Thread Boris Pavlovic
Mark, But for a variety of reasons, I do not consider the general thrust of "use oslo db code" to be approved. Instead, lets continue to consider features from olso db on a case by case basis, and see what the right resolution is in each case. Absolutely agree with this point (e.g. we removed sha

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-19 Thread John Bresnahan
> All I'm saying is that we should be careful not to swap one set of > problems for another. My 2 cents: I am in agreement with Jay. I am leery of NoSQL being a direct sub in and I fear that this effort can be adding a large workload for little benefit. A somewhat related post: http://www.joelo

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-19 Thread Mark Washenberger
Thanks for refocusing the discussion on your original questions! Also thanks for this additional summary. I consider the patches you have up for review in glance to have a general direction-level green light at this point (though I've got a question on the specifics in the ultimate review). But f

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-19 Thread Joshua Harlow
+1 for trying things differently :) On 8/19/13 12:14 AM, "Jay Pipes" wrote: >On 08/18/2013 10:33 PM, Joe Gordon wrote: >> An alternative I think would be better would be to scrap the >>use of >> the SQLAlchemy ORM; keep using the DB engine abstraction >>support. >> >> +1, I am ho

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-19 Thread Boris Pavlovic
Mark, Main part of oslo is: 1) common migration testing 2) common sqla.models 3) common hacks around sqla and sqla-migrate 4) common work around engines and sessions All these points are implemented in Glance almost in the same way as in Oslo. Also we are able to use only part of this code in Gl

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-19 Thread Flavio Percoco
On 19/08/13 02:51 -0400, Jay Pipes wrote: On 08/19/2013 12:56 AM, Joshua Harlow wrote: Another good article from an ex-coworker that keeps on making more and more sense the more projects I get into... http://seldo.com/weblog/2011/08/11/orm_is_an_antipattern Your mileage/opinion though may vary

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-19 Thread Flavio Percoco
On 18/08/13 18:47 -0400, Jay Pipes wrote: On 08/18/2013 06:28 PM, Joe Gordon wrote: On Aug 18, 2013 3:58 PM, "Jay Pipes" mailto:jaypi...@gmail.com>> wrote: > > On 08/18/2013 03:53 AM, Joshua Harlow wrote: >> >> I always just liked SQL as the database abstraction layer ;) >> >> On a more serious

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-19 Thread Julien Danjou
On Mon, Aug 19 2013, Jay Pipes wrote: > 2) I highly caution folks who think a No-SQL store is a good storage > solution for any of the data currently used by Nova, Glance (registry), > Cinder (registry), Ceilometer, and Quantum. All of the data stored and > manipulated in those projects is HIGHLY

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-19 Thread Jay Pipes
OK, cool. I'm in agreement with your explained storage/logic separation below. Cheers, -jay On 08/19/2013 03:12 AM, Robert Collins wrote: On 19 August 2013 18:35, Jay Pipes wrote: http://www.codinghorror.com/blog/2006/06/object-relational-mapping-is-the-vietnam-of-computer-science.html The

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-19 Thread Jay Pipes
On 08/18/2013 10:33 PM, Joe Gordon wrote: An alternative I think would be better would be to scrap the use of the SQLAlchemy ORM; keep using the DB engine abstraction support. +1, I am hoping this will provide noticeable performance benefits while being agnostic of what DB back-e

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-19 Thread Robert Collins
On 19 August 2013 18:35, Jay Pipes wrote: >> http://www.codinghorror.com/blog/2006/06/object-relational-mapping-is-the-vietnam-of-computer-science.html >> >> There is no proper use of an ORM. > > > I'm not a super-fan of ORMs, Robert. I'm not sure why you're insisting on > taking me down this roa

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-18 Thread Jay Pipes
On 08/19/2013 12:56 AM, Joshua Harlow wrote: Another good article from an ex-coworker that keeps on making more and more sense the more projects I get into... http://seldo.com/weblog/2011/08/11/orm_is_an_antipattern Your mileage/opinion though may vary :) I don't disagree with most of that ar

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-18 Thread Jay Pipes
On 08/18/2013 11:07 PM, Robert Collins wrote: On 19 August 2013 14:22, Jay Pipes wrote: I'm completely with Joshua here - the ORM layer is more often than not a source of bugs and performance issues. If used improperly, yep. http://www.codinghorror.com/blog/2006/06/object-relational-mappin

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-18 Thread Joshua Harlow
Another good article from an ex-coworker that keeps on making more and more sense the more projects I get into... http://seldo.com/weblog/2011/08/11/orm_is_an_antipattern Your mileage/opinion though may vary :) Sent from my really tiny device... On Aug 18, 2013, at 8:12 PM, "Robert Collins" m

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-18 Thread Joshua Harlow
It will be I interesting to see how it works out in nova, correct me if I am wrong but nova has even more onion layers than other openstack projects. For ex: Nova compute <->unified object model <->rpc<->conductor<->sqlalchemy ORM model<->SQL<->your db Is nova moving away from the ORM model or

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-18 Thread Robert Collins
On 19 August 2013 14:22, Jay Pipes wrote: >> I'm completely with Joshua here - the ORM layer is more often than not >> a source of bugs and performance issues. > > > If used improperly, yep. http://www.codinghorror.com/blog/2006/06/object-relational-mapping-is-the-vietnam-of-computer-science.htm

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-18 Thread Joe Gordon
On Sun, Aug 18, 2013 at 10:22 PM, Jay Pipes wrote: > On 08/18/2013 09:56 PM, Robert Collins wrote: > >> On 19 August 2013 10:43, Jay Pipes wrote: >> >>> On 08/18/2013 06:08 PM, Joshua Harlow wrote: >>> In my opinion (and just an opinion that I know everyone doesn't share) ORM

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-18 Thread Jay Pipes
On 08/18/2013 09:56 PM, Robert Collins wrote: On 19 August 2013 10:43, Jay Pipes wrote: On 08/18/2013 06:08 PM, Joshua Harlow wrote: In my opinion (and just an opinion that I know everyone doesn't share) ORM layers are bulky, restrictive and overly complicate and confuse the reader of the cod

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-18 Thread Jay Pipes
On 08/18/2013 07:44 PM, Joshua Harlow wrote: Using an ORM how does the ORM know what attributes u might access (forgive me if this is a documented sqlalchemy pattern/solution). Doesn't it have to give u back the full model since the ORM layer can't predict what u might do with the model object

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-18 Thread Robert Collins
On 19 August 2013 10:43, Jay Pipes wrote: > On 08/18/2013 06:08 PM, Joshua Harlow wrote: >> >> In my opinion (and just an opinion that I know everyone doesn't share) ORM >> layers are bulky, restrictive and overly complicate and confuse the reader >> of the code (code is read more often than writt

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-18 Thread Joshua Harlow
Using an ORM how does the ORM know what attributes u might access (forgive me if this is a documented sqlalchemy pattern/solution). Doesn't it have to give u back the full model since the ORM layer can't predict what u might do with the model object? Sent from my really tiny device... On Aug 1

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-18 Thread Joshua Harlow
It would be neat to see what would happen if just the "raw" models were just used directly. Of course this must be treaded careful since I could see it spreading db logic all over. +1 for turning off deferred loads, I think this encourages and actually hides bugs when lazy loads occur on demand

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-18 Thread Jay Pipes
On 08/18/2013 06:28 PM, Joe Gordon wrote: On Aug 18, 2013 3:58 PM, "Jay Pipes" mailto:jaypi...@gmail.com>> wrote: > > On 08/18/2013 03:53 AM, Joshua Harlow wrote: >> >> I always just liked SQL as the database abstraction layer ;) >> >> On a more serious note I think novas new object model

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-18 Thread Jay Pipes
On 08/18/2013 06:08 PM, Joshua Harlow wrote: In my opinion (and just an opinion that I know everyone doesn't share) ORM layers are bulky, restrictive and overly complicate and confuse the reader of the code (code is read more often than written) and require another layer of understanding (a la

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-18 Thread Joe Gordon
On Aug 18, 2013 3:58 PM, "Jay Pipes" wrote: > > On 08/18/2013 03:53 AM, Joshua Harlow wrote: >> >> I always just liked SQL as the database abstraction layer ;) >> >> On a more serious note I think novas new object model might be a way to go but in all honesty there won't be a one size fits all sol

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-18 Thread Joshua Harlow
In my opinion (and just an opinion that I know everyone doesn't share) ORM layers are bulky, restrictive and overly complicate and confuse the reader of the code (code is read more often than written) and require another layer of understanding (a layer is useful if it adds good value, I am not s

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-18 Thread Jay Pipes
On 08/18/2013 03:53 AM, Joshua Harlow wrote: I always just liked SQL as the database abstraction layer ;) On a more serious note I think novas new object model might be a way to go but in all honesty there won't be a one size fits all solution. I just don't think sqlalchemy is that solution pe

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-18 Thread Joshua Harlow
I always just liked SQL as the database abstraction layer ;) On a more serious note I think novas new object model might be a way to go but in all honesty there won't be a one size fits all solution. I just don't think sqlalchemy is that solution personally (maybe if we just use sqlalchemy core

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-16 Thread Jay Pipes
On 08/16/2013 02:41 PM, Mark Washenberger wrote: I think the issue here for glance is whether or not oslo common code makes it easier or harder to make other planned improvements. In particular, using openstack.common.db.api will make it harder to refactor away from a giant procedural interface f

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-16 Thread Mark Washenberger
I would prefer to pick and choose which parts of oslo common db code to reuse in glance. Most parts there look great and very useful. However, some parts seem like they would conflict with several goals we have. 1) To improve code sanity, we need to break away from the idea of having one giant db

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-16 Thread Flavio Percoco
On 16/08/13 11:42 -0400, Monty Taylor wrote: On 08/16/2013 09:31 AM, Victor Sergeyev wrote: Hello All. Glance cores (Mark Washenberger, Flavio Percoco, Iccha Sethi) have some questions about Oslo DB code, and why is it so important to use it instead of custom implementation and so on. As ther

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-16 Thread Eric Windisch
On Fri, Aug 16, 2013 at 9:31 AM, Victor Sergeyev wrote: > Hello All. > > Glance cores (Mark Washenberger, Flavio Percoco, Iccha Sethi) have some > questions about Oslo DB code, and why is it so important to use it instead > of custom implementation and so on. As there were a lot of questions it wa

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-16 Thread Monty Taylor
On 08/16/2013 09:31 AM, Victor Sergeyev wrote: > Hello All. > > Glance cores (Mark Washenberger, Flavio Percoco, Iccha Sethi) have some > questions about Oslo DB code, and why is it so important to use it > instead of custom implementation and so on. As there were a lot of > questions it was rea

[openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-16 Thread Victor Sergeyev
Hello All. Glance cores (Mark Washenberger, Flavio Percoco, Iccha Sethi) have some questions about Oslo DB code, and why is it so important to use it instead of custom implementation and so on. As there were a lot of questions it was really hard to answer on all this questions in IRC. So we decide