Re: [openstack-dev] [Nova] sqlalchemy-migrate vs alembic for new database

2014-12-05 Thread Johannes Erdfelt
On Fri, Dec 05, 2014, Andrew Laski wrote: > The cells v2 effort is going to be introducing a new database into > Nova. This has been an opportunity to rethink and approach a few > things differently, including how we should handle migrations. There > have been discussions for a long time now abou

Re: [openstack-dev] [Nova] Spring cleaning nova-core

2014-12-07 Thread Johannes Erdfelt
On Mon, Dec 08, 2014, Michael Still wrote: > There are other things happening behind the scenes as well -- we have > a veto process for current cores when we propose a new core. It has > been made clear to me that several current core members believe we > have reached "the maximum effective size"

Re: [openstack-dev] [hacking] proposed rules drop for 1.0

2014-12-09 Thread Johannes Erdfelt
On Tue, Dec 09, 2014, Sean Dague wrote: > This check should run on any version of python and give the same > results. It does not, because it queries python to know what's in stdlib > vs. not. Just to underscore that it's difficult to get right, I found out recently that hacking doesn't do a grea

Re: [openstack-dev] [hacking] proposed rules drop for 1.0

2014-12-09 Thread Johannes Erdfelt
On Tue, Dec 09, 2014, Sean Dague wrote: > I'd like to propose that for hacking 1.0 we drop 2 groups of rules entirely. > > 1 - the entire H8* group. This doesn't function on python code, it > functions on git commit message, which makes it tough to run locally. It > also would be a reason to prev

Re: [openstack-dev] [nova] Adding new features to Kilo and future releases - DB upgrades

2015-01-26 Thread Johannes Erdfelt
On Thu, Jan 22, 2015, Kekane, Abhishek wrote: > With online schema changes/No downtime DB upgrades things would be > much lot easier for OpenStack deployments. > Big kudos to Johannes who initiated this feature. But as a service > provider, I'm curious to understand what is the development process

Re: [openstack-dev] design question : green thread model

2015-01-28 Thread Johannes Erdfelt
On Wed, Jan 28, 2015, murali reddy wrote: > I am trying to understand how a nova component can be run parallely on a > host. From the developer reference documentation it seems to indicate that > all the openstack services use green thread model of threading. Is it the > only model of parallelism

Re: [openstack-dev] design question : green thread model

2015-01-28 Thread Johannes Erdfelt
On Wed, Jan 28, 2015, murali reddy wrote: > On hosts with multi-core processors, it does not seem optimal to run a > single service instance with just green thread. I understand that on > controller node, we can run one or more nova services but still it does not > seem to utilize multi-core proce

Re: [openstack-dev] [oslo.db] PyMySQL review

2015-01-28 Thread Johannes Erdfelt
On Wed, Jan 28, 2015, Mike Bayer wrote: > I can envision turning this driver into a total monster, adding > C-speedups where needed but without getting in the way of async > patching, adding new APIs for explicit async, and everything else. > However, I’ve no idea what the developers have an appet

Re: [openstack-dev] [oslo.db] PyMySQL review

2015-01-28 Thread Johannes Erdfelt
On Wed, Jan 28, 2015, Clint Byrum wrote: > As is often the case with threading, a reason to avoid using it is > that libraries often aren't able or willing to assert thread safety. > > That said, one way to fix that, is to fix those libraries that we do > want to use, to be thread safe. :) I flo

Re: [openstack-dev] [oslo.db] PyMySQL review

2015-01-28 Thread Johannes Erdfelt
On Wed, Jan 28, 2015, Vishvananda Ishaya wrote: > On Jan 28, 2015, at 4:03 PM, Doug Hellmann wrote: > > I hope someone who was around at the time will chime in with more detail > > about why green threads were deemed better than regular threads, and I > > look forward to seeing your analysis of a

Re: [openstack-dev] [all][tc] SQL Schema Downgrades and Related Issues

2015-01-29 Thread Johannes Erdfelt
On Thu, Jan 29, 2015, Morgan Fainberg wrote: > The concept that there is a utility that can (and in many cases > willfully) cause permanent, and in some cases irrevocable, data loss > from a simple command line interface sounds crazy when I try and > explain it to someone. > > The more I work wit

Re: [openstack-dev] [nova] The strange case of osapi_compute_unique_server_name_scope

2015-02-20 Thread Johannes Erdfelt
On Thu, Feb 19, 2015, Matthew Booth wrote: > Assuming this configurability is required, is there any way we can > instead use it to control a unique constraint in the db at service > startup? This would be something akin to a db migration. How do we > manage those? Ignoring if this particular fea

Re: [openstack-dev] [stable][all] Revisiting the 6 month release cycle

2015-02-24 Thread Johannes Erdfelt
On Mon, Feb 23, 2015, Joe Gordon wrote: > What this actually means: > >- Stop approving blueprints for specific stable releases, instead just >approve them and target them to milestones. > - Milestones stop becoming Kilo-1, Kilo-2, Kilo-3 etc. and just > become 1,2,3,4,5,6,7,8

Re: [openstack-dev] [stable][all] Revisiting the 6 month release cycle

2015-02-24 Thread Johannes Erdfelt
On Tue, Feb 24, 2015, Thierry Carrez wrote: > Joe Gordon wrote: > > [...] > > I think a lot of the frustration with our current cadence comes out of > > the big stop everything (development, planning etc.), and stabilize the > > release process. Which in turn isn't really making things more stable

Re: [openstack-dev] [stable][all] Revisiting the 6 month release cycle

2015-02-24 Thread Johannes Erdfelt
On Tue, Feb 24, 2015, Thierry Carrez wrote: > Agree on the pain of maintaining milestone plans though, which is why I > propose we get rid of most of it in Liberty. That will actually be > discussed at the cross-project meeting today: > > https://wiki.openstack.org/wiki/Release_Cycle_Management/L

Re: [openstack-dev] [stable][all] Revisiting the 6 month release cycle

2015-02-25 Thread Johannes Erdfelt
On Tue, Feb 24, 2015, Jeremy Stanley wrote: > On 2015-02-24 10:00:51 -0800 (-0800), Johannes Erdfelt wrote: > [...] > > Recently, I have spent a lot more time waiting on reviews than I > > have spent writing the actual code. > > That's awesome, assuming what you m

Re: [openstack-dev] [neutron][all] switch from mysqldb to another eventlet aware mysql client

2014-09-12 Thread Johannes Erdfelt
On Fri, Sep 12, 2014, Doug Hellmann wrote: > I don’t think we will want to retroactively change the migration scripts > (that’s not something we generally like to do), We don't allow semantic changes to migration scripts since people who have already run it won't get those changes. However, we ha

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

2014-03-10 Thread Johannes Erdfelt
On Mon, Mar 10, 2014, Joshua Harlow wrote: > Sounds like a good idea to me. I generally think this is a good idea too. > I've never understood why we treat the DB as a LOG (keeping deleted == 0 > records around) when we should just use a LOG (or similar system) to > begin with instead. > > Does

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

2014-03-11 Thread Johannes Erdfelt
On Tue, Mar 11, 2014, Mike Wilson wrote: > Undeleting things is an important use case in my opinion. We do this in our > environment on a regular basis. In that light I'm not sure that it would be > appropriate just to log the deletion and git rid of the row. I would like > to see it go to an arch

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

2014-03-12 Thread Johannes Erdfelt
On Wed, Mar 12, 2014, CARVER, PAUL wrote: > I have personally witnessed someone (honestly, not me) select "Terminate > Instance" when they meant "Reboot Instance" and that mistake is way too > easy. I'm not sure if it was a brain mistake or mere slip of the mouse, > but it's enough to make people

Re: [openstack-dev] Does DB schema hygiene warrant long migrations?

2013-10-24 Thread Johannes Erdfelt
On Fri, Oct 25, 2013, Michael Still wrote: > Because I am a grumpy old man I have just -2'ed > https://review.openstack.org/#/c/39685/ and I wanted to explain my > rationale. Mostly I am hoping for a consensus to form -- if I am wrong > then I'll happy remove my vote from this patch. > > This pat

Re: [openstack-dev] Does DB schema hygiene warrant long migrations?

2013-10-24 Thread Johannes Erdfelt
On Fri, Oct 25, 2013, Michael Still wrote: > On Fri, Oct 25, 2013 at 8:19 AM, Johannes Erdfelt > wrote: > > On Fri, Oct 25, 2013, Michael Still wrote: > > >> However, when I run it with medium sized (30 million instances) > >> databases, the change does ca

Re: [openstack-dev] [nova] changing old migrations is verboten

2013-10-31 Thread Johannes Erdfelt
On Thu, Oct 31, 2013, Sean Dague wrote: > So there is a series of patches starting with - > https://review.openstack.org/#/c/53417/ that go back and radically > change existing migration files. > > This is really a no-no, unless there is a critical bug fix that > absolutely requires it. Changing

Re: [openstack-dev] [nova] changing old migrations is verboten

2013-11-01 Thread Johannes Erdfelt
On Fri, Nov 01, 2013, Sean Dague wrote: > It's trading one source of bugs for another. I'd love to say we can > have our cake and eat it to, but we really can't. And I very much > fall on the side of "getting migrations is hard, updating past > migrations without ever forking the universe is reall

Re: [openstack-dev] [oslo] Logging exceptions and Python 3

2014-05-16 Thread Johannes Erdfelt
On Fri, May 16, 2014, Igor Kalnitsky wrote: > > According to http://legacy.python.org/dev/peps/pep-0352/ the message > > attribute of BaseException is deprecated since Python 2.6 and was > > dropped in Python 3.0. > > Some projects have custom exception hierarchy, with strictly defined > attribut

Re: [openstack-dev] [oslo] Logging exceptions and Python 3

2014-05-16 Thread Johannes Erdfelt
On Thu, May 15, 2014, Victor Stinner wrote: > I'm trying to define some rules to port OpenStack code to Python 3. I just > added a section in the "Port Python 2 code to Python 3" about formatting > exceptions and the logging module: > https://wiki.openstack.org/wiki/Python3#logging_module_and_fo

Re: [openstack-dev] [oslo] Logging exceptions and Python 3

2014-05-17 Thread Johannes Erdfelt
On Fri, May 16, 2014, Victor Stinner wrote: > See my documentation: > https://wiki.openstack.org/wiki/Python3#logging_module_and_format_exceptions > > " six.text_type(exc): always use Unicode. It may raise unicode error > depending > on the exception, be careful. Example of such error in python

Re: [openstack-dev] [oslo] Logging exceptions and Python 3

2014-05-17 Thread Johannes Erdfelt
On Fri, May 16, 2014, Igor Kalnitsky wrote: > > unicode(exc) (or six.text_type(exc)) works for all exceptions, built-in or > custom. > > That's too much of a statement. Sometimes exceptions implement their own > __str__ / __unitcode__ > methods, that return too many rubbish information or not eno

Re: [openstack-dev] [Neutron] Consistency between models and migrations

2014-05-19 Thread Johannes Erdfelt
On Tue, May 20, 2014, Collins, Sean wrote: > I've been looking at two reviews that Ann Kamyshnikova has proposed > > https://review.openstack.org/#/c/82073/ > > https://review.openstack.org/#/c/80518/ > > I think the changes are fundamentally a Good Thing™ - they appear to > reduce the differe

Re: [openstack-dev] [oslo] Logging exceptions and Python 3

2014-05-21 Thread Johannes Erdfelt
On Wed, May 21, 2014, John Dennis wrote: > But that's a bug in the logging implementation. Are we supposed to write > perverse code just to avoid coding mistakes in other modules? Why not > get the fundamental problem fixed? It has been fixed, by making Python 3 :) This is a problem in the Pytho

[openstack-dev] Gate and Skipped Tests

2014-05-22 Thread Johannes Erdfelt
I noticed recently that some tests are being skipped in the Nova gate. Some will always be skipped, but others are conditional. In particular the ZooKeeper driver tests are being skipped because an underlying python module is missing. It seems to me that we should want no tests to be conditional

Re: [openstack-dev] Gate and Skipped Tests

2014-05-23 Thread Johannes Erdfelt
On Fri, May 23, 2014, Rick Harris wrote: > On Thu, May 22, 2014 at 7:31 PM, Johannes Erdfelt wrote: > > > I noticed recently that some tests are being skipped in the Nova gate. > > > > Some will always be skipped, but others are conditional. > > I'd like to

Re: [openstack-dev] Promoting healing script to scheme migration script?

2014-06-09 Thread Johannes Erdfelt
On Mon, Jun 09, 2014, Jakub Libosvar wrote: > I'd like to get some opinions on following idea: > > Because currently we have (thanks to Ann) WIP of healing script capable > of changing database scheme by comparing tables in the database to > models in current codebase, I started to think whether

Re: [openstack-dev] [nova] formally distinguish server "desired state" from "actual state"?

2014-10-01 Thread Johannes Erdfelt
On Wed, Oct 01, 2014, Chris Friesen wrote: > Currently in nova we have the "vm_state", which according to the > code comments is supposed to represent "a VM's current stable (not > transition) state", or "what the customer expect the VM to be". > > However, we then added in an ERROR state. How d

Re: [openstack-dev] [all][oslo][db][docs] RFC: drop support for libpq < 9.1

2014-10-06 Thread Johannes Erdfelt
On Mon, Oct 06, 2014, Ihar Hrachyshka wrote: > Maybe it is indeed wasteful, I don't have numbers; though the fact is > we don't allow any migrations for databases with any non utf8 tables > as of [1]. The code was copied in multiple projects (Nova, Glance > among other things). Also, the same chec

Re: [openstack-dev] [nova] Do any hyperviors allow disk reduction as part of resize ?

2014-06-13 Thread Johannes Erdfelt
On Fri, Jun 13, 2014, Russell Bryant wrote: > On 06/13/2014 09:22 AM, Day, Phil wrote: > > I guess the question I’m really asking here is: “Since we know resize > > down won’t work in all cases, and the failure if it does occur will be > > hard for the user to detect, should we just block it at t

Re: [openstack-dev] [nova] Do any hyperviors allow disk reduction as part of resize ?

2014-06-13 Thread Johannes Erdfelt
On Fri, Jun 13, 2014, Andrew Laski wrote: > > On 06/13/2014 10:53 AM, Johannes Erdfelt wrote: > >On Fri, Jun 13, 2014, Russell Bryant wrote: > >>On 06/13/2014 09:22 AM, Day, Phil wrote: > >>>I guess the question I’m really asking here is: “Since we know resize

Re: [openstack-dev] [nova][vmware] Convert to rescue by adding the rescue image and booting from it

2014-07-14 Thread Johannes Erdfelt
On Mon, Jul 14, 2014, Daniel P. Berrange wrote: > I think that I'd probably say there is an expectation that the rescue > image will be different from the primary image the OS was booted from. So every image would now need a corresponding rescue image? JE __

Re: [openstack-dev] [nova] fair standards for all hypervisor drivers

2014-07-16 Thread Johannes Erdfelt
On Wed, Jul 16, 2014, Mark McLoughlin wrote: > No, there are features or code paths of the libvirt 1.2.5+ driver that > aren't as well tested as the "class A" designation implies. And we have > a proposal to make sure these aren't used by default: > > https://review.openstack.org/107119 > > i.

Re: [openstack-dev] [nova] fair standards for all hypervisor drivers

2014-07-17 Thread Johannes Erdfelt
On Thu, Jul 17, 2014, Daniel P. Berrange wrote: > On Wed, Jul 16, 2014 at 09:44:55AM -0700, Johannes Erdfelt wrote: > > So that means the libvirt driver will be a mix of tested and untested > > features, but only the tested code paths will be enabled by default? > > > &

Re: [openstack-dev] [nova] fair standards for all hypervisor drivers

2014-07-17 Thread Johannes Erdfelt
On Thu, Jul 17, 2014, Russell Bryant wrote: > On 07/17/2014 02:31 PM, Johannes Erdfelt wrote: > > It kind of helps. It's still implicit in that you need to look at what > > features are enabled at what version and determine if it is being > > tested. > > >

[openstack-dev] [Nova] [Spec freeze exception] Online Schema Changes

2014-07-18 Thread Johannes Erdfelt
I'm requestion a spec freeze exception for online schema changes. https://review.openstack.org/102545 This work is being done to try to minimize the downtime as part of upgrades. Database migrations have historically been a source of long periods of downtime. The spec is an attempt to start optim

Re: [openstack-dev] [oslo.config] Config files overriding CLI: The path of most surprise.

2013-07-01 Thread Johannes Erdfelt
On Mon, Jul 01, 2013, Clint Byrum wrote: > I am writing today to challenge that notion, and also to suggest that even > if that is the case, it is inappropriate to have oslo.config operate in > such a profoundly different manner than basically any other config library > or system software in gener

Re: [openstack-dev] Work around DB in OpenStack (Oslo, Nova, Cinder, Glance)

2013-07-03 Thread Johannes Erdfelt
On Wed, Jul 03, 2013, Michael Still wrote: > On Wed, Jul 3, 2013 at 3:50 AM, Boris Pavlovic wrote: > > > Question: > > Why we should put in oslo slqlalchemy-migrate monkey patches, when we are > > planing to switch to alembic? > > > > Answer: > >If we don’t put in oslo sqlalchemy-migrate m

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-20 Thread Johannes Erdfelt
On Tue, Aug 20, 2013, Flavio Percoco wrote: > There are a couple of things that would worry me about an hypothetic > support for NoSQL but I guess one that I'd consider very critical is > migrations. Some could argue asking whether we'd really need them or > not - when talking about NoSQL databas

Re: [openstack-dev] [Nova] Frustrations with review wait times

2013-08-28 Thread Johannes Erdfelt
On Wed, Aug 28, 2013, Russell Bryant wrote: > On 08/28/2013 12:25 PM, Davanum Srinivas wrote: > > If each "little group" had at least one active Nova core member, i think > > it would speed things up way faster IMHO. > > Agreed, in theory. However, we should not add someone just for the sake >