Re: [openstack-dev] [glance] need to pin jsonschema version for glance?
Actually, when I build out a virtual environment and install python-glanceclient, I get jsonschema 2.0.0. So maybe the problem is elsewhere? I also get python-glanceclient 0.9.0, but I notice that tempest requires python-glanceclient 0.5.0 ( https://github.com/openstack/tempest/blob/master/requirements.txt#L11 ). What version of python-glanceclient do you have installed in the environment where there is a problem? On Wed, Jul 17, 2013 at 9:52 AM, Mark Washenberger mark.washenber...@markwash.net wrote: On Wed, Jul 17, 2013 at 7:16 AM, Matt Riedemann mrie...@us.ibm.comwrote: I recently synched up on the latest glance and ran tempest on my RHEL 6.3 box and the image v2 tests all started failing due to json schema validation errors: *http://paste.openstack.org/show/40684/*http://paste.openstack.org/show/40684/ I found that the version of jsonschema on the system is 0.7, probably because of the dependency from warlock in python-glanceclient: * https://github.com/openstack/python-glanceclient/blob/master/requirements.txt#L8 *https://github.com/openstack/python-glanceclient/blob/master/requirements.txt#L8 I started looking at what recent changes in glance might be causing the issue and I found this one: *https://review.openstack.org/#/c/35134/*https://review.openstack.org/#/c/35134/ As pointed out in the test output from that patch, since there is no version constraint on jsonschema in glance or tempest, it's getting the latest version from pypi (2.0.0 in this case). When I updated my test box to jsonschema 1.3.0, I got past the schema validation error. So this leads me to believe that we need to pin the jsonschema version in glance and tempest to = 1.3.0. Thoughts? This sounds correct. Another alternative would be to switch back to the old syntax and pin 1.3.0, which sounds like its not really forward progress, but might be easier. Thanks, *MATT RIEDEMANN* Advisory Software Engineer Cloud Solutions and OpenStack Development -- *Phone:* 1-507-253-7622 | *Mobile:* 1-507-990-1889* E-mail:* *mrie...@us.ibm.com* mrie...@us.ibm.com [image: IBM] 3605 Hwy 52 N Rochester, MN 55901-1407 United States ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev image/gif___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Glance] Team Meeting Reminder July 18 14:00 UTC
Hi folks, As usual, we'll be having a Glance team meeting during our early slot in less than 24 hours at 14:00 UTC [1]. For this meeting, we'll be trying to wade in to the process of cleaning up our Havana 3 targeted blueprints, hopefully finishing completely after next week's meeting. Thanks again to everyone for the strong last-minute push to complete Havana 2! Cheers, markwash [1] - http://www.timeanddate.com/worldclock/fixedtime.html?msg=Glance+Meetingiso=20130718T14ah=1 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Review] Use of exception for non-exceptional cases
I'm curious if folks know about this?
import sys
def foo():
raise Exception('Foo Exception')
def bar():
try:
foo()
except Exception:
exc_info = sys.exc_info()
raise Exception('Bar Exception'), None, exc_info[2]
bar()
Traceback (most recent call last):
File test.py, line 15, in module
bar()
File test.py, line 9, in bar
foo()
File test.py, line 4, in foo
raise Exception('Foo Exception')
Exception: Bar Exception
On Fri, Jul 12, 2013 at 3:53 PM, Doug Hellmann
doug.hellm...@dreamhost.comwrote:
On Fri, Jul 12, 2013 at 2:40 PM, Brant Knudson b...@acm.org wrote:
Somewhat offtopic, but others might find it interesting. On another
project I used a different technique for exceptions, where the original
exception is enhanced with new fields as it propagates up the stack to
where it's handled. In this way all the information needed to handle the
exception is available (even if it's just to log it).
Often there's some information that would be useful for the exception
handler that isn't available at the place where the exception is raised.
The classic example is an error writing to a file where you'd like to know
the filename but all you've got is the file descriptor. An example from an
OpenStack REST API might be that you get an exception and you'd like to log
a message that includes the resource path, but the resource path isn't
known at this point.
So rather than logging the exception when it happens, enhance the
exception, re-raise it, and only once it's got all the information where
you can print out a useful log message you log it.
Here's a short example to illustrate. An exception is raised by f1(), but
at this point we don't know the status code that the server should respond
with or the request path which would be useful in a log message. These bits
of information are added as the exception propagates and then where it's
caught we've got all the information for a useful log message.
def f1():
raise Exception('something')
def f2():
try:
f1()
except Exception as e:
e.status_code = 403
raise
def do_tokens():
try:
f2()
except Exception as e:
e.req_url = '/v3/tokens'
raise
status_code = 200
try:
do_tokens()
except Exception as e:
status_code = getattr(e, 'status_code', 500)
req_url = getattr(e, 'req_url', 'unknown')
if status_code != 200:
print 'Got %s from %s' % (status_code, req_url)
# build an error document for the response.
One problem with this approach is it spreads knowledge of the error
construction up and down the stack through different layers of the
application, and that brings with it assumptions about the implementation
at the different layers. For example, should the application code above
know that do_tokens() is making web requests to URLs? Why does it need that
information?
SRP-ly,
Doug
On Fri, Jul 12, 2013 at 12:25 PM, Nachi Ueno na...@ntti3.com wrote:
Hi folks
Monty
Thank you for your adding good topic for this.
I agree, hiding true cause by exception get hard to debug
For, example, Sqlalchemy gives me this error for general sql errors..
exceptions must be old-style classes or derived from BaseException, not
str
(It should be hidden from REST API users though)
Also, I agree with you about log policies.
sometimes 404 get warn log..
IMO, The log more than warn level should help Operators.
also If the exception is truly, the exceptional case which needs help
of operators.
It should be logged as error level.
Thanks
Nachi
2013/7/12 Dolph Mathews dolph.math...@gmail.com:
On Fri, Jul 12, 2013 at 10:09 AM, Monty Taylor mord...@inaugust.com
wrote:
On 07/11/2013 05:20 AM, Mark McLoughlin wrote:
On Wed, 2013-07-10 at 19:49 -0400, Monty Taylor wrote:
I'd like top-post and hijack this thread for another exception
related
thing:
a) Anyone writing code such as:
try:
blah()
except SomeException:
raise SomeOtherExceptionLeavingOutStackContextFromSomeException
should be mocked ruthlessly.
Ok, mock me ruthlessly then.
Mock. Mock. Mock. Mock.
Part of designing any API is specifying what predictable exceptions
it
will raise. For any predictable error condition, you don't want
callers
to have to catch random exceptions from the underlying libraries you
might be calling into.
Absolutely. But you don't want to go so overboard that you remove the
ability for a developer to debug it. More importantly, INSIDE of
server
code, we're not designing fun apis for the consumption of people we
don't know. There is clearly an API, but we are the consumers of our
own
API.
Lies! Write everything to be intuitive for new contributors or we
won't have
any :(
Say if I was designing an image downloading API, I'd do something
like
this:
https://gist.github.com/markmc/5973868
Assume there's a
Re: [openstack-dev] [oslo.config] Config files overriding CLI: The path of most surprise.
I think I had a different takeaway from that thread. My understanding was that people were agreeing with you that CLI args *should* have highest precedence and override everything else. The talk about permanence confuses me, unless we mean that more permanent values are overridden by less permanent ones. I was also confused by the ordering in this list, though when I read more carefully it seems to agree with me: - Default value in source code - Overridden by value in config file - Overridden by value in environment variable - Overridden by value given as command line option I'd like to rewrite that list as - Value given as a command line option - Failing that, value in environment variable - Failing that, value in config file - Failing that, default value in source code which I believe to be completely equivalent to Monty's original formulation. On Mon, Jul 1, 2013 at 2:52 PM, Clint Byrum cl...@fewbar.com wrote: Last week I went to use oslo.config in a utility I am writing called os-collect-config[1]... While running unit tests on the main() method that is used for the CLI, I was surprised to find that my unit tests were picking up values from a config file I had created just as a test. The tests can be fixed to disable config file lookups, but what was more troublesome was that the config file was overriding values I was passing in as sys.argv. I have read the thread[2] which suggest that CLI should defer to config file because config files are somehow less permanent than the CLI. I am writing today to challenge that notion, and also to suggest that even if that is the case, it is inappropriate to have oslo.config operate in such a profoundly different manner than basically any other config library or system software in general use. CLI options are _for config files_ and if packagers are shipping configurations in systemd unit files, upstart jobs, or sysvinits, they are doing so to control the concerns of that particular invocation of whatever command they are running, and not to configure the software entirely. CLI args are by definition ephemeral, even if somebody might make them permanent in their system, I doubt any packager would then expect that these CLI args would be overridden by any config files. This default is just wrong, and needs to be fixed. [1] https://github.com/SpamapS/os-collect-config.git [2] http://lists.openstack.org/pipermail/openstack-dev/2013-May/008691.html Clint Byrum HP Cloud Services ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Build-time unit tests failing when building glance 2013.2.b1
Hi Thomas, We switched to using entrypoints for creating our binaries. Unfortunately, this means you need to have the glance version installed in some sense in your path when you want to run those tests. This felt like a no-no when we made the change, but it worked fine without any changes to our normal tox and run_tests.sh (with virtualenv) so it made it through. Please let me know if this is really hard to deal with on your side; I can imagine a few other possible fixes we might pursue. Thanks On Sat, Jun 29, 2013 at 3:07 AM, Thomas Goirand z...@debian.org wrote: Hi, I got a bunch of unit test failures with the output: STDERR: /bin/sh: 1: glance-manage: not found glance-manage used to be in bin/ in the sources (at least until Grizzly), though this changed for Havana. I used find to search for it, and couldn't find it in the source code, even though it really is installed at the correct place after my Debian package is built. Where has it been moved? How to fix the unit tests? Cheers, Thomas Goirand (zigo) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Build-time unit tests failing when building glance 2013.2.b1
Munging together some replies: On Sat, Jun 29, 2013 at 12:26 PM, Monty Taylor mord...@inaugust.com wrote: On 06/29/2013 02:14 PM, Thomas Goirand wrote: On 06/30/2013 12:16 AM, Mark Washenberger wrote: Hi Thomas, We switched to using entrypoints for creating our binaries. Can you explain this a bit more? Very specifically, https://github.com/openstack/glance/blob/master/setup.py#L55 This means rather than keeping a /bin directory in our project, we're asking setup.py to generate scripts and install them in an appropriate location when setup.py install is run. Unfortunately, this means you need to have the glance version installed in some sense in your path when you want to run those tests. This felt like a no-no when we made the change, but it worked fine without any changes to our normal tox and run_tests.sh (with virtualenv) so it made it through. Please let me know if this is really hard to deal with on your side; I can imagine a few other possible fixes we might pursue. Thanks Using virtualenv is indeed not an option on the packaging side. Having glance build-depends: on itself is also not an option. What can I do? Just for future reference for me, what kinds of dependencies can you establish for your test step in maintaining your package repos? Do you pretty much only have access to a source checkout / tarball? And is the test step completely contained within the build step? I believe we should fix the unittests to not shell out to glance-manage in that manner. One of the nice things about moving the code from bin/ to glance.cmd is that it's available inside of the source tree for unittests! :) What we want to do is call the glance.cmd.manage:main() function directly. So, for instance, in: glance/tests/functional/__init__.py and glance/tests/functional/test_glance_manage.py We can call the function itself, rather than the shell wrapper. I'll make a patch. Are you seeing problems with anything other than glance-manage? Thanks Monty! This was exactly what I had in mind as an alternative approach. I believe glance-manage is the only binary where glance had such an install dependency for testing--but I can't answer for Thomas. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] OpenStack Programs
Thanks for kicking off this discussion! I think the idea of programs has fantastic promise. On Mon, Jun 24, 2013 at 2:50 AM, Thierry Carrez thie...@openstack.orgwrote: Hi everyone, Official OpenStack projects are those under the oversight of the Technical Committee, and contributing to one grants you ATC status (which in turn you use to elect the Technical Committee members). The list of official projects used to be simple (Swift+Nova) but nowadays it is rather convoluted, with categories like integrated, incubated, library, gating and supporting, as described in [1]. That complexity derived from the need to special-case some projects because their PTL would automatically get a TC seat. Now that we simplified the TC membership, we can also simplify official projects nomenclature by blessing the goals rather than the specific repositories. [1] https://wiki.openstack.org/wiki/Projects This is why Monty had the idea of programs which would be blessed by the TC (and then any code under an official program becomes official). Rather than trying to come up with categories that would cover all the stuff that the Infrastructure team is working on (gating, supporting, libraries...), just say that Infrastructure is a program and let them add any repo that they need. The TC would bless the *mission statement* of the program rather than the specific set of projects implemented to reach that goal. That sounds like a pretty nice idea, so could we consider everything falls under the realm of a program ? Like having an integrated release program that would contain all integrated projects ? I think we need to keep special-casing a concept of projects, separated from programs, since those are accepted one by one by the TC and go through an incubation period. Those projects would contain at least one repo that wants to be part of the integrated, common OpenStack release, plus anything the same team works on (like the corresponding python client project). To match with the current state we would end up with: * Projects (Nova, Neutron, Swift, Glance, Keystone, Horizon, Cinder, Ceilometer, Heat) * Incubated projects (Trove, Ironic) * Programs (Oslo, Infrastructure, Documentation, QA) New programs would draft a clear mission statement and apply to the TC for consideration. Programs should also expect to have a specific topic at the Design Summit (most of them already have), and should probably designate a lead/ambassador as a clear go-to person. A few questions I had left: * There are efforts that span multiple projects but work directly on the project code repositories, like integrated release, or stable maintenance, or vulnerability management (collectively called for the convenience of this thread horizontal efforts). Should they be considered separate programs (without repos) ? Be lumped together into some catch-all integration or production program ? Or ignored as far as ATC status goes ? I've mixed feelings about that. On one hand I'd like those efforts visible and official to be more widely seen as a good way to contribute to OpenStack. On the other hand it's hard to tie ATC membership to those since we can't trace that back to commits to a specific repo Can you clarify this concern? Overall, folks would still be ATCs of the projects that they were committing on in the name of the given program, so their OpenStack ATC status would be successfully tracked regardless. Is the problem organizing leadership votes *within* the program? If so, could we settle on horizontal votes being extended to all OpenStack ATCs? or is that just too wide open? , and I'd like the programs mission statements to be precise rather than vague, so that the TC can bless them... Can you give an example of the kind of vagueness you're worried about? Perhaps it is just a failure of my imagination, but I can't seem to conceive of a way a program mission statement would be distinctly more vague than a project mission statement. * Where would devstack fall ? QA program ? Infrastructure program ? * Where would openstack/requirements fall ? -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [PTLs] Proposed simplification around blueprint tracking
Perfect. On Thu, Jun 20, 2013 at 5:20 AM, Thierry Carrez thie...@openstack.orgwrote: Mark Washenberger wrote: This sounds like a fantastic improvement. I'd really like to have a next milestone as well as an Ongoing milestone, for tracking the kinds of long, drawn out refactorings / code improvements that cannot fit in a single milestone but still need to be communicated to developers (though not necessarily to the project update coordinators). We could setup across the board a future series with a next milestone and an ongoing milestone. That way we could still give proper priority to stuff we want in the next cycle, and somewhere to place ongoing efforts that are basically never quite finished. When a new cycle starts we'd review the next milestone and move most of it to the milestones of the newly-created cycle. Would that work for you ? -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
