On 09/05/2014 06:32 AM, Sean Dague wrote:
While reviewing this zookeeper service group fix in Nova -
https://review.openstack.org/#/c/102639/ it was exposed that the
zookeeper tests aren't running in infra.
The crux of the issue is that zookeeper python modules are C extensions.
So you have to either install from packages (which we don't do in unit
tests) or install from pip, which means forcing zookeeper dev packages
locally. Realistically this is the same issue we end up with for mysql
and pg, but given their wider usage we just forced that pain on developers.
But it seems like a bad stand off between testing upstream and testing
normal path locally.
Big picture it would be nice to not require a ton of dev libraries
locally for optional components, but still test them upstream. So that
in the base case I'm not running zookeeper locally, but if it fails
upstream because I broke something in zookeeper, it's easy enough to
spin up that dev env that has it.
Which feels like we need some decoupling on our requirements vs. tox
targets to get there. CC to Monty and Clark as our super awesome tox
hackers to help figure out if there is a path forward here that makes sense.
Funny story - I've come to dislike what we're doing here, so I've been
slowly working on an idea in this area:
https://github.com/emonty/dox
The tl;dr is "it's like tox, except it uses docker instead of
virtualenv" - which means we can express all of our requirements, not
just pip ones.
It's not quite ready yet - although I'd be happy to move it in to
stackforge or even openstack-dev and get other people hacking on it with
me until it is. The main problem that needs solving, I think, is how to
sanely express multiple target environments (like py26,py27) without
making a stupidly baroque config file. OTOH, tox's insistence of making
a new virtualenv for each "environment" is heavyweight and has led to
some pretty crazy hacks across the project. Luckily, docker itself does
an EXCELLENT job at handling caching and reuse - so I think we can have
a set of containers that something in infra (waves hands) publishes to
dockerhub, like:
infra/py27
infra/py26
And then have things like nova build on those, like:
infra/nova/py27
Which would have zookeeper as well
The _really_ fun part, again, if we can figure out how to express it in
config without reimplementing make accidentally, is that we could start
to have things like:
infra/mysql
infra/postgres
infra/mongodb
And have dox files say things like:
Nova unittests want a python27 environment, this means we want an
infra/mysql container, an infra/postgres container and for those to be
linked to the infra/nova/py27 container where the tests will run.
Since those are all reusable, the speed should be _Excellent_ and we
should be able to more easily get more things runnable locally without
full devstack.
Thoughts? Anybody wanna hack on it with me? I think it could wind up
being a pretty useful tool for folks outside of OpenStack too if we get
it right.
Monty
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
