Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-29 Thread Daniel P. Berrange
On Tue, Aug 06, 2013 at 11:36:44PM -0300, Monty Taylor wrote: > > > On 08/06/2013 11:14 PM, Robert Collins wrote: > > On 7 August 2013 11:22, Jay Buffington wrote: > > > >> ln -s /usr/lib64/python2.6/site-packages/libvirtmod_qemu.so > >> $(VENV)/lib/python2.6/site-packages/ > >> > >> Why is

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-08 Thread Ian Wienand
On Thu, Aug 08, 2013 at 10:10:09AM -0300, Monty Taylor wrote: > I don't think we will gain much by auto-generating packages. What really is the difference between devstack auto-generating a package and having a human basically doing the same thing and sticking it in a repo? It just seems unreliab

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-08 Thread Joshua Harlow
Just an idea how this could go (without saying just use anvil). Taking from how anvil is doing it the following might work: 1. In devstack pass all the requirement files to multipip to get a unified list back. 2. Use yumfind (maybe a similar utility that integrates with apt-get/apt also?) to scan

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-08 Thread Joshua Harlow
Seems pretty similar. Has anyone been using that in production scenarios?? On 8/8/13 4:10 AM, "Pádraig Brady" wrote: >On 08/07/2013 06:54 PM, Monty Taylor wrote: >> >> >> On 08/07/2013 12:53 PM, Joshua Harlow wrote: >>> I agree triple-o will help a lot here although I would disagree that >>> p

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-08 Thread Dean Troyer
On Thu, Aug 8, 2013 at 11:34 AM, Monty Taylor wrote: > I'd be more happy about depending on RDO - we already said that we're ok > with depending on Ubuntu Cloud Archive for Ubuntu flavors. Basically, I > don't want devstack to depend on repos that we don't think is reasonable > to tell end users w

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-08 Thread Monty Taylor
On 08/08/2013 12:58 PM, Pádraig Brady wrote: > On 08/08/2013 02:10 PM, Monty Taylor wrote: >> >> >> On 08/05/2013 02:03 PM, Dean Troyer wrote: >>> [Moving a discussion from https://review.openstack.org/40019 to the ML >>> to get a wider audience] >>> >>> We've been around this block more than onc

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-08 Thread Pádraig Brady
On 08/08/2013 02:10 PM, Monty Taylor wrote: > > > On 08/05/2013 02:03 PM, Dean Troyer wrote: >> [Moving a discussion from https://review.openstack.org/40019 to the ML >> to get a wider audience] >> >> We've been around this block more than once so let's get it all >> documented in one place and s

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-08 Thread Monty Taylor
On 08/05/2013 02:03 PM, Dean Troyer wrote: > [Moving a discussion from https://review.openstack.org/40019 to the ML > to get a wider audience] > > We've been around this block more than once so let's get it all > documented in one place and see where to go next. Skip down to > # for

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-08 Thread Pádraig Brady
On 08/07/2013 06:54 PM, Monty Taylor wrote: > > > On 08/07/2013 12:53 PM, Joshua Harlow wrote: >> I agree triple-o will help a lot here although I would disagree that >> package rollback is an illusion. I would call it more of a hard >> problem instead since nothing is really impossible :) > > i

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-08 Thread Daniel P. Berrange
On Tue, Aug 06, 2013 at 11:36:44PM -0300, Monty Taylor wrote: > > > On 08/06/2013 11:14 PM, Robert Collins wrote: > > On 7 August 2013 11:22, Jay Buffington wrote: > > > >> ln -s /usr/lib64/python2.6/site-packages/libvirtmod_qemu.so > >> $(VENV)/lib/python2.6/site-packages/ > >> > >> Why is

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-08 Thread Daniel P. Berrange
On Wed, Aug 07, 2013 at 05:43:19PM +, Joshua Harlow wrote: > Interesting, I should look into conary. > > I was hoping that DNF (the yum rewrite) would be a little better at this > stage, but it still seems a ways off. I don't know about what future plans the developers may have, but currently

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-07 Thread Joshua Harlow
Ya, something like that would be super awesome. Yum has something like a LVM snapshot plugin, idk if it works. I was talking with angus about this, and I still think there is hope that DNF might have something like that (peasee). Making a package manager that has interaction with a commit lo

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-07 Thread Monty Taylor
On 08/07/2013 12:53 PM, Joshua Harlow wrote: > I agree triple-o will help a lot here although I would disagree that > package rollback is an illusion. I would call it more of a hard > problem instead since nothing is really impossible :) illusion with current packaging systems yum/rpm and apt/dp

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-07 Thread Joshua Harlow
Interesting, I should look into conary. I was hoping that DNF (the yum rewrite) would be a little better at this stage, but it still seems a ways off. And I'm also not sure if DNF is attempting to even address this kind of stuff, although I don't quite see how RH if they plan on selling RDO can a

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-07 Thread Jeremy Stanley
On 2013-08-07 15:53:54 + (+), Joshua Harlow wrote: > I agree triple-o will help a lot here although I would disagree > that package rollback is an illusion. I would call it more of a > hard problem instead since nothing is really impossible :) [...] SCO OpenServer (shudder) for all its iss

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-07 Thread Joshua Harlow
I agree triple-o will help a lot here although I would disagree that package rollback is an illusion. I would call it more of a hard problem instead since nothing is really impossible :) If say yum had a git "like" log then I don't think it would be impossible to yum "checkout" a previous syste

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-07 Thread Monty Taylor
On 08/06/2013 11:39 PM, Robert Collins wrote: > On 7 August 2013 14:36, Monty Taylor wrote: >> >> >> On 08/06/2013 11:14 PM, Robert Collins wrote: >>> On 7 August 2013 11:22, Jay Buffington wrote: >>> ln -s /usr/lib64/python2.6/site-packages/libvirtmod_qemu.so $(VENV)/lib/python2.

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-06 Thread Clint Byrum
Excerpts from Joshua Harlow's message of 2013-08-06 21:03:58 -0700: > It does seem sad that the state of package management is this bad. > > It'd would be equally interesting to hear how others rollback changes > (another thing yum doesn't do so well, since it doesn't have a good ability > to r

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-06 Thread Joshua Harlow
It does seem sad that the state of package management is this bad. It'd would be equally interesting to hear how others rollback changes (another thing yum doesn't do so well, since it doesn't have a good ability to rollback dependent changes, especially when said rollback would alter a lot of

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-06 Thread Robert Collins
On 7 August 2013 14:36, Monty Taylor wrote: > > > On 08/06/2013 11:14 PM, Robert Collins wrote: >> On 7 August 2013 11:22, Jay Buffington wrote: >> >>> ln -s /usr/lib64/python2.6/site-packages/libvirtmod_qemu.so >>> $(VENV)/lib/python2.6/site-packages/ >>> >>> Why isn't libvirt-python on pypi

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-06 Thread Monty Taylor
On 08/06/2013 11:14 PM, Robert Collins wrote: > On 7 August 2013 11:22, Jay Buffington wrote: > >> ln -s /usr/lib64/python2.6/site-packages/libvirtmod_qemu.so >> $(VENV)/lib/python2.6/site-packages/ >> >> Why isn't libvirt-python on pypi? AFAICT, nothing is stopping us from >> uploading it

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-06 Thread Robert Collins
On 6 August 2013 05:03, Dean Troyer wrote: > * Continue to factor the prereq setup out of stack.sh such that > requirements.txt is satisfied one way or another before it begins to > install OpenStack. tools/install_prereqs.sh and tools/install_pip.sh > are the prototypes for this. Each distro g

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-06 Thread Robert Collins
On 7 August 2013 11:22, Jay Buffington wrote: > ln -s /usr/lib64/python2.6/site-packages/libvirtmod_qemu.so > $(VENV)/lib/python2.6/site-packages/ > > Why isn't libvirt-python on pypi? AFAICT, nothing is stopping us from > uploading it. Maybe we should just stick it on there and this issue

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-06 Thread Jay Buffington
On Tue, Aug 6, 2013 at 12:00 PM, Monty Taylor wrote: > On 08/06/2013 02:44 PM, Mate Lakat wrote: > > I would say, use a separated virtual environment in devstack - without > > the --system-site-packages switch, of course, and set it up as a user. > > Install the packages that are needed in order

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-06 Thread Monty Taylor
On 08/06/2013 02:44 PM, Mate Lakat wrote: > Hi, > > I would say, use a separated virtual environment in devstack - without > the --system-site-packages switch, of course, and set it up as a user. > Install the packages that are needed in order to be able to pip install > them (like libxslt-dev).

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-06 Thread Clint Byrum
Excerpts from Dean Troyer's message of 2013-08-05 10:03:07 -0700: > * about-face on all-packages to all-PyPI (mtaylor): I'm still on the > use packages where possible side but DevStack specifically is not in > the packaging business. If it were we'd do what Java folk (*sorry*) > have long taken th

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-06 Thread Mate Lakat
Hi, I would say, use a separated virtual environment in devstack - without the --system-site-packages switch, of course, and set it up as a user. Install the packages that are needed in order to be able to pip install them (like libxslt-dev). It's a development environment. I think my email is equ

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-06 Thread Joshua Harlow
k.org>> Date: Tuesday, August 6, 2013 9:40 AM To: OpenStack Development Mailing List mailto:openstack-dev@lists.openstack.org>> Subject: Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages On Tue, Aug 6, 2013 at 8:35 AM, Joshua Harlow mailto:harlo...@yahoo-

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-06 Thread Jay Buffington
On Tue, Aug 6, 2013 at 8:35 AM, Joshua Harlow wrote: > I think jay your usage also was before anvil started to build all the > *missing* dependencies automatically (something u inspired me to get going > in the first place) so hopefully said updates to rhel.yaml are only now > needed for exceptio

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-06 Thread John Dennis
On 08/06/2013 11:19 AM, Dean Troyer wrote: > And that is the crux of the problem. When both can be installed > side-by-side and sys.path is in control of the order, things work. > This is such a fundamental problem in the distro that I am beginning > to thing that we need to address it ourselves.

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-06 Thread Joshua Harlow
I think jay your usage also was before anvil started to build all the *missing* dependencies automatically (something u inspired me to get going in the first place) so hopefully said updates to rhel.yaml are only now needed for exceptions and not the common path :) Sent from my really tiny devi

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-06 Thread Dean Troyer
On Tue, Aug 6, 2013 at 3:50 AM, Bob Ball wrote: > I think we need a further constraint: > > We must ensure that yum/etc believes that the python-* packages are installed. If we want the rest of the system to use them, yes. > If we want to install a newer version of the python-* packages then we

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-06 Thread Jay Buffington
On Tue, Aug 6, 2013 at 4:36 AM, Ian Wienand wrote: > On Mon, Aug 05, 2013 at 03:37:24PM -0700, Jay Buffington wrote: > > I used Anvil for the first three months, but it required constant > > updating of dependency versions and it didn't support quantum. > > What do you mean by "updating" here? T

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-06 Thread Dean Troyer
On Mon, Aug 5, 2013 at 8:37 PM, Ian Wienand wrote: > I think Anvil is working with the package management system so that > scenario doesn't happen. The "fine, those can be re-installed with > pip" bit is where the problem occurs. Agreed. > The "whole lot" bit is important, because you can't hav

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-06 Thread Ian Wienand
On Mon, Aug 05, 2013 at 03:37:24PM -0700, Jay Buffington wrote: > I used Anvil for the first three months, but it required constant > updating of dependency versions and it didn't support quantum. What do you mean by "updating" here? The rpm packages being updated causing a lot of churn, or some

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-06 Thread Bob Ball
I think we need a further constraint: We must ensure that yum/etc believes that the python-* packages are installed. There are a number of packages that we are currently purging in devstack due to the redhat/pip conflict which we cannot purge (e.g. python-lxml, python-c rypto) as they will remo

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-05 Thread Ian Wienand
On Mon, Aug 05, 2013 at 12:03:07PM -0500, Dean Troyer wrote: > * proposals to use a tool to automatically decide between package and > PyPI (harlowja, sdague): this works well on the surface, but anything > that does not take in to account the dependencies in these packages > going BOTH ways is go

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-05 Thread Jay Buffington
s?? Be interesting to look at > :-) > > From: Jay Buffington > Reply-To: OpenStack Development Mailing List < > openstack-dev@lists.openstack.org> > Date: Monday, August 5, 2013 3:37 PM > To: OpenStack Development Mailing List > Subject: Re: [openstack-dev] [De

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-05 Thread Joshua Harlow
e: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages I've been doing continuous deployment via rpms on RHEL 6u3 of glance, keystone, neutron and nova for about six months now. I used Anvil for the first three months, but it required constant updating of dependency versi

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-05 Thread Jay Buffington
I've been doing continuous deployment via rpms on RHEL 6u3 of glance, keystone, neutron and nova for about six months now. I used Anvil for the first three months, but it required constant updating of dependency versions and it didn't support quantum. Also, yum is terrible at managing dependency

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

2013-08-05 Thread Joshua Harlow
Just so people know how anvil (the other tool) is doing it, and which might be useful. Anvil goes through a few stages to get a working openstack system up and running. 1. Downloaded all the git repos for the desired components 2. Extract all the python requirements from said repos 3. Collapse al