Re: [openstack-dev] [all] New setuptools release, and the world is broken
Excerpts from Jeremy Stanley's message of 2017-07-14 16:18:04 +: > On 2017-07-14 16:05:36 + (+), Jesse Pretorius wrote: > > On 7/14/17, 4:54 PM, "Doug Hellmann" wrote: > [...] > > > I wonder if we could convince the PyPA folks to allow get-pip.py > > >to take a version argument, so we could specify which version we want > > > in > > >our jobs. We would still need a way to manage that version number, but > > >modifying get-pip.py would solve the bootstrapping issue. > > > > It has been capable of this for quite some time. You can provide > > both requirements And constraints. > > > > python /opt/get-pip.py pip==9.0.1 setuptools==33.1.1 wheel==0.29.0 > > > > Or, far better, is to use constraints because it’ll ensure that > > any dependent packages are also the right versions. > > > > python /opt/get-pip.py pip setuptools wheel –constraint > > http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt > > Is there a mechanism to leverage this in tox or when invoking > virtualenv? We don't run get-pip.py in most jobs because our images > have pip/setuptools preinstalled to get around bootstrapping issues, > though I suppose that could with some effort be moved into job > runtime as a (very early) builder macro. Using constraints to > control these during image generation doesn't make a whole lot of > sense though as images are only rebuilt once a day and so tracking > these in the constraints list won't be self-testing in that regard > anyway. I was thinking we would use an early stage builder to do it, too. Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] New setuptools release, and the world is broken
On 7/14/17, 5:18 PM, "Jeremy Stanley" wrote: >Is there a mechanism to leverage this in tox or when invoking >virtualenv? We don't run get-pip.py in most jobs because our images >have pip/setuptools preinstalled to get around bootstrapping issues, >though I suppose that could with some effort be moved into job >runtime as a (very early) builder macro. Using constraints to >control these during image generation doesn't make a whole lot of >sense though as images are only rebuilt once a day and so tracking >these in the constraints list won't be self-testing in that regard >anyway. Fair enough, although that’s what we have today so I don’t see it as worse. In fact, I think it’s an improvement because our constraints can be updated when we choose to (through the normal requirements management process) rather than whenever the new package releases. I think it may make sense to do a pre-tox macro which preps pip/setuptools/wheel using get-pip.py, then places the relevant wheels into the virtualenv_support directory [1] so that tox uses them when building virtualenvs. Another option could be to use a configuration file [2] to specify a directory to find the right wheels in. [1] https://virtualenv.pypa.io/en/stable/userguide/#the-extra-search-dir-option [2] https://virtualenv.pypa.io/en/stable/reference/#configuration-file Rackspace Limited is a company registered in England & Wales (company registered number 03897010) whose registered office is at 5 Millington Road, Hyde Park Hayes, Middlesex UB3 4AZ. Rackspace Limited privacy policy can be viewed at www.rackspace.co.uk/legal/privacy-policy - This e-mail message may contain confidential or privileged information intended for the recipient. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at ab...@rackspace.com and delete the original message. Your cooperation is appreciated. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] New setuptools release, and the world is broken
On 2017-07-14 15:55:40 + (+), Jesse Pretorius wrote: > I did a little spelunking to try and figure out where it is we put > pip onto the nodepool nodes. It appears that d-i-b for Ubuntu > Xenial uses packages to put it there, but that doesn’t explain how > the latest version got there. Does anyone have any insight into > where exactly the latest version is installed? This has changed slightly since we excised the remaining Puppet setup from our images, so it's no longer done by the puppet-pip module. It seems the pip-and-virtualenv element is currently bringing that in. http://git.openstack.org/cgit/openstack/diskimage-builder/tree/diskimage_builder/elements/pip-and-virtualenv/ > -- Jeremy Stanley signature.asc Description: Digital signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] New setuptools release, and the world is broken
On 2017-07-14 16:05:36 + (+), Jesse Pretorius wrote: > On 7/14/17, 4:54 PM, "Doug Hellmann" wrote: [...] > > I wonder if we could convince the PyPA folks to allow get-pip.py > >to take a version argument, so we could specify which version we want in > >our jobs. We would still need a way to manage that version number, but > >modifying get-pip.py would solve the bootstrapping issue. > > It has been capable of this for quite some time. You can provide > both requirements And constraints. > > python /opt/get-pip.py pip==9.0.1 setuptools==33.1.1 wheel==0.29.0 > > Or, far better, is to use constraints because it’ll ensure that > any dependent packages are also the right versions. > > python /opt/get-pip.py pip setuptools wheel –constraint > http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt Is there a mechanism to leverage this in tox or when invoking virtualenv? We don't run get-pip.py in most jobs because our images have pip/setuptools preinstalled to get around bootstrapping issues, though I suppose that could with some effort be moved into job runtime as a (very early) builder macro. Using constraints to control these during image generation doesn't make a whole lot of sense though as images are only rebuilt once a day and so tracking these in the constraints list won't be self-testing in that regard anyway. -- Jeremy Stanley signature.asc Description: Digital signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] New setuptools release, and the world is broken
On 7/14/17, 4:54 PM, "Doug Hellmann" wrote: > I wonder if we could convince the PyPA folks to allow get-pip.py >to take a version argument, so we could specify which version we want in >our jobs. We would still need a way to manage that version number, but >modifying get-pip.py would solve the bootstrapping issue. It has been capable of this for quite some time. You can provide both requirements And constraints. python /opt/get-pip.py pip==9.0.1 setuptools==33.1.1 wheel==0.29.0 Or, far better, is to use constraints because it’ll ensure that any dependent packages are also the right versions. python /opt/get-pip.py pip setuptools wheel –constraint http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt Rackspace Limited is a company registered in England & Wales (company registered number 03897010) whose registered office is at 5 Millington Road, Hyde Park Hayes, Middlesex UB3 4AZ. Rackspace Limited privacy policy can be viewed at www.rackspace.co.uk/legal/privacy-policy - This e-mail message may contain confidential or privileged information intended for the recipient. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at ab...@rackspace.com and delete the original message. Your cooperation is appreciated. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] New setuptools release, and the world is broken
On 7/14/17, 4:17 PM, "Jeremy Stanley" wrote: >Chicken and egg. Once you get to the point where pip can enforce > constraints, you already have a version of setuptools installed. And >as evidenced by, for example, this current bug you would just end up >breaking on the downgrade trying to replace your existing broken >version with whatever version is requested. Also you would need a >separate phase to upgrade/downgrade setuptools separate from other >packages using it. Which is why I think we should do it right at the point where we put pip there. This can be done with the get-pip script which only requires python to be there. I did a little spelunking to try and figure out where it is we put pip onto the nodepool nodes. It appears that d-i-b for Ubuntu Xenial uses packages to put it there, but that doesn’t explain how the latest version got there. Does anyone have any insight into where exactly the latest version is installed? Rackspace Limited is a company registered in England & Wales (company registered number 03897010) whose registered office is at 5 Millington Road, Hyde Park Hayes, Middlesex UB3 4AZ. Rackspace Limited privacy policy can be viewed at www.rackspace.co.uk/legal/privacy-policy - This e-mail message may contain confidential or privileged information intended for the recipient. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at ab...@rackspace.com and delete the original message. Your cooperation is appreciated. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] New setuptools release, and the world is broken
Excerpts from Jeremy Stanley's message of 2017-07-14 15:17:52 +:
> On 2017-07-14 10:50:50 -0400 (-0400), Doug Hellmann wrote:
> > Excerpts from Jesse Pretorius's message of 2017-07-14 08:32:48 +:
> > > FYI if you see the following error in your job logs, you have the new
> > > setuptools to thank:
> > >
> > > AttributeError: Distribution instance has no attribute 'install_requires'
> > >
> > > I’ve registered https://github.com/pypa/setuptools/issues/1086 to track
> > > the issue, and very quickly got a response and there’s a PR up to resolve
> > > it.
> > >
> > > In our case we saw this when downgrading setuptools to our known, good
> > > working version.
> > >
> > > I’d like to suggest that we include setuptools, pip, wheel and other core
> > > packages in the upper constraints management process and that all the
> > > images built for jobs make use of it. The number of times that a new
> > > release of pip/setuptools has completely ground development to a halt for
> > > a day, sometimes more, is a little too frequent for my liking.
> > >
> > > IIRC we’d need to just change the u-c generation output from ‘pip freeze’
> > > to ‘pip freeze –all’ for the output to include the versions for pip,
> > > setuptools and wheel. Then, with that spec, pip can be installed using
> > > u-c like so:
> > >
> > > CURL_CMD="curl --silent --show-error --retry 5"
> > > OUTPUT_FILE="get-pip.py"
> > > ${CURL_CMD} https://bootstrap.pypa.io/get-pip.py > ${OUTPUT_FILE} ||\
> > > ${CURL_CMD}
> > > https://raw.githubusercontent.com/pypa/get-pip/master/get-pip.py >
> > > ${OUTPUT_FILE}
> > >
> > > python ${OUTPUT_FILE} pip setuptools wheel -c upper-constraints.txt
> > >
> > > That will ensure a consistent, known good version set is installed and
> > > will also cater for the situation where the primary URL for get-pip.py is
> > > down (as happens sometimes).
> > >
> >
> > I know we made the explicit decision not to pin setuptools, but I don't
> > remember the motivation. Was it a technical decision (we can't) or
> > because it seemed like a bad idea (we want to ensure we have the
> > latest)?
>
> Chicken and egg. Once you get to the point where pip can enforce
> constraints, you already have a version of setuptools installed. And
> as evidenced by, for example, this current bug you would just end up
> breaking on the downgrade trying to replace your existing broken
> version with whatever version is requested. Also you would need a
> separate phase to upgrade/downgrade setuptools separate from other
> packages using it.
That makes sense.
I wonder if we could convince the PyPA folks to allow get-pip.py
to take a version argument, so we could specify which version we want in
our jobs. We would still need a way to manage that version number, but
modifying get-pip.py would solve the bootstrapping issue.
Doug
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] New setuptools release, and the world is broken
On 2017-07-14 10:50:50 -0400 (-0400), Doug Hellmann wrote:
> Excerpts from Jesse Pretorius's message of 2017-07-14 08:32:48 +:
> > FYI if you see the following error in your job logs, you have the new
> > setuptools to thank:
> >
> > AttributeError: Distribution instance has no attribute 'install_requires'
> >
> > I’ve registered https://github.com/pypa/setuptools/issues/1086 to track the
> > issue, and very quickly got a response and there’s a PR up to resolve it.
> >
> > In our case we saw this when downgrading setuptools to our known, good
> > working version.
> >
> > I’d like to suggest that we include setuptools, pip, wheel and other core
> > packages in the upper constraints management process and that all the
> > images built for jobs make use of it. The number of times that a new
> > release of pip/setuptools has completely ground development to a halt for a
> > day, sometimes more, is a little too frequent for my liking.
> >
> > IIRC we’d need to just change the u-c generation output from ‘pip freeze’
> > to ‘pip freeze –all’ for the output to include the versions for pip,
> > setuptools and wheel. Then, with that spec, pip can be installed using u-c
> > like so:
> >
> > CURL_CMD="curl --silent --show-error --retry 5"
> > OUTPUT_FILE="get-pip.py"
> > ${CURL_CMD} https://bootstrap.pypa.io/get-pip.py > ${OUTPUT_FILE} ||\
> > ${CURL_CMD}
> > https://raw.githubusercontent.com/pypa/get-pip/master/get-pip.py >
> > ${OUTPUT_FILE}
> >
> > python ${OUTPUT_FILE} pip setuptools wheel -c upper-constraints.txt
> >
> > That will ensure a consistent, known good version set is installed and will
> > also cater for the situation where the primary URL for get-pip.py is down
> > (as happens sometimes).
> >
>
> I know we made the explicit decision not to pin setuptools, but I don't
> remember the motivation. Was it a technical decision (we can't) or
> because it seemed like a bad idea (we want to ensure we have the
> latest)?
Chicken and egg. Once you get to the point where pip can enforce
constraints, you already have a version of setuptools installed. And
as evidenced by, for example, this current bug you would just end up
breaking on the downgrade trying to replace your existing broken
version with whatever version is requested. Also you would need a
separate phase to upgrade/downgrade setuptools separate from other
packages using it.
--
Jeremy Stanley
signature.asc
Description: Digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] New setuptools release, and the world is broken
Excerpts from Jesse Pretorius's message of 2017-07-14 08:32:48 +:
> FYI if you see the following error in your job logs, you have the new
> setuptools to thank:
>
> AttributeError: Distribution instance has no attribute 'install_requires'
>
> I’ve registered https://github.com/pypa/setuptools/issues/1086 to track the
> issue, and very quickly got a response and there’s a PR up to resolve it.
>
> In our case we saw this when downgrading setuptools to our known, good
> working version.
>
> I’d like to suggest that we include setuptools, pip, wheel and other core
> packages in the upper constraints management process and that all the images
> built for jobs make use of it. The number of times that a new release of
> pip/setuptools has completely ground development to a halt for a day,
> sometimes more, is a little too frequent for my liking.
>
> IIRC we’d need to just change the u-c generation output from ‘pip freeze’ to
> ‘pip freeze –all’ for the output to include the versions for pip, setuptools
> and wheel. Then, with that spec, pip can be installed using u-c like so:
>
> CURL_CMD="curl --silent --show-error --retry 5"
> OUTPUT_FILE="get-pip.py"
> ${CURL_CMD} https://bootstrap.pypa.io/get-pip.py > ${OUTPUT_FILE} ||\
> ${CURL_CMD}
> https://raw.githubusercontent.com/pypa/get-pip/master/get-pip.py >
> ${OUTPUT_FILE}
>
> python ${OUTPUT_FILE} pip setuptools wheel -c upper-constraints.txt
>
> That will ensure a consistent, known good version set is installed and will
> also cater for the situation where the primary URL for get-pip.py is down (as
> happens sometimes).
>
I know we made the explicit decision not to pin setuptools, but I don't
remember the motivation. Was it a technical decision (we can't) or
because it seemed like a bad idea (we want to ensure we have the
latest)?
Doug
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
