Re: [Distutils] Disabling non HTTPS access to APIs on PyPI

[Alex Domoradov]

Yes, I know that. But unfortunately it's a legacy, so there is no way :(

I have resolved my issue with the following line in Dockerfile

RUN echo "[easy_install]\n\nindex-url = https://pypi.python.org/simple/"; >
~/.pydistutils.cfg

On Mon, Oct 30, 2017 at 3:21 PM, Marius Gedminas  wrote:

> On Sat, Oct 28, 2017 at 12:22:32AM +0300, Alex Domoradov wrote:
> > I got it. And what I should do with old system? For e.g. we still use
> ubuntu
> > 12.04. Is there any way to upgrade pip/setuptools?
>
> If you're using Ubuntu 12.04, then presumably you're paying Canoncial
> for extended support, so ask them to provide a pip/setuptools SRU.
>
> (If you're not paying Canonical, then you're not getting security
> updates and should upgrade ASAP.)
>
> Marius Gedminas
> --
> Favorite MS-DOS error message: "Drive C: not ready, close door."
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Disabling non HTTPS access to APIs on PyPI

[Marius Gedminas]

On Sat, Oct 28, 2017 at 12:22:32AM +0300, Alex Domoradov wrote:
> I got it. And what I should do with old system? For e.g. we still use ubuntu
> 12.04. Is there any way to upgrade pip/setuptools?

If you're using Ubuntu 12.04, then presumably you're paying Canoncial
for extended support, so ask them to provide a pip/setuptools SRU.

(If you're not paying Canonical, then you're not getting security
updates and should upgrade ASAP.)

Marius Gedminas
-- 
Favorite MS-DOS error message: "Drive C: not ready, close door."


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Disabling non HTTPS access to APIs on PyPI

[Wes Turner]

- Are there issue tickets which contain the search-indexed ERROR_STRINGS
users may encounter due to this change?

- Does it make sense to add an update regarding this necessary security
upgrade to https://status.python.org (which can be subscribed to and
followed on http://www.twitter.com/PythonStatus )?

On Thursday, October 26, 2017, Donald Stufft  wrote:

> Historically PyPI was only available over either HTTP or unvalidated
> HTTPS, and over time we’ve been pushing more and more traffic onto HTTPS.
> In Warehouse the decision was made to *not* redirect “API” URLs from HTTP
> to HTTPS, but to rather return an error accessing them from HTTP. This is
> because while logged in views have HSTS to ensure HTTPS in the browser (and
> with humans manually entering them into the URL bar regularly they are more
> error prone) APIs which are typically accessed by automated clients with an
> URL configured or hardcoded typically do not respect HSTS, so if you had a
> script that did ``curl http://pypi.python.org/simple/``, it would
> silently get redirects to https and appear to “work”, but you wouldn’t get
> any of the security properties of TLS because an attacker would just
> intercept the request prior to the redirect happening.
>
> Today I’ve backported this changed to the current production deployment of
> PyPI, which means that you can no longer access /simple/ and /packages/
> over HTTP and you will have to directly go to HTTPS. For most people this
> should have no effect, because most tooling should be defaulting to HTTPS
> anyways, however if you’re using a significantly old version of tooling, it
> may still be defaulting to the HTTP url and will now stop functioning.
>
> The recommended remediation is to upgrade your tooling to versions that
> support verified TLS connections and which default to the proper HTTPS URLs.
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Disabling non HTTPS access to APIs on PyPI

[Wes Turner]

On Friday, October 27, 2017, Paul Moore  wrote:

> On 27 October 2017 at 22:22, Alex Domoradov  > wrote:
> > I got it. And what I should do with old system? For e.g. we still use
> ubuntu
> > 12.04. Is there any way to upgrade pip/setuptools?
>
> Well, if Ubuntu aren't offering an upgrade, you can do
>
> pip install -i https://pypi.python.org/simple/ ...
>
> Or install your own copy of pip/setuptools, I guess (get-pip --user,
> see https://pip.pypa.io/en/stable/installing/#installing-with-get-pip-py).
> Paul


Does this upgrade system pip, regardless of the package checksums:

$ sudo pip install -i https://pypi.python.org/simple/ -U pip setuptools


If that doesn't work (or **isn't advisable because `sudo pip` is
dangerous**),
you can also configure the index URL with a config file or an environment
variable:

https://pip.pypa.io/en/latest/user_guide/#config-file lists the paths for
Windows, MacOS, and Linux.

/etc/pip.conf
~/.pip/pip.conf
$VIRTUAL_ENV/pip.conf

```
[global]

index-url = https://pypi.python.org/simple/
```

$ export PIP_INDEX_URL="https://pypi.python.org/simple/";

Setuptools (easy_install) uses ~/.pydistutils.cfg :

```
[easy_install]
index_url = https://pypi.python.org/simple/
```

Buildout uses buildout.cfg and ~/.buildout/default.cfg :

```
[buildout]
index = https://pypi.python.org/simple/
```

"What to do when PyPi goes down" (2010)
https://jacobian.org/writing/when-pypi-goes-down/

This URL also works as the -i/--index-url?

https://pypi.org/simple/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Disabling non HTTPS access to APIs on PyPI

[Alex Domoradov]

I got it. And what I should do with old system? For e.g. we still use
ubuntu 12.04. Is there any way to upgrade pip/setuptools?

On Fri, Oct 27, 2017 at 11:17 PM, Paul Moore  wrote:

> On 27 October 2017 at 21:05, Leonardo Rochael Almeida
>  wrote:
> > By the way, some pages on pypi itself are pointing to http:
> >
> > See, e.g. the "Download URL:" link here:
> >
> > https://pypi.python.org/pypi/google-appengine
>
> That's the project itself doing that, I believe (it's metadata from
> setup.py) so you'd need to report it to them.
>
> Paul
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Disabling non HTTPS access to APIs on PyPI

[Paul Moore]

On 27 October 2017 at 22:22, Alex Domoradov  wrote:
> I got it. And what I should do with old system? For e.g. we still use ubuntu
> 12.04. Is there any way to upgrade pip/setuptools?

Well, if Ubuntu aren't offering an upgrade, you can do

pip install -i https://pypi.python.org/simple/ ...

Or install your own copy of pip/setuptools, I guess (get-pip --user,
see https://pip.pypa.io/en/stable/installing/#installing-with-get-pip-py).
Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Disabling non HTTPS access to APIs on PyPI

[Paul Moore]

On 27 October 2017 at 21:05, Leonardo Rochael Almeida
 wrote:
> By the way, some pages on pypi itself are pointing to http:
>
> See, e.g. the "Download URL:" link here:
>
> https://pypi.python.org/pypi/google-appengine

That's the project itself doing that, I believe (it's metadata from
setup.py) so you'd need to report it to them.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Disabling non HTTPS access to APIs on PyPI

[Leonardo Rochael Almeida]

By the way, some pages on pypi itself are pointing to http:

See, e.g. the "Download URL:" link here:

https://pypi.python.org/pypi/google-appengine

On 27 October 2017 at 17:22, Nathaniel Smith  wrote:

> On Oct 27, 2017 11:49, "Alex Domoradov"  wrote:
>
> RUN pip install --upgrade pip
>
> Try upgrading setuptools here too.
>
> -n
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Disabling non HTTPS access to APIs on PyPI

[Nathaniel Smith]

On Oct 27, 2017 11:49, "Alex Domoradov"  wrote:

RUN pip install --upgrade pip

Try upgrading setuptools here too.

-n
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Disabling non HTTPS access to APIs on PyPI

[Alex Domoradov]

> This will probably (unfortunately) break some things for some people,
> which is worrying.  But it is the right thing to do and good advice in
> general.

Did I miss something, but I can't install some package. I use docker
and get the following

$ cat Dockerfile
FROM ubuntu:16.04

RUN apt-get update
RUN apt-get install -y python-pip

RUN pip install --upgrade pip

$ docker run -it --rm pip bash

root@a633a43c5df6:/# pip -V
pip 9.0.1 from /usr/local/lib/python2.7/dist-packages (python 2.7)

root@a633a43c5df6:/# pip install google-appengine
Collecting google-appengine
  Downloading google-appengine-1.5.1.tar.gz (897kB)
100% || 901kB 745kB/s
Complete output from command python setup.py egg_info:
Downloading
http://pypi.python.org/packages/source/d/distribute/distribute-0.6.14.tar.gz
Traceback (most recent call last):
  File "", line 1, in 
  File "/tmp/pip-build-hd5GWR/google-appengine/setup.py", line 3,
in 
ez_setup.use_setuptools()
  File "/usr/local/lib/python2.7/dist-packages/ez_setup.py", line
145, in use_setuptools
return _do_download(version, download_base, to_dir, download_delay)
  File "/usr/local/lib/python2.7/dist-packages/ez_setup.py", line
124, in _do_download
to_dir, download_delay)
  File "/usr/local/lib/python2.7/dist-packages/ez_setup.py", line
193, in download_setuptools
src = urlopen(url)
  File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 435, in open
response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 473, in error
return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
*urllib2.HTTPError: HTTP Error 403: SSL is required*


Command "python setup.py egg_info" failed with error code 1 in
/tmp/pip-build-hd5GWR/google-appengine/

Do I need to make some additional upgrades? I just thought that pip is
the only one tool that I have to upgrade
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Disabling non HTTPS access to APIs on PyPI

[Alex Walters]

> -Original Message-
> From: Distutils-SIG [mailto:distutils-sig-bounces+tritium-
> list=sdamon@python.org] On Behalf Of Erik Bray
> Sent: Friday, October 27, 2017 11:16 AM
> To: Distutils 
> Subject: Re: [Distutils] Disabling non HTTPS access to APIs on PyPI
> 
> On Thu, Oct 26, 2017 at 5:11 PM, Donald Stufft  wrote:
> > Historically PyPI was only available over either HTTP or unvalidated HTTPS,
> > and over time we’ve been pushing more and more traffic onto HTTPS. In
> > Warehouse the decision was made to *not* redirect “API” URLs from HTTP
> to
> > HTTPS, but to rather return an error accessing them from HTTP. This is
> > because while logged in views have HSTS to ensure HTTPS in the browser
> (and
> > with humans manually entering them into the URL bar regularly they are
> more
> > error prone) APIs which are typically accessed by automated clients with an
> > URL configured or hardcoded typically do not respect HSTS, so if you had a
> > script that did ``curl http://pypi.python.org/simple/``, it would silently
> > get redirects to https and appear to “work”, but you wouldn’t get any of
> the
> > security properties of TLS because an attacker would just intercept the
> > request prior to the redirect happening.
> >
> > Today I’ve backported this changed to the current production deployment
> of
> > PyPI, which means that you can no longer access /simple/ and /packages/
> over
> > HTTP and you will have to directly go to HTTPS. For most people this should
> > have no effect, because most tooling should be defaulting to HTTPS
> anyways,
> > however if you’re using a significantly old version of tooling, it may still
> > be defaulting to the HTTP url and will now stop functioning.
> >
> > The recommended remediation is to upgrade your tooling to versions that
> > support verified TLS connections and which default to the proper HTTPS
> URLs.
> 
> +1
> 
> This will probably (unfortunately) break some things for some people,
> which is worrying.  But it is the right thing to do and good advice in
> general.

Might want to post a message on the front page of pypi.python.org when this 
becomes effective (if it's not there already.)

> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Disabling non HTTPS access to APIs on PyPI

[Erik Bray]

On Thu, Oct 26, 2017 at 5:11 PM, Donald Stufft  wrote:
> Historically PyPI was only available over either HTTP or unvalidated HTTPS,
> and over time we’ve been pushing more and more traffic onto HTTPS. In
> Warehouse the decision was made to *not* redirect “API” URLs from HTTP to
> HTTPS, but to rather return an error accessing them from HTTP. This is
> because while logged in views have HSTS to ensure HTTPS in the browser (and
> with humans manually entering them into the URL bar regularly they are more
> error prone) APIs which are typically accessed by automated clients with an
> URL configured or hardcoded typically do not respect HSTS, so if you had a
> script that did ``curl http://pypi.python.org/simple/``, it would silently
> get redirects to https and appear to “work”, but you wouldn’t get any of the
> security properties of TLS because an attacker would just intercept the
> request prior to the redirect happening.
>
> Today I’ve backported this changed to the current production deployment of
> PyPI, which means that you can no longer access /simple/ and /packages/ over
> HTTP and you will have to directly go to HTTPS. For most people this should
> have no effect, because most tooling should be defaulting to HTTPS anyways,
> however if you’re using a significantly old version of tooling, it may still
> be defaulting to the HTTP url and will now stop functioning.
>
> The recommended remediation is to upgrade your tooling to versions that
> support verified TLS connections and which default to the proper HTTPS URLs.

+1

This will probably (unfortunately) break some things for some people,
which is worrying.  But it is the right thing to do and good advice in
general.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Disabling non HTTPS access to APIs on PyPI

[Nate Coraor]

Heads up to anyone running pypiserver, this breaks the default redirection
from pypiserver to PyPI. There's an open issue[1] to fix it. In the
meantime, you can use the pypiserver option `--fallback-url
https://pypi.python.org/simple/` to work around it.

--nate

[1] https://github.com/pypiserver/pypiserver/issues/179

On Thu, Oct 26, 2017 at 11:11 AM, Donald Stufft  wrote:

> Historically PyPI was only available over either HTTP or unvalidated
> HTTPS, and over time we’ve been pushing more and more traffic onto HTTPS.
> In Warehouse the decision was made to *not* redirect “API” URLs from HTTP
> to HTTPS, but to rather return an error accessing them from HTTP. This is
> because while logged in views have HSTS to ensure HTTPS in the browser (and
> with humans manually entering them into the URL bar regularly they are more
> error prone) APIs which are typically accessed by automated clients with an
> URL configured or hardcoded typically do not respect HSTS, so if you had a
> script that did ``curl http://pypi.python.org/simple/``, it would
> silently get redirects to https and appear to “work”, but you wouldn’t get
> any of the security properties of TLS because an attacker would just
> intercept the request prior to the redirect happening.
>
> Today I’ve backported this changed to the current production deployment of
> PyPI, which means that you can no longer access /simple/ and /packages/
> over HTTP and you will have to directly go to HTTPS. For most people this
> should have no effect, because most tooling should be defaulting to HTTPS
> anyways, however if you’re using a significantly old version of tooling, it
> may still be defaulting to the HTTP url and will now stop functioning.
>
> The recommended remediation is to upgrade your tooling to versions that
> support verified TLS connections and which default to the proper HTTPS URLs.
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Disabling non HTTPS access to APIs on PyPI

[Donald Stufft]

Historically PyPI was only available over either HTTP or unvalidated HTTPS, and 
over time we’ve been pushing more and more traffic onto HTTPS. In Warehouse the 
decision was made to *not* redirect “API” URLs from HTTP to HTTPS, but to 
rather return an error accessing them from HTTP. This is because while logged 
in views have HSTS to ensure HTTPS in the browser (and with humans manually 
entering them into the URL bar regularly they are more error prone) APIs which 
are typically accessed by automated clients with an URL configured or hardcoded 
typically do not respect HSTS, so if you had a script that did ``curl 
http://pypi.python.org/simple/`` , it 
would silently get redirects to https and appear to “work”, but you wouldn’t 
get any of the security properties of TLS because an attacker would just 
intercept the request prior to the redirect happening.

Today I’ve backported this changed to the current production deployment of 
PyPI, which means that you can no longer access /simple/ and /packages/ over 
HTTP and you will have to directly go to HTTPS. For most people this should 
have no effect, because most tooling should be defaulting to HTTPS anyways, 
however if you’re using a significantly old version of tooling, it may still be 
defaulting to the HTTP url and will now stop functioning.

The recommended remediation is to upgrade your tooling to versions that support 
verified TLS connections and which default to the proper HTTPS URLs.___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig