Re: [Distutils] What to do about the PyPI mirrors

[holger krekel]

On Mon, Aug 05, 2013 at 23:31 -0700, Noah Kantrowitz wrote:
> On Aug 5, 2013, at 11:11 PM, Christian Theune  wrote:
> 
> > Two more things:
> > 
> > why is the CDN not suffering from the security problems you describe for 
> > the mirrors?
> > 
> > a) Fastly seems to be the one owning the certificate for pypi.python.org. 
> > What?!?
> 
> They have a delegated SAN for it, which digicert (the CA) authorizes with the 
> domain contact (the board in this case).
> 
> > b) What does stop Fastly from introducing incorrect/rogue code in package 
> > downloads?
> 
> Basically this one boils down to personal trust from me to the Fastly team 
> combined with the other companies using them being very reputable. At the end 
> of the day, there is not currently any cryptographic mechanism preventing 
> Fastly from doing bad things.

The problem is not so much trusting individuals but that the companies
in question are based in the US.  If its government wants to temporarily
serve backdoored packages to select regions, they could silently force Fastly
to do it.  I guess the only way around this is to work with pypi- and
eventually author/maintainer-signatures and verification.

best,
holger
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What to do about the PyPI mirrors

[Donald Stufft]

On Aug 6, 2013, at 2:49 AM, Noah Kantrowitz  wrote:

> I am also hoping that pypi-mirrors.org will continue to operate as a 
> community project (side note, I would be happy to assist with hosting for it 
> if Ken reads this list and if thats a concern of his) and that the mirror 
> operators can develop policies for things like this.

Additionally if anyone else wants to maintain a list like this I think it would 
be more than appropriate to link to it in addition to pypi-mirrors.org on the 
page about mirroring.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What to do about the PyPI mirrors

[Noah Kantrowitz]

On Aug 5, 2013, at 11:09 PM, Christian Theune  wrote:

> Hi,
> 
> looks like I'm late to the party to figure out that I'm going to be hurt 
> again.
> 
> I'd like to suggest explicitly considering what is going to break due to this 
> and how much work you are forcefully inflicting on others. My whole 
> experience around the packaging (distribute/setuptools) and mirroring/CDN in 
> this year estimates cost for my company somewhere between 10k-20k EUR just 
> for keeping up with the breakage those changes incure. It might be that we're 
> wonderfully stupid (..enough to contribute) and all of this causes no 
> headaches for anybody else …. Overall, guessing that the packaging 
> infrastructure is used by probably multiple thousands of companies then I'd 
> expect that at least 100 of them might be experiencing problems like us. 
> Juggling arbritrary numbers I can see that we're inflicting around a million 
> EURs of cost that nobody asked for. 
> 
> More specific statements below.
> 
> On 2013-08-04 22:25:01 +, Donald Stufft said:
> 
> Here's my PEP for Deprecating and  Removing the Official Public Mirrors
> 
> It's source is at: 
> https://github.com/dstufft/peps/blob/master/mirror-removal.rst
> 
> Abstract
> ===
> This PEP provides a path to deprecate and ultimately remove the official
> public mirroring infrastructure for `PyPI`_. It does not propose the removal
> of mirroring support in general.
> 
> -1 - maybe I don't have the right to speak up on CDN usage, but personally I 
> feel it's a bad idea to delegate overall PyPI availability exclusively to a 
> commercial third party. It's OK for me that we're using them to improve PyPI 
> availability, but completely putting our faith in their hands, doesn't sound 
> right to me.
> 
> Rationale
> 
> The PyPI mirroring infrastructure (defined in `PEP381`_) provides a means to
> mirror the content of PyPI used by the automatic installers. It also provides
> a method for autodiscovery of mirrors and a consistent naming scheme.
> 
> There are a number of problems with the official public mirrors:
> 
> * They give control over a \*.python.org domain name to a third party,
>   allowing that third party to set or read cookies on the pypi.python.org and
>   python.org domain name.
> 
> Agreed, that's a problem.
> 
> * The use of a sub domain of pypi.python.org means that the mirror operators
>   will never be able to get a certificate of their own, and giving them
>   one for a python.org domain name is unlikely to happen.
> 
> Agreed.
> 
> * They are often out of date, most often by several hours to a few days, but
>   regularly several days and even months.
> 
> That's something that the mirroring infrastructure should have been 
> constructed for. I completely agree that the way the mirroring was 
> established was way sub-optimal. I think we can do better.
> 
> * With the introduction of the CDN on PyPI the public mirroring infrastructure
>   is not as important as it once was as the CDN is also a globally distributed
>   network of servers which will function even if PyPI is down.
> 
> Well, now we have one breakage point more which keeps annoying me. This 
> argument is not completely true. They may be getting better over time but we 
> have invested heavily to accomodate the breakage - that needs to be balanced 
> with some benefit in the near future.

To be clear, the CDN and other server-side improvements are not a hard-HA 
replacement like a local company mirror. You are exactly the use case that can 
and should be using a mirror for your own use. We are doing _nothing_ that 
disrupts this use case and will support is exactly as before.

> 
> * Although there is provisions in place for it, there is currently no known
>   installer which uses the authenticity checks discussed in `PEP381`_ which
>   means that any download from a mirror is subject to attack by a malicious
>   mirror operator, but further more due to the lack of TLS it also means that
>   any download from a mirror is also subject to a MITM attack.
> 
> Again, I think that was a mistake during the introduction of the mirroring 
> infrastructure: too few people, too confusing PEP. 
> 
> * They have only ever been implemented by one installer (pip), and its
>   implementation, besides being insecure, has serious issues with performance
>   and is slated for removal with it's next release (1.5).
> 
> Only if you consider the mirror auto-discovery protocol. I'm not sure whether 
> using DNS was such a smart move. A simple HTTP request to find mirrors would 
> have been nice. I think we can still do that.
> 
> Also, not everyone wants or needs auto-detection the way that the protocol 
> describes it. I personally just hand-pick a mirror (my own, hah) and keep 
> using that. 
> 
> We are also thinking about providing system-level default configuration to 
> hint tools like PIP and setuptools to a different default index that is 
> closer from a network perspective. From a custom

Re: [Distutils] What to do about the PyPI mirrors

[Donald Stufft]

On Aug 6, 2013, at 2:31 AM, Noah Kantrowitz  wrote:

> 
> On Aug 5, 2013, at 11:11 PM, Christian Theune  wrote:
> 
>> Two more things:
>> 
>> why is the CDN not suffering from the security problems you describe for the 
>> mirrors?
>> 
>> a) Fastly seems to be the one owning the certificate for pypi.python.org. 
>> What?!?
> 
> They have a delegated SAN for it, which digicert (the CA) authorizes with the 
> domain contact (the board in this case).
> 
>> b) What does stop Fastly from introducing incorrect/rogue code in package 
>> downloads?
> 
> Basically this one boils down to personal trust from me to the Fastly team 
> combined with the other companies using them being very reputable. At the end 
> of the day, there is not currently any cryptographic mechanism preventing 
> Fastly from doing bad things.

To further expand on this answer, you need to trust *someone*. If we cut out 
Fastly here you could say, well what prevents Dyn Inc (DNS host) from simply 
redirecting the DNS to a different host? What prevents OSUOL from simply 
accessing the machines stored there and doing bad things (™). Hell, how many 
people here know the entire infrastructure team and has personally decided to 
trust them?

At the end of the day you need to pick and choose who you trust. Right now 
we're working on narrowing down the number of people trusted. The Python 
Infrastructure has decided it is willing to extend trust to Fastly to cover 
PyPI the same as it was willing to extend trust to Dyn, and OSOUL, and even the 
members of the Infra team.

Now that being said narrowing the list of people you need to trust is an 
ongoing goal, and one that isn't going to stop with limiting the number of 
places able to publish at varying python.org domain names who don't need to be. 
We're not in a particularly well off position yet but we are getting better all 
the time.

> 
> --Noah
> 
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> http://mail.python.org/mailman/listinfo/distutils-sig


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What to do about the PyPI mirrors

[Noah Kantrowitz]

On Aug 5, 2013, at 11:11 PM, Christian Theune  wrote:

> Two more things:
> 
> why is the CDN not suffering from the security problems you describe for the 
> mirrors?
> 
> a) Fastly seems to be the one owning the certificate for pypi.python.org. 
> What?!?

They have a delegated SAN for it, which digicert (the CA) authorizes with the 
domain contact (the board in this case).

> b) What does stop Fastly from introducing incorrect/rogue code in package 
> downloads?

Basically this one boils down to personal trust from me to the Fastly team 
combined with the other companies using them being very reputable. At the end 
of the day, there is not currently any cryptographic mechanism preventing 
Fastly from doing bad things.

--Noah



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What to do about the PyPI mirrors

[Christian Theune]

Two more things:

why is the CDN not suffering from the security problems you describe 
for the mirrors?


a) Fastly seems to be the one owning the certificate for 
pypi.python.org. What?!?


b) What does stop Fastly from introducing incorrect/rogue code in 
package downloads?


Christian


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What to do about the PyPI mirrors

[Christian Theune]

Hi,

looks like I'm late to the party to figure out that I'm going to be hurt again.

I'd like to suggest explicitly considering what is going to break due 
to this and how much work you are forcefully inflicting on others. My 
whole experience around the packaging (distribute/setuptools) and 
mirroring/CDN in this year estimates cost for my company somewhere 
between 10k-20k EUR just for keeping up with the breakage those changes 
incure. It might be that we're wonderfully stupid (..enough to 
contribute) and all of this causes no headaches for anybody else …. 
Overall, guessing that the packaging infrastructure is used by probably 
multiple thousands of companies then I'd expect that at least 100 of 
them might be experiencing problems like us. Juggling arbritrary 
numbers I can see that we're inflicting around a million EURs of cost 
that nobody asked for. 


More specific statements below.

On 2013-08-04 22:25:01 +, Donald Stufft said:


Here's my PEP for Deprecating and  Removing the Official Public Mirrors

It's source is at: 
https://github.com/dstufft/peps/blob/master/mirror-removal.rst


Abstract
===
This PEP provides a path to deprecate and ultimately remove the official
public mirroring infrastructure for `PyPI`_. It does not propose the removal
of mirroring support in general.


-1 - maybe I don't have the right to speak up on CDN usage, but 
personally I feel it's a bad idea to delegate overall PyPI availability 
exclusively to a commercial third party. It's OK for me that we're 
using them to improve PyPI availability, but completely putting our 
faith in their hands, doesn't sound right to me.



Rationale

The PyPI mirroring infrastructure (defined in `PEP381`_) provides a means to
mirror the content of PyPI used by the automatic installers. It also provides
a method for autodiscovery of mirrors and a consistent naming scheme.

There are a number of problems with the official public mirrors:

* They give control over a \*.python.org domain name to a third party,
  allowing that third party to set or read cookies on the pypi.python.org and
  python.org domain name.


Agreed, that's a problem.


* The use of a sub domain of pypi.python.org means that the mirror operators
  will never be able to get a certificate of their own, and giving them
  one for a python.org domain name is unlikely to happen.


Agreed.


* They are often out of date, most often by several hours to a few days, but
  regularly several days and even months.


That's something that the mirroring infrastructure should have been 
constructed for. I completely agree that the way the mirroring was 
established was way sub-optimal. I think we can do better.



* With the introduction of the CDN on PyPI the public mirroring infrastructure
  is not as important as it once was as the CDN is also a globally distributed
  network of servers which will function even if PyPI is down.


Well, now we have one breakage point more which keeps annoying me. This 
argument is not completely true. They may be getting better over time 
but we have invested heavily to accomodate the breakage - that needs to 
be balanced with some benefit in the near future.



* Although there is provisions in place for it, there is currently no known
  installer which uses the authenticity checks discussed in `PEP381`_ which
  means that any download from a mirror is subject to attack by a malicious
  mirror operator, but further more due to the lack of TLS it also means that
  any download from a mirror is also subject to a MITM attack.


Again, I think that was a mistake during the introduction of the 
mirroring infrastructure: too few people, too confusing PEP. 


* They have only ever been implemented by one installer (pip), and its
  implementation, besides being insecure, has serious issues with performance
  and is slated for removal with it's next release (1.5).


Only if you consider the mirror auto-discovery protocol. I'm not sure 
whether using DNS was such a smart move. A simple HTTP request to find 
mirrors would have been nice. I think we can still do that.


Also, not everyone wants or needs auto-detection the way that the 
protocol describes it. I personally just hand-pick a mirror (my own, 
hah) and keep using that. 

We are also thinking about providing system-level default configuration 
to hint tools like PIP and setuptools to a different default index that 
is closer from a network perspective. From a customer perspective this 
should be "PyPI".


I'd like to avoid breakage. Again, if you don't let me choose where to 
spend my time, I'd rather invest the time I need for cleaning up the 
breakage into something constructive.


The indices are in active use. f.pypi.python.org is seeing between 
150-300GB of traffic per month, the patterns widely ranging over the 
last month. This is traffic that is not used internally from gocept.



Due to the number of issues, some of them very serious, and the CDN which more
or less provid

Re: [Distutils] What to do about the PyPI mirrors

[Nick Coghlan]

On 5 August 2013 08:25, Donald Stufft  wrote:
> Here's my PEP for Deprecating and  Removing the Official Public Mirrors
>
> It's source is at: 
> https://github.com/dstufft/peps/blob/master/mirror-removal.rst

Donald's proposal is now PEP 449: http://www.python.org/dev/peps/pep-0449/

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] New PyPI mirror in China

[Nick Coghlan]

On 6 August 2013 14:24, Hexchain Tong  wrote:
> On 08/05/2013 02:29 PM, Donald Stufft wrote:
>> Just to give you a response. There are currently discussions going on
>> about deprecating the official public mirroring infrastructure.
>
> Thank you! I didn't even realized that :-P

Those discussions only turned into a concrete plan in the last couple
of days, so it's not surprising you hadn't heard about it :)

The details are up at http://www.python.org/dev/peps/pep-0449/

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] setuptools' install_script overwrites symlinked files rather than replacing symlinks

[Nick Coghlan]

On 4 August 2013 21:12, Michał Górny  wrote:
> I'm willing to write a patch. Please just tell me which solution would
> you prefer.

The standard library has switched to atomic replacement for writing
.pyc files, which seems like the appropriate solution for script
writing as well (note that os.rename isn't atomic on Windows - on
3.3+, os.replace provides atomic renaming on all supported platforms)

However, a "not a regular file or symlink" sanity check similar to the
one now performed by pycompile (see http://bugs.python.org/issue17222)
may also be appropriate here.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] setuptools' install_script overwrites symlinked files rather than replacing symlinks

[Michał Górny]

Hello,

We've got a pretty specific setup where various Python scripts
in /usr/bin are symlinked to a common wrapper. For example:

  /usr/bin/easy_install -> python-exec

As a result, calling 'setup.py install' in a package that installs
setuptools' legacy script wrappers (e.g. setuptools itself) rewrites
python-exec (which -- shortly saying -- breaks a lot) rather than
replacing the 'easy_install' symlink.

I believe that the core issue is in command/easy_install.py. There,
the write_script() command writes directly onto the destination file
with no precautions:

if not self.dry_run:
ensure_directory(target)
f = open(target,"w"+mode)
f.write(contents)
f.close()

Therefore, if 'target' is a symlink, the symlink target is opened
rather than the expected path.

distutils itself is free of this issue since it removes the target
before writing (distutils/file_util.py):

if os.path.exists(dst):
try:
os.unlink(dst)
except os.error, (errno, errstr):
raise DistutilsFileError(
  "could not delete '%s': %s" % (dst, errstr))

This suffers a race condition but is better than nothing. Other tools 
usually create a temporary file in the target directory and use
rename() to atomically replace the target.

I'm willing to write a patch. Please just tell me which solution would
you prefer.

(please keep python@g.o in CC when replying)

-- 
Best regards,
Michał Górny


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] New PyPI mirror in China

[Hexchain Tong]

On 08/05/2013 02:29 PM, Donald Stufft wrote:
> Just to give you a response. There are currently discussions going on
> about deprecating the official public mirroring infrastructure.

Thank you! I didn't even realized that :-P
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Who administers bugs.python.org, and can we get a notice on the setuptools tracker?

[PJ Eby]

On Mon, Aug 5, 2013 at 11:10 AM, Nick Coghlan  wrote:
> I filed the request on the metatracker:
> http://psf.upfronthosting.co.za/roundup/meta/issue522

Thanks!  That should keep me from having to keep telling people their
princess is in another castle.  ;-)
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Who administers bugs.python.org, and can we get a notice on the setuptools tracker?

[Nick Coghlan]

I filed the request on the metatracker:
http://psf.upfronthosting.co.za/roundup/meta/issue522
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Who administers bugs.python.org, and can we get a notice on the setuptools tracker?

[PJ Eby]

Hi.  Not sure who this should go to, but it would be really good if we
could get a prominent notice on the old setuptools tracker (at
bugs.python.org), specifically on the issue creation screen, to inform
people that this tracker is only for setuptools 0.6, and that issues
for later versions should go to
https://bitbucket.org/pypa/setuptools/issues instead (with a link, of
course).

Right now, what's happening is that, despite all the other prominent
links to the correct tracker, there are still people going to the old
tracker and posting issues for the newer versions of setuptools.  This
means they then have to wait for me to tell them they're in the wrong
place, and then resubmit to the correct tracker.

Setuptools 0.6 is still supported, so closing the tracker entirely
isn't appropriate just yet, but putting up a prominent notice on the
issue submission screen saying, "If you're reporting an issue for
setuptools 0.7 or higher, please use".

Thanks in advance!
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] [issue157] setuptools installation instructions fail on OS X: no wget

[Ned Deily]

New submission from Ned Deily:

The current installation instructions for "Unix-based Systems including Mac OS 
X" do not work on vanilla OS X systems because OS X does not ship with "wget".  
It does, however, ship with "curl".  (The Distribute web page had been updated 
a long time ago to use "curl" instead of "wget"; I guess that never happened 
for the "setuptools" page.)

https://pypi.python.org/pypi/setuptools/0.9.8#unix-based-systems-including-mac-os-x

--
messages: 743
nosy: nad
priority: bug
status: unread
title: setuptools installation instructions fail on OS X: no wget

___
Setuptools tracker 

___
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig