Re: [Distutils] Migrating Hashes from MD5 to SHA256

2013-07-26 Thread Nick Coghlan
On 27 July 2013 06:59, PJ Eby  wrote:
> On Fri, Jul 26, 2013 at 3:14 PM, Donald Stufft  wrote:
>> Does the hashlib backport I added to
>> setuptools 0.9 for Python 2.4 work on 2.3? It's a pure python
>> implementation of hashlib.
>
> Ah, didn't know about that!  I can't imagine what problems there would
> be; not much changed in 2.4 that can't be emulated in 2.3.
>
> Anyway, I'll have a look.  Thanks!

FWIW, I expect the intersection of "running RHEL 4" and "downloading
software directly from PyPI" to be a vanishingly small subset of
humanity - anybody conservative enough to be running a version of RHEL
that old is going to have *very* strict rules about how software gets
onto their production servers and are also likely to be using
something far more recent to talk to the outside world.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig


Re: [Distutils] Migrating Hashes from MD5 to SHA256

2013-07-26 Thread Donald Stufft

On Jul 26, 2013, at 8:55 PM, zooko  wrote:

> On Fri, Jul 26, 2013 at 12:25:36PM -0400, Donald Stufft wrote:
>> PyPI has historically used MD5 in order to verify the downloads. However MD5 
>> is severely broken and is generally regarded as something that should be 
>> migrated away from ASAP. From speaking with a number of cryptographers 
>> they've more or less said that the major reason they believe that MD5 hasn't 
>> had a published pre-image attack is just because it's so broken that most 
>> researchers have moved on to newer hashes.
> 
> Who said that? That contradicts my beliefs.
> 

It's possible I misunderstood the exact implications of what they were
saying. I am not a cryptographer and it was a month or two ago we
spoke. It was stressed to me that PyPI should be moving off of MD5.

I do believe however that we don't know for sure if MD5 is going to be
have a practical pre-image attack tomorrow, or if it will last another 10
years. Given that all security systems are not infallible and are generally
designed so that you have margins of security so there is time to migrate.

The safety margins on MD5 have long since gone so by continuing to use
it we are ignoring prudence (especially at a fairly ideal time where we are
at a transitioning from unverified HTTPS/HTTP to HTTPS so we do not
need to regard backwards compatibility as highly). As far as I am aware
these attacks tend to come all of a sudden and without warning. I would
much rather have already migrated to something that still has it's safety
margins than be caught with our proverbial pants down and need to
scramble *if* an attack is discovered.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig


Re: [Distutils] Migrating Hashes from MD5 to SHA256

2013-07-26 Thread zooko
On Fri, Jul 26, 2013 at 12:25:36PM -0400, Donald Stufft wrote:
> PyPI has historically used MD5 in order to verify the downloads. However MD5 
> is severely broken and is generally regarded as something that should be 
> migrated away from ASAP. From speaking with a number of cryptographers 
> they've more or less said that the major reason they believe that MD5 hasn't 
> had a published pre-image attack is just because it's so broken that most 
> researchers have moved on to newer hashes.

Who said that? That contradicts my beliefs.

Thanks!

Regards,

Zooko
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig


Re: [Distutils] Migrating Hashes from MD5 to SHA256

2013-07-26 Thread Donald Stufft

On Jul 26, 2013, at 4:59 PM, PJ Eby  wrote:

> On Fri, Jul 26, 2013 at 3:14 PM, Donald Stufft  wrote:
>> Does the hashlib backport I added to
>> setuptools 0.9 for Python 2.4 work on 2.3? It's a pure python
>> implementation of hashlib.
> 
> Ah, didn't know about that!  I can't imagine what problems there would
> be; not much changed in 2.4 that can't be emulated in 2.3.
> 
> Anyway, I'll have a look.  Thanks!

Here's the relevant commits in the new setuptools stuff:

https://bitbucket.org/pypa/setuptools/commits/330b628f38c9380c95a818e65fd56812cbc854c4
https://bitbucket.org/pypa/setuptools/commits/b1d4e48beebdcc3cf7cb06fae4c4005a85dfc9bc
https://bitbucket.org/pypa/setuptools/commits/12dd4b89148a225856a060cbee1137fc4cf79736

The implementations are taken from PyPy so they are made
to work on Python 2.7, but they worked just fine on 2.4 after
removing a single b"".

> 
> 
>> I don't have a Python 2.3 available to attempt to test. To be honest I've
>> never even used Python 2.3.
> 
> Heh.  Noob.  ;-)  (j/k)

:) I was still in high school when the last 2.3 was released :/

> 
> 2.3 is basically 2.4 minus decorators, generator expressions, various
> builtins and stdlib features.  Unless you used set types, reversed(),
> or various other odds and ends, I should be able to backport it.

Great!

> 
> 
>> [stuff about RHEL support]
> 
> If there's a 2.4 hashlib backport, that addresses my concerns just
> fine.  If I need to, I'll backport it to setuptools 0.6.  Thanks.


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig


Re: [Distutils] Migrating Hashes from MD5 to SHA256

2013-07-26 Thread PJ Eby
On Fri, Jul 26, 2013 at 3:14 PM, Donald Stufft  wrote:
> Does the hashlib backport I added to
> setuptools 0.9 for Python 2.4 work on 2.3? It's a pure python
> implementation of hashlib.

Ah, didn't know about that!  I can't imagine what problems there would
be; not much changed in 2.4 that can't be emulated in 2.3.

Anyway, I'll have a look.  Thanks!


> I don't have a Python 2.3 available to attempt to test. To be honest I've
> never even used Python 2.3.

Heh.  Noob.  ;-)  (j/k)

2.3 is basically 2.4 minus decorators, generator expressions, various
builtins and stdlib features.  Unless you used set types, reversed(),
or various other odds and ends, I should be able to backport it.


> [stuff about RHEL support]

If there's a 2.4 hashlib backport, that addresses my concerns just
fine.  If I need to, I'll backport it to setuptools 0.6.  Thanks.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig


Re: [Distutils] Shebang lines, /usr/bin/python, and PEP394

2013-07-26 Thread Tres Seaver
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 07/26/2013 03:32 PM, Barry Warsaw wrote:
> If you're installing a script into a virtualenv, it's better to
> rewrite the shebang to use the executable that was used to install 
> it.

Exactly -- the script likely won't run at all outside the environment
where its dependencies are installed.


Tres.
- -- 
===
Tres Seaver  +1 540-429-0999  tsea...@palladion.com
Palladion Software   "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlHy4E0ACgkQ+gerLs4ltQ57XgCg1JsuCk2zxOpJyWNoHv1V/0h/
Y1EAoND2clmyfdHjqVY/3p+PafWXM0Lp
=uzfB
-END PGP SIGNATURE-

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig


Re: [Distutils] Migrating Hashes from MD5 to SHA256

2013-07-26 Thread Donald Stufft

On Jul 26, 2013, at 2:33 PM, PJ Eby  wrote:

> Anyway, this is all somewhat moot since the hashes only matter when
> the download is hosted somewhere besides PyPI, since SSL verification
> is available for the PyPI part.  Even so, I'd suggest that moving to
> SHA1 might be a good intermediate step: it's available on Python 2.3,
> so I could backport the relevant support to the 0.6 branch.  (IIUC,
> Python 2.3 is still the default version for many Linux distros that
> have not reached end-of-life support.)

I think RHEL is the one that will support things the longest. As far as I
can tell Python 2.3 was default in RHEL4 and RHEL5 used Python 2.4

ELS support for RHEL4 ends Feb of 2015, so that's roughly a year and
a half till not even Red Hat supports Python 2.3 anymore that I can tell?
It also appears that support for new installations ended roughly a year
and a half ago.

Many (most?) projects no longer support Python 2.3 on PyPI, and I
seriously doubt that there is a significant number of people who both
want the stability provided by RHEL and is willing to continue using a
release that is this close to being EOL'd while simultaneously wanting
to download things from PyPI. CPython hasn't supported Python 2.3
in years.

Basically If RHEL customers want the security updates they should
bother Red Hat for them, that's part of why they pay for RHEL instead
of going with a free system. I don't think it's appropriate to allow a
handful of people who might still be on a version of python first
released 10 years ago and last released 8 years ago to negatively
impact everyone else.

Note: I don't use RHEL so my understanding of it's life cycle is from
reading their support page.


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig


Re: [Distutils] Shebang lines, /usr/bin/python, and PEP394

2013-07-26 Thread Barry Warsaw
On Jul 26, 2013, at 08:38 PM, Paul Moore wrote:

>There are cases where it's useful and appropriate

Sure, I don't disagree.  Just that I think the general rule should be:

* Use /usr/bin/env in your source tree
* Use /usr/bin/$python when installed

I think those rules cover the majority of cases.  Any automatic shebang
rewriting must be selectable.  I'd argue for default-rewrite with an option to
disable.

-Barry


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig


Re: [Distutils] Shebang lines, /usr/bin/python, and PEP394

2013-07-26 Thread Paul Moore
On 26 July 2013 20:32, Barry Warsaw  wrote:

> I love `#!/usr/bin/env python` *for development* but I really think its a
> bad
> thing to have for installed scripts.  Certainly, for distro installed
> scripts,
> it's (usually) terrible.  I think virtualenv installs are generally in the
> same boat though - if you're installing a script into a virtualenv, it's
> better to rewrite the shebang to use the executable that was used to
> install
> it.
>

There are cases where it's useful and appropriate - the best example "in
the wild" is distil, which uses #!/usr/bin/env python to allow it to run
with "the current Python" specifically, so that it can be used to install
packages whatever Python you're using and without needing multiple copies
of the script installed. I've written similar types of scripts which
benefit in the same way from a /user/bin/env shebang line.

In fact, support for /usr/bin/env was added to the Windows launcher just
recently to provide precisely this functionality for scripts on Windows.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig


Re: [Distutils] Shebang lines, /usr/bin/python, and PEP394

2013-07-26 Thread Barry Warsaw
On Jul 26, 2013, at 09:58 PM, Nick Coghlan wrote:

>Not everybody uses generated script wrappers, though - if there is a
>hardcoded "/usr/bin/env python" or "/usr/bin/python" in a shebang
>line, the Python build tools won't touch it. There's also a whole lot
>of software that isn't packaged at all - it's sitting around in user
>and admin home directories, or maybe in a shared directory on a file
>server or even in a private source control repo.

I actually think installing a script via setuptools should rewrite shebangs
even in the case of /usr/bin/env.

I love `#!/usr/bin/env python` *for development* but I really think its a bad
thing to have for installed scripts.  Certainly, for distro installed scripts,
it's (usually) terrible.  I think virtualenv installs are generally in the
same boat though - if you're installing a script into a virtualenv, it's
better to rewrite the shebang to use the executable that was used to install
it.

-Barry


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig


Re: [Distutils] Migrating Hashes from MD5 to SHA256

2013-07-26 Thread Donald Stufft

On Jul 26, 2013, at 3:24 PM, Christian Heimes  wrote:

> A couple of months ago I suggested a schema that includes MD5, SHA-2
> and file size:
> 
>   file.tar.gz#MD5=1234&SHA-256=abcd&filesize=5023
> 
> That should work for old versions of setuptool and can easily be
> supported in new versions of pip and setuptools.

It won't work for old versions, it explicitly includes the end of line 
terminator and the #.

> 
> A new hash sum scheme must include the possibility to add multiple and
> new hash algorithms. A download tool shall check the hash sum for all
> supported algorithms, too. I also like to see the file size in the
> scheme. It's useful to know the file size in preparation of the
> download. The file size validation mitigates some attack possibilities.

Right now that would break too much. I agree this is where we need to
get too but It'll likely need to wait for the new API in Warehouse.

> 
> Christian
> 
> 


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig


Re: [Distutils] Migrating Hashes from MD5 to SHA256

2013-07-26 Thread Christian Heimes
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Am 26.07.2013 18:25, schrieb Donald Stufft:
> PyPI has historically used MD5 in order to verify the downloads.
> However MD5 is severely broken and is generally regarded as
> something that should be migrated away from ASAP. From speaking
> with a number of cryptographers they've more or less said that the
> major reason they believe that MD5 hasn't had a published pre-image
> attack is just because it's so broken that most researchers have
> moved on to newer hashes.
> 
> Since versions 1.2 pip has supported md5, sha1, and any of the sha2
> family. Additionally it has only supported SSL verification since
> 1.3. This means there is no version of pip which both verifies SSL
> and only allows MD5.
> 
> Since version 0.9 setuptools has supported md5, sha1, and any of
> the sha2 family and it has only supported SSL verification since
> 0.7.
> 
> I propose we switch PyPI from using MD5 to using SHA256. There is
> no security lost from using a hash that pip prior to version 1.2
> doesn't understand as it didn't verify SSL so an attacker could
> simply modify the hashes if they wanted. Additionally there is no
> security list from setuptools versions earlier than 0.7.

A couple of months ago I suggested a schema that includes MD5, SHA-2
and file size:

  file.tar.gz#MD5=1234&SHA-256=abcd&filesize=5023

That should work for old versions of setuptool and can easily be
supported in new versions of pip and setuptools.

A new hash sum scheme must include the possibility to add multiple and
new hash algorithms. A download tool shall check the hash sum for all
supported algorithms, too. I also like to see the file size in the
scheme. It's useful to know the file size in preparation of the
download. The file size validation mitigates some attack possibilities.

Christian

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iQIcBAEBCAAGBQJR8s0HAAoJEMeIxMHUVQ1FO90P+gM6Lx3WZA94Tg8bEco6ckLI
m00Yt+Gwn1HgnF+wSAaxIsThy2C/yltTPqfJiwZGErlzt0tAIdFZhYMbkhzO1//Z
N6i1O+seH5eqMSgUd7K1mgiRIAsKpXH6SEt/U3VzPNm/qVvIV0FFIUTEIx9xXpkD
HYKbup7dkcBkIkhreUpIG4TGEK22/Vcs+G4NjR8UllcqRS4iWrkiKzXuLfnto++t
9fnYfz0uxh2nG3doFGr2gzypLtctrRzAqy28AtlbgGEKaK5E2/hoGrRE8VIBZg+f
SEWKLctTLoOcXHVTaxoAcp+3XzwKZPpGoJzjyLtPFDrH55kZFLA2a25vB51xteLA
A7Kz60eccHe7Io76incJiL+RmorcpUTTp6FRoTCdqDUW2rSmTM1z8tUtenJNAQYG
UnuyRrRbTeQ1JlImdakqXA1X5/qsYLy7kcaf4Xb9SXxIdlEk//0o3tiB4B92oIgF
If5yx65KoKPUCg1pXA/ZawTuH/d1aJQWOjz0eP7Wn+GnEnHxoKIwYMP65xVyNCXU
0afS5lRs7gxtOKlXofWoXO1u7H7EHJQzFbbgdJkSl65mz+hOVMu1w7RQwPb7LzeO
16gnUtvIpXFaab/NCM4UmXuHWLx07jWB4ZJ45zsXuyXa3m4kdt9aMS0oVaSYgA/a
Zq84rJiWc17eItR9iyU5
=ZRue
-END PGP SIGNATURE-
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig


Re: [Distutils] Migrating Hashes from MD5 to SHA256

2013-07-26 Thread Donald Stufft

On Jul 26, 2013, at 2:33 PM, PJ Eby  wrote:

> On Fri, Jul 26, 2013 at 12:25 PM, Donald Stufft  wrote:
>> Additionally there is no security list from setuptools versions earlier than 
>> 0.7.
> 
> Not true, actually.  Setuptools 0.6 dev releases supported SSL
> verification since mid-May, but don't support any hashes besides MD5.
> Anybody who updated their setuptools between then and the release of
> 0.7 would have that version.  Unfortunately, it's hard to tell how
> many people that is, though I could try and dig through my server logs
> to find out.
> 
> There's also another issue with jumping to SHA256: Python prior to 2.5
> didn't support it.
> 
> Which brings up another point: the setuptools 0.6 series is the only
> setuptools available for Python 2.3.  That's one of the reasons it's
> still available for download.  If you want SSL verification on 2.3,
> it's the only thing available.  (Meanwhile, a lot of people are still
> downloading 0.6c11; probably I should package up an 0.6c12 so those
> folks pick it up instead of 0.6c11.)
> 
> Anyway, this is all somewhat moot since the hashes only matter when
> the download is hosted somewhere besides PyPI, since SSL verification
> is available for the PyPI part.  Even so, I'd suggest that moving to
> SHA1 might be a good intermediate step: it's available on Python 2.3,
> so I could backport the relevant support to the 0.6 branch.  (IIUC,
> Python 2.3 is still the default version for many Linux distros that
> have not reached end-of-life support.)


I don't have a Python 2.3 available to attempt to test. To be honest I've 
never even used Python 2.3. Does the hashlib backport I added to
setuptools 0.9 for Python 2.4 work on 2.3? It's a pure python
implementation of hashlib.

Sha1 is better but it already has weaknesses so if at all possible it
would be much preferred and significantly better to switch to sha256.

Setuptools doesn't appear to include the python version in it's user
agent so I can't get any sort of information about Python 2.3 usage.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig


Re: [Distutils] Migrating Hashes from MD5 to SHA256

2013-07-26 Thread Alex Gaynor
PJ Eby  telecommunity.com> writes:

> 
> There's also another issue with jumping to SHA256: Python prior to 2.5
> didn't support it.


I'm not sure this is a particularly relevant concern. Python's prior to 2.5 are
no longer supported by the people who wrote them, or almost any major packages
(SQLAlchemy, Django, pip, virtualenv, the list goes on and on). I'm not sure
why we should expend any effort to support them either.

Alex

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig


Re: [Distutils] Migrating Hashes from MD5 to SHA256

2013-07-26 Thread PJ Eby
On Fri, Jul 26, 2013 at 12:25 PM, Donald Stufft  wrote:
> Additionally there is no security list from setuptools versions earlier than 
> 0.7.

Not true, actually.  Setuptools 0.6 dev releases supported SSL
verification since mid-May, but don't support any hashes besides MD5.
Anybody who updated their setuptools between then and the release of
0.7 would have that version.  Unfortunately, it's hard to tell how
many people that is, though I could try and dig through my server logs
to find out.

There's also another issue with jumping to SHA256: Python prior to 2.5
didn't support it.

Which brings up another point: the setuptools 0.6 series is the only
setuptools available for Python 2.3.  That's one of the reasons it's
still available for download.  If you want SSL verification on 2.3,
it's the only thing available.  (Meanwhile, a lot of people are still
downloading 0.6c11; probably I should package up an 0.6c12 so those
folks pick it up instead of 0.6c11.)

Anyway, this is all somewhat moot since the hashes only matter when
the download is hosted somewhere besides PyPI, since SSL verification
is available for the PyPI part.  Even so, I'd suggest that moving to
SHA1 might be a good intermediate step: it's available on Python 2.3,
so I could backport the relevant support to the 0.6 branch.  (IIUC,
Python 2.3 is still the default version for many Linux distros that
have not reached end-of-life support.)
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig


[Distutils] Migrating Hashes from MD5 to SHA256

2013-07-26 Thread Donald Stufft
PyPI has historically used MD5 in order to verify the downloads. However MD5 is 
severely broken and is generally regarded as something that should be migrated 
away from ASAP. From speaking with a number of cryptographers they've more or 
less said that the major reason they believe that MD5 hasn't had a published 
pre-image attack is just because it's so broken that most researchers have 
moved on to newer hashes.

Since versions 1.2 pip has supported md5, sha1, and any of the sha2 family. 
Additionally it has only supported SSL verification since 1.3. This means there 
is no version of pip which both verifies SSL and only allows MD5.

Since version 0.9 setuptools has supported md5, sha1, and any of the sha2 
family and it has only supported SSL verification since 0.7.

I propose we switch PyPI from using MD5 to using SHA256. There is no security 
lost from using a hash that pip prior to version 1.2 doesn't understand as it 
didn't verify SSL so an attacker could simply modify the hashes if they wanted. 
Additionally there is no security list from setuptools versions earlier than 
0.7.

However setuptools versions 0.7-0.8 will lose their hashes. I believe this is a 
ok thing to happen because the uptake on 0.7-0.8 is fairly low. Most people 
will use the setuptools bundled with virtualenv which has only ever bundled 0.6 
or 0.9.

So essentially moving from MD5 to SHA256 will only negatively affect the 
security of a handful of users while positively impacting the security of 
everyone else.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig


Re: [Distutils] Shebang lines, /usr/bin/python, and PEP394

2013-07-26 Thread Nick Coghlan
On 26 July 2013 21:31, Alex Burke  wrote:
> On 26 July 2013 00:37, Philip Jenvey  wrote:
>> -BEGIN PGP SIGNED MESSAGE-
>> Hash: SHA512
>>
>>
>> On Jul 25, 2013, at 9:04 AM, Toshio Kuratomi wrote:
>>
>>> Over on python-dev we're talking about Linux Distributions switching from
>>> python2 to python3, what steps they need to take and in what order.  One of
>>> the things that's come up [1]_ is that a very early step in the process is 
>>> making
>>> sure that shebang lines use /usr/bin/python2 or /usr/bin/python3 as noted in
>>> PEP394 [2]_.  Faced with the prospect of patching a whole bunch of scripts
>>> in the distribution, I'm wondering what distutils, distlib, setuptools, etc
>>> do with shebang lines.
>>> * Do they rewrite shebang lines?
>>
>> distutils, distlib and setuptools all do.
>
> Hi,
>
> It was interesting that discussion came up on python-dev but I admit
> to being surprised by the suggestion the shebang lines may need to be
> rewritten in end user code.
>
> This may be a callous over-simplification but if #!python is rewritten
> by the python packaging infrastructure, would it not be changed for
> python2/python3 as appropriate at installation time? Thus a python 2
> package (whatever it is named) would be generated by calling a python2
> executable + setuptools while the same is true for v3 except using
> python3. The result is then packaged by rpm/dpkg.
>
> Keen to understand why it can't work this way if that's the case.

It actually occurs to me that the following example (showing how
symlinks affect "sys.executable") illustrates both the problem and the
solution for cases where users are relying on generated script
wrappers or the shebang line rewriting in wheel:

$ ln -s /usr/bin/python foo
$ ./foo -c "import sys; print sys.executable"
/home/ncoghlan/foo

So long as the distro build systems are updated to invoke setup.py
through an appropriately versioned symlink (rather than through
/usr/bin/python), setuptools et al should all automatically do the
right thing when generating script wrappers.

Not everybody uses generated script wrappers, though - if there is a
hardcoded "/usr/bin/env python" or "/usr/bin/python" in a shebang
line, the Python build tools won't touch it. There's also a whole lot
of software that isn't packaged at all - it's sitting around in user
and admin home directories, or maybe in a shared directory on a file
server or even in a private source control repo.

Published software is actually the vanishingly small tip of a very
large iceberg, especially for languages like Python that handle
scripting use cases fairly well and are thus popular for personal
automation tasks amongst developers and system administrators.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig


Re: [Distutils] Shebang lines, /usr/bin/python, and PEP394

2013-07-26 Thread Alex Burke
On 26 July 2013 00:37, Philip Jenvey  wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA512
>
>
> On Jul 25, 2013, at 9:04 AM, Toshio Kuratomi wrote:
>
>> Over on python-dev we're talking about Linux Distributions switching from
>> python2 to python3, what steps they need to take and in what order.  One of
>> the things that's come up [1]_ is that a very early step in the process is 
>> making
>> sure that shebang lines use /usr/bin/python2 or /usr/bin/python3 as noted in
>> PEP394 [2]_.  Faced with the prospect of patching a whole bunch of scripts
>> in the distribution, I'm wondering what distutils, distlib, setuptools, etc
>> do with shebang lines.
>> * Do they rewrite shebang lines?
>
> distutils, distlib and setuptools all do.
>

Hi,

It was interesting that discussion came up on python-dev but I admit
to being surprised by the suggestion the shebang lines may need to be
rewritten in end user code.

This may be a callous over-simplification but if #!python is rewritten
by the python packaging infrastructure, would it not be changed for
python2/python3 as appropriate at installation time? Thus a python 2
package (whatever it is named) would be generated by calling a python2
executable + setuptools while the same is true for v3 except using
python3. The result is then packaged by rpm/dpkg.

Keen to understand why it can't work this way if that's the case.

Thanks, Alex J Burke.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig