Re: [Python-Dev] issue 9807 - a glitch in coexisting builds of different types

2010-10-02 Thread Martin v. Löwis 
>>> With my branch, you'll end up with this in /tmp/python:
>>>
>>> bin/python3.2m   - the normal build binary
>>> bin/python3.2dmu - the wide+pydebug build binary
>>> bin/python3.2m-config
>>> bin/python3.2dmu-config
>>
>> Do users really want to see such idiosyncratic suffixes?
> 
> Ordinary users won't be building Python from source. Developers won't
> care so long as we clearly document the sundry suffixes and describe
> them in the README (or in a PEP, with a pointer from the README).

I think this is not true. Developers *will* care, and they will cry
foul very loudly, asking what nonsense this is. Antoine is proof of
that: he is a developer, and he understands the motivation well,
but it still goes against his notions of beauty (channeling him here).

> Having multiple parallel "altinstall" installations be genuinely
> non-interfering out of the box certainly seems like a desirable
> feature to me.

I think this should not use automatically generated suffixes, though.
Perhaps I want an altinstall that is in some kind restrict?
Or one where user "peter" has write access into site-packages?

I could accept that a suffix is parameter to configure (or some such),
and then gets used throughout. By default, Python will not add a suffix.
However, I still wonder why people couldn't just install Python in a
different prefix if they want separate installations.

Regards,
Martin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] issue 9807 - a glitch in coexisting builds of different types

2010-10-02 Thread Georg Brandl 
Am 02.10.2010 00:06, schrieb Barry Warsaw:

> Let's say I build the foo module, which has an _foo extension for both the
> 3.2m and 3.2dmu builds.  Everything gets installed correctly, and you'll see:
> 
> lib/python3.2/site-packages/foo-...egg/_foo.cpython-32m.so
> lib/python3.2/site-packages/foo-...egg/_foo.cpython-32dmu.so
> 
> If you now run bin/python3.2dmu and try to import _foo, the right thing
> happens (foreshadowing) assuming that the directory listing for the foo.egg is
> alphabetical.
> 
> But what happens if you run bin/python3.2m and import _foo?  Yeah, well, even
> though there's a _foo.cpython-32m.so sitting right there waiting for you to
> load it, what actually happens is that Python tries to dynload
> _foo.cpython-32dmu.so and crashes miserably.  Why did it do that?
> 
> The reason is that the import.c logic that uses the struct filedescr tables
> built from _PyImport_DynLoadFiletab are just not smart enough to handle this
> case.  All it knows about are suffix, and for backwards compatibility, we have
> dynload_shlib.c matching both .SOABI.so *and* bare .so.  So when it's
> searching the directories for .cpython-32m.so files, it finds the
> ..cpython-32dmu.so first based on the bare .so match.

I don't understand -- wouldn't "foo.sometag.so" (where sometag is not SOABI)
only be found a match for a suffix of ".so" if the module name requested is
"foo.sometag"?  (And if not, isn't implementing that the easiest solution?)

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] issue 9807 - a glitch in coexisting builds of different types

2010-10-02 Thread Antoine Pitrou 
On Sat, 02 Oct 2010 09:44:16 +0200
"Martin v. Löwis"  wrote:
> 
> I could accept that a suffix is parameter to configure (or some such),
> and then gets used throughout. By default, Python will not add a suffix.
> However, I still wonder why people couldn't just install Python in a
> different prefix if they want separate installations.

Indeed. It might make sense to inquire what other packages do here.
Apparently they don't do anything special, and they let users install
in different prefixes if necessary.

Besides, mingling different installations together makes uninstalling
much more difficult.

Regards

Antoine.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] the "PPC Leopard" buildbot is back online

2010-10-02 Thread Bill Janssen 
I've finished upgrading the PPC Leopard builder to a dual 2 GHz G5
machine.  Tests should run a bit faster now that the eMac is out of the
loop :-).  I'm looking for a faster machine for the PPC Tiger buildbot.

Bill
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] sad state of OS X Python testing...

2010-10-02 Thread Bill Janssen 
Folks, I was looking at the buildbots again.  Do you realize that we
have no OS X Snow Leopard buildbot?  No Intel Leopard buildbot?  Well
over half of Mac users are using Snow Leopard, and we're not testing on
that platform.  In fact, the only Intel OS X machine we're testing on
is a Core Duo, not a Core 2 Duo, and that's running Tiger.

Surely someone could volunteer an old Intel Mac to run some tests?
(Actually, two someones -- we'd need separate machines for Leopard and
Snow Leopard.)  I'd be happy to stick it in a server room at PARC and
keep an eye on it.  (Right now I've got a rack full of old eMacs and a
G5 running PPC buildbots.)  Perhaps we could get Apple to contribute
some "seconds"?  I believe donations to the PSF are deductible.

Bill
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Rietveld integration into Roundup

2010-10-02 Thread Martin v. Löwis 
Following up to the recent thread, I have now integrated Rietveld
into Roundup. This is a rough draft still, and highly experimental.
Please try it out, but expect that it may be necessary to discard
all data (including comments) to start over (of course, I'll try
to avoid having to do so).

The Rietveld integration currently works this way:
- the installation lives at http://bugs.python.org/review/
- for each roundup user, a rietveld user is created;
  log into roundup to get access to rietveld
- for each roundup issue, a rietveld issue is created with the
  same issue id. Please don't create additional Rietveld issues.
- regularly (currently every hour), each patch is considered.
  Patches are skipped if:
  * they have been added to Rietveld already
  * they have no clear base version (i.e. they don't originate
from svn diff)
  * they belong to no or a closed issue
  * they apply to a patch that is not currently supported
(only trunk, 2.6, 2.7, 3.1, py3k are currently supported)
- for each such patch, a patchset is created
- if that is successful, a "review" link is added to roundup

Feel free to discuss this here; bug reports and feature requests
should go to the meta tracker. Contributions are welcome;
I won't be able to work on this much for the next four days.

Regards,
Martin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] sad state of OS X Python testing...

2010-10-02 Thread Martin v. Löwis 
> Surely someone could volunteer an old Intel Mac to run some tests?
> (Actually, two someones -- we'd need separate machines for Leopard and
> Snow Leopard.)  I'd be happy to stick it in a server room at PARC and
> keep an eye on it.  (Right now I've got a rack full of old eMacs and a
> G5 running PPC buildbots.)  Perhaps we could get Apple to contribute
> some "seconds"?  I believe donations to the PSF are deductible.

The issue isn't really to get the hardware. The issue is to find
somebody to volunteer running a build slave.

Regards,
Martin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Catching exceptions and convert them to warnings in C level code (was: Re: Pronouncement needed in issue9675)

2010-10-02 Thread Jesus Cea 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 28/09/10 18:00, Antoine Pitrou wrote:
> By "fails" you mean "crashes the interpreter".
> While the deprecation warning can be discussed, bsddb shouldn't crash
> when PyCObject_FromVoidPtr() returns NULL, but instead bail out
> cleanly (or perhaps even ignore the error and simply display a warning
> that the C API won't be available).

I plan to deploy two patches for this issue. One, bsddb surviving the
warning->error flag. The other, changing the deprecation warning in
"CObject" to a py3k warning.

While doing this, I think would be useful if bsddb could intercept the
exception and provide sensible feedback without raising an exception at
import time. I have checked
. "PyErr_Fetch" is a bit
overkill, it seems.

My idea is to convert the CObject exception in bsddb to print a warning
like "CObject creation failed. The bsddb.cAPI will be not available.
Reason is: ".

Is there a "canonical way" of doing this?. Sorry if the answer is trivial...

- -- 
Jesus Cea Avion _/_/  _/_/_/_/_/_/
[email protected] - http://www.jcea.es/ _/_/_/_/  _/_/_/_/  _/_/
jabber / xmpp:[email protected] _/_/_/_/  _/_/_/_/_/
.  _/_/  _/_/_/_/  _/_/  _/_/
"Things are not so easy"  _/_/  _/_/_/_/  _/_/_/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/_/_/_/  _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQCVAwUBTKeSpJlgi5GaxT1NAQIMkgP9FCFewuhLHzExwi+aMSuevb56aoFOrBSE
q/XKN+9JxE59n2BwxgaJtUBrBdUraLe17SswgVgjxzReHKI+eNgC1fTwvDV+KqGa
9AMvlNGBfNCLP/TcSRzIGc9w0MsZaS8MvBKJWw+4hINclcp4a/AgxkCxc9sbm/dM
ATijPsG0RPY=
=qAIG
-END PGP SIGNATURE-
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] sad state of OS X Python testing...

2010-10-02 Thread Alan McIntyre 
I have an Intel Core 2 Duo Mac that I was going to convert into a home
file server in the next couple of weeks, and I'd be glad to set it up
as a build slave as well.  I don't remember what version of OS X it
has on it (it's still packed up in the box), but it certainly won't be
the latest one.

On Sat, Oct 2, 2010 at 1:08 PM, "Martin v. Löwis"  wrote:
>> Surely someone could volunteer an old Intel Mac to run some tests?
>> (Actually, two someones -- we'd need separate machines for Leopard and
>> Snow Leopard.)  I'd be happy to stick it in a server room at PARC and
>> keep an eye on it.  (Right now I've got a rack full of old eMacs and a
>> G5 running PPC buildbots.)  Perhaps we could get Apple to contribute
>> some "seconds"?  I believe donations to the PSF are deductible.
>
> The issue isn't really to get the hardware. The issue is to find
> somebody to volunteer running a build slave.
>
> Regards,
> Martin
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/alan.mcintyre%40gmail.com
>
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] sad state of OS X Python testing...

2010-10-02 Thread Alan McIntyre 
On Sat, Oct 2, 2010 at 1:16 PM, Alan McIntyre  wrote:
> I have an Intel Core 2 Duo Mac that I was going to convert into a home
> file server in the next couple of weeks, and I'd be glad to set it up
> as a build slave as well.  I don't remember what version of OS X it
> has on it (it's still packed up in the box), but it certainly won't be
> the latest one.

I just looked and the install disc says it's 10.4.7.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Rietveld integration into Roundup

2010-10-02 Thread Giampaolo Rodolà 
Thanks for this. It looks very nice.


2010/10/2 "Martin v. Löwis" :
> Following up to the recent thread, I have now integrated Rietveld
> into Roundup. This is a rough draft still, and highly experimental.
> Please try it out, but expect that it may be necessary to discard
> all data (including comments) to start over (of course, I'll try
> to avoid having to do so).
>
> The Rietveld integration currently works this way:
> - the installation lives at http://bugs.python.org/review/
> - for each roundup user, a rietveld user is created;
>  log into roundup to get access to rietveld
> - for each roundup issue, a rietveld issue is created with the
>  same issue id. Please don't create additional Rietveld issues.
> - regularly (currently every hour), each patch is considered.
>  Patches are skipped if:
>  * they have been added to Rietveld already
>  * they have no clear base version (i.e. they don't originate
>    from svn diff)
>  * they belong to no or a closed issue
>  * they apply to a patch that is not currently supported
>    (only trunk, 2.6, 2.7, 3.1, py3k are currently supported)
> - for each such patch, a patchset is created
> - if that is successful, a "review" link is added to roundup
>
> Feel free to discuss this here; bug reports and feature requests
> should go to the meta tracker. Contributions are welcome;
> I won't be able to work on this much for the next four days.
>
> Regards,
> Martin
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/g.rodola%40gmail.com
>
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Catching exceptions and convert them to warnings in C level code (was: Re: Pronouncement needed in issue9675)

2010-10-02 Thread Antoine Pitrou 
On Sat, 02 Oct 2010 22:14:28 +0200
Jesus Cea  wrote:
> 
> My idea is to convert the CObject exception in bsddb to print a warning
> like "CObject creation failed. The bsddb.cAPI will be not available.
> Reason is: ".
> 
> Is there a "canonical way" of doing this?. Sorry if the answer is trivial...

Try PyErr_WriteUnraisable().

Regards

Antoine.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Rietveld integration into Roundup

2010-10-02 Thread Antoine Pitrou 

Hello,

On Sat, 02 Oct 2010 22:03:57 +0200
"Martin v. Löwis"  wrote:
> Following up to the recent thread, I have now integrated Rietveld
> into Roundup. This is a rough draft still, and highly experimental.
> Please try it out, but expect that it may be necessary to discard
> all data (including comments) to start over (of course, I'll try
> to avoid having to do so).

Thank you! This looks promising.

I have some questions:

1) who is the notification e-mail sent to when a review is published? I
just tried on one of my patches and the e-mail came with the following
headers:

From:   [email protected]
Reply-to:   [email protected], [email protected]
To: [email protected]
Cc: [email protected]
Sujet:  Fast path for small int-indexing of lists and tuples
(issue9800)

Am I right in assuming that nobody except me (even other people
nosied in the Roundup issue) received the notification?

(is n...@psf some kind of /dev/null?)

2) if I look at http://bugs.python.org/issue9962, only the second patch
of all three has been enabled for review. Yet they were all created
through "svn diff" against a recent py3k checkout.

Thanks again

Antoine.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Pythonmac-SIG] sad state of OS X Python testing...

2010-10-02 Thread Martin v. Löwis 
> I'm already running a Jython buildslave on an Intel Mac Pro which is
> pretty underused - I'd be happy to run a CPython one there too, if
> it'd be worthwhile.

I think Bill was specifically after Snow Leopard - what system are you
using?

Regards,
Martin

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] We should be using a tool for code reviews

2010-10-02 Thread Jesus Cea 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 30/09/10 22:41, Brett Cannon wrote:
> Don't see why not, but those of us who use OpenID would need to start
> caring about a password which would be unfortunate.

+1. OpenID or OAuth is a must.

Moreover, I am a bit worried of needing a google account. Google already
knows too much about me, I don't want to navigate with a google cookie
in my browser.

Guido says that Rietveld can run outside of Google. That is not a bad
option. I could host it, I guess, but my server uptime is only around
99.7% (half an hour per week).

- -- 
Jesus Cea Avion _/_/  _/_/_/_/_/_/
[email protected] - http://www.jcea.es/ _/_/_/_/  _/_/_/_/  _/_/
jabber / xmpp:[email protected] _/_/_/_/  _/_/_/_/_/
.  _/_/  _/_/_/_/  _/_/  _/_/
"Things are not so easy"  _/_/  _/_/_/_/  _/_/_/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/_/_/_/  _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQCVAwUBTKeZI5lgi5GaxT1NAQLcRgP/YldvoSubtVYUC3IxdXtC+XjiKWaC28eJ
xIp3CxWwtuzAGrYl/kfiyrxLpk40jL/T6xEdU8r/lXXxKttbBBThLsNf93LrECCB
4uxmIdEY+SkK5Mj2gU3FWTQ4PQfRspAwYtfjGvnaPEbVLTWOccqAK4SsyEpkZ2n9
wQRUNYURv44=
=qoXI
-END PGP SIGNATURE-
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] We should be using a tool for code reviews

2010-10-02 Thread Martin v. Löwis 
Am 02.10.2010 22:42, schrieb Jesus Cea:
> On 30/09/10 22:41, Brett Cannon wrote:
>> Don't see why not, but those of us who use OpenID would need to start
>> caring about a password which would be unfortunate.
> 
> +1. OpenID or OAuth is a must.
> 
> Moreover, I am a bit worried of needing a google account. Google already
> knows too much about me, I don't want to navigate with a google cookie
> in my browser.
> 
> Guido says that Rietveld can run outside of Google. That is not a bad
> option. I could host it, I guess, but my server uptime is only around
> 99.7% (half an hour per week).

Don't worry. It's hosted on bugs.python.org now.

Regards,
Martin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Rietveld integration into Roundup

2010-10-02 Thread Antoine Pitrou 
Le samedi 02 octobre 2010 à 22:55 +0200, "Martin v. Löwis" a écrit :
> 
> > 2) if I look at http://bugs.python.org/issue9962, only the second patch
> > of all three has been enabled for review. Yet they were all created
> > through "svn diff" against a recent py3k checkout.
> 
> They had both the same problem: it could figure out the revision number
> the patch applied to (e.g. 85039), but not the branch that this revision
> is for. That's because the revision is 85039, and in r85039, only
> /peps got modified.
> 
> I see... so the revision number is mostly useless when trying to
> identify the branch.
> 
> You should be able to fix this by manually filling out the branch on
> the file. I'll have to come up with a better way to determine the branch
> which a patch was created on. Perhaps going back in history and taking
> the first branch where the patch cleanly applies can do the trick.

I think a heuristic would be to try py3k HEAD first. That's what most
patches are supposed to work against.

Regards

Antoine.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Rietveld integration into Roundup

2010-10-02 Thread Martin v. Löwis 
> 1) who is the notification e-mail sent to when a review is published?

I haven't figured that out yet. I'm not even sure whether email gets
sent at all. I'll look into this a week from now, unless someone
resolves that earlier. But...

> I
> just tried on one of my patches and the e-mail came with the following
> headers:
> 
> From: [email protected]
> Reply-to: [email protected], [email protected]
> To:   [email protected]
> Cc:   [email protected]
> Sujet:Fast path for small int-indexing of lists and tuples
> (issue9800)
> 
> Am I right in assuming that nobody except me (even other people
> nosied in the Roundup issue) received the notification?

So email did get send ... good.

It certainly doesn't consider the nosy list at all at the moment.

> (is n...@psf some kind of /dev/null?)

I have no idea where that came from.

> 2) if I look at http://bugs.python.org/issue9962, only the second patch
> of all three has been enabled for review. Yet they were all created
> through "svn diff" against a recent py3k checkout.

They had both the same problem: it could figure out the revision number
the patch applied to (e.g. 85039), but not the branch that this revision
is for. That's because the revision is 85039, and in r85039, only
/peps got modified.

I see... so the revision number is mostly useless when trying to
identify the branch.

You should be able to fix this by manually filling out the branch on
the file. I'll have to come up with a better way to determine the branch
which a patch was created on. Perhaps going back in history and taking
the first branch where the patch cleanly applies can do the trick.

Regards,
Martin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Pythonmac-SIG] sad state of OS X Python testing...

2010-10-02 Thread Nicholas Riley 
On Sat, Oct 02, 2010 at 10:08:09PM +0200, "Martin v. Löwis" wrote:
> > Surely someone could volunteer an old Intel Mac to run some tests?
> > (Actually, two someones -- we'd need separate machines for Leopard and
> > Snow Leopard.)  I'd be happy to stick it in a server room at PARC and
> > keep an eye on it.  (Right now I've got a rack full of old eMacs and a
> > G5 running PPC buildbots.)  Perhaps we could get Apple to contribute
> > some "seconds"?  I believe donations to the PSF are deductible.
> 
> The issue isn't really to get the hardware. The issue is to find
> somebody to volunteer running a build slave.

I'm already running a Jython buildslave on an Intel Mac Pro which is
pretty underused - I'd be happy to run a CPython one there too, if
it'd be worthwhile.

-- 
Nicholas Riley 
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Patch making the current email package (mostly) support bytes

2010-10-02 Thread R. David Murray 
A while back on some issue or another I remember telling someone that
if there was any sort of clever hack that would allow the current email
package (email5) to work with bytes we would have implemented it.

Well, I've come up with a clever hack.

The idea came out of a conversation with Antoine.  I was saying that it
was ironic that Unicode could only be used as a 7bit-clean data
transmission channel for email, and he remarked that by using
surrogate escape you *could* use unicode as a transmission channel
for 8bit data.  At first I dismissed this observation as irrelevant
to email, since email has to transform the 8bit data at some point.

But I started thinking.  And then I started experimenting.  And it turns
out that it works.

The clever hack (thanks ultimately to Martin) is to accept 8bit data
by encoding it using the ASCII codec and the surrogateescape error
handler.  Then, inside the email module at any point where bytes might
be meaningful or might be about to escape, it can check to see if there
are any surrogates and act accordingly.

The API additions are few, and in fact for most programs (he says bravely,
not really knowing) there are really only two changes you need to make
when converting a program that handles bytes data to py3k.  The first
is the encoding of binary input data as mentioned.  The second is that
when you want to get the bytes back out, you use the new BytesGenerator
instead of Generator.  BytesGenerator is just like Generator except
that it writes bytes to its file argument instead of strings, and it
recovers any bytes that were in the original input.

So given this sequence:

msg = email.msg_from_file(open('myfile',
   encoding='ascii',
   errors='surrogateescape'))
email.generator.BytesGenerator(open('myfile2', 'wb')).flatten(msg)

myfile and myflie2 will theoretically be identical (modulo universal
newline and _mangle_from issues).

I've additionally added a 'message_from_bytes' convenience function.

One nice feature of this patch is that once you've got the model built
from surrogateescaped input, if you do a get_payload() on a message body
whose ContentTransferEncoding is '8bit' you will get the body decoded
to unicode using the charset declared in the Content-Type header
(assuming Python supports that charset).

You can always get at the bytes version of the body of a message part by
using get_payload(decode=True) [*].  You can't really get at the bytes
version of message headers, though...for safety if you access a header
whose value contains non-ASCII chars (that aren't RFC2047 encoded to be
ASCII) the 8bit characters get replaced with '?'s.  (But BytesGenerator
will emit the original 8bit characters if the headers haven't been
modified.)

I do not propose that this is a *good* API, since it has the classic
problem that if there are coding bugs in the email module strings may
"escape" that have surrogates in them and we end up with programs that
work most of the timeexcept when they fail with mysterious errors
because of unusual bytes input data.  On the other hand you always
*know* when you have bytes data in an unknown encoding (because they
are surrogate escaped), so it is ever so much better than the Python2
situation.

The advantage of this patch is that it means Python3.2 can have an
email module that is capable of handling a significant proportion of the
applications where the ability to process binary email data is required.

I've uploaded the patch to issue 4661 (http://bugs.python.org/issue4661).
I uploaded it to rietveld as well just before Martin's announcement.
After the announcement I uploaded the svn patch to the tracker, so
hopefully there will be an automated review button as well.  Here
is your chance to exercise the new review tools :)

This patch does break two of Barry's patch-for-review rules: it is
more than 800 lines of diff (but not a lot more, and less than 800
if you count only code diff and not docs), and it did not have a very
extensive design discussion beforehand.  I did talk with people on IRC,
particularly Barry, before finishing the patch, and I did post a summary
to the email-sig mailing list (but got no response).

Now it is time to see what the wider community thinks.  There is some
question of whether this is a bending of the string/bytes separation
that doesn't belong as part of the standard library, but after working
my way through it I think it is a fairly clean hack[**], and most
likely a case where practicality beats purity.

Regardless of whether or not this patch or a descendant thereof is
accepted I still intend to continue working on email6.  There are many
other bugs in the current email package that require a rewrite of parts
of its infrastructure, and the email-sig is agreed that the email API
needs revision quite apart from the bytes/string issues.  However, there
is something pleasing about the simplicity of this way of handling bytes
tha

Re: [Python-Dev] Patch making the current email package (mostly) support bytes

2010-10-02 Thread Benjamin Peterson 
2010/10/2 R. David Murray :
> Regardless of whether or not this patch or a descendant thereof is
> accepted I still intend to continue working on email6.  There are many
> other bugs in the current email package that require a rewrite of parts
> of its infrastructure, and the email-sig is agreed that the email API
> needs revision quite apart from the bytes/string issues.  However, there
> is something pleasing about the simplicity of this way of handling bytes
> that I intend to consider carefully while we work further on email6.

And how would this addition interact with changes in email6?



-- 
Regards,
Benjamin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] os.path.normcase rationale?

2010-10-02 Thread Nir Soffer 
On Sat, Sep 25, 2010 at 1:36 AM, James Y Knight  wrote:
>
> An OSX code sketch is available here (summary: call FSPathMakeRef to get an
> FSRef from a path string, then FSRefMakePath to make it back into a path,
> which will then have the correct case). And note that it only works if the
> file actually exists.
>
>
> http://stackoverflow.com/questions/370186/how-do-i-find-the-correct-case-of-a-filename
>
> It would indeed be useful to have that be available in Python.
>

There is a much simpler way:

>>> from Carbon import File
>>> File.FSRef('/tmp/foo').as_pathname()
'/private/tmp/Foo'

Note that this is much slower compared to os.path.exists.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] sad state of OS X Python testing...

2010-10-02 Thread Ivan Krstić 

On Oct 2, 2010, at 11:50 AM, Bill Janssen wrote:

Perhaps we could get Apple to contribute some "seconds"?


If you don't get a good solution soon, let me know off-list and I'll  
see if Apple can help.


Cheers,

--
Ivan Krstić  | http://radian.org

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Patch making the current email package (mostly) support bytes

2010-10-02 Thread R. David Murray 
On Sat, 02 Oct 2010 19:15:57 -0500, Benjamin Peterson  
wrote:
> 2010/10/2 R. David Murray :
> > Regardless of whether or not this patch or a descendant thereof is
> > accepted I still intend to continue working on email6. =C2=A0There are ma=
> ny
> > other bugs in the current email package that require a rewrite of parts
> > of its infrastructure, and the email-sig is agreed that the email API
> > needs revision quite apart from the bytes/string issues. =C2=A0However, t=
> here
> > is something pleasing about the simplicity of this way of handling bytes
> > that I intend to consider carefully while we work further on email6.
> 
> And how would this addition interact with changes in email6?

It will be no harder to do the backward compatibility support for this
than for the rest of the email5 API, if that's what you are asking.
Assuming my plan for backward compatibility works at all (which it
should).

--David
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Patch making the current email package (mostly) support bytes

2010-10-02 Thread Nick Coghlan 
On Sun, Oct 3, 2010 at 9:00 AM, R. David Murray  wrote:
> I do not propose that this is a *good* API, since it has the classic
> problem that if there are coding bugs in the email module strings may
> "escape" that have surrogates in them and we end up with programs that
> work most of the timeexcept when they fail with mysterious errors
> because of unusual bytes input data.  On the other hand you always
> *know* when you have bytes data in an unknown encoding (because they
> are surrogate escaped), so it is ever so much better than the Python2
> situation.

It's a similar concept to one Antoine and I (and some others) have
been considering in the tracker for making urllib.parse able to handle
ASCII-compatible bytes-encodings. I've already implemented a version
of that patch which has parallel bytes and str versions of all the
ASCII constants, and the result is pretty ugly. My next goal is to
implement a version that uses the same trick you have here for email
and see how the code complexity compares.

We do need to tread carefully to make sure the pseudo strings don't
escape, but the other approach requires similar care all the way
through the internal algorithms to make sure they aren't assuming
bytes or str instances anywhere.

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Pythonmac-SIG] sad state of OS X Python testing...

2010-10-02 Thread Nicholas Riley 
On Sat, Oct 02, 2010 at 10:37:40PM +0200, "Martin v. Löwis" wrote:
> > I'm already running a Jython buildslave on an Intel Mac Pro which is
> > pretty underused - I'd be happy to run a CPython one there too, if
> > it'd be worthwhile.
> 
> I think Bill was specifically after Snow Leopard - what system are you
> using?

It's still on Leopard but I could use another (slower, but still
probably fine) machine that has Snow Leopard on it.

-- 
Nicholas Riley 

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com