Re: [Python-Dev] len(chr(i)) = 2?

2010-11-20 Thread Martin v. Löwis 
Am 20.11.2010 05:11, schrieb Stephen J. Turnbull:
> "Martin v. Löwis" writes:
> 
>  > The term "UCS-2" is a character set that can encode only encode 65536
>  > characters; it thus refers to Unicode 1.1. According to the Unicode
>  > Consortium's FAQ, the term UCS-2 should be avoided these days.
> 
> So what do you propose we call the Python implementation?

A technical correct description would be to say that Python uses either
16-bit code units or 32-bit code units; for brevity, these can be called
narrow and wide code units.

> Strictly speaking, internally Python only encodes 65536 characters in
> 2-octet builds.  Its (Unicode) string-handling code does not know
> about surrogates at all, AFAIK

Here you are mistaken: it does indeed know about UTF-16 and surrogates
in several places, e.g. in the UTF-8 codec, or in the repr()
implementation; likewise in the parser.

> and therefore is not UTF-16 conforming.

I disagree. Python does "conform" to "UTF-16" (certainly in the
sense that no UTF-16 specification ever mandates a certain Python
API, and that Python follows all general requirements of the
UTF-16 specification).

> AFAIK this was not supposed to change in Python 3; indexing and
> slicing go by code unit (isomorphic to UCS-n), not character, and due
> to PEP 383 4-octet builds do not conform (internally) to UTF-32, and
> can produce output that conforms to Unicode not at all (as a user
> option, of course, but it's still non-conformant).

What behavior specifically do you consider non-conforming, and what
specific specification do you think it is not conforming to? For
example, it *is* fully conforming with UTF-8.

Regards,
Martin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Web servers, bytes, str, documentation, Python 3.2a4

2010-11-20 Thread Éric Araujo 
Hello

> cgitb.enable(0,"d:\temp")

Isn’t that expanded to “d:emp”?
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] pymigr: Build identification patch is updated, but only for Unix.

2010-11-20 Thread Nick Coghlan 
On Sat, Nov 20, 2010 at 6:02 PM, georg.brandl
 wrote:
> georg.brandl pushed abd0dc1328ce to pymigr:
>
> http://hg.python.org/pymigr/rev/abd0dc1328ce
> changeset:   70:abd0dc1328ce
> tag:         tip
> user:        Georg Brandl 
> date:        Sat Nov 20 09:01:03 2010 +0100
> summary:     Build identification patch is updated, but only for Unix.
> files:       todo.txt

Does this repository use the same set of hooks as distutils2? (I'm
hoping not, since if it does, my change to the email hook didn't
work...)

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial Schedule

2010-11-20 Thread Nick Coghlan 
On Sat, Nov 20, 2010 at 2:51 AM, Georg Brandl  wrote:
> I'm at it.  In fact, I think I will merge both todo.txt and tasks.txt
> into the PEP.  It's not more of a burden to update it there, and it's
> more visible to the developer community.

The latest checkin was definitely an improvement (especially the
updated timeline).

According to the PEP, the .hgeol rules aren't currently enforced
server side - having such a hook in place before Hg went live was
definitely one of the things we agreed on before the hgeol extension
even existed in a usable form.

For fixing whitespace issues (another open question mentioned in the
PEP), "make patchcheck" can continue to handle that - no need to
create a Hg specific extension for it.

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] r86566 - in python/branches/py3k: Doc/glossary.rst Doc/library/inspect.rst Lib/inspect.py Lib/test/test_inspect.py Misc/NEWS Misc/python-wing4.wpr

2010-11-20 Thread Nick Coghlan 
On Sun, Nov 21, 2010 at 1:07 AM, michael.foord
 wrote:
> +Fetching attributes statically
> +--
> +
> +Both :func:`getattr` and :func:`hasattr` can trigger code execution when
> +fetching or checking for the existence of attributes. Descriptors, like
> +properties, will be invoked and :meth:`__getattr__` and 
> :meth:`__getattribute__`
> +may be called.
> +
> +For cases where you want passive introspection, like documentation tools, 
> this
> +can be inconvenient. `getattr_static` has the same signature as 
> :func:`getattr`
> +but avoids executing code when it fetches attributes.

This description feels a little strong to me - getattr_static still
executes all those things on the metaclass as it retrieves the
information it needs to do the "static" lookup. Leaving this original
description (which assumes metaclass=type) alone and adding a note
near the end of the section to say that metaclass code is still
executed might be an improvement.

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] r86566 - in python/branches/py3k: Doc/glossary.rst Doc/library/inspect.rst Lib/inspect.py Lib/test/test_inspect.py Misc/NEWS Misc/python-wing4.wpr

2010-11-20 Thread Michael Foord 

On 20/11/2010 15:21, Nick Coghlan wrote:

On Sun, Nov 21, 2010 at 1:07 AM, michael.foord
  wrote:

+Fetching attributes statically
+--
+
+Both :func:`getattr` and :func:`hasattr` can trigger code execution when
+fetching or checking for the existence of attributes. Descriptors, like
+properties, will be invoked and :meth:`__getattr__` and 
:meth:`__getattribute__`
+may be called.
+
+For cases where you want passive introspection, like documentation tools, this
+can be inconvenient. `getattr_static` has the same signature as :func:`getattr`
+but avoids executing code when it fetches attributes.

This description feels a little strong to me - getattr_static still
executes all those things on the metaclass as it retrieves the
information it needs to do the "static" lookup. Leaving this original
description (which assumes metaclass=type) alone and adding a note
near the end of the section to say that metaclass code is still
executed might be an improvement.
Can you give an example of code in a metaclass that may be executed by 
getattr_static? It's not that I don't believe you I just can't think of 
an example. Looking up the class and the mro are the only two examples I 
can think of (klass.__mro__ and instance.__class__ - and they are noted 
in the docs?) but aren't metaclass specific.


Michael


Cheers,
Nick.




--

http://www.voidspace.org.uk/

READ CAREFULLY. By accepting and reading this email you agree,
on behalf of your employer, to release me from all obligations
and waivers arising from any and all NON-NEGOTIATED agreements,
licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap,
confidentiality, non-disclosure, non-compete and acceptable use
policies (”BOGUS AGREEMENTS”) that I have entered into with your
employer, its partners, licensors, agents and assigns, in
perpetuity, without prejudice to my ongoing rights and privileges.
You further represent that you have the authority to release me
from any BOGUS AGREEMENTS on behalf of your employer.

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] r86570 - in python/branches/py3k: Lib/unittest/case.py Lib/unittest/test/test_case.py Misc/NEWS

2010-11-20 Thread Antoine Pitrou 
On Sat, 20 Nov 2010 16:34:26 +0100 (CET)
michael.foord  wrote:
> +
> +def testPickle(self):
> +# Issue 10326
> +
> +# Can't use TestCase classes defined in Test class as
> +# pickle does not work with inner classes
> +test = unittest.TestCase('run')
> +for protocol in range(pickle.HIGHEST_PROTOCOL + 1):
> +
> +# blew up prior to fix
> +pickled_test = pickle.dumps(test, protocol=protocol)

You must also check that the object can be unpickled, otherwise
making TestCase picklable is not only pointless, but misleading the
user.  Other classes which claim to be picklable (such as e.g.
io.BytesIO) are careful to check that unpickling works fine and
produces an usable object.

Regards

Antoine.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] r86570 - in python/branches/py3k: Lib/unittest/case.py Lib/unittest/test/test_case.py Misc/NEWS

2010-11-20 Thread Michael Foord 

On 20/11/2010 15:42, Antoine Pitrou wrote:

On Sat, 20 Nov 2010 16:34:26 +0100 (CET)
michael.foord  wrote:

+
+def testPickle(self):
+# Issue 10326
+
+# Can't use TestCase classes defined in Test class as
+# pickle does not work with inner classes
+test = unittest.TestCase('run')
+for protocol in range(pickle.HIGHEST_PROTOCOL + 1):
+
+# blew up prior to fix
+pickled_test = pickle.dumps(test, protocol=protocol)

You must also check that the object can be unpickled, otherwise
making TestCase picklable is not only pointless, but misleading the
user.  Other classes which claim to be picklable (such as e.g.
io.BytesIO) are careful to check that unpickling works fine and
produces an usable object.


Well, given the *particular* bug it is fixing, ensuring that the 
TestCase instances can be pickled is enough. If they fail to unpickle 
that is a bug in pickle and not in unittest. *However*, the test is very 
easy to extend to what you suggest so I have done it.


All the best,

Michael


Regards

Antoine.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk



--

http://www.voidspace.org.uk/

READ CAREFULLY. By accepting and reading this email you agree,
on behalf of your employer, to release me from all obligations
and waivers arising from any and all NON-NEGOTIATED agreements,
licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap,
confidentiality, non-disclosure, non-compete and acceptable use
policies (”BOGUS AGREEMENTS”) that I have entered into with your
employer, its partners, licensors, agents and assigns, in
perpetuity, without prejudice to my ongoing rights and privileges.
You further represent that you have the authority to release me
from any BOGUS AGREEMENTS on behalf of your employer.

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] r86570 - in python/branches/py3k: Lib/unittest/case.py Lib/unittest/test/test_case.py Misc/NEWS

2010-11-20 Thread Antoine Pitrou 
Le samedi 20 novembre 2010 à 15:48 +, Michael Foord a écrit :
> On 20/11/2010 15:42, Antoine Pitrou wrote:
> > On Sat, 20 Nov 2010 16:34:26 +0100 (CET)
> > michael.foord  wrote:
> >> +
> >> +def testPickle(self):
> >> +# Issue 10326
> >> +
> >> +# Can't use TestCase classes defined in Test class as
> >> +# pickle does not work with inner classes
> >> +test = unittest.TestCase('run')
> >> +for protocol in range(pickle.HIGHEST_PROTOCOL + 1):
> >> +
> >> +# blew up prior to fix
> >> +pickled_test = pickle.dumps(test, protocol=protocol)
> > You must also check that the object can be unpickled, otherwise
> > making TestCase picklable is not only pointless, but misleading the
> > user.  Other classes which claim to be picklable (such as e.g.
> > io.BytesIO) are careful to check that unpickling works fine and
> > produces an usable object.
> 
> Well, given the *particular* bug it is fixing, ensuring that the 
> TestCase instances can be pickled is enough. If they fail to unpickle 
> that is a bug in pickle and not in unittest.

It wouldn't be, no.  pickle provides several different APIs to ensure
that state gets correctly stored *and* restored, but it's up to
application classes such as TestCase to ensure that they implement those
APIs correctly for the intended behaviour.  Therefore, checking that
pickling "works" fine (or, rather, seems to work) is only half ot the
job.

(for example, if you define a __getstate__, chances are you must define
a __setstate__ too, and it is your job to make it work properly)

Antoine.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] r86566 - in python/branches/py3k: Doc/glossary.rst Doc/library/inspect.rst Lib/inspect.py Lib/test/test_inspect.py Misc/NEWS Misc/python-wing4.wpr

2010-11-20 Thread Michael Foord 

On 20/11/2010 16:01, Nick Coghlan wrote:

On Sun, Nov 21, 2010 at 1:29 AM, Michael Foord
  wrote:

Can you give an example of code in a metaclass that may be executed by
getattr_static? It's not that I don't believe you I just can't think of an
example. Looking up the class and the mro are the only two examples I can
think of (klass.__mro__ and instance.__class__ - and they are noted in the
docs?) but aren't metaclass specific.

The description heavily implies that arbitrary Python code won't be
executed by calling getattr_static, and that isn't necessarily true.
It's almost certain to be true in the case when the metaclass is type,
but can't be guaranteed otherwise.


Given the way that member lookups are done by getattr_static I don't 
think any assumptions about the metaclass are made. I'm happy to be 
proven wrong (but would rather fix it than document it as an exception). 
(Actually we assume the metaclass doesn't use __slots__, but only 
because it isn't *possible* for a metaclass to use __slots__.)



The retrieval of __class__ is a
normal lookup on the object, so it can trigger all of the things
getattr_static is trying to avoid (unavoidable if you want to support
proxy classes at all), and the lookup of __mro__ invokes all of those
things on the metaclass.


__class__ and mro lookup are noted in the docs as being exceptions. We 
could actually remove the __class__ lookup from the list of exceptions 
by using type(...) instead of obj.__class__.



I'll see if I'm still of the same opinion after I sleep on it, but my
first impression of the docs was that they slightly oversold the
strength of the "doesn't execute arbitrary code" aspect of the new
function. The existing caveats were all relating to when getattr() and
getattr_static() might give different answers, while the additional
caveats I was suggesting related to cases where arbitrary code may
still be executed.

I'm happy to change the wording to make the promise less strong.

All the best,

Michael


Cheers,
Nick.




--

http://www.voidspace.org.uk/

READ CAREFULLY. By accepting and reading this email you agree,
on behalf of your employer, to release me from all obligations
and waivers arising from any and all NON-NEGOTIATED agreements,
licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap,
confidentiality, non-disclosure, non-compete and acceptable use
policies (”BOGUS AGREEMENTS”) that I have entered into with your
employer, its partners, licensors, agents and assigns, in
perpetuity, without prejudice to my ongoing rights and privileges.
You further represent that you have the authority to release me
from any BOGUS AGREEMENTS on behalf of your employer.

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] r86570 - in python/branches/py3k: Lib/unittest/case.py Lib/unittest/test/test_case.py Misc/NEWS

2010-11-20 Thread Michael Foord 

On 20/11/2010 15:59, Antoine Pitrou wrote:

Le samedi 20 novembre 2010 à 15:48 +, Michael Foord a écrit :

On 20/11/2010 15:42, Antoine Pitrou wrote:

On Sat, 20 Nov 2010 16:34:26 +0100 (CET)
michael.foord   wrote:

+
+def testPickle(self):
+# Issue 10326
+
+# Can't use TestCase classes defined in Test class as
+# pickle does not work with inner classes
+test = unittest.TestCase('run')
+for protocol in range(pickle.HIGHEST_PROTOCOL + 1):
+
+# blew up prior to fix
+pickled_test = pickle.dumps(test, protocol=protocol)

You must also check that the object can be unpickled, otherwise
making TestCase picklable is not only pointless, but misleading the
user.  Other classes which claim to be picklable (such as e.g.
io.BytesIO) are careful to check that unpickling works fine and
produces an usable object.

Well, given the *particular* bug it is fixing, ensuring that the
TestCase instances can be pickled is enough. If they fail to unpickle
that is a bug in pickle and not in unittest.

It wouldn't be, no.  pickle provides several different APIs to ensure
that state gets correctly stored *and* restored, but it's up to
application classes such as TestCase to ensure that they implement those
APIs correctly for the intended behaviour.  Therefore, checking that
pickling "works" fine (or, rather, seems to work) is only half ot the
job.

(for example, if you define a __getstate__, chances are you must define
a __setstate__ too, and it is your job to make it work properly)


Yes, but unittest.TestCase doesn't implement any of those APIs (and if 
we did we would *definitely* need to test unpickling). That aside I have 
extended the test in the way you suggest.


Actually it would be nice to implement custom pickling / unpickling 
methods to allow Python 2.7 / 3.2 pickled TestCases to be unpickled on 
earlier versions of Python. I couldn't see how to change the class name 
in the pickle using the pickle protocol methods. Suggestions welcomed.


Michael


Antoine.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk



--

http://www.voidspace.org.uk/

READ CAREFULLY. By accepting and reading this email you agree,
on behalf of your employer, to release me from all obligations
and waivers arising from any and all NON-NEGOTIATED agreements,
licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap,
confidentiality, non-disclosure, non-compete and acceptable use
policies (”BOGUS AGREEMENTS”) that I have entered into with your
employer, its partners, licensors, agents and assigns, in
perpetuity, without prejudice to my ongoing rights and privileges.
You further represent that you have the authority to release me
from any BOGUS AGREEMENTS on behalf of your employer.

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] r86566 - in python/branches/py3k: Doc/glossary.rst Doc/library/inspect.rst Lib/inspect.py Lib/test/test_inspect.py Misc/NEWS Misc/python-wing4.wpr

2010-11-20 Thread Nick Coghlan 
On Sun, Nov 21, 2010 at 1:29 AM, Michael Foord
 wrote:
> Can you give an example of code in a metaclass that may be executed by
> getattr_static? It's not that I don't believe you I just can't think of an
> example. Looking up the class and the mro are the only two examples I can
> think of (klass.__mro__ and instance.__class__ - and they are noted in the
> docs?) but aren't metaclass specific.

The description heavily implies that arbitrary Python code won't be
executed by calling getattr_static, and that isn't necessarily true.
It's almost certain to be true in the case when the metaclass is type,
but can't be guaranteed otherwise. The retrieval of __class__ is a
normal lookup on the object, so it can trigger all of the things
getattr_static is trying to avoid (unavoidable if you want to support
proxy classes at all), and the lookup of __mro__ invokes all of those
things on the metaclass.

I'll see if I'm still of the same opinion after I sleep on it, but my
first impression of the docs was that they slightly oversold the
strength of the "doesn't execute arbitrary code" aspect of the new
function. The existing caveats were all relating to when getattr() and
getattr_static() might give different answers, while the additional
caveats I was suggesting related to cases where arbitrary code may
still be executed.

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] r86566 - in python/branches/py3k: Doc/glossary.rst Doc/library/inspect.rst Lib/inspect.py Lib/test/test_inspect.py Misc/NEWS Misc/python-wing4.wpr

2010-11-20 Thread Michael Foord 

On 20/11/2010 16:06, Michael Foord wrote:

On 20/11/2010 16:01, Nick Coghlan wrote:
[snip...]

The retrieval of __class__ is a
normal lookup on the object, so it can trigger all of the things
getattr_static is trying to avoid (unavoidable if you want to support
proxy classes at all), and the lookup of __mro__ invokes all of those
things on the metaclass.


__class__ and mro lookup are noted in the docs as being exceptions. We 
could actually remove the __class__ lookup from the list of exceptions 
by using type(...) instead of obj.__class__.




Done.


I'll see if I'm still of the same opinion after I sleep on it, but my
first impression of the docs was that they slightly oversold the
strength of the "doesn't execute arbitrary code" aspect of the new
function. The existing caveats were all relating to when getattr() and
getattr_static() might give different answers, while the additional
caveats I was suggesting related to cases where arbitrary code may
still be executed.

I'm happy to change the wording to make the promise less strong.


I've also removed the __mro__ exception. This is done with:

type.__dict__['__mro__'].__get__(klass)

If you can think of any other exceptions then please let me know.

Michael


All the best,

Michael


Cheers,
Nick.







--

http://www.voidspace.org.uk/

READ CAREFULLY. By accepting and reading this email you agree,
on behalf of your employer, to release me from all obligations
and waivers arising from any and all NON-NEGOTIATED agreements,
licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap,
confidentiality, non-disclosure, non-compete and acceptable use
policies (”BOGUS AGREEMENTS”) that I have entered into with your
employer, its partners, licensors, agents and assigns, in
perpetuity, without prejudice to my ongoing rights and privileges.
You further represent that you have the authority to release me
from any BOGUS AGREEMENTS on behalf of your employer.

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Web servers, bytes, str, documentation, Python 3.2a4

2010-11-20 Thread Glenn Linderman 

On 11/20/2010 3:38 AM, Éric Araujo wrote:

Hello


cgitb.enable(0,"d:\temp")

Isn’t that expanded to “d:emp”?



Oops.  Yes, that fixes the problem with creation of the temp file, 
thanks for catching that.  I  now get a complete report of the original 
error in the temp file (below).  I am a bit less confused now... but it 
seems that there are still a number of issues.  Here is an enumeration 
of problems I was hard pressed to make before you removed my confusion 
on this issue.


1. cgitb should expect to report to a binary stdout, using whatever 
encoding (possibly ASCII) that seems appropriate for the output that in 
generates.


2. Some appropriate documentation or API or both should be provided to 
enable a script to set "binary" mode for stdout for CGI scripts. This 
link 
 
demonstrates the confusion (wish I had found it earlier) that is 
encountered by such lack.  One must tell msvcrt the stream is binary (I 
had figured that out early on), one must also sidestep the use of the 
cp1252 default when printing binary, one must also choose a proper text 
encoding corresponding to the HTTP headers sent.  My second email in 
this thread, sent a few hours after the first, shows a convenient set of 
cures for all but msvcrt (as long as only "write" is used for writing.  
"print" support could be added, similarly).  Likely something along this 
line is needed for stdin as well, I haven't yet experimented with 
uploading binary content to a CGI.


One could speculate about having the Python runtime auto-detect CGI 
mode, but I don't know of any foolproof technique for that, and the 
selection of the "proper" text encoding depends on the details of the 
CGI, so having instead an API or two that assists with doing this sort 
of thing would be better; the need for documentation, at least, seems 
imperative.


3. subprocess documentation could be improved to point out  that when 
using subprocess.PIPE to talk to a Python subprocess, that the 
communications will be in binary.  Again, I don't know of any way to 
autodetect the subprocess environment, but if it were possible to select 
an appropriate encoding and use it consistently on both sides of the 
PIPE, that would be a convenience to its use; if not possible, 
documenting the issue, and providing an API to use to easily select such 
encodings both in client and server, would be helpful.


While the layers are all there, and ".buffer" is documented for 
TextIOWrapper, the use of sys.stdout.buffer and the fact that it has a 
full set of operations isn't immediately obvious from the reference 
material; perhaps it is in a tutorial I haven't found, but... I was 
looking, and didn't find it.


Of course, subprocess may launch non-Python programs; they will have 
their own ideas of binary vs text encoding, so it is important that it 
is convenient to match them on the Python side.


It would be nice if subprocess had a mechanism for providing no-deadlock 
stdout data to the parent prior to the child terminating.  A CGI 
implementation via subprocess shouldn't accumulate all of stdout (or all 
of stderr, for that matter, although less important).  I don't (yet) 
know enough about Python threading to know if this is possible, but it 
certainly would be useful.


4. http.server has a number of bugs and limitations.
4a. _url_collapse_path_split seems inefficient (although I have to 
benchmark it against what I think would be more efficient), and for its 
only use within http.server it produces the wrong information, so the 
information has to be recombined and resplit to make it function 
properly, adding to the perception of inefficiency.
4b. Detection of "executable" on Windows is simply wrong.  Unix 
execution bits do not exist.
4c. is_cgi doesn't properly handle PATHINFO parts of the path, this is 
the other half of 4a.  The Python2.x CGIHTTPServer.py had this right, 
but the introduction and use of _url_collapse_path_split broke it.
4d. Searching for a ? to find an explicit query string should use 
.find('?') rather than .rfind('?') as there is no prohibition on using 
'?' within a query string, AFAIK.
4e. doesn't set the REQUEST_URI, HTTP_HOST, or HTTP_PORT  environment 
variables for the CGI.
4f. Should not send the 200 response until it sees if the CGI sends a 
Status: header.
4g. Should not buffer all of stdout: subprocess.communicate is 
inappropriate for a web server CGI interface.  The data should stream 
through to avoid consuming inordinate amounts of memory.  The only 
solution within the current limitations of subprocess is to abandon 
stderr, force the CGI to do its own error logging, and use 
shutil.copyfileobj to hook up p.stdout to self.wfile once the Status: 
message processing has happened.
4h. Doesn't seem to close p.stdin (I'm not sure if that is necessary, it 
may happ

Re: [Python-Dev] len(chr(i)) = 2?

2010-11-20 Thread Alexander Belopolsky 
On Sat, Nov 20, 2010 at 4:05 AM, "Martin v. Löwis"  wrote:
..
> A technical correct description would be to say that Python uses either
> 16-bit code units or 32-bit code units; for brevity, these can be called
> narrow and wide code units.

+1

PEP 261 introduced terms "wide Py_UNICODE" and "narrow Py_UNICODE,"
but when discussion is at Python level, I don't think we should use
names of C typedefs.   I think "wide/narrow Unicode" builds describe
the two options clearly and unambiguously.   I prefer Python-specific
terminology to Unicode terms because in Python reference documentation
we often discuss details that are outside of the scope of Unicode
Standard.  For example, interpretation of lone surrogates on narrow
builds is one such detail.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Reminder: Distutils vs Distutils2

2010-11-20 Thread Tarek Ziadé 
Hello,

I have seen some efforts recently to improve Distutils in the standard library,

Just a quick reminder of the status of Distutils: it's frozen and is
just being bug fixed at this time. The work I done last year was
reverted and pushed to Distutils2.
A lot of work has been done since then, and we had 4 GSOC students
working this summer on Distutils2.

It's backward-incompatible, so we can remove the things we don't like
and add new things w/o suffering from backward compatibility pains.

So if you want to improve the tool, or if you have some pending
changes to Distutils, I would encourage you to join the Distutils2
effort and not to waste time on Distutils anymore.  The patches that
did not make it to Distutils can still be added in Distutils2, for
most of them.

The workflow we currently use to change the code is as follow and make
it easy for everyone to contribute:

1. clone http://bitbucket.org/tarek/distutils2
2. discuss / propose a patch on IRC (#distutils - Freenode) or on the
dedicated mailing list
(http://groups.google.com/group/the-fellowship-of-the-packaging)
3. I review and merge all changes at bitbucket, then push them on
http://hg,python.org/distutils2

Crazy ideas are welcome. "setup.py" is gone in d2 for instance ;)

Thanks !

Regards.
Tarek

-- 
Tarek Ziadé | http://ziade.org
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reminder: Distutils vs Distutils2

2010-11-20 Thread Tarek Ziadé 
On Sun, Nov 21, 2010 at 12:05 AM, Tarek Ziadé  wrote:
..
> Crazy ideas are welcome. "setup.py" is gone in d2 for instance ;)

But you can still use a similar form if you want - just to mention
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] r86566 - in python/branches/py3k: Doc/glossary.rst Doc/library/inspect.rst Lib/inspect.py Lib/test/test_inspect.py Misc/NEWS Misc/python-wing4.wpr

2010-11-20 Thread Nick Coghlan 
On Sun, Nov 21, 2010 at 2:06 AM, Michael Foord
 wrote:
>> I'll see if I'm still of the same opinion after I sleep on it, but my
>> first impression of the docs was that they slightly oversold the
>> strength of the "doesn't execute arbitrary code" aspect of the new
>> function. The existing caveats were all relating to when getattr() and
>> getattr_static() might give different answers, while the additional
>> caveats I was suggesting related to cases where arbitrary code may
>> still be executed.
>
> I'm happy to change the wording to make the promise less strong.

Your latest changes may have actually made the stronger wording
accurate (I certainly can't think of any loopholes off the top of my
head). If you did still want to soften the wording, I'd be inclined to
replace the word "avoids" with "minimises" in the appropriate places.

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] r86566 - in python/branches/py3k: Doc/glossary.rst Doc/library/inspect.rst Lib/inspect.py Lib/test/test_inspect.py Misc/NEWS Misc/python-wing4.wpr

2010-11-20 Thread Nick Coghlan 
On Sun, Nov 21, 2010 at 1:07 AM, michael.foord
 wrote:
> Author: michael.foord
> Date: Sat Nov 20 16:07:30 2010
> New Revision: 86566
>
> Log:
> Issue 9732: addition of getattr_static to the inspect module
>
> Modified:
>   python/branches/py3k/Doc/glossary.rst
>   python/branches/py3k/Doc/library/inspect.rst
>   python/branches/py3k/Lib/inspect.py
>   python/branches/py3k/Lib/test/test_inspect.py
>   python/branches/py3k/Misc/NEWS
>   python/branches/py3k/Misc/python-wing4.wpr

Unrelated to my previous comment - when adding
inspect.getgeneratorstate, I noticed that inspect.getattr_static isn't
mentioned in the 3.2 What's New yet (I put a XXX placeholder in for
you/Raymond).


-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Web servers, bytes, str, documentation, Python 3.2a4

2010-11-20 Thread Glenn Linderman 

On 11/20/2010 10:19 AM, Glenn Linderman wrote:
Oops.  Yes, that fixes the problem with creation of the temp file, 
thanks for catching that.  I  now get a complete report of the 
original error in the temp file (below).  I am a bit less confused 
now... but it seems that there are still a number of issues.  Here is 
an enumeration of problems I was hard pressed to make before you 
removed my confusion on this issue.


Related issues, regarding binary stream requirements for cgi interface.  
Perhaps the cgi module should have the API to set binary mode.


http://bugs.python.org/issue1610654
http://bugs.python.org/issue8077
http://bugs.python.org/issue4953

Sadly, cgi.py input handling seems to depend on the email module, 
thought to be fixed for 3.2, but it is not clear if that has been 
achieved, or if the surrogate encode workaround is sufficient for this.  
More testing needed, but I don't have such a test case developed yet.



1. cgitb should expect to report to a binary stdout, using whatever 
encoding (possibly ASCII) that seems appropriate for the output that 
in generates.


Maybe cgi.py should have an API to set the stdin and stdout to binary 
streams.   Although cgi.py deals more with stdin than stdout, cgitb 
deals more with stdout.


Created http://bugs.python.org/issue10479



2. Some appropriate documentation or API or both should be provided to 
enable a script to set "binary" mode for stdout for CGI scripts. This 
link 
 
demonstrates the confusion (wish I had found it earlier) that is 
encountered by such lack.  One must tell msvcrt the stream is binary 
(I had figured that out early on), one must also sidestep the use of 
the cp1252 default when printing binary, one must also choose a proper 
text encoding corresponding to the HTTP headers sent.  My second email 
in this thread, sent a few hours after the first, shows a convenient 
set of cures for all but msvcrt (as long as only "write" is used for 
writing.  "print" support could be added, similarly).  Likely 
something along this line is needed for stdin as well, I haven't yet 
experimented with uploading binary content to a CGI.


One could speculate about having the Python runtime auto-detect CGI 
mode, but I don't know of any foolproof technique for that, and the 
selection of the "proper" text encoding depends on the details of the 
CGI, so having instead an API or two that assists with doing this sort 
of thing would be better; the need for documentation, at least, seems 
imperative.


Created http://bugs.python.org/issue10480



3. subprocess documentation could be improved to point out  that when 
using subprocess.PIPE to talk to a Python subprocess, that the 
communications will be in binary.  Again, I don't know of any way to 
autodetect the subprocess environment, but if it were possible to 
select an appropriate encoding and use it consistently on both sides 
of the PIPE, that would be a convenience to its use; if not possible, 
documenting the issue, and providing an API to use to easily select 
such encodings both in client and server, would be helpful.


While the layers are all there, and ".buffer" is documented for 
TextIOWrapper, the use of sys.stdout.buffer and the fact that it has a 
full set of operations isn't immediately obvious from the reference 
material; perhaps it is in a tutorial I haven't found, but... I was 
looking, and didn't find it.


Of course, subprocess may launch non-Python programs; they will have 
their own ideas of binary vs text encoding, so it is important that it 
is convenient to match them on the Python side.


It would be nice if subprocess had a mechanism for providing 
no-deadlock stdout data to the parent prior to the child terminating.  
A CGI implementation via subprocess shouldn't accumulate all of stdout 
(or all of stderr, for that matter, although less important).  I don't 
(yet) know enough about Python threading to know if this is possible, 
but it certainly would be useful.


http://bugs.python.org/issue1048 for subprocess to document that 
communicate produces byte stream output.


http://bugs.python.org/issue10482 for subprocess enhancements to handle 
more cases without deadlock.


Found http://bugs.python.org/issue4571 which documents how to switch 
stdin/stdout/stderr to binary mode, and even back!   I couldn't track 
the documented change to the actual documentation, though, but I did 
find it in section 26.1, under the documentation for the three stdio 
streams:


def  make_streams_binary():
sys.stdin  =  sys.stdin.detach()
sys.stdout  =  sys.stdout.detach()



4. http.server has a number of bugs and limitations.
4a. _url_collapse_path_split seems inefficient (although I have to 
benchmark it against what I think would be more efficient), and for 
its only use within http.server it produces