Re: [Python-Dev] Update to Python Documentation Website Request

[Jean-Paul Calderone]

On Sat, 25 Jul 2009 10:40:52 +1000, Ben Finney  
wrote:

[snip]

If that is not your intent, then your application shouldn't be mentioned
in standard Python documentation.



Hm.  But docutils isn't part of the standard library, and the documentation
refers to it.  And the docs link to ActivePython and Enthought's Python
distribution.  And they link to things like "Collin Winter’s functional
module" and Django and PyOpenGL and so forth.

I don't think anyone would seriously suggest removing all references to
third-party software from the Python documentation.  Perhaps the criterion
that David is looking to have is explained is that people would probably
prefer for his app to get a bit more popular before Python itself starts
pointing people in its direction.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile, and WriteFile

[Jean-Paul Calderone]

On Thu, 23 Jul 2009 14:23:56 +0200, Christian Heimes  wrote:

Nick Coghlan wrote:

I see ctypes as largely useful when you want to call a native DLL but
don't have any existing infrastructure for accessing native code from
your project. A few lines of ctypes code is then a much better solution
than adding a C or C++ compilation dependency just to access a couple of
functions.

Of course, that definitely isn't the case for CPython - we not only have
plenty of existing C infrastructure, but in the specific case of
subprocess on Windows we already have a dedicated extension module
(PC/_subprocess.c).


You've hit the nail on the head! That's it.



True, CPython has C infrastructure.  What about the other Python runtimes,
though?  At the language summit, there was a lot of discussion (and, I
thought, agreement) about moving the standard library to be a collaborative
project between several of the major runtime projects.  A ctypes-based
solution seems more aligned with this goal than more C code.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile, and WriteFile

[Jean-Paul Calderone]

On Thu, 23 Jul 2009 14:21:38 +0200, Christian Heimes  wrote:

Michael Foord wrote:

A big advantage of using ctypes is that it works cross-implementation -
on IronPython and PyPy already and on Jython soon. I'd like to see more
standard library modules use it. Distributions that choose not to
include it are crippling their Python distribution.


Interesting, I didn't know that IronPython supports ctypes, too. I still
find ctypes a bit problematic because it doesn't us header files for its
types, structs and function definitions.


This is indeed a big problem with ctypes.  Fortunately, a project exists
to correct it:

   http://pypi.python.org/pypi/ctypes_configure/0.1

Anyone writing code with ctypes should be looking at ctypes_configure as
well.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pthread sem PyThread_acquire_lock

[Jean-Paul Calderone]

On Thu, 2 Jul 2009 15:47:48 -0700, "Gregory P. Smith"  wrote:

On Mon, Jun 29, 2009 at 2:28 PM, "Martin v. Löwis" wrote:

AFAIK, ignoring EINTR doesn't preclude the calling of signal handlers.


This is my understanding as well - so I don't think Python actually
"swallows" the signal.


A great example is reading from a socket. Whether or not it can be
interrupted depends on the platform, so catching Ctrl+C often requires
a timeout loop.

Also, remember that signals are asynchronous in the sense that they
are handled outside the normal execution flow of a program. Checking
for EINTR probably isn't the best way to determine if a signal has
been sent to the program.


I think it would be reasonable to support "asynchronous" exceptions,
and Python supports SIGINT fairly well most of the time.

It might be possible to support keyboard interrupts throughout
the system, but changing Python to do so could also cause
incompatibilities. So any change must be done with greatest care,
but simultaneously, should also try to arrange to cover all cases.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/greg%40krypto.org


If you want signals to actually be handled in a timely manner, its
best to leave the main thread of the program alone as a signal
handling thread that just spends its time in a loop of time.sleep(123)
calls rather than blocking on any sort of lock.  Spawn other threads
to do the actual work in your program.  Signals are delivered
indirectly in the existing CPython implementation by setting an
internal flag that the main interpreter thread polls on occasion so
blocking calls that do not interrupt and return early being called
from the main thread will effectively block signals.


Yes, this is all true now.  The question is why the implementation
works that way, and whether it is desirable to keep it working that
way.

Considering *some* of the lock implementations make themselves not
interruptable by threads while others don't bother, it seems like
*some* change to the status quo is desirable.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] python sendmsg()/recvmsg() implementation

[Jean-Paul Calderone]

On Tue, 09 Jun 2009 16:46:54 +0200, Kálmán Gergely  
wrote:

Hello, my name is Greg.

I've just started using python after many years of C programming, and I'm 
also new to the list. I wanted to clarify this

first, so that maybe I will get a little less beating for my stupidity :)



Welcome!



[snip]

Browsing the net I've found a patch to the python core 
(http://bugs.python.org/issue1194378), dated 2005.


First of all, I would like to ask you guys, whether you know of any way of 
doing this FD passing magic, or that you know

of any 3rd party module / patch / anything that can do this for me.


Aside from the patch in the tracker, there are several implementations of
these APIs as third-party extension modules.



Since I'm fairly familiar with C and (not that much, but I feel the power) 
python, I would take the challenge of writing
it, given that the above code is still somewhat usable. If all else fails I 
would like to have your help to guide me through

this process.



What would be great is if you could take the patch in the tracker and get
it into shape so that it is suitable for inclusion.  This would involve
three things, I think:

 1. Write unit tests for the functionality (since the patch itself provides
none)
 2. Update the patch so that it again applies cleanly against trunk
 3. Add documentation for the new APIs

Once this is done, you can get a committer to look at it and either provide
more specific feedback or apply it.

Thanks,

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Issues with Py3.1's new ipaddr

[Jean-Paul Calderone]

On Tue, 02 Jun 2009 19:34:11 +0200, "\"Martin v. Löwis\""  
wrote:

[snip]


You seem comfortable with these quirks, but then you're not planning
to write software with this library. Developers who do intend to write
meaningful network applications do seem concerned, yet we're ignored.


I don't hear a public outcry - only a single complainer.


Clay repeatedly pointed out that other people have objected to ipaddr and
been ignored.  It's really, really disappointing to see you continue to
ignore not only them, but the repeated attempts Clay has made to point
them out.

I don't have time to argue this issue, but I agree with essentially
everything Clay has said in this thread, and I commented about these
problems on the ticket months ago, before ipaddr was added.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] "setuptools has divided the Python community"

[Jean-Paul Calderone]

On Wed, 25 Mar 2009 11:34:43 -0400, Tres Seaver  wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Antoine Pitrou wrote:

Tarek Ziadé  gmail.com> writes:

But I agree that the sizes of the packages are too small now, and it has gone
to far. Installing a web app like Plone is scary (+100 packages)


I am working on a TurboGears2-based app and I just did a count of the .egg
packages in the virtualenv. There are 45 of them

People should really stop splitting their work into micro-libraries (with such
ludicrous names as "AddOns" or "Extremes", I might add (*)), and myriads of
separately-packaged plugins (the repoze stuff). The Twisted approach is much
saner, where you have a cohesive whole in a single package.


Monoliths have downsides:  consider that fact that the WSGI-complieant
HTTP server for Twisted languished for *years* outside the released
versions of Twisted:  IIRC, the server was released as a separate
distribution, but it was not compatible with the released versions of
the main Twisted distribution:  you had to install a snapshot / alpha of
Twisted to get the 'web2' server to work.


Maybe monoliths have downsides, but please pick a different example to
support this.  The manner in which the WSGI server in Twisted Web2 was
made available has very little to do with large vs small packages and
much to do with our (the Twisted developers) decision /not/ to release
Twisted Web2 at all.  I could go into lots more detail about that
decision, but I don't think any of i  would be relevant to the topic at
hand.

If anything, Twisted's example shows how monolithic packages are easier
all-around than micro-packages.  We basically have the release infrastructure
to release Twisted in many smaller pieces, and we even do - but we only
make releases of all the smaller pieces simultaneously, we encourage people
to use the "Twisted" package which includes all the pieces, and we don't
do any testing of mixed versions of the smaller pieces because it would be
very difficult.

Further, we *have* done point releases of /all/ of Twisted to supply some
critical bug fix (these generally take the form of a X.Y.1 release, we
rarely go to .2 or higher these days).  And we've done these quite rapidly
when the need arises - the monolithic nature of Twisted isn't really a
hindrance here (updating our website, a manual process, is far more time
consuming than actually doing the release - and a big part of that cost
comes from the fact that we have web pages for each smaller piece, even
though we don't encourage people to use these!).

So, as long as we're just talking about the vagueries of "monolithic" vs
"micro", I'll weigh in in favr of monolithic (but I'm open to discussion
about specific issues which are much more interesting than abstractions).

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Test failures under Windows?

[Jean-Paul Calderone]

On Tue, 24 Mar 2009 13:49:28 + (UTC), Antoine Pitrou  
wrote:

Hello,

[snip]
By the way, what happened to the Windows buildbots?


It looks like some of them are suffering from problems which I think are
common with buildbot on Windows - primarily difficulty dealing with runaway
processes or timeouts.  Perhaps BuildBot/Windows improvements would make a
good GSoC project? :)

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] asyncore fixes in Python 2.6 broke Zope's version of medusa

[Jean-Paul Calderone]

On Wed, 4 Mar 2009 10:54:19 -0800, Guido van Rossum  wrote:

On Wed, Mar 4, 2009 at 10:51 AM, Jean-Paul Calderone  wrote:

On Wed, 4 Mar 2009 10:46:28 -0800, Guido van Rossum 
wrote:


On Wed, Mar 4, 2009 at 10:27 AM, Jean-Paul Calderone 
wrote:
[snip]


So, as a disinterested party in this specific case, I'd say revert to
the pre-2.6 behavior.  It does less harm than leaving the current
behavior.


Sorry, but I really do think that we should maintain backward
compatibility *within* the 2.6 series as well. If that makes it
impossible to also maintain the 2.5 behavior, perhaps some flag could
be added to restore 2.5 compatibility, e.g.

import asyncore
asyncore.python_25_compat = True

Note that this "API" is designed to work in 2.5 as well. :-)



But why?  The argument I made had the objective of minimizing developer
effort.  What's the objective of maintaining backward compatibility within
the 2.6 series in this case (sorry if it appeared earlier in this thread
and I missed it)?


The same as always. We don't change APIs in bugfix releases.



Okay.  Thanks for explaining.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] asyncore fixes in Python 2.6 broke Zope's version of medusa

[Jean-Paul Calderone]

On Wed, 4 Mar 2009 10:46:28 -0800, Guido van Rossum  wrote:

On Wed, Mar 4, 2009 at 10:27 AM, Jean-Paul Calderone  wrote:
[snip]


So, as a disinterested party in this specific case, I'd say revert to
the pre-2.6 behavior.  It does less harm than leaving the current
behavior.


Sorry, but I really do think that we should maintain backward
compatibility *within* the 2.6 series as well. If that makes it
impossible to also maintain the 2.5 behavior, perhaps some flag could
be added to restore 2.5 compatibility, e.g.

import asyncore
asyncore.python_25_compat = True

Note that this "API" is designed to work in 2.5 as well. :-)



But why?  The argument I made had the objective of minimizing developer
effort.  What's the objective of maintaining backward compatibility within
the 2.6 series in this case (sorry if it appeared earlier in this thread
and I missed it)?

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] asyncore fixes in Python 2.6 broke Zope's version of medusa

[Jean-Paul Calderone]

On Wed, 4 Mar 2009 10:21:26 -0800, Guido van Rossum  wrote:

On Wed, Mar 4, 2009 at 10:14 AM, Sidnei da Silva
 wrote:

On Wed, Mar 4, 2009 at 3:04 PM, Guido van Rossum  wrote:

Sounds like it's not so much the code that's future proof but the
process used for evolving it. That seems to be missing for asyncore.
:-(


Turning the issue around a bit, has anyone considered polishing up the
current fix to restore it's backwards compatibility, instead of
starting a discussion about a full-blown replacement?

I think that would be enough for most asyncore users (or even the
couple few affected) for the moment, and then we can think about a
possible future replacement.


If it can be done while maintaining backwards compatibility with both
the 2.6 version and the pre-2.6 version, that would be great of
course! But can it?



Is it really necessary to retain compatibility with the Python 2.6
version?  Python 2.6.0 and Python 2.6.1 contain a regression (as
compared to basically all previous versions of Python) which prevents
asyncore-based programs which are years old from working on them.
Restoring the pre-2.6 behavior will fix these old, presumably stable,
widely used programs for users who install 2.6.2 and newer.

The downside (which you were imagining, I'm sure) is that any new
software developed against the Python 2.6.0 or 2.6.1 behavior will
then break in 2.6.2 and later.  While this is unfortunate, it is
clearly the far lesser of two evils.  The choice must be made, though.
Either leave old software broken or break new software.  Just because
the "leave old software broken" choice is made through inaction doesn't
make it the better choice (though obviously since it requires action,
someone will have to do it, and I'm not volunteering - if inaction is
the choice because no one wants to do the work, fine, but that's a
different motivation than avoiding breaking newly written software).

So, as a disinterested party in this specific case, I'd say revert to
the pre-2.6 behavior.  It does less harm than leaving the current
behavior.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 372 -- Adding an ordered directory to collections ready for pronouncement

[Jean-Paul Calderone]

On Sun, 1 Mar 2009 19:13:27 + (UTC), Armin Ronacher 
 wrote:

Hi everybody,

PEP 372 was modified so that it provides a simpler API (only the dict API
to be exact) and it was decided to start with a Python-only implementation
and replace it with a C version later if necessary.

Annotated changes from earlier versions of the PEP:

-   the extra API for ordered dict was dropped to keep the interface
   simple and clean.  Future versions can still be expanded but it's
   impossible to drop features later on.


Keeping the API simple and clean sounds great.  I'm all in favor of this.
However, it does no one a service to continue to propagate the idea that
all the code written for Python always has to be perfect.  It's feasible
and even simple to drop features later on, should it turn out to be that
they are not desirable.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Choosing a best practice solution for Python/extension modules

[Jean-Paul Calderone]

On Sat, 21 Feb 2009 11:07:07 -0800, Brett Cannon  wrote:

On Sat, Feb 21, 2009 at 09:17, Jean-Paul Calderone wrote:


On Fri, 20 Feb 2009 13:45:26 -0800, Brett Cannon  wrote:


On Fri, Feb 20, 2009 at 12:53, Aahz  wrote:

 On Fri, Feb 20, 2009, Brett Cannon wrote:

> On Fri, Feb 20, 2009 at 12:37, Brett Cannon  wrote:
>> On Fri, Feb 20, 2009 at 12:31, Daniel Stutzbach <
>> dan...@stutzbachenterprises.com> wrote:
>>>
>>> A slight change would make it work for modules where only key
functions
>>> have been rewritten.  For example, pickle.py could read:
>>>
>>> from _pypickle import *
>>> try: from _pickle import *
>>> except ImportError: pass
>>
>> True, although that still suffers from the problem of overwriting
things
>> like __name__, __file__, etc.
>
> Actually, I take that back; the IMPORT_STAR opcode doesn't pull in
anything
> starting with an underscore. So while this alleviates the worry above,
it
> does mean that anything that gets rewritten needs to have a name that
does
> not lead with an underscore for this to work. Is that really an
acceptable
> compromise for a simple solution like this?

Doesn't __all__ control this?




If you define it, yes.

But there is another issue with this: the pure Python code will never call
the extension code because the globals will be bound to _pypickle and not
_pickle. So if you have something like::

 # _pypickle
 def A(): return _B()
 def _B(): return -13

 # _pickle
 def _B(): return 42

 # pickle
 from _pypickle import *
 try: from _pickle import *
 except ImportError: pass

If you import pickle and call pickle.A() you will get -13 which is not
what
you are after.



If pickle and _pypickle are both Python modules, and _pypickle.A is
intended
to be used all the time, regardless of whether _pickle is available, then
there's not really any reason to implement A in _pypickle.  Just implement
it
in pickle.  Then import whatever optionally fast thing it depends on from
_pickle, if possible, and fall-back to the less fast thing in _pypickle
otherwise.

This is really the same as any other high-level/low-level library split.
 It
doesn't matter that in this case, one low-level implementation is provided
as
an extension module.  Importing the low-level APIs from another module and
then using them to implement high-level APIs is a pretty common, simple,
well-understood technique which is quite applicable here.



But that doesn't provide a clear way, short of screwing with sys.modules, to
get at just the pure Python implementation for testing when the extensions
are also present. The key point in trying to figure this out is to
facilitate testing since the standard library already uses the import *
trick in a couple of places.


"screwing with sys.modules" isn't a goal.  It's a means of achieving a goal,
and not a particularly good one.

I guess I overedited my message, sorry about that.  Originally I included
an example of how to parameterize the high-level API to make it easier to
test (or use) with any implementation one wants.  It went something like
this:

   try:
   import _pickle as _lowlevel
   except ImportError:
   import _pypickle as _lowlevel

   class Pickler:
   def __init__(self, implementation=None):
   if implementation is None:
   implementation = _lowlevel
   self.dump = implementation.dump
   self.load = implementation.load
   ...

Perhaps this isn't /exactly/ how pickle wants to work - I haven't looked at
how the C extension and the Python code fit together - but the general idea
should apply regardless of those details.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Choosing a best practice solution for Python/extension modules

[Jean-Paul Calderone]

On Fri, 20 Feb 2009 13:45:26 -0800, Brett Cannon  wrote:

On Fri, Feb 20, 2009 at 12:53, Aahz  wrote:


On Fri, Feb 20, 2009, Brett Cannon wrote:
> On Fri, Feb 20, 2009 at 12:37, Brett Cannon  wrote:
>> On Fri, Feb 20, 2009 at 12:31, Daniel Stutzbach <
>> dan...@stutzbachenterprises.com> wrote:
>>>
>>> A slight change would make it work for modules where only key functions
>>> have been rewritten.  For example, pickle.py could read:
>>>
>>> from _pypickle import *
>>> try: from _pickle import *
>>> except ImportError: pass
>>
>> True, although that still suffers from the problem of overwriting things
>> like __name__, __file__, etc.
>
> Actually, I take that back; the IMPORT_STAR opcode doesn't pull in
anything
> starting with an underscore. So while this alleviates the worry above, it
> does mean that anything that gets rewritten needs to have a name that
does
> not lead with an underscore for this to work. Is that really an
acceptable
> compromise for a simple solution like this?

Doesn't __all__ control this?



If you define it, yes.

But there is another issue with this: the pure Python code will never call
the extension code because the globals will be bound to _pypickle and not
_pickle. So if you have something like::

 # _pypickle
 def A(): return _B()
 def _B(): return -13

 # _pickle
 def _B(): return 42

 # pickle
 from _pypickle import *
 try: from _pickle import *
 except ImportError: pass

If you import pickle and call pickle.A() you will get -13 which is not what
you are after.


If pickle and _pypickle are both Python modules, and _pypickle.A is intended
to be used all the time, regardless of whether _pickle is available, then
there's not really any reason to implement A in _pypickle.  Just implement it
in pickle.  Then import whatever optionally fast thing it depends on from
_pickle, if possible, and fall-back to the less fast thing in _pypickle
otherwise.

This is really the same as any other high-level/low-level library split.  It
doesn't matter that in this case, one low-level implementation is provided as
an extension module.  Importing the low-level APIs from another module and
then using them to implement high-level APIs is a pretty common, simple,
well-understood technique which is quite applicable here.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Warnings

[Jean-Paul Calderone]

On Thu, 5 Feb 2009 08:35:30 -0800, Raymond Hettinger  wrote:

import os

os.tmpnam()

RuntimeWarning: tmpnam is a potential security risk to your program



This warning is a reflection of the fact that (at least) the glibc authors
think you shouldn't be using tmpnam(2).  If you compile a C program that
uses it, you'll see a warning about it.  Since you can write a Python
program that uses tmpnam(2) without ever compiling such a C program,
you get a RuntimeWarning instead.  It's not quite analogous, since
you don't get the warning from the C program every time you run it,
but it's about as close as you can do in Python without resorting to
crazy tricks.



Are these runtime warnings necessary?  Suppressing these warnings is a pita 
for one-off uses of os.tmpnam() or os.tempnam().


Why are you using them?  Why not just use one of the many, many, many
other APIs for generating temporary files that Python exposes?  One of
the ones that doesn't emit any warnings?

I would hate for this sort of thing to propagate throughout the standard 
library.  Some folks think eval() should never be used and the same for 
input().  Some folks think md5 should be removed.  Some folks think pickles 
are the ultimate security threat.  IMO, it is enough to note potential 
vulnerabilities in the docs.  Even then, I'm not too keen on the docs being 
filled with lots of red-outlined pink-boxed warning signs, effectively 
communicating that Python itself is dangerous and unreliable.




I agree.  The best thing to do would be to deprecate the Python wrappers
around insecure C functions and then remove them after a couple releases.
It's not as though these functions fill a critical niche - the tempfile
module and even os.tmpfile are more than enough.  Why does Python offer
this attractive nuisance?

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Missing operator.call

[Jean-Paul Calderone]

On Wed, 4 Feb 2009 10:50:47 -0800, Brett Cannon  wrote:

On Wed, Feb 4, 2009 at 10:43, Steven Bethard  wrote:

[snip]

Not sure I follow you here. It's not the __init__ that allows you to
do ``x()``, it's the fact that the class declares a __call__, right?


class C(object):

... pass
...

C.__call__()

<__main__.C object at 0x01A3C370>

C()

<__main__.C object at 0x02622EB0>

str.__call__()

''

str()

''



I don't think so::


Foo.__call__



Foo.__call__ = lambda: None
Foo.__call__

>

Foo()

<__main__.Foo object at 0xf7f90e8c>


That's because the __call__ special on an instance is ignored, as many
specials on new-style instances are ignored.  If you change the method
where it counts - on type(Foo) in this case - then you see something
different.

   >>> class X(type):
   ... def __call__(self, *a, **kw):
   ... print 'X.__call__', a, kw
   ... return super(X, self).__call__(*a, **kw)
   ...
   >>> class Y(object):
   ... __metaclass__ = X
   ... 
   >>> Y.__call__

   >
   >>> Y()
   X.__call__ () {}
   <__main__.Y object at 0xb7d0706c>
   >>> Y.__call__ = lambda: None
   >>> Y.__call__
   >
   >>> Y()
   X.__call__ () {}
   <__main__.Y object at 0xb7d0706c>
   >>> X.__call__ = lambda: None
   >>> Y()
   Traceback (most recent call last):
 File "", line 1, in 
   TypeError: () takes no arguments (1 given)
   >>>

As far as I know, Steven Bethard's point is correct.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Summary of Python tracker Issues

[Jean-Paul Calderone]

On Fri, 30 Jan 2009 18:06:48 +0100 (CET), Python tracker 
 wrote:

[snip]

Average duration of open issues: 697 days.
Median duration of open issues:   6 days.


It seems there's a bug in the summary tool.  I thought it odd a few
weeks ago when I noticed the median duration of open issues was one
day.  I just went back and checked and the week before it was one
day it was 2759 days.  Perhaps there is some sort of overflow problem
when computing this value?

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.0.1 (io-in-c)

[Jean-Paul Calderone]

On Wed, 28 Jan 2009 18:52:41 +, Paul Moore  wrote:

2009/1/28 "Martin v. Löwis" :

Well, first try to understand what the error *is*:

py> unicodedata.name('\u0153')
'LATIN SMALL LIGATURE OE'
py> unicodedata.name('£')
'POUND SIGN'
py> ascii('£')
"'\\xa3'"
py> ascii('£'.encode('cp850').decode('cp1252'))
"'\\u0153'"

So when Python reads the file, it uses cp1252. This is sensible - just
that the console uses cp850 doesn't change the fact that the "common"
encoding of files on your system is cp1252. It is an unfortunate fact
of Windows that the console window uses a different encoding from the
rest of the system (namely, the console uses the OEM code page, and
everything else uses the ANSI code page).


Ah, I see. That is entirely obvious. The key bit of information is
that the default io encoding is cp1252, not cp850. I know that in
theory, I see the consequences often enough (:-)), but it isn't
"instinctive" for me. And the simple "default encoding is system
dependent" comment is not very helpful in terms of warning me that
there could be an issue.


It probably didn't help that the exception raised told you that the
error was in the "charmap" codec.  This should have said "cp850"
instead.  The fact that cp850 is implemented in terms of "charmap"
isn't very interesting.  The fact that while encoding some text
using "cp850" is.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] r68547 - in python/trunk/Lib/test: test_datetime.py test_os.py

[Jean-Paul Calderone]

On Mon, 12 Jan 2009 19:09:28 +0100 (CET), "kristjan.jonsson" 
 wrote:

Author: kristjan.jonsson
Date: Mon Jan 12 19:09:27 2009
New Revision: 68547

Log:
Add tests for invalid format specifiers in strftime, and for handling of 
invalid file descriptors in the os module.

Modified:
  python/trunk/Lib/test/test_datetime.py
  python/trunk/Lib/test/test_os.py


Several of the tests added to test_os.py are invalid and fail.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RELEASED Python 3.0 final

[Jean-Paul Calderone]

On Thu, 4 Dec 2008 22:05:05 -0800, Guido van Rossum <[EMAIL PROTECTED]> wrote:

On Thu, Dec 4, 2008 at 9:40 PM,  <[EMAIL PROTECTED]> wrote:

The default case, the case of the user without the wherewithal
to understand the nuances of the distinction between 2.x and 3.x, is a user
who should use 2.x.


Not at all clear. If they're not sensitive to those nuances it's just
as likely that they're a casual developer (e.g. a student just
learning to program). Such users are unlikely to start using major 3rd
party packages like Twisted or Django, which would be completely
overwhelming to someone just learning.


That seems like it would be right to me, but two or three times a month
someone shows up in the Twisted IRC channel who is learning both Python
and Twisted at the same time.  So apparently there are a lot of people
for whom this isn't overwhelming.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RELEASED Python 3.0 final

[Jean-Paul Calderone]

On Thu, 4 Dec 2008 20:20:34 +, Paul Moore <[EMAIL PROTECTED]> wrote:

2008/12/4 Barry Warsaw <[EMAIL PROTECTED]>:
[snip]

One thing I'd like to see more clearly stated is that there's no
reason NOT to use Python 3.0 for new code. I don't think that message
has really come across yet - in spite of the warnings being all about
compatibility issues, no-one has stressed the simple point that if
your code is new, it doesn't have compatibility concerns!


New code that wouldn't be more easily written with a dependency on a
library that hasn't been ported, you mean.

Although beyond that, there may be reasons (for example, the significant
performance degradation in the I/O library currently being discussed on
python-list).

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] My patches

[Jean-Paul Calderone]

On Thu, 30 Oct 2008 17:17:02 -0400, "A.M. Kuchling" <[EMAIL PROTECTED]> wrote:

[snip]

On some of my issues (esp. ones relating to curses and mailbox.py), I
feel paralyzed because problems are occurring on platforms I don't
have access to (e.g. FreeBSD).  The buildbots will report problems,
but then you have to debug them by committing changes, triggering a
build, and observing the results.  And all of these actions will send
e-mail to python-checkins.  (Imagine if every 'print "reached here!"'
you added while debugging was e-mailed to everyone...)


I do that when I need to.  People whose lives would be ruined by the
receipt of such an email always have the option of leaving the checkin
list.

However, there is a buildbot feature named "try" which lets you submit
a patch (subject to authentication) and performs a build with the patch
applied.  This lets you try lots of little changes without getting your
VCS involved.  It needs to be enabled in the buildmaster configuration
and credentials created for any user who will be given access.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Filename as byte string in python 2.6 or 3.0?

[Jean-Paul Calderone]

On Mon, 29 Sep 2008 14:34:07 +0200, Ulrich Eckhardt <[EMAIL PROTECTED]> wrote:

On Monday 29 September 2008, [EMAIL PROTECTED] wrote:

Also, what about MacOS X?


AFAIK, OS X guarantees UTF-8 for filesystem encodings. So the OS also provides
Unicode filenames and how it deals with broken or legacy media is left up to
the OS.


Read Jack Jansen's recent email about NFC vs NFD.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What this code should do?

[Jean-Paul Calderone]

On Fri, 19 Sep 2008 18:26:05 +0200, Amaury Forgeot d'Arc <[EMAIL PROTECTED]> 
wrote:

Hello Maciej,

Maciej Fijalkowski wrote:

Hello,

I'm a little clueless about exact semantics of following snippets:

http://paste.pocoo.org/show/85698/

is this fine?
or shall I fill the bug?
(the reason to ask is because a) django is relying on this b) pypy
implements it differently)


Note that python 3.0 has a different behaviour; in the first sample, it prints:
A ( ...
B (, ...

See the subtle differences between
http://docs.python.org/dev/library/sys.html#sys.exc_info
http://docs.python.org/dev/3.0/library/sys.html#sys.exc_info



The second example changes its behavior, too.  It gives back the NameError
from the exc_info call.  I'm having a hard time reconciling this with the
Python 3.0 documentation.  Can you shed some light?

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] ssl module, non-blocking sockets and asyncore integration

[Jean-Paul Calderone]

On Wed, 17 Sep 2008 10:40:01 PDT, Bill Janssen <[EMAIL PROTECTED]> wrote:

Ah, now I remember.  It seems that sometimes when SSL_ERROR_WANT_READ
was returned, things would block; that is, the "handle_read" method on
asyncore.dispatcher was never called again, so the SSLSocket.recv()
method was never re-called.  There are several levels of buffering going
on, and I never figured out just why that was.  This (very rare) re-call
of "read" is to handle that.



You certainly do need to call read again if OpenSSL fails an SSL_read with
a want-read error, but in asyncore, you don't want to do it right away,
you want to wait until the socket becomes readable again, otherwise you *do*
block waiting for bytes from the network.  See the SSL support in Twisted
for an example of the correct way to handle this.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.6 rc1 performance results

[Jean-Paul Calderone]

On Sat, 13 Sep 2008 08:03:50 -0400, "A.M. Kuchling" <[EMAIL PROTECTED]> wrote:

Three weeks ago, Antoine Pitrou posted the pybench results
for 2.6 trunk:
http://mail.python.org/pipermail/python-dev/2008-August/081951.html

The big discovery in those results were TryExcept being 48% slower,
but there was a patch in the bug tracker to improve things.  I've
re-run the tests to check the results.

Disclaimer: these results are probably not directly comparable.
Antoine was using a "32-bit Linux installation on an Athlon 3600+ X2";
I'm on a Macbook.

Good news: TryExcept is now only 10% slower than 2.5, not 48%.

Bad news: the big slowdowns are:

CompareFloats:   117ms98ms  +19.2%   118ms99ms  +19.0%
  CompareIntegers:   110ms   104ms   +5.6%   110ms   105ms   +4.9%
   DictWithStringKeys:   118ms   105ms  +12.8%   133ms   108ms  +22.7%
   NestedForLoops:   125ms   116ms   +7.7%   127ms   118ms   +8.0%
Recursion:   193ms   159ms  +21.5%   197ms   163ms  +20.8%
 SecondImport:   139ms   129ms   +8.4%   143ms   130ms   +9.9%
  SecondPackageImport:   150ms   139ms   +8.6%   152ms   140ms   +8.1%
SecondSubmoduleImport:   211ms   191ms  +10.5%   214ms   195ms   +9.4%
  SimpleComplexArithmetic:   130ms   119ms   +9.4%   131ms   120ms   +9.2%



I see similar results for some of these.  The complete results from a run
on an AMD Athlon(tm) 64 Processor 3200+ are attached.

Jean-Paul

---
PYBENCH 2.0
---
* using CPython 2.6rc1 (trunk:66421M, Sep 12 2008, 21:05:52) [GCC 4.2.3 (Ubuntu 
4.2.3-2ubuntu7)]
* disabled garbage collection
* system check interval set to maximum: 2147483647
* using timer: time.time

---
Benchmark: p26.pybench
---

Rounds: 10
Warp:   10
Timer:  time.time

Machine Details:
   Platform ID:Linux-2.6.24-19-generic-i686-with-debian-lenny-sid
   Processor:  

Python:
   Implementation: CPython
   Executable: /home/exarkun/Projects/python/trunk//python
   Version:2.6.0
   Compiler:   GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)
   Bits:   32bit
   Build:  Sep 12 2008 21:05:52 (#trunk:66421M)
   Unicode:UCS2


---
Comparing with: p25.pybench
---

Rounds: 10
Warp:   10
Timer:  time.time

Machine Details:
   Platform ID:Linux-2.6.24-19-generic-i686-with-debian-lenny-sid
   Processor:  

Python:
   Implementation: n/a
   Executable: 
/home/exarkun/Projects/python/branches/release25-maint/python
   Version:2.5.3a0
   Compiler:   GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)
   Bits:   32bit
   Build:  Sep 13 2008 09:32:41 (#release25-maint:66444)
   Unicode:UCS2


Test minimum run-timeaverage  run-time
 thisother   diffthisother   diff
---
  BuiltinFunctionCalls:   178ms   187ms   -4.5%   184ms   193ms   -4.6%
   BuiltinMethodLookup:   151ms   165ms   -8.5%   155ms   167ms   -7.2%
 CompareFloats:   150ms   146ms   +2.9%   153ms   150ms   +1.9%
 CompareFloatsIntegers:   143ms   147ms   -2.8%   150ms   150ms   +0.4%
   CompareIntegers:   180ms   182ms   -1.0%   182ms   190ms   -4.3%
CompareInternedStrings:   159ms   160ms   -1.1%   163ms   166ms   -2.0%
  CompareLongs:   135ms   136ms   -0.7%   136ms   139ms   -1.5%
CompareStrings:   142ms   150ms   -5.4%   146ms   153ms   -4.5%
CompareUnicode:   148ms   135ms   +9.6%   151ms   137ms  +10.6%
ComplexPythonFunctionCalls:   155ms   226ms  -31.4%   158ms   229ms  -30.9%
 ConcatStrings:   197ms   203ms   -2.8%   202ms   215ms   -6.4%
 ConcatUnicode:   179ms   168ms   +6.6%   182ms   184ms   -0.8%
   CreateInstances:   159ms   157ms   +1.4%   162ms   161ms   +0.7%
CreateNewInstances:   119ms   141ms  -15.4%   121ms   144ms  -16.2%
   CreateStringsWithConcat:   189ms   173ms   +9.3%   195ms   177ms  +10.2%
   CreateUnicodeWithConcat:   116ms   113ms   +2.3%   118ms   115ms   +2.6%
  DictCreation:   109ms   140ms  -22.2%   112ms   143ms  -21.8%
 DictWithFloatKeys:   202ms   199ms   +1.6%   208ms   204ms   +1.6%
   DictWithIntegerKeys:   158ms   156ms   +1.0%   161m

Re: [Python-Dev] Further PEP 8 compliance issues in threading and multiprocessing

[Jean-Paul Calderone]

On Mon, 1 Sep 2008 09:42:06 -0500, Benjamin Peterson <[EMAIL PROTECTED]> wrote:

On Mon, Sep 1, 2008 at 9:36 AM, Antoine Pitrou <[EMAIL PROTECTED]> wrote:

Nick Coghlan  gmail.com> writes:


Is this just intended to discourage subclassing? If so, why give the
misleading impression that these things can be subclassed by naming them
as if they were classes?

How should this be handled when it comes to the addition of PEP 8
compliant aliases?


I don't see a problem for trivial functional wrappers to classes to be
capitalized like classes. So I'd suggest option 3: leave it as-is. Otherwise
option 2 (replace the wrappers with the actual classes) has my preference.


Yes, I believe that pretending that functions are classes is a fairly
common idiom in the stdlib and out, so I see no problem leaving them
alone. We haven't had any complaints about the threading Event
function yet either. :)


Here's a complaint.  It's surprising that you can't use Event et al with
isinstance.  This is something I'm sure a lot of people run into (I did,
many years ago) when they start to use these APIs.  Once you do figure
out why it doesn't work, it's not clear how to do what you want, since
_Event is private.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subprocess insufficiently platform-independent?

[Jean-Paul Calderone]

On Mon, 25 Aug 2008 10:13:32 -0700, Guido van Rossum <[EMAIL PROTECTED]> wrote:

Several people at Google seem to have independently discovered that
despite all of the platform-independent goodness in subprocess.py, you
still need to be platform aware. One of my colleagues summarized it
like this:

"""
Given a straightforward command list like:

   cmd = ['svn', 'ls', 'http://rietveld.googlecode.com/svn/trunk']



Launching child processes on Windows is actually pretty hard to do.
The idea of an array of arguments is more like a group hallucination
than a reality there.  If you assume certain things about how the
launched process will interpret its command line (something each
program gets to decide for itself on Windows), and many programs do
actually use the same rules, then you can get something together that
mostly works and is mostly general.

Twisted's process support does this and presents a cross-platform API
(at least as far as arguments are concerned).  Perhaps the subprocess
module should borrow that implementation?

http://twistedmatrix.com/trac/browser/trunk/twisted/python/win32.py#L66

This doesn't handle all possible inputs correctly (you can read about its
flaws in some detail at ) but
it does handle *most* inputs.  It also isn't appropriate for all programs,
but for the programs which don't follow these quoting rules, it's probably
a mistake to have a list-based API anyway: they should be invoked using one
string giving the entire command line.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Think a dead import finding script would be handy?

[Jean-Paul Calderone]

On Sun, 17 Aug 2008 15:04:58 -0700, Brett Cannon <[EMAIL PROTECTED]> wrote:

On Sun, Aug 17, 2008 at 1:40 PM, Georg Brandl <[EMAIL PROTECTED]> wrote:

Brett Cannon schrieb:

After Christian mentioned how we could speed up interpreter start-up
by removing some dead imports he found, I decided to write up a quick
script that generates the AST for a source file and (very roughly)
tries to find imports that are never used. People think it's worth
tossing into Tools, even if it is kind of rough? Otherwise I might
toss it into the sandbox or make a quick Google code project out of
it.

Regardless, one interesting side-effect of the script is that beyond
finding some extraneous imports in various places, it also found some
holes in __all__. I have the script look for the definition of __all__
and consider an import used if it is listed there.


pylint already finds unused imports. It finds tons of other, relatively
useless, stuff in the default configuration, but I'm sure it can be
coaxed into only showing unused imports too.



Does anyone ever run pylint over the stdlib on occasion?



Buildbot includes a pyflakes step.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Memory tests in Unicode

[Jean-Paul Calderone]

On Sat, 16 Aug 2008 13:01:33 -0300, Facundo Batista <[EMAIL PROTECTED]> wrote:

2008/8/16 Antoine Pitrou <[EMAIL PROTECTED]>:


If the test does allocate the very large string, it means MemoryError isn't
raised, which defeats the purpose of the test.


I do *not* want to remove the test.


Antoine wasn't suggesting removing it.  He's suggesting that the test
is not accomplishing its goal if the "except" suite isn't executed,
and so the test should be changed to make this failure noticable.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] unittest Suggestions

[Jean-Paul Calderone]

On Wed, 13 Aug 2008 15:35:15 + (UTC), Antoine Pitrou <[EMAIL PROTECTED]> 
wrote:

Barry Warsaw  python.org> writes:

The goal
should be to produce something like a unittest-ng, distribute it via
the Cheeseshop, and gather consensus around it for possible inclusion
in Python 2.7/3.1.


There is already unittest, nose, py.test, trial... perhaps others I don't know
of. I fear writing yet another testing framework from the ground-up will lead to
more bikeshedding and less focussed discussion (see some testing-in-python
threads for an example :-)).

nose itself is not a completely independent piece of work but "a discovery-based
unittest extension" (although a very big extension!). For that reason, Michael
Foord's suggestion to gradually modernize and improve the stdlib unittest sounds
reasonable to me: it allows to be more focussed, keep backwards compatibility,
and also to decide and implement changes piecewise - avoiding the blank sheet
effect where people all push for wild ideas and radically new concepts (tm).

(however, nose is LGPL-licensed so it would not be suitable for direct reuse of
large chunks of code in the stdlib, unless the authors agree for a relicensing)



trial is also an extension of the stdlib unittest module (less and less
over time as more and more stdlib unittest changes break it).  Incremental
improvements with backwards compatibility are a great thing.  I very
strongly encourage that course of action.  It has already happened a number
of times in this thread that some proposed functionality already exists in
some third-party unittest extension and could easily be moved into the
stdlib unittest module.  That's a good thing: it shows that the functionality
is actually valuable and it makes it easy to include, since it's already
implemented.

For what it's worth, trial is MIT licensed; anyone should feel free to grab
any part of it they like for any purpose.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] unittest Suggestions

[Jean-Paul Calderone]

On Tue, 12 Aug 2008 11:05:57 -0400, Barry Warsaw <[EMAIL PROTECTED]> wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Aug 12, 2008, at 10:30 AM, Sebastian Rittau wrote:

[I just saw the other post about unit testing, while I was writing  this.
A strange conincidence.]


Indeed.  I've played around (again) recently with both nose and  py.test, so 
I'd like to make a meta comment.


I would really like to see some of the people who are interested in  Python 
unit testing to get together and work on an updated testing  framework that 
incorporates the best ideas from all the existing  frameworks.  I'd like to 
see good integration with setuptools, both  for running the tests and for 
packaging.  I'd like to see good doctest  support, with the ability to hook 
in setups and teardowns.  I'd like  to see some of useful things like layers 
taken from zope.testing.


This doesn't belong on python-dev, and probably not on python-ideas  either, 
but I'd be willing to start a testing SIG on python.org if  others are 
interested in getting together for this purpose.  The goal  should be to 
produce something like a unittest-ng, distribute it via  the Cheeseshop, and 
gather consensus around it for possible inclusion  in Python 2.7/3.1.


A SIG might be a good idea.  There's also already the "testing in python"
list, too:

 http://lists.idyll.org/listinfo/testing-in-python

A lot of this discussion would be appropriate there.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposed unittest changes

[Jean-Paul Calderone]

On Sun, 13 Jul 2008 23:51:44 +0100, Michael Foord <[EMAIL PROTECTED]> wrote:

Ben Finney wrote:

Howdy Michael,

I'm interested in the changes you're proposing for Python's 'unittest' 
module. I am (like, I suspect, many Python coders) maintaining my own set 
of extensions to the module across many projects, so I'd really like to see 
many of the improvements you discuss actually in the standard library.


What assistance can I offer to help on this issue?


I intend to start working on them in August, after I have finished my 
current writing commitments.


The full list of changes proposed (feel free to start - but ping me or the 
list) and not shot down was something like:


Documenting that the assert method names are to be preferred over the 
'FailUnless' names (this stirred up some controversy this weekend so should 
probably not happen).



Adding the following new asserts:

   assertIn(member, container, msg=None)
   assertNotIn (member, container, msg=None)
   assertIs (first, second, msg=None)
   assertNotIs   (first, second, msg=None)
   assertRaisesWithMessage(exc_class, message, callable, *args, 
**keywargs)




Several of these are implemented in other libraries (Twisted, at least).
You might save some time by grabbing them and their unit tests, rather
than re-implementing them.  Twisted calls `assertIs´ `assertIdentical´,
by the way.


[snip]

Other suggestions that weren't controversial but I might not get to:

   assertRaisesWithMessage taking a regex to match the error message



Actually, I remember that someone raised an object to this as being not
as flexible as some might want - an objection I agree with.  Perhaps that
was overruled, but I didn't want this to slip by as "not controversial".

   expect methods that collect failures and report at the end of the test 
(allowing an individual test method to raise several errors without 
stopping)


   assertIsInstance and assertIsSubclass



The former of these is also in Twisted already, if you want to copy it.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Community buildbots and Python release quality metrics

[Jean-Paul Calderone]

On Thu, 26 Jun 2008 21:46:48 +0200, Georg Brandl <[EMAIL PROTECTED]> wrote:

[snip]

As for reverting changes that break, I'd support this
only for changes that break *all* of them. For example, I only use one
platform to develop on (and I guess it's the same for many others), having
the buildbots go red on another platform means I can try to fix the issue.


BuildBot has two ways to let you run your code on all builders before you
commit it to trunk.  You can force a build on a branch or you can try a
build with a patch.  I don't know if these options are enabled on Python's
buildmaster.  If they are, then if you want, you can use them to make sure
your code works on all platforms before you put it into trunk, where it may
cause problems for someone else.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Py3k DeprecationWarning in stdlib

[Jean-Paul Calderone]

On Fri, 27 Jun 2008 01:52:18 +1000, Nick Coghlan <[EMAIL PROTECTED]> wrote:

Jean-Paul Calderone wrote:

I don't particularly care about the details, I just want some public
API for this.  Making warn_explicit public seems better to me, since
it was already there in previous versions of Python, and it lets you
completely ignore both the filters list and the global registry, but
if others would rather make the filters and global registry a part of
the public API, that's fine by me as well.


Why do you say warn_explicit isn't public? It's in both the 2.5 and 2.6 API 
docs for the warnings module.


Brett told me it was private (on this list several weeks or a month or so
ago).  It's also no longer called in 2.6 by the C implementation of the
warning system.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Py3k DeprecationWarning in stdlib

[Jean-Paul Calderone]

On Fri, 27 Jun 2008 01:14:36 +1000, Nick Coghlan <[EMAIL PROTECTED]> wrote:

Jean-Paul Calderone wrote:
[snip]

The real problem with testing many uses of the warning system is that it
doesn't expose enough public APIs for this to be possible.  You *have*
to use APIs which are, apparently, private (such as warn_explicit).


Hmm, I think the bigger problem is that there is no documented way to save 
the warnings filter and restore it to a previous state - the 'filters' 
attribute (which holds the actual list of filters) isn't documented and 
isn't included in __all__. This makes it hard to write an officially 
supported test that fiddles with the warning settings then puts them back 
the way they were.


It sounds like you're agreeing that there aren't enough public APIs.
Making warn_explicit public addresses this particular problem in one
way - by letting applications hook into the warning system before
filters are applied.  Making the filter list public is another way,
since it would let applications clear and then restore the filters.

I don't particularly care about the details, I just want some public
API for this.  Making warn_explicit public seems better to me, since
it was already there in previous versions of Python, and it lets you
completely ignore both the filters list and the global registry, but
if others would rather make the filters and global registry a part of
the public API, that's fine by me as well.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Py3k DeprecationWarning in stdlib

[Jean-Paul Calderone]

On Thu, 26 Jun 2008 23:56:23 +1000, Nick Coghlan <[EMAIL PROTECTED]> wrote:


[snip]


Ok, then we're back to there being no supported way to write tests that 
need to
intercept warnings.  Twisted has already suffered from this (JP reports 
that

Twisted's assertWarns is broken in 2.6), and I doubt it's alone.

So I guess I am filing a bug after all... :)


Yeah - Brett's correct that everything under "test.test_support" should 
really be formally undocumented. It's mostly a place for code that reflects 
"things we do a lot in our unit tests and are tired of repeating" rather 
than "this is a good API that we want to support forever and encourage other 
people to use".


However, if other folks turn out to have similar needs, then it may be 
possible to add something to unittest to support it. However, given that the 
beta deadline has already passed, you may need to use similar hackery to 
that used by catch_warning and replace warnings.showwarning with a test 
function that saves the raised exception (it also wouldn't be hard to 
enhance it a bit to handle more than a single warning).


We don't use showwarning because in order to reliably catch warnings that
way, it's necessary to rely on even more private implementation details of
the warning system.

Consider this:

   from warnings import warn
   from test.test_support import catch_warning

   def f():
   warn("foo")

   def test():
   with catch_warning() as w:
   f()
   assert str(w.message) == "foo", "%r != %r" % (w.message, "foo")

   test()
   test()

The first assertion passes (by the way, I don't understand why w.message
isn't the message passed to warn, but is instead an instance of UserWarning)
but the second assertion fails.  A more subtle example might include two
functions, the first of which is deprecated and called by the second, and
one test for each of them.  Now the test for the warning will only pass if
it runs before the other test; if they accidentally run in the other order,
you won't see the warning, so as far as I can tell, you can't reliably write
a unit test for warnings using catch_warning.

The real problem with testing many uses of the warning system is that it
doesn't expose enough public APIs for this to be possible.  You *have*
to use APIs which are, apparently, private (such as warn_explicit).

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Py3k DeprecationWarning in stdlib

[Jean-Paul Calderone]

On Tue, 24 Jun 2008 23:03:33 -, [EMAIL PROTECTED] wrote:

On 10:05 pm, [EMAIL PROTECTED] wrote:
We need to be especially careful with the unit test suite itself - changing 
the test code to avoid the warning will normally be the right answer, but 
when the code is actually setting out to test the deprecated feature we 
need to suppress the warning in the test suite instead.


This is a dangerous road to go down.  If you suppress warnings in the test 
suite, you might suppress additional warnings which should actually be 
reported.  Or, if the API gets modified in some way that the warning is 
supposed to be emitted but isn't any longer, it will be silent.  It's easy 
to accidentally suppress too much or not enough.


The way we've dealt with this in Twisted is adding an 'assertWarns' method 
so that we can invoke an API that is supposed to generate a warning, and (A) 
that warning and only that *specific* warning will not be emitted, and (B) 
if the API stops emitting the warning in the future, the test will fail and 
we will notice.


It's also nice to have this facility in the test harness itself, so that you 
don't run the additional risk of accidentally (and silently) leaving warning 
suppression in place for subsequent tests.


It would be *extra* nice to have this facility added to the standard
library, since assertWarns in Twisted is broken by changes in Python
2.6 (ie, our tests for warnings all fail with [EMAIL PROTECTED]).  For now,
we will probably address this by switching to a different warning
API.  In the long term, it'd be better for us, other Python developers,
and the standard library if there were an API in the standard library
which facilitated testing of warnings.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] test_multiprocessing: test_listener_client flakiness

[Jean-Paul Calderone]

On Wed, 18 Jun 2008 18:35:26 +0200, Amaury Forgeot d'Arc <[EMAIL PROTECTED]> 
wrote:

[snip]

I just found the cause of the problem ten minutes ago:
It seems that when a socket listens on the address "127.0.0.1" or
"localhost", another process cannot connect to it using the machine's
name (even from the same machine).


That's because when you try to connect to A:B you won't connect to a
server listening on X:B - somewhat by design.  Changing the test to
listen on A:B and X:B might fix it, but so would changing it to connect
to the same address it listens to.


The best seems to listen with the empty address "".


This will cause it to listen on all available interfaces, including
public ones.  It's rather unlikely that someone from the internet
will connect to the port while the test suite is running and use it
to do terrible things to you, but it's not /impossible/.

I'd suggest changing the client to connect to "127.0.0.1" or "localhost",
instead.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Interesting blog post by Ben Sussman-Collins

[Jean-Paul Calderone]

On Fri, 13 Jun 2008 18:22:42 -0400, Barry Warsaw <[EMAIL PROTECTED]> wrote:

[snip]

* small branches - we have a strict limit on diffs no greater than 800 
lines.  Yes we have exceptions, but they are rare and pre-arranged.   Having 
such a strict limit really forces you to be disciplined,  organized and very 
effectively diffuses code bombs.


* everyone can see (lots of) everyone else's code - this is great  because 
everyone needs some advice or guidance along the way.  If you  get stuck, 
you can push a branch and I can pull it and look at it, run  it, test it, 
even modify it and push my own branch for you to see.   This is /much/ more 
effective than trading patches, and I don't see  how this could even work 
without a dvcs.


* nothing lands without being reviewed - this is a hard and fast rule,  no 
exceptions.  Someone else has to review your code, and most  developers are 
also reviewers (we have a mentoring program to train  new reviewers).  You 
get over the fear pretty quickly, and learn /a  lot/ both by reviewing and 
getting reviewed.  Coding standards emerge,  best practices are established, 
and overall team productivity goes way  up.  Small branches are critical to 
this process, as is our goal of  reviewing every branch within 24 hours of 
its submission.


* nothing lands without passing all tests - speaking from experience,  this 
is the one thing I wish Python would adopt!  This means the trunk  is 
/always/ releasable and stable.  The trade-off is that it can take  quite a 
while for your branch to land once it's been approved, since  this process 
is serialized and is dependent on full test suite  execution time.  Python's 
challenge here is that what passes on one  platform does not necessarily 
pass on another.  Still, if this week is  any indication, passing on /any/ 
platform would be nice. ;)


I'm not saying Python can or should adopt these guidelines.  An open  source 
volunteer project is different than a corporate environment,  even if the 
latter is very open-source-y.  But it is worthwhile to  continually evaluate 
and improve the process because over time, you  definitely improve 
efficiency in ways that are happily adopted by the  majority of the 
community.


A big +1 on all these points.  I can also add that Twisted is developed
following many of these rules so it *can* work for open source volunteer
projects, if the developers want it to.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] segfault in struct module

[Jean-Paul Calderone]

   [EMAIL PROTECTED]:~$ ~/Projects/python/trunk/python
   Python 2.6a3+ (trunk:63964, Jun  5 2008, 16:49:12)
   [GCC 4.0.3 (Ubuntu 4.0.3-1ubuntu5)] on linux2
   Type "help", "copyright", "credits" or "license" for more information.
   >>> import struct
   >>> struct.pack("357913941c", 'a')
   Segmentation fault
   [EMAIL PROTECTED]:~$

The unit test for exactly this case was deleted in r60892.  I would like to
suggest that just deleting unit tests isn't a very good idea.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] GIL cpu usage problem, confirm me

[Jean-Paul Calderone]

On Sun, 8 Jun 2008 08:37:20 -0500, Benjamin Peterson <[EMAIL PROTECTED]> wrote:


Certainly not in core Python. Have a look
http://code.google.com/p/python-threadsafe/, though.



http://code.google.com/p/python-safethread/

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Addition of "pyprocessing" module to standard lib.

[Jean-Paul Calderone]

On Wed, 21 May 2008 20:57:33 +0200, "\"Martin v. Löwis\"" <[EMAIL PROTECTED]> 
wrote:

As said before, PyOpenGL is an example of an extension that moved from C
code to Python/ctypes, luckily we don't use it, but what if the
maintainers of MySQL-Python or cx_Oracle decide to move to ctypes.
Having the ctypes extension in the stdlib doesn't imply it runs on any
platform where python runs. Extension writers should keep this in mind
when they decide to use ctypes. They should document, that their
extension depends on ctypes and therefore doesn't run on platforms where
ctypes doesn't work.


Plus, even if ctypes works, the code might be incorrect, because they
had been assuming structure layouts and symbolic constants that have
just a different definition on some other platform, causing the
extension module to crash.

Writing portable ctypes modules is really hard - significantly harder
than writing portable C code (although writing non-portable ctypes
code is apparently easier than writing non-portable C code).


True.  There's some room for improvement in ctypes here, fortunately.

For example, PyPy has some tools which resolve the particular problem
you're talking about; the library is even available separately and can
(and probably should) be used by anyone writing a ctypes module.

Sample usage and installation instructions available here:

 http://codespeak.net/~fijal/configure.html

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Symbolic errno values in error messages

[Jean-Paul Calderone]

On Sat, 17 May 2008 00:15:23 +1000, Nick Coghlan <[EMAIL PROTECTED]> wrote:

Alexander Belopolsky wrote:

Yannick Gingras  ygingras.net> writes:

2) Where can I find the symbolic name in C?


Use standard C library char* strerror(int errnum) function.   You can see
an example usage in Modules/posixmodule.c (posix_strerror).


I don't believe that would provide adequate Windows support.



It's not C, but maybe it's interesting to look at anyway:

http://twistedmatrix.com/trac/browser/trunk/twisted/python/win32.py?rev=21682#L94

However, neither strerror nor the linked code gives out symbolic names for
errnos.  They both produce messages like "Interrupted system call", whereas
the symbolic name would be "EINTR".  Modules/errnomodule.c might be worth
looking at, although its solution is somewhat disappointing.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Community buildbots

[Jean-Paul Calderone]

Hi,

I just wanted to point out a few things:

Community 2.5 bots, 6 out of 8 offline, of the remaining two (which are both
red), one is actually using Python 2.6, not Python 2.5:

 http://python.org/dev/buildbot/community/2.5/

Community 2.6 bots, 6 out of 8 offline, but at least the remaining two (both
of which are red) seem to be using the correct Python version:

 http://python.org/dev/buildbot/community/trunk/

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] warnings.showwarning (was Re: [Python-3000] Reminder: last alphas next Wednesday 07-May-2008)

[Jean-Paul Calderone]

On Thu, 1 May 2008 19:31:20 -0700, Brett Cannon <[EMAIL PROTECTED]> wrote:


[snip]

I just closed the release blocker I created (the
backwards-compatibility issue with warnings.showwarning() ). I would
like to add a PendingDeprecationWarning (or stronger) to 2.6 for
showwarning() implementations that don't support the optional 'line'
argument. I guess the best way to do it in C code would be to see if
PyFunction_GetDefaults() returns a tuple of length two (since
showwarning() already has a single optional argument as it is).


Hi Brett,

I'm still seeing some strange behavior from the warnings module,  This
can be observed on the community buildbot for Twisted, for example:

http://python.org/dev/buildbot/community/trunk/x86%20Ubuntu%20Hardy%20trunk/builds/171/step-Twisted.zope.stable/0

The log ends with basically all of the warning-related tests in Twisted
failing, reporting that no warnings happened.

There is also some strange behavior that can be easily observed in the REPL:

   [EMAIL PROTECTED]:~/Projects/python/trunk$ ./python 
/home/exarkun/Projects/Divmod/trunk/Combinator/combinator/xsite.py:7: DeprecationWarning: the sets module is deprecated

 from sets import Set
   Python 2.6a2+ (trunk:62636M, May  2 2008, 09:19:41) 
   [GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)] on linux2

   Type "help", "copyright", "credits" or "license" for more information.
   >>> import warnings
   >>> warnings.warn("foo")
   :1: UserWarning: foo   # Where'd the module name go?
   >>> def f(*a):
   ... print a
   ... 
   >>> warnings.showwarning = f

   >>> warnings.warn("foo")
   >>># Where'd the warning go?

Any ideas on this?

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cStringIO buffer interface

[Jean-Paul Calderone]

On Wed, 30 Apr 2008 09:51:25 -0700, Guido van Rossum <[EMAIL PROTECTED]> wrote:

On Wed, Apr 30, 2008 at 9:36 AM, Farshid Lashkari <[EMAIL PROTECTED]> wrote:

 I was just curious as to why cStringIO objects don't implement the
 buffer interface.  cStringIO objects seem similar to string and array
 objects, and those support the buffer protocol. Is there a reason
 against allowing cStringIO to support at least the read buffer
 interface, or is just that nobody has considered it until now?


Well, for one, it would mean you could no longer exchange a StringIO
instance for a cStringIO instance.


It would probably only mean that there is one further incompatibility
between cStringIO and StringIO - you already can't exchange them in a
number of cases.  They handle unicode differently, they have different
methods, etc.

Maybe making them diverge even further is a step in the wrong direction,
though.


Also, what's the compelling use case you're thinking of?


I'm not sure what use-case Farshid Lashkari had.  For Twisted, it has
been considered as a way to reduce peak memory usage (by reducing the
need for memory copying, which also speeds things up).  I'm not sure
if anyone has benchmarked this yet, so I don't know if it's a real win
or not.  I think Thomas Hervé has a patch to cStringIO which implements
the feature, though.

For reference, .

This isn't high on my priority list, but I thought I'd point out the
potential use-case.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Encoding detection in the standard library?

[Jean-Paul Calderone]

On Mon, 21 Apr 2008 17:50:43 +0100, Michael Foord <[EMAIL PROTECTED]> wrote:
>[EMAIL PROTECTED] wrote:
>> David> Is there some sort of text encoding detection module is the
>> David> standard library?  And, if not, is there any reason not to add
>> David> one?
>>
>> No, there's not.  I suspect the fact that you can't correctly determine the
>> encoding of a chunk of text 100% of the time mitigates against it.
>>
>
>The only approach I know of is a heuristic based approach. e.g.
>
>http://www.voidspace.org.uk/python/articles/guessing_encoding.shtml
>
>(Which was 'borrowed' from docutils in the first place.)

This isn't the only approach, although you're right that in general you
have to rely on heuristics.  See the charset detection features of ICU:

  http://www.icu-project.org/userguide/charsetDetection.html

I think OSAF's pyicu exposes these APIs:

  http://pyicu.osafoundation.org/

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] __eq__ vs hash

[Jean-Paul Calderone]

On Fri, 4 Apr 2008 07:38:04 -0700, Guido van Rossum <[EMAIL PROTECTED]> wrote:
>On Fri, Apr 4, 2008 at 2:46 AM, Ralf Schmitt <[EMAIL PROTECTED]> wrote:
>> the news file for python 2.6 does not mention that you need to define
>> __hash__ in case you define __eq__ for a class.
>> This breaks some code (for me: mercurial and pyparsing).
>> Shouldn't this be documented somewhere (I also cannot find it in the
>> whatsnew file).
>
>Well, technically this has always been the requirement.
>
>What specific code breaks? Maybe we need to turn this into a warning
>in order to be more backwards compatible?
>

There was some code in Twisted (one class, specifically) which was broken/
revealed to be broken by this Python 2.6 change.  The code assumed identity
hashing if no __hash__ method was implemented.  This ended up only working
if you only had a singleton instance of the class, but the class also went
out of its way to make sure that was the case.  We have since changed the
code to work on Python 2.6.

If you're curious about the details, here's the code after the fix:

http://twistedmatrix.com/trac/browser/trunk/twisted/web2/dav/element/base.py?rev=22305#L345

Here's the changeset that fixed it:

http://twistedmatrix.com/trac/changeset/22305

And here's the same class before the fix:

http://twistedmatrix.com/trac/browser/trunk/twisted/web2/dav/element/base.py?rev=22304#L344

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] socket.SOL_REUSEADDR: different semantics between Windows vs Unix (or why test_asynchat is sometimes dying on Windows)

[Jean-Paul Calderone]

On Fri, 4 Apr 2008 13:24:49 -0700, Trent Nelson <[EMAIL PROTECTED]> wrote:
>Interesting results!  I committed the patch to test_socket.py in r62152.  I 
>was expecting all other platforms except for Windows to behave consistently 
>(i.e. pass).  That is, given the following:
>
>import socket
>host = '127.0.0.1'
>sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>sock.bind((host, 0))
>port = sock.getsockname()[1]
>sock.close()
>del sock
>
>sock1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>sock1.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
>sock1.bind((host, port))
>sock2 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>sock2.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
>sock2.bind((host, port))
>
>
>the second bind should fail with EADDRINUSE, at least according to the 
>'SO_REUSEADDR and SO_REUSEPORT Socket Options' section in chapter 7.5 of 
>Stevens' UNIX Network Programming Volume 1 (2nd Ed):
>
>"With TCP, we are never able to start multiple servers that bind
> the same IP address and same port: a completely duplicate binding.
> That is, we cannot start one server that binds 198.69.10.2 port 80
> and start another that also binds 198.69.10.2 port 80, even if we
> set the SO_REUSEADDR socket option for the second server."
>
>The results: both Windows *and* Linux fail the patched test; none of the 
>buildbots for either platform encountered an EADDRINUSE socket.error after the 
>second bind().  FreeBSD, OS X, Solaris and Tru64 pass the test -- EADDRINUSE 
>is raised on the second bind.  (Interesting that all the ones that passed have 
>a BSD lineage.)

Notice that the quoted text explains that you cannot start multiple servers
that etc.  Since you didn't call listen on either socket, it's arguable that
you didn't start any servers, so there should be no surprise regarding the
behavior.  Try adding listen calls at various places in the example and
you'll see something different happen.

FWIW, AIUI, SO_REUSEADDR behaves just as described in the above quote on
Linux/BSD/UNIX/etc.  On Windows, however, that option actually means
something quite different.  It means that the address should be stolen from
any process which happens to be using it at the moment.

There is another option, SO_EXCLUSIVEADDRUSE, only on Windows I think,
which, AIUI, makes it impossible for another process to steal the port
using SO_REUSEADDR.

Hope this helps,

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-3000] the release gods are angry at python

[Jean-Paul Calderone]

On Thu, 27 Mar 2008 11:43:25 -0700, Neal Norwitz <[EMAIL PROTECTED]> wrote:
>On Wed, Mar 26, 2008 at 5:52 PM,  <[EMAIL PROTECTED]> wrote:
>> >> The next releases of 2.6/3.0 are planned for April 2, just over a
>> >> week from now.  There is much work that needs to be done.  The
>> >> buildbots are in a pretty sad state and the gods are seeing too much
>> >> red.
>> >>
>> >> http://www.python.org/dev/buildbot/stable/
>> >> http://www.python.org/dev/buildbot/all/
>>
>>  Is there some reason the "g4 osx.4 trunk" buildbot isn't running?  When I
>>  checked it this morning it was idle with two pending.  I forced a run.  I
>>  checked later today and it showed five pending.  Now it shows 10 pending.
>>  It looks like its last run was about 18 hours ago.  You would think by now
>>  it would have been able to make another run.  It pings fine.
>
>I logged in to the machine an the buildbot is running.  I think what
>happens is that sometimes the master loses track of the slaves.  The
>fix requires restarting the master, but I didn't want to do that last
>night while there were still builds going on.  I'll try to find a
>quiet time tonight and restart the master.  I expect that will fix the
>problem.
>

There's a bug in some configurations (I have never managed to track down
the details) where the ping action actually prevents any further builds
from happening on that slave until the master is restarted.  Not sure if
this is related to the problem you're seeing here or not.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: from __future__ import unicode_string_literals

[Jean-Paul Calderone]

On Mon, 24 Mar 2008 00:14:13 +0100, "\"Martin v. Löwis\"" <[EMAIL PROTECTED]> 
wrote:
>> You are still only seeing this as a case of libraries with a small
>> number of people developing them and making regular well defined
>> releases. That is not how the world I am talking about looks.
>
>Can you give me examples of such software? Are you perhaps talking
>about closed source software?

I'm not sure what software he was talking about.  I can say that for
the work I do on both Twisted and Divmod software, I'd be quite happy
to see this feature.  As either part of a migration path towards 3k
_or_ as a feature entirely on its own merits, this would be very useful
to me.

I'm a bit curious about why Thomas said this sort of thing results in
fragile code.  Twisted has been using __future__ imports for years and
they've never been a problem.  Twisted currently supports Python 2.3
through Python 2.5, and the only thing that's really difficult about
that is subtle changes in library behavior, not syntax.

I'm also curious about why Lennart thinks that this would only be relevant
for large projects with lots of developers making regular releases.  Sure,
I'd use it in that scenario, but that's because it's a subset of "all
development". ;)

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Buildbot failures

[Jean-Paul Calderone]

On Thu, 7 Feb 2008 09:08:26 -0500, "A.M. Kuchling" <[EMAIL PROTECTED]> wrote:
>On Wed, Feb 06, 2008 at 08:34:21PM -0500, Raymond Hettinger wrote:
>> Also, test_docxmlrpc hasn't been happy. One of the tests isn't
>> getting the exact response string it expected.  Any ideas what is
>> causing this?
>
>My fault; it should be fixed now.
>
>> There is also a recurring failure in SocketServer.py returning
>> "ValueError: list.remove(x): x not in list" during attempts to
>> remove a PID from the list of active_children.  Any ideas about what
>> is causing this?
>
>I couldn't find a current build that was showing this error, but
>searching python.org turned up one that had been indexed:
>
>http://www.python.org/dev/buildbot/trunk/ppc%20Debian%20unstable%20trunk/builds/726/step-test/0
>
>I don't see what could be causing this failure, though; the test isn't
>starting any subprocesses outside of what the ForkingServer class
>does.  I don't see how this could be an artifact of the buildbot
>environment, either.
>
>It would be easy to add an 'if pid in self.active_children' to the
>code, but I don't want to do that without understanding the problem.

You could instrument fork() so that it logs the call stack and the child
PID and instrument ForkingServer so that it reports which PID it is about
to try to remove from active_children.  Perhaps this will point to the
problem.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 370, open questions

[Jean-Paul Calderone]

On Thu, 17 Jan 2008 13:09:34 +0100, Christian Heimes <[EMAIL PROTECTED]> wrote:
>Jean-Paul Calderone wrote:
>> If it should, I think the PEP should explain the attack this defends
>> against in more detail.  The current brief mention of "security issues"
>> is a bit hand-wavey.  For example, what is the relationship between
>> security, this feature, and the PYTHONPATH environment variable?  Isn't
>> the attack of putting malicious code into a user site-packages directory
>> the same as the attack of putting it into a directory in PYTHONPATH?
>
>The PYTHONPATH env var has the same security implications. However a
>user has multiple ways to avoid problems. For example the user can use
>the -E flag or set up sudo to ignore the environment.

I'm not sure how sudo gets involved.  sudo doesn't set the euid, it sets
the uid.  This is about programs with the setuid bit set.  (I assume this
doesn't also apply to Python programs that explicitly make use of the
seteuid() call, since this will probably only be checked at interpreter
startup before any Python application code has run.)

>
>The uid and gid tests aren't really required. They just provide an extra
>safety net if a user forgets to add the -s flag to a suid app.

It's not much of a safety net if PYTHONPATH still allows injection of
arbitrary code.  It's just needless additional complexity for no benefit.

On the other hand, if all of the other mechanisms for modifying how
imports work is also made to behave this way, then maybe there's a point.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 370, open questions

[Jean-Paul Calderone]

On Thu, 17 Jan 2008 08:55:51 +0100, Christian Heimes <[EMAIL PROTECTED]> wrote:
>
>* Should the site package directory also be ignored if process
>  gid != effective gid?

If it should, I think the PEP should explain the attack this defends
against in more detail.  The current brief mention of "security issues"
is a bit hand-wavey.  For example, what is the relationship between
security, this feature, and the PYTHONPATH environment variable?  Isn't
the attack of putting malicious code into a user site-packages directory
the same as the attack of putting it into a directory in PYTHONPATH?

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bug day tasks

[Jean-Paul Calderone]

On Fri, 04 Jan 2008 16:53:46 +0100, Christian Heimes <[EMAIL PROTECTED]> wrote:
>A.M. Kuchling wrote:
>> Another task is to get logging set up for the #python-dev IRC channel.
>> Searching didn't find any existing archive; we could run it on
>> python.org somewhere, but does anyone here already run an IRC logging
>> bot?  Maybe someone could just add #python-dev to their existing
>> setup.
>
>It'd be nice if we can also get a bot into #python-dev to broadcast svn
>commits and bug tracker changes. The Twisted guys have good bot with
>decent msg coloring but IIRC it's tight into TRAC. For svn we could
>probably use CIA bot and tie it into a svn post commit hook.

The trac integration is entirely optional, so don't let that discourage
you.

If anyone wants to investigate setting this up,

  svn://divmod.org/svn/Divmod/sandbox/exarkun/commit-bot

The code has no unit tests and there is no documentation.  Also notice
"sandbox" in the SVN URL.

The only real advantage that it has over CIA that I can point out is that
you don't have to write an XML or have a SQL server running in order to use
it.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Does Python need a file locking module (slightly higher level)?

[Jean-Paul Calderone]

On Tue, 23 Oct 2007 01:11:39 +0100, Jon Ribbens <[EMAIL PROTECTED]> wrote:
>On Tue, Oct 23, 2007 at 12:29:35PM +1300, Greg Ewing wrote:
>> [EMAIL PROTECTED] wrote:
>> > Does fcntl.flock work over NFS and SMB and on Windows?
>>
>> I don't think file locking will ever work over NFS, since
>> it's a stateless protocol by design, and locking would
>> require maintaining state on the server.
>
>You can do file locking over NFS, that's one of the reasons people
>use fcntl. It uses an RPC side channel separate to the main NFS
>protocol.

You can do it.  It just doesn't work.  (You could say the same about
regular read and write operations for many NFS implementations, though)

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] incompatible unittest changes

[Jean-Paul Calderone]

On Fri, 19 Oct 2007 15:51:51 -0700, Collin Winter <[EMAIL PROTECTED]> wrote:
>On 10/19/07, Jean-Paul Calderone <[EMAIL PROTECTED]> wrote:
>> In trunk after 2.5, equality and hashing for TestCase were added, changing 
>> the behavior so that two instances of TestCase for the same test method hash 
>> the same and compare equal.  This means two instances of TestCase for the 
>> same test method cannot be added to a single set.
>>
>> Here's the change:
>>
>> http://svn.python.org/view/python/trunk/Lib/unittest.py?rev=54199&r1=42115&r2=54199
>>
>> The implementations aren't even very good, since they prevent another type 
>> from deciding that it wants to customize comparison against TestCase (or 
>> TestSuite,
>> or FunctionTestCase) instances.
>
>The implementations have been changed in a more recent revision.

Not in http://svn.python.org/projects/python/trunk/Lib/[EMAIL PROTECTED]

>
>> Is there a real use case for this functionality?  If not, I'd like it to be
>> removed to restore the old behavior.
>
>The use-case was problems I encountered when writing the test suite
>for unittest. If you can find a way to implement the functionality you
>want *and* keep the test suite reasonably straightforward, I'll be
>happy to review your patch.

The test suite can implement the comparison which is currently on the
unittest classes and invoke that functionality instead of using == and
!=.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] incompatible unittest changes

[Jean-Paul Calderone]

In trunk after 2.5, equality and hashing for TestCase were added, changing the 
behavior so that two instances of TestCase for the same test method hash the 
same and compare equal.  This means two instances of TestCase for the same test 
method cannot be added to a single set.

Here's the change:

http://svn.python.org/view/python/trunk/Lib/unittest.py?rev=54199&r1=42115&r2=54199

The implementations aren't even very good, since they prevent another type from 
deciding that it wants to customize comparison against TestCase (or TestSuite,
or FunctionTestCase) instances.

Is there a real use case for this functionality?  If not, I'd like it to be
removed to restore the old behavior.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Removing the GIL (Me, not you!)

[Jean-Paul Calderone]

On Fri, 14 Sep 2007 17:43:39 -0400, James Y Knight <[EMAIL PROTECTED]> wrote:
>
>On Sep 14, 2007, at 3:30 PM, Jean-Paul Calderone wrote:
>>On Fri, 14 Sep 2007 14:13:47 -0500, Justin Tulloss  <[EMAIL PROTECTED]> 
>>wrote:
>>>Your idea can be combined with the maxint/2 initial refcount for
>>>>non-disposable objects, which should about eliminate thread-count 
>>>>updates
>>>>for them.
>>>>--
>>>
>>>I don't really like the maxint/2 idea because it requires us to
>>>differentiate between globals and everything else. Plus, it's a  hack. I'd
>>>like a more elegant solution if possible.
>>
>>It's not really a solution either.  If your program runs for a couple
>>minutes and then exits, maybe it won't trigger some catastrophic  behavior
>>from this hack, but if you have a long running process then you're  almost
>>certain to be screwed over by this (it wouldn't even have to be *very*
>>long running - a month or two could do it on a 32bit platform).
>
>Not true: the refcount becoming 0 only calls a dealloc function.. For 
>objects which are not deletable, the dealloc function should simply  set the 
>refcount back to maxint/2. Done.
>

So, eg, replace the Py_FatalError in none_dealloc with an assignment to
ob_refcnt?  Good point, sounds like it could work (I'm pretty sure you
know more about deallocation in CPython than I :).

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Removing the GIL (Me, not you!)

[Jean-Paul Calderone]

On Fri, 14 Sep 2007 14:13:47 -0500, Justin Tulloss <[EMAIL PROTECTED]> wrote:
>Your idea can be combined with the maxint/2 initial refcount for
>> non-disposable objects, which should about eliminate thread-count updates
>> for them.
>> --
>>
>
> I don't really like the maxint/2 idea because it requires us to
>differentiate between globals and everything else. Plus, it's a hack. I'd
>like a more elegant solution if possible.

It's not really a solution either.  If your program runs for a couple
minutes and then exits, maybe it won't trigger some catastrophic behavior
from this hack, but if you have a long running process then you're almost
certain to be screwed over by this (it wouldn't even have to be *very*
long running - a month or two could do it on a 32bit platform).

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-3000] test_asyncore fails intermittently on Darwin

[Jean-Paul Calderone]

On Sun, 29 Jul 2007 23:40:29 -0700, Hasan Diwan <[EMAIL PROTECTED]> wrote:
>The issue seems to be in the socket.py close method. It needs to sleep
>socket.SO_REUSEADDR seconds before returning. Yes, it is a simple fix
>in python, but the socket code is C. I found some code in socket.py
>and made the changes. Patch is available at
>http://sourceforge.net/tracker/index.php?func=detail&aid=1763387&group_id=5470&atid=305470
>-- enjoy your week.

Uh, no, that's basically totally wrong.  Details on the ticket.

>--
>Cheers,
>Hasan Diwan <[EMAIL PROTECTED]>
>___
>Python-3000 mailing list
>[EMAIL PROTECTED]
>http://mail.python.org/mailman/listinfo/python-3000
>Unsubscribe: 
>http://mail.python.org/mailman/options/python-3000/exarkun%40divmod.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Fwd: [ python-Patches-1744382 ] Read Write lock

[Jean-Paul Calderone]

On Fri, 6 Jul 2007 10:47:16 -0700, Mike Klaas <[EMAIL PROTECTED]> wrote:
>On 6-Jul-07, at 6:45 AM, Yaakov Nemoy wrote:
>
>>
>> I can do the other three parts, but I am wondering, how do I write a
>> deterministic test unit for my patch?  How is it done with the
>> threading model in python in general?
>
>I don't know how it is done in general, but for reference, here are
>some of the unittests for my read/write lock class:
>
> [snip]
>
> read.release()
> self.assertEqual(wrlock.readerCount, 0)
> time.sleep(.1)
> self.assertTrue(writer.gotit)

Not exactly deterministic.  Instead of a flag attribute, try using an Event or
a Condition.  Either of these will let you know exactly when the necessary
operation has completed.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Adding NetworkIOError for bug 1706815

[Jean-Paul Calderone]

On Tue, 3 Jul 2007 23:58:44 -0700, "Gregory P. Smith" <[EMAIL PROTECTED]> wrote:
>In response to bug 1706815 and seeing messy code to catch errors in
>network apps I've implemented most of the ideas in the bug and added a
>NetworkIOError exception (child of IOError).  With this, socket.error
>would now inherit from NetworkIOError instead of being its own thing
>(the old one didn't even declare a parent!).
>
>Complete patch attached to the bug.  All unit tests pass.
>Documentation updates included.
>
> http://sourceforge.net/tracker/index.php?func=detail&aid=1706816&group_id=5470&atid=105470
>

FWIW, that page does not seem to be generally accessible.  It's difficult
to know what you're talking about without being able to see it.

Artifact: Invalid ArtifactID; this Tracker item may have moved to a
different Tracker since this URL was generated -- [Find the new location
of this Tracker item]

Following [Find the new location ...]:

Artifact: This Artifact Has Been Made Private. Only Group Members Can
View Private ArtifactTypes.

>Any thoughts?  I'm happy with it and would like to commit it if folks
>agree.
>

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] updated for gdbinit

[Jean-Paul Calderone]

On Tue, 15 May 2007 09:26:55 -0500, [EMAIL PROTECTED] wrote:
>Christian> I tried to use gdbinit today and found that the "fragile"
>Christian> pystacks macro didn't work anymore. I don't know gdb very
>Christian> well, but this turned out to work a bit more reliably:
>
>...
>
>Thanks.  I'll give it a try and check it in if it checks out.
>

It would also be nice if it handled non-main threads.  This is accomplished
by additionally checking if the pc is in t_bootstrap (ie, between that and
thread_PyThread_start_new_thread).

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.5.1

[Jean-Paul Calderone]

On Sat, 28 Apr 2007 09:32:57 -0400, Raghuram Devarakonda <[EMAIL PROTECTED]> 
wrote:
>On 4/28/07, Calvin Spealman <[EMAIL PROTECTED]> wrote:
>> Index: test_os.py
>> ===
>> --- test_os.py  (revision 54982)
>> +++ test_os.py  (working copy)
>> @@ -6,6 +6,7 @@
>>  import unittest
>>  import warnings
>>  import sys
>> +import tempfile
>>  from test import test_support
>>
>>  warnings.filterwarnings("ignore", "tempnam", RuntimeWarning, __name__)
>> @@ -241,13 +242,18 @@
>>  self.assertEquals(os.stat(self.fname).st_mtime, t1)
>>
>>  def test_1686475(self):
>> +fn = tempfile.mktemp()
>> +openfile = open(fn, 'w')
>>  # Verify that an open file can be stat'ed
>>  try:
>> -os.stat(r"c:\pagefile.sys")
>> +os.stat(fn)
>>  except WindowsError, e:
>>  if e == 2: # file does not exist; cannot run test
>>  return
>>  self.fail("Could not stat pagefile.sys")
>> +finally:
>> +openfile.close()
>> +os.remove(fn)
>>
>>  from test import mapping_tests
>
>mktemp() is deprecated. You may want to use mkstemp(). There will be
>no need for explicit open as well as mkstemp() also returns open
>descriptor.

You still need fdopen() though, since os.stat() won't take a file
descriptor.

The patch is incomplete though, since it should remove the ENOENT
handling and the remaining reference to pagefile.sys.

As for mktemp() being deprecated - the docstring warns users away,
but actually calling it emits no warning.  Sure, using it can lead
to insecurities, but there's hardly any worry of that here.  If the
function were actually deprecated (that is, if calling it emitted a
DeprecationWarning), that would be a good reason to avoid calling it,
though.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.5 branch unfrozen

[Jean-Paul Calderone]

On Wed, 25 Apr 2007 21:28:10 +0200, Georg Brandl <[EMAIL PROTECTED]> wrote:
>Lars Gustäbel schrieb:
>> On Sat, Apr 21, 2007 at 04:45:37PM +1000, Anthony Baxter wrote:
>>> Ok, things seem to be OK. So the release25-maint branch is unfrozen.
>>> Go crazy. Well, a little bit crazy.
>>
>> I'm afraid that I went crazy a little too early. Sorry for that.
>> Won't happen again.
>
>BTW, svn provides a "lock" mechanism by which the branch freezing could
>be enforced more strictly...

It doesn't work on directories, though.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] object.__init__

[Jean-Paul Calderone]

As a data point, I thought I'd point out that the recent object.__init__ change
broke a handful of Twisted unit tests.  The fix for this was simple, and I've
already implemented it, but it would have been nice if the old behavior had been
deprecated first and then removed, instead of just disappearing.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] new metaclass error condition checks

[Jean-Paul Calderone]

I just noticed r53997 (from some unit tests it broke), which disallowed things
like this:

class X(object):
def __repr__(self):
return "blah"

class Y(X, type):
pass

class Z:
__metaclass__ = Y

Making X classic eliminates the TypeError, and is probably an acceptable fix
in a lot of cases (at least as long as classic classes are available).  I
wonder if the ability to override type special methods like this was considered
when the change was made, though?

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] functools additions

[Jean-Paul Calderone]

On Sun, 15 Apr 2007 18:18:16 -0400, SevenInchBread <[EMAIL PROTECTED]> wrote:
>>>Do you have commit access? What's your real name?
>I prefer to remain pseudonymous, and I don't have commit access.
>
>Yeah... they're not terribly useful - more or less there for the sake of
>being there. Batteries included and all that
>

Please discuss this on the python-ideas list before bringing it up on
python-dev.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] test_pty.py hangs in verbose mode on Mac OS X?

[Jean-Paul Calderone]

On Fri, 13 Apr 2007 11:02:01 -0400, Barry Warsaw <[EMAIL PROTECTED]> wrote:
>-BEGIN PGP SIGNED MESSAGE-
>Hash: SHA1
>
>On Apr 13, 2007, at 10:57 AM, Jean-Paul Calderone wrote:
>>>I don't know if this is caused by a bug in the Mac's pty
>>>implementation or something we're doing wrong on that platform.  I
>>>played around with several modifications to pty.fork() on the Mac,
>>>including letting it drop down to the openpty()/os.fork() code, even
>>>adding an explicit ioctl(slave_fd, TIOCSCTTY) call which Stevens
>>>chapter 19 recommends for 4.3+BSD. I can't get it to not block.
>>
>>What about reading from the child in the parent before calling  waitpid?
>
>Yep, this is what I suggested below.  Porting the same change over to  Linux 
>produced an OSError, but that's probably just because I wasn't  as careful 
>as I should have been late last night.
>>>Barring a fix to pty.fork() (or possibly os.forkpty()) for the Mac,
>>>then I would like to at least make test_pty.py not block when run in
>>>verbose mode.  A very simple hack would add something like this to
>>>the "if pid == pty.CHILD" stanza: "def debug(msg): pass", possibly
>>>protected by a "if verbose:".  A less icky hack would be to read the
>>>output from the master_fd in the parent, though you have to be
>>>careful with that on Linux else the read can throw an input/output
>>>error.
>>>
>>>Disabling debug output is band-aid yes, and any application on the
>>>Mac like the above snippet will still fail.  If anybody has any
>>>suggestions, I'm all ears, but I've reached the limit of my pty-fu.
>>
>>I don't think this is an OS X PTY bug.  Writing to a blocking file
>>descriptor can block.  Programs that do this need to account for the
>>possibility.
>
>Why doesn't it block on Linux then?
>

Likely differing buffering behavior.  Prior to Linux 2.6, the pipe
implementation allowed only a single buffer (that is, the bytes from
a single write call) in a pipe at a time, and blocked subsequent
writes until that buffer was read.  Recently this has changed to allow
multiple buffers up to 4k total length, so multiple short writes won't
block anymore.  OS X may have some other buffering behavior which is
causing writes to block where they don't on Linux.  All these details
are left to the platform, and there are a variety of behaviors which
can be considered valid.

Of course, I don't actually /know/ the cause of the problem here, but
this explanation seems plausible to me, and I'd investigate it before
looking for platform-specific pty bugs (although OS X is a good platform
on which to go looking for those ;).

Jean-Paul
___
Python-Dev mailing list
[EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] test_pty.py hangs in verbose mode on Mac OS X?

[Jean-Paul Calderone]

On Fri, 13 Apr 2007 10:32:28 -0400, Barry Warsaw <[EMAIL PROTECTED]> wrote:
>-BEGIN PGP SIGNED MESSAGE-
>Hash: SHA1
>
>I've been getting some test failures in Python 2.5 svn head on Mac OS
>X 10.4.9 which I'm not getting on Linux (Ubuntu feisty beta).
>test_sqlite and test_zipimport both fail, however, when run in
>verbose mode (e.g. ./python.exe Lib/test/test_sqlite.py) both pass.
>
>But that's not exactly why I'm writing this email .  In the
>course of trying to debug this, I ran the following on my Mac:
>
>make TESTOPTS=-v test
>
>This runs the entire test suite in verbose mode, and you do get a lot
>of output.  However the test suite hangs on test_pty.py.  In fact, if
>you run that test alone:
>
>./python.exe Lib/test/test_pty.py
>
>it too hangs for me.  The reason is that in verbose mode, debug()
>actually prints stuff to stdout and on the Mac, when the child of the
>pty.fork() writes to its stdout, it blocks and so the parent's waitpid
>() never returns.  This doesn't happen on Linux though; the child's
>stdout prints don't block, it exits, and the parent continues after
>the waitpid().
>
>Here's a very simple program that reproduces the problem:
>
>- -snip snip-
>import os, pty, sys
>
>pid, fd = pty.fork()
>print >> sys.stderr, pid, fd
>if pid:
> os.waitpid(pid, 0)
>else:
> os._exit(0)
>- -snip snip-
>
>stderr, stdout, doesn't matter.  This hangs on the Mac but completes
>successfully on Linux.  Of course, in neither case do you see the
>child's output.
>
>I don't know if this is caused by a bug in the Mac's pty
>implementation or something we're doing wrong on that platform.  I
>played around with several modifications to pty.fork() on the Mac,
>including letting it drop down to the openpty()/os.fork() code, even
>adding an explicit ioctl(slave_fd, TIOCSCTTY) call which Stevens
>chapter 19 recommends for 4.3+BSD. I can't get it to not block.

What about reading from the child in the parent before calling waitpid?

>
>Barring a fix to pty.fork() (or possibly os.forkpty()) for the Mac,
>then I would like to at least make test_pty.py not block when run in
>verbose mode.  A very simple hack would add something like this to
>the "if pid == pty.CHILD" stanza: "def debug(msg): pass", possibly
>protected by a "if verbose:".  A less icky hack would be to read the
>output from the master_fd in the parent, though you have to be
>careful with that on Linux else the read can throw an input/output
>error.
>
>Disabling debug output is band-aid yes, and any application on the
>Mac like the above snippet will still fail.  If anybody has any
>suggestions, I'm all ears, but I've reached the limit of my pty-fu.
>

I don't think this is an OS X PTY bug.  Writing to a blocking file
descriptor can block.  Programs that do this need to account for the
possibility.

Jean-Paul
___
Python-Dev mailing list
[EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] About SSL tests

[Jean-Paul Calderone]

On Thu, 29 Mar 2007 00:22:23 + (UTC), Facundo Batista <[EMAIL PROTECTED]> 
wrote:
>Jean-Paul Calderone wrote:
>
>> Take a look at "openssl s_server".  This is still a pretty terrible way
>> to test the SSL functionality, but it's loads better than connecting to
>> a site on the public internet.
>
>How would you deal with the deployment and maintenance of this server in
>all buildbot's machines?
>
>Or we just can ask to see if we have the server available, and then run
>the tests if yes?
>

If the openssl binary is available, when the test starts, launch it in
a child process, talk to it for the test, then kill it when the test is
done.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] About SSL tests

[Jean-Paul Calderone]

On Wed, 28 Mar 2007 16:38:45 -0700, Brett Cannon <[EMAIL PROTECTED]> wrote:
>On 3/28/07, Facundo Batista <[EMAIL PROTECTED]> wrote:
>> There's this bug (#451607) about the needing of tests for socket SSL...
>>
>> Last interesting update in the tracker is five years ago, and since a
>> lot of work has been done in test_socket_ssl.py (Brett, Neal, Tim,
>> George Brandl).
>>
>> Do you think is useful to leave this bug opened?
>
>Having a bug left open because a module needs more test is not really
>needed.  It's rather obvious when a module needs more tests.  =)
>
>I say close it.  I just wish we had a more reliable web site to
>connect to for SSL tests.

How about something even better?

Take a look at "openssl s_server".  This is still a pretty terrible way
to test the SSL functionality, but it's loads better than connecting to
a site on the public internet.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking calls to object.__init__/__new__

[Jean-Paul Calderone]

On Wed, 21 Mar 2007 15:45:16 -0700, Guido van Rossum <[EMAIL PROTECTED]> wrote:
>See python.org/sf/1683368. I'd like to invite opinions on whether it's
>worth breaking an unknown amount of user code in 2.6 for the sake of
>stricter argument checking for object.__init__ and object.__new__. I
>think it probably isn't; but the strict version could be added to 3.0
>and a warning issued in 2.6 in -Wpy3k mode. Alternatively, we could
>introduce the stricter code in 2.6, fix the stdlib modules that it
>breaks, and hope for the best. Opinions?
>

Perhaps I misunderstand the patch, but it would appear to break not just
some inadvisable uses of super(), but an actual core feature of super().
Maybe someone can set me right.  Is this correct?

  class Base(object):
  def __init__(self, important):
  # Don't upcall with `important` because object is the base
  # class and its __init__ doesn't care (or won't accept) it
  super(Base, self).__init__()
  self.a = important

If so, what are the implications for this?

  class Other(object):
  def __init__(self, important):
  # Don't upcall with `important` because object is the base
  # class and its __init__ doesn't care (or won't accept) it
  super(Other, self).__init__()
  self.b = important

  class Derived(Base, Other):
  pass


(A similar example could be given where Base and Other take differently
named arguments with nothing to do with each other.  The end result is
the same either way, I think.)

I think I understand the desire to pull keyword arguments out at each
step of the upcalling process, but I don't see how it can work, since
"up" calling isn't always what's going on - given a diamond, there's
arbitrary side-calling, so for cooperation to work every method has to
pass on every argument, so object.__init__ has to take arbitrary args,
since no one knows when their "up" call will actually hit object.

Since without diamonds, naive "by-name" upcalling works, I assume that
super() is actually intended to be used with diamonds, so this seems
relevant.

I hope I've just overlooked something.  Writing this email feels very
strange.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of thread cancellation

[Jean-Paul Calderone]

On Thu, 15 Mar 2007 09:41:31 -0500, [EMAIL PROTECTED] wrote:
>
>>> I just proposed to implement thread cancellation for the SoC.  Is
>>> there any prior work where one could start?
>
>Jean-Paul> The outcome of some prior work, at least:
>
>Jean-Paul>   
> http://java.sun.com/j2se/1.4.2/docs/guide/misc/threadPrimitiveDeprecation.html
>
>I responded to that.  I got the impression reading that page that the killed
>thread doesn't regain control so it can't clean up its potentially
>inconsistent data structures.

The second question on the page:

  Couldn't I just catch the ThreadDeath exception and fix the damaged object?

Addresses this.

>I inferred from Martin's proposal that he
>expected the thread to be able to catch the exception.  Perhaps he can
>elaborate on what cleanup actions the dying thread will be allowed to
>perform.

Perhaps he can.  Hopefully, he can specifically address these points:

   1. A thread can throw a ThreadDeath exception almost anywhere. All
  synchronized methods and blocks would have to be studied in great
  detail, with this in mind.

   2. A thread can throw a second ThreadDeath exception while cleaning up
  from the first (in the catch or finally clause). Cleanup would have
  to repeated till it succeeded. The code to ensure this would be quite
  complex. 

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of thread cancellation

[Jean-Paul Calderone]

On Thu, 15 Mar 2007 14:34:15 +0100, "\"Martin v. Löwis\"" <[EMAIL PROTECTED]> 
wrote:
>I just proposed to implement thread cancellation for the SoC.
>Is there any prior work where one could start?

The outcome of some prior work, at least:

  http://java.sun.com/j2se/1.4.2/docs/guide/misc/threadPrimitiveDeprecation.html
  
Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] unittest enhancement for TestCase classes hierarchies

[Jean-Paul Calderone]

On Sat, 10 Mar 2007 09:13:28 -0600, Collin Winter <[EMAIL PROTECTED]> wrote:
>In my continuing trawl through the SF patch tracker, I came across
>#1244929 (http://python.org/sf/1244929), which causes
>TestLoader.loadTestsFromModule() to skip classes whose name starts
>with an underscore. This addresses the warning in that method's docs:
>
>"""
>While using a hierarchy of
>TestCase-derived classes can be convenient in sharing
>fixtures and helper functions, defining test methods on base classes
>that are not intended to be instantiated directly does not play well
>with this method.  Doing so, however, can be useful when the
>fixtures are different and defined in subclasses.
>"""
>
>"Does not play well", in this case, means that your base classes will
>be picked up against your will if they subclass TestCase.
>
>I like the patch and have worked up tests and doc changes for it. Any
>objections to including this in 2.6?

This use case is what mixins are for.  You don't have to include TestCase
in your ancestry until you get to a class which you actually want to run
tests.

The current rule of loading anything that subclasses TestCase is simple
and straightforward.  Complicating it to provide a feature which is already
available through a widely used standard Python idiom doesn't seem worth
while.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] with_traceback

[Jean-Paul Calderone]

On Wed, 28 Feb 2007 18:29:11 -0700, Adam Olsen <[EMAIL PROTECTED]> wrote:
>On 2/28/07, Greg Ewing <[EMAIL PROTECTED]> wrote:
>> Adam Olsen wrote:
>>
>> > It sounds like we should always copy the exception given to raise,
>>
>> I don't like that either, for all the reasons that
>> make it infeasible to copy an arbitrary object in a
>> general way.
>
>Exceptions aren't arbitrary objects though.  The requirement that they
>inherit from BaseException is specifically to create a common
>interface.  Copying would be an extension of that interface.
>
>I believe calling copy.copy() would be sufficient.
>

Does copying every exception given to `raise' solve the problem being
discussed here?

Consider the current Python behavior: no copying is performed, most code
instantiates a new exception instance for each raise statement, some code
creates a single exception and re-raises it repeatedly.

And the new behavior?  Every raise statement copies an exception instance,
some code will create a new exception instance for each raise statement,
some code will create a single exception and re-raise it repeatedly.

That doesn't sound like an improvement to me.  Normal code will be more
wasteful.  Code which the author has gone out of his way to tune will be
as wasteful as /average/ code currently is, and more wasteful than tuned
code now is.

Plus you now have the added mental burden of keeping track of which objects
are copies of what (and if you throw in the refcount=1 optimization, then
this burden is increased - was something accidentally relying on copying or
non-copying?  Did a debugger grab a reference to the exception object, thus
changing the programs behavior?  Did a third-party hang on to an exception
for longer than the raising code expected?  etc).

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PATCH] Handling of scripts / substitution of python executable path

[Jean-Paul Calderone]

On Fri, 23 Feb 2007 15:36:50 +0100, Hans Meine <[EMAIL PROTECTED]> wrote:
>Hi!
>
> [snip - distutils should leave #!/usr/bin/env python alone]
>
>Comments?  (I first posted this to distutils-sig but was told that distutils
>is a bit neglected there, so I decided to try to push these simple patches in
>via python-dev.)

How about a distutils installation command line option to tell it to use this
behavior?  People with unusual environments can select it.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)

[Jean-Paul Calderone]

On Thu, 15 Feb 2007 13:55:31 -0800, Josiah Carlson <[EMAIL PROTECTED]> wrote:
>
>Jean-Paul Calderone <[EMAIL PROTECTED]> wrote:
> [snip]
>>
>> Now if we can only figure out a way for everyone to benefit from this without
>> tying too many brains up in knots. :)
>
>Whenever I need to deal with these kinds of things (in wxPython
>specifically), I usually set up a wxTimer to signal
>asyncore.poll(timeout=0), but I'm lazy, and rarely need significant
>throughput in my GUI applications.

And I guess you also don't mind that on OS X this is often noticably broken?
:)

> [snip]
>
>Protocol support is hit and miss.  NNTP in Python could be better, but
>that's not an asyncore issue (being that nntplib isn't implemented using
>asyncore), that's an "NNTP in Python could be done better" issue.  Is it
>worth someone's time to patch it, or should they just use Twisted?  Well,
>if we start abandoning stdlib modules, "because they can always use
>Twisted", then we may as well just ship Twisted with Python.
>

We could always replace the stdlib modules with thin compatibility layers
based on the Twisted protocol implementations.  It's trivial to turn an
asynchronous API into a synchronous one.  I think you are correct in marking
this an unrelated issue, though.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] microthreading vs. async io

[Jean-Paul Calderone]

On Thu, 15 Feb 2007 10:36:21 -0600, [EMAIL PROTECTED] wrote:
> [snip]
>
>def fetchSequence(...):
>  fetcher = Fetcher()
>  yield fetcher.fetchHomepage()
>  firstData = yield fetcher.fetchPage('http://...')
>  if someCondition(firstData):
>while True:
>  secondData = yield fetcher.fetchPage('http://...')
>  # ...
>  if someOtherCondition(secondData): break
>  else:
># ...

Ahem:

from twisted.internet import reactor
from twisted.internet.defer import inlineCallbacks
from twisted.web.client importt getPage

@inlineCallbacks
def fetchSequence(...):
homepage = yield getPage(homepage)
firstData = yield getPage(anotherPage)
if someCondition(firstData):
while:
secondData = yield getPage(wherever)
if someOtherCondition(secondData):
break
else:
...

So as I pointed out in another message in this thread, for several years it
has been possible to do this with Twisted.  Since Python 2.5, you can do it
exactly as I have written above, which looks exactly the same as your example
code.

Is the only problem here that this style of development hasn't had been made
visible enough?

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)

[Jean-Paul Calderone]

On Thu, 15 Feb 2007 10:46:05 -0500, "A.M. Kuchling" <[EMAIL PROTECTED]> wrote:
>On Thu, Feb 15, 2007 at 09:19:30AM -0500, Jean-Paul Calderone wrote:
>> >That feels like 6 layers too many, given that
>> > _logrun(selectable, _drdw, selectable, method, dict)
>> > return context.call({ILogContext: newCtx}, func, *args, **kw)
>> > return self.currentContext().callWithContext(ctx, func, *args, **kw)
>> > return func(*args, **kw)
>> > getattr(selectable, method())
>> > klass(number, string)
>> >
>> >are all generic calls.
>>
>> I know function calls are expensive in Python, and method calls even more
>> so... but I still don't understand this issue.  Twisted's call stack is too
>> deep?  It is fair to say it is deep, I guess, but I don't see how that is a
>> problem.  If it is, I don't see how it is specific to this discussion.
>
>It's hard to debug the resulting problem.  Which level of the *12*
>levels in the stack trace is responsible for a bug?  Which of the *6*
>generic calls is calling the wrong thing because a handler was set up
>incorrectly or the wrong object provided?  The code is so 'meta' that
>it becomes effectively undebuggable.

I've debugged plenty of Twisted applications.  So it's not undebuggable. :)

Application code tends to reside at the bottom of the call stack, so Python's
traceback order puts it right where you're looking, which makes it easy to
find.  For any bug which causes something to be set up incorrectly and only
later manifests as a traceback, I would posit that whether there is 1 frame or
12, you aren't going to get anything useful out of the traceback.  Standard
practice here is just to make exception text informative, I think, but this is
another general problem with Python programs and event loops, not one specific
to either Twisted itself or the particular APIs Twisted exposes.

As a personal anecdote, I've never once had to chase a bug through any of the
6 "generic calls" singled out.  I can't think of a case where I've helped any
one else who had to do this, either.  That part of Twisted is very old, it is
_very_ close to bug-free, and application code doesn't have very much control
over it at all.  Perhaps in order to avoid scaring people, there should be a
way to elide frames from a traceback (I don't much like this myself, I worry
about it going wrong and chopping out too much information, but I have heard
other people ask for it)?

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)

[Jean-Paul Calderone]

On Thu, 15 Feb 2007 02:36:22 -0700, Andrew Dalke <[EMAIL PROTECTED]> wrote:
>I was the one on the Stackless list who last September or so
>proposed the idea of monkeypatching and I'm including that
>idea in my presentation for PyCon.  See my early rough draft
>at http://www.stackless.com/pipermail/stackless/2007-February/002212.html
>which contains many details about using Stackless, though
>none on the Stackless implementation. (A lot on how to tie things together.)
>
>So people know, I am an applications programmer and not a
>systems programmer.  Things like OS-specific event mechanisms
>annoy and frustrate me.  If I could do away with hardware and
>still write useful programs I would.

What a wonderful world it would be. :)

>
> [snip]
>
>In all three cases I've found it hard to use Twisted because
>the code didn't do as I expected it to do and when something
>went wrong I got results which were hard to interpret.  I
>believe others have similar problems and is one reason Twisted
>is considered to be "a big, complicated, inseparable hairy mess."
>
>I find the Stackless code also hard to understand.  Eg,
>I don't know where the watchdog code is for the "run()"
>command.  It uses several layers of macros and I haven't
>been able get it straight in my head.  However, so far
>I've not run into strange errors in Stackless that I
>have in Twisted.
>

As you point out below, however, Twisted and stackless achieve different goals.

>I find the normal Python code relatively easy to understand.
>
>Stackless only provides threadlets.  It does no I/O.
>Richard Tew developed a "stacklesssocket" module which emulates
>the API for the stdlib "socket" module.  I tweaked it a
>bit and showed that by doing the monkeypatch
>
>  import stacklesssocket
>  import sys
>  sys.modules["socket"] = stacklesssocket
>
>then code like "urllib.urlopen" became Stackless compatible.
>Eg, in my PyCon talk draft I show something like
>

It may be of interest to you to learn that a Twisted developer implement this
model several years ago.  It has not been further developed for a handful of
reasons, at the core of which is the fact that it is very similar to
pre-emptive threading in terms of application-level complexity.

You gave several examples of the use of existing code which expects a blocking
socket interface and which "just works" when the socket module is changed in
this way.

However, this is a slight simplification.  Code written without expecting a
context switch (exactly what happens when a socket operation is performed in
this model) is not necessarily correct when context switches are suddenly
introduced.

Consider this extremely trivial example:

  x = 0
  def foo(conn):
  global x
  a = x + 1
  b = ord(conn.recv(1))
  x = a + b
  return x

Clearly, foo is not threadsafe.  Global mutable state is a terrible, terrible
thing.  The point to note is that by introducing a context switch at the
conn.recv(1) call, the same effect is achieved as by any other context switch:
it becomes possible for foo to return an inconsistent result or otherwise
corrupt its own state if another piece of code violates its assumptions and
changes x while it is waiting for the recv call to complete.

Is urllib2 threadsafe?  I have heard complaints that it is not.  I have looked
at the code, and at least in its support for caching, it appears not to be.
Perhaps it can be made threadsafe, but in requiring that, the advantage of
having a whole suite of modules which will "just work" with a transparently
context switching socket module are mostly lost.

>
> [snip - urllib2/tasklet example]
>
>The choice of asyncore is, I think, done because 1) it
>prevents needing an external dependency,

But if some new event loop is introduced into the standard library, then using
it also will not require an external dependency. ;)

>2) asyncore is
>smaller and easier to understand than Twisted,

While I hear this a lot, applications written with Twisted _are_ shorter and
contain less irrelevant noise in the form of boilerplate than the equivalent
asyncore programs.  This may not mean that Twisted programs are easier to
understand, but it is at least an objectively measurable metric.

>and
>3) it was for demo/proof of concept purposes.
>While
>tempting to improve that module I know that Twisted
>has already gone though all the platform-specific crap
>and I don't want to go through it again myself.  I don't
>want to write a reactor to deal with GTK, and one for
>OS X, and one for ...
>

Now if we can only figure out a way for everyone to benefit from this without
tying too many brains up in knots. :)

>
>Another reason I think Twisted is considered "tangled-up
>Deep Magic, only for Wizards Of The Highest Order" is because
>it's infused with event-based processing.  I've done a lot
>of SAX processing and I can say that few people think that
>way or want to go through the process of learning how.
>
>Compare, for example, the following
>
>  f = urllib2.

Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)

[Jean-Paul Calderone]

On Thu, 15 Feb 2007 16:18:40 +1300, Greg Ewing <[EMAIL PROTECTED]> wrote:
> [snip]
>
>This is where my vision is fundamentally different:
>you shouldn't have to *make* a decision in the first
>place. All event-driven libraries should be made to
>use the same substrate on any given platform. Then
>they can coexist without the need for any top-level
>choices.
>
>I know that will be hard to do, but it's the only
>way out of this mess that I can see.
>

Thomas already pointed this out, but I'm repeating it
anyway.  This vision represents an impossible reality
at present.  You will not get Gtk or Qt or wxWidgets
to use Python's event notification API.  If you are
really very interested in solving this problem, go to
the developers of each platform those toolkits run on
and sell them on a unified event notification API. Once
they have adopted, implemented, and deployed it, you
can go to the Gtk, Qt, and wxWidgets teams and tell them
to port all of their code to that new API.  Then, you
can have a unified model in Python.

Until then, the practical compromise with almost zero
negative consequences (sometimes, one extra piece of
configuration will be required - compare this to how
the logging module works ;) is to optionally allow
explicit reactor selection.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)

[Jean-Paul Calderone]

On Thu, 15 Feb 2007 15:47:39 +1300, Greg Ewing <[EMAIL PROTECTED]> wrote:
>Steve Holden wrote:
>
>> If the borrowed code takes a reactor parameter then presumably the
>> top-level code can pass the appropriate reactor type in.
>
>Since there should only be one reactor at a time in
>any given application, it shouldn't have to be passed
>in -- it could be held in a global variable deep
>inside the library. Only the code which creates the
>reactor initially needs to know about that variable,
>or even that there is such a thing as a reactor.

Whether or not the premise here is accurate may be out
of scope for this thread.  Or it may not be.  I dunno.
However, I do want to point out that it is not necessarily
correct that there should be only one reactor at a time in
a given application.  PJE has already explained that
peak.events can have multiple reactors.  Twisted is tied to
one, but this may not always be the case.  Whether there is
a default reactor for applications that don't care about the
ability to have more than one at a time is yet another
question which may be worth examining.  These are the kinds
of things which should be spelled out in a PEP, including
the rationale for any particular policy decisions (which
should be kept to an absolute minimum) are made.

>
>> "Incorporating some piece of event-driven code written by someone else"
>> implies specific assumptions about event types and delivery, surely.
>
>It requires agreement on how to specify the event types
>and what to do in response, but that's all it should
>require.
>
>The way I envisage it, setting up an event callback
>should be like opening a file -- there's only one way
>to do it, and you don't have to worry about what the
>rest of the application is doing. You don't have to
>get passed an object that knows how to open files --
>it's a fundamental service provided by the system.
>You just use it.

If we suppose that files and sockets are supported in
roughly the same way, and we suppose that sockets are
supported in the way that Twisted supports them, then
there is no difficulty supporting files in this way. :)

>
>> That's why it's difficult to port code between GUI toolkits, for
>> example, and even more so to write code that runs on several toolkits
>> without change.
>
>Just in case it's not clear, the events I'm talking
>about are things like file and socket I/O, not GUI
>events. Trying to use two different GUIs at once is
>not something I'm addressing.

Alright, good.  Getting two different GUI libraries to
play together is a pretty hairy task indeed, and well
worth keeping separate from this one. :)

>
>Rather, you should be able to write code that does
>e.g. some async socket I/O, and embed it in a GUI
>app using e.g. gtk, without having to modify it to
>take account of the fact that it's working in a
>gtk environment, or having to parameterise it to
>allow for such things.

Excellent.  To be clear, this is how the Twisted model
works, with respect to integration with GUI toolkits.
I would not enjoy working with a system in which this
was not the case.

>
>> You seem to be arguing for libraries that contain platform dependencies
>> to handle multiple platforms.
>
>I'm arguing that as much of the platform dependency
>as possible should be in the asyncore library (or
>whatever replaces it).

Certainly.  Library code doesn't care if the event
loop is driven by select or poll or epoll or /dev/poll
or kqueue or aio or iocp or win32 events or realtime
signals or kaio or whatever gnarly thing is hidden in
gtk or whatever gnarly thing is hidden inside qt or
whatever gnarly thing is hidden inside COM or whatever
gnarly thing is hidden inside wxWidgets.  It cares
about what features are available.  It requests them
somehow, and uses them.  If they are unavailable, then
it can decide whether the lack is catastrophic and give
up or if it can be worked around somehow.  The way a
Twisted application does this is based on interfaces.
Assuming interfaces continue to not be present in the
stdlib, a stdlib event loop would have to find some
other API for presenting this information, but it is not
a very hard problem to solve.

>The main application code
>*might* have to give it a hint such as "this app
>uses gtk", but no more than that. And ideally, I'd
>prefer it not to even have to do that -- pygtk
>should do whatever is necessary to hook itself
>into asyncore if at all possible, not the other
>way around.

There is some advantage to declaring things up front,
lest you get into the situation where you are partway
through using code which will suddenly begin to demand
Gtk at the same time as you are partway through using
code which will suddenly begin to demand Qt, at which
point you are in trouble.  But this is another minor
point.

>
>> Since Glyph has already stated his opinion that Twisted isn't yet ready
>> for adoption as-is this doesn't add to the discussion.
>
>Okay, but one of the suggestions made seemed to be
>"why not just use the Tw

Re: [Python-Dev] Trial balloon: microthreads library in stdlib

[Jean-Paul Calderone]

On Wed, 14 Feb 2007 15:20:13 +1300, Greg Ewing <[EMAIL PROTECTED]> wrote:
>

Greg, productive discussion is not furthered by the
unsupported statement of one position or another.

Instead of only stating what you believe to be a problem,
explain why you believe it is a problem.  A sentence like:

>
>The need for different event-driven mechanisms to compete
>with each other is the very problem that needs to be
>addressed.
>

Invites a response which merely contradicts it (for example,
"you are wrong"), an exchange which hasn't helped anyone to
understand anything better.  If you present supporting evidence
for the position, then the validity and the weight of that
evidence can be discussed, and one position or another might
be shown to have greater validity.

Also, show that you have fully understood the position you are
arguing against.  For example, if you respond to a message in
which someone claims to welcome something, don't respond by
saying that requiring that thing is bad.  As you know, welcoming
something is not the same as requiring that thing, so by making
this statement alone, you give the impression of talking past the
person to whom you are responding and it may seem to readers that
you haven't understood the other person's position.

>If Twisted is designed so that it absolutely *has* to
>use its own special event mechanism, and everything else
>needs to be modified to suit its requirements, then it's
>part of the problem, not part of the solution.
>

Here, you've built on your unsupported premise to arrive at
a conclusion which may be controversial.  Again, instead of
couching the debate in terms of what you might see as self
evidence problems, explain why you hold the position you do.
That way, the possibility is created for other people to come
to understand why you believe the conclusion to be valid.

You have presented what could be the beginning of supporting
evidence here, in saying that requiring "everything else" to
be modified is undesirable.  This is only a place to start,
not to end, though.  You may want to discuss the real scope
of modifications required (because "everything" is obviously
hyperbole, focusing on what changes are actually necessary
would be beneficial) and why you think that modifications are
necessary (it may not be clear to others why they are, or it
may be the case that others can correct misconceptions you
have).

For example, you might give a case in which you have needed to
integrate Twisted (or a different event framework) with another
event loop and describe difficulties you discovered.  This will
help advance the discussion around practical, specific concerns.
Without this focus, it is hard for a discussion to be productive,
since it will involve only vague handwaving.

Finally, it is often beneficial to avoid bringing up phrases such
as "the problem".  Particularly in a context such as this, where
the existing discussion is focusing on a specific issue, such as
the necessity or utility of adding a new set of functionality to
the Python standard library, the relevance of "the problem" may
not be apparent to readers.  In this case, some may not find it
obvious how a third party library can be "the problem" with such
new functionality.  If you explicitly spell out the detrimental
consequences of an action, instead of waving around "the problem",
the resulting discussion can be that much more productive and
focused.

Thanks

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Summary of "dynamic attribute access" discussion

[Jean-Paul Calderone]

On Tue, 13 Feb 2007 11:27:48 -0800, Mike Klaas <[EMAIL PROTECTED]> wrote:
>On 2/13/07, Josiah Carlson <[EMAIL PROTECTED]> wrote:
>
>> As for people who say, "but getattr, setattr, and delattr aren't used";
>> please do some searches of the Python standard library.  In a recent
>> source checkout of the trunk Lib, there are 100+ uses of setattr, 400+
>> uses of getattr (perhaps 10-20% of which being the 3 argument form), and
>> a trivial number of delattr calls.  In terms of applications where
>> dynamic attribute access tends to happen; see httplib, urllib, smtpd,
>> the SocketServer variants, etc.
>
>Another data point:  on our six-figure loc code base, we have 123
>instances of getattr, 30 instances of setattr, and 0 instances of
>delattr.  There are 5 instances of setattr( ... getattr( ... ) ) on
>one line (and probably a few more that grep didn't pick up that span
>multiple lines).

Another data point: in our six-figure loc code base, we have 469
instances of getattr, 91 instances of setattr, and 0 instances of
delattr.  There is one instances of setattr(..., getattr(...)), and
one instance of setattr(getattr(...), ...).

>
>+1 on .[] notation and the idea in general.
>

-1 on a syntax change for this.  Somewhere between -0 and +0 for a
builtin or library function like attrview().

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Summary of "dynamic attribute access" discussion

[Jean-Paul Calderone]

On Tue, 13 Feb 2007 17:20:02 +0100, "\"Martin v. Löwis\"" <[EMAIL PROTECTED]> 
wrote:
>Anthony Baxter schrieb:
>>> and the "wrapper class" idea of Nick Coghlan:
>>>attrview(obj)[foo]
>>
>> This also appeals - partly because it's not magic syntax 
>
>I also like this. I would like to spell it attrs, and
>I think its specification is
>
>class attrs:
>   def __init__(self, obj):
> self.obj = obj
>   def __getitem__(self, name):
> return getattr(self.obj, name)
>   def __setitem__(self, name, value):
> return setattr(self.obj, name, value)
>   def __delitem__(self, name):
> return delattr(self, name)
>   def __contains__(self, name):
> return hasattr(self, name)
>
>It's so easy people can include in their code for backwards
>compatibility; in Python 2.6, it could be a highly-efficient
>builtin (you still pay for the lookup of the name 'attrs',
>of course).

This looks nice.  The simplicity of the implementation is
great too.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Importing .pyc in -O mode and vice versa

[Jean-Paul Calderone]

On Tue, 07 Nov 2006 12:20:00 +1300, Greg Ewing <[EMAIL PROTECTED]> wrote:
>
>I think I'd be happy with having to do that explicitly.
>I expect the vast majority of Python programs don't
>need to track changes to the set of importable modules
>during execution. The exceptions would be things like
>IDEs, and they could do a cache flush before reloading
>a module, etc.

Another questionable optimization which changes application-
level semantics.

No, please?

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Importing .pyc in -O mode and vice versa

[Jean-Paul Calderone]

On Sun, 05 Nov 2006 14:21:34 +1300, Greg Ewing <[EMAIL PROTECTED]> wrote:
>Fredrik Lundh wrote:
>
>> well, from a performance perspective, it would be nice if Python looked
>> for *fewer* things, not more things.
>
>Instead of searching for things by doing a stat call
>for each possible file name, would it perhaps be
>faster to read the contents of all the directories
>along sys.path into memory and then go searching
>through that?

Bad for large directories.  There's a cross-over at some number
of entries.  Maybe Python should have a runtime-tuned heuristic
for selecting a filesystem traversal mechanism.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Path object design

[Jean-Paul Calderone]

On Wed, 01 Nov 2006 11:06:14 +0100, Georg Brandl <[EMAIL PROTECTED]> wrote:
>[EMAIL PROTECTED] wrote:
>> On 03:14 am, [EMAIL PROTECTED] wrote:
>>
>>  >One thing is sure -- we urgently need something better than os.path.
>>  >It functions well but it makes hard-to-read and unpythonic code.
>>
>> I'm not so sure.  The need is not any more "urgent" today than it was 5
>> years ago, when os.path was equally "unpythonic" and unreadable.  The
>> problem is real but there is absolutely no reason to hurry to a
>> premature solution.
>>
>> I've already recommended Twisted's twisted.python.filepath module as a
>> possible basis for the implementation of this feature.  I'm sorry I
>> don't have the time to pursue that.  I'm also sad that nobody else seems
>> to have noticed.  Twisted's implemenation has an advantage that it
>> doesn't seem that these new proposals do, an advantage I would really
>> like to see in whatever gets seriously considered for adoption:
>
>Looking at
>,
>it seems as if FilePath was made to serve a different purpose than what we're
>trying to discuss here:
>
>"""
>I am a path on the filesystem that only permits 'downwards' access.
>
>Instantiate me with a pathname (for example,
>FilePath('/home/myuser/public_html')) and I will attempt to only provide access
>to files which reside inside that path. [...]
>
>The correct way to use me is to instantiate me, and then do ALL filesystem
>access through me.
>"""
>
>What a successor to os.path needs is not security, it's a better (more 
>pythonic,
>if you like) interface to the old functionality.

No.  You've misunderstood the code you looked at.  FilePath serves exactly
the purpose being discussed here.  Take a closer look.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The "lazy strings" patch

[Jean-Paul Calderone]

On Mon, 23 Oct 2006 09:07:51 -0700, Josiah Carlson <[EMAIL PROTECTED]> wrote:
>
>"Paul Moore" <[EMAIL PROTECTED]> wrote:
>> I had picked up on this comment, and I have to say that I had been a
>> little surprised by the resistance to the change based on the "code
>> would break" argument, when you had made such a thorough attempt to
>> address this. Perhaps others had missed this point, though.
>
>I'm also concerned about future usability.

Me too (perhaps in a different way though).

>Word in the Py3k list is
>that Python 2.6 will be just about the last Python in the 2.x series,
>and by directing his implementation at only Python 2.x strings, he's
>just about guaranteeing obsolescence.

People will be using 2.x for a long time to come.  And in the long run,
isn't all software obsolete? :)

>By building with unicode and/or
>objects with a buffer interface in mind, Larry could build with both 2.x
>and 3.x in mind, and his code wouldn't be obsolete the moment it was
>released.

(I'm not sure what the antecedent of "it" is in the above, I'm going to
assume it's Python 3.x.)

Supporting unicode strings and objects providing the buffer interface seems
like a good idea in general, even disregarding Py3k.  Starting with str is
reasonable though, since there's still plenty of code that will benefit from
this change, if it is indeed a beneficial change.

Larry, I'm going to try to do some benchmarks against Twisted using this
patch, but given my current time constraints, you may be able to beat me
to this :)  If you're interested, Twisted [EMAIL PROTECTED] plus this trial 
plugin:

  http://twistedmatrix.com/trac/browser/sandbox/exarkun/merit/trunk

will let you do some gross measurements using the Twisted test suite.  I can
give some more specific pointers if this sounds like something you'd want to
mess with.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The "lazy strings" patch

[Jean-Paul Calderone]

On Mon, 23 Oct 2006 07:58:25 -0700, Larry Hastings <[EMAIL PROTECTED]> wrote:
> [snip]
>If external Python extension modules are as well-behaved as the shipping 
>Python source tree, there simply wouldn't be a problem.  Python source is 
>delightfully consistent about using the macro PyString_AS_STRING() to get at 
>the creamy char *center of a PyStringObject *.  When code religiously uses 
>that macro (or calls PyString_AsString() directly), all it needs is a 
>recompile with the current stringobject.h and it will Just Work.
>
>I genuinely don't know how many external Python extension modules are well- 
>behaved in this regard.  But in case it helps: I just checked PIL, NumPy, 
>PyWin32, and SWIG, and all of them were well-behaved.

FWIW, http://www.google.com/codesearch?q=+ob_sval

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Massive test_sqlite failure on Mac OSX ... sometimes

[Jean-Paul Calderone]

On Sun, 22 Oct 2006 07:51:27 -0500, [EMAIL PROTECTED] wrote:
>
>Ronald> According to a comment in (IIRC) the pyOpenGL sources GLUT on
>Ronald> OSX does a chdir() during initialization, that could be the
>Ronald> problem here.
>
>How would that explain that it fails on my g5 but not on my powerbook?  They
>are at the same revision of the operating system and compiler.  The
>checksums on the libraries are different though the file sizes are the same.
>The dates on the files are different as well.  I suspect the checksum
>difference is caused by the different upgrade dates of the two machines and
>the resulting different times the two systems were "optimized".
>
>Is there anyone else with a g5 who can do a vanilla Unix (not framework)
>build on an up-to-date g5 from an up-to-date Subversion repository?  It
>would be nice if someone else could at least confirm or not confirm this
>problem.

Robert Gravina has seen a problem which bears some resemblance to this one
while using PySQLite in a real application on OS X.  I've pointed him to
this thread; hopefully it's the same issue and a second way of producing
the issue will shed some more light on the matter.

The top of that thread is available here:

http://divmod.org/users/mailman.twistd/pipermail/divmod-dev/2006-October/000707.html

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Caching float(0.0)

[Jean-Paul Calderone]

On Sun, 1 Oct 2006 13:54:31 -0400, Terry Reedy <[EMAIL PROTECTED]> wrote:
>
>"Nick Craig-Wood" <[EMAIL PROTECTED]> wrote in message
>news:[EMAIL PROTECTED]
>> On Fri, Sep 29, 2006 at 12:03:03PM -0700, Guido van Rossum wrote:
>>> I see some confusion in this thread.
>>>
>>> If a *LITERAL* 0.0 (or any other float literal) is used, you only get
>>> one object, no matter how many times it is used.
>>
>> For some reason that doesn't happen in the interpreter which has been
>> confusing the issue slightly...
>>
>> $ python2.5
> a=0.0
> b=0.0
> id(a), id(b)
>> (134737756, 134737772)
>
>Guido said *a* literal (emphasis shifted), reused as in a loop or function
>recalled, while you used *a* literal, then *another* literal, without
>reuse.  Try a=b=0.0 instead.

Actually this just has to do with, um, "compilation units", for lack of a
better term:

  [EMAIL PROTECTED]:~$ python
  Python 2.4.3 (#2, Apr 27 2006, 14:43:58) 
  [GCC 4.0.3 (Ubuntu 4.0.3-1ubuntu5)] on linux2
  Type "help", "copyright", "credits" or "license" for more information.
  >>> a = 0.0
  >>> b = 0.0
  >>> print a is b
  False
  >>> ^D
  [EMAIL PROTECTED]:~$ cat > test.py
  a = 0.0
  b = 0.0
  print a is b
  ^D
  [EMAIL PROTECTED]:~$ python test.py
  True
  [EMAIL PROTECTED]:~$ cat > test_a.py
  a = 0.0
  ^D
  [EMAIL PROTECTED]:~$ cat > test_b.py
  b = 0.0
  ^D
  [EMAIL PROTECTED]:~$ cat > test.py
  from test_a import a
  from test_b import b
  print a is b
  ^D
  [EMAIL PROTECTED]:~$ python test.py
  False
  [EMAIL PROTECTED]:~$ python
  Python 2.4.3 (#2, Apr 27 2006, 14:43:58) 
  [GCC 4.0.3 (Ubuntu 4.0.3-1ubuntu5)] on linux2
  Type "help", "copyright", "credits" or "license" for more information.
  >>> a = 0.0; b = 0.0
  >>> print a is b
  True
  >>> 
  [EMAIL PROTECTED]:~$

Each line in an interactive session is compiled separately, like modules
are compiled separately.  With the current implementation, literals in a
single compilation unit have a chance to be "cached" like this.  Literals
in different compilation units, even for the same value, don't.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] deja-vu .. python locking

[Jean-Paul Calderone]

On Mon, 18 Sep 2006 17:06:47 +0200, Martin Devera <[EMAIL PROTECTED]> wrote:
>Martin v. Löwis wrote:
>> Martin Devera schrieb:
>>> RCU like locking
>>>   Solution I have in mind is similar to RCU. In Python we have quiscent
>>>   state - when a thread returns to main loop of interpreter.
>>
>> There might be a terminology problem here. RCU is read-copy-update,
>> right? I fail to see the copy (copy data structure to be modified)
>> and update (replace original pointer with pointer to copy) part.
>> Do this play a role in that scheme? If so, what specific structure
>> is copied for, say, a list or a dict?
>>
>> This confusion makes it very difficult for me to understand your
>> proposal, so I can't comment much on it. If you think it could work,
>> just go ahead and create an implementation.
>
>It is why I used a word "similar". I see the similarity in a way to archieve
>safe "delete" phase of RCU. Probably I selected bad title for the text. It
>is because I was reading about RCU implementation in Linux kernel and
>I discovered that the idea of postponing critical code to some safe point in
>future might work in Python interpreter.
>
>So that you are right. It is not RCU. It only uses similar technique as RCU
>uses for free-ing old copy of data.
>
>It is based on assumption that an object is typicaly used by single thread. 

Which thread owns builtins?  Or module dictionaries?  If two threads are
running the same function and share no state except their globals, won't
they constantly be thrashing on the module dictionary?  Likewise, if the
same method is running in two different threads, won't they thrash on the
class dictionary?

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Testsuite fails on Windows if a space is in the path

[Jean-Paul Calderone]

On Sat, 16 Sep 2006 19:22:34 +0200, "\"Martin v. Löwis\"" <[EMAIL PROTECTED]> 
wrote:
>The test suite currently (2.5) has two failures on Windows
>if Python is installed into a directory with a space in it
>(such as "Program Files"). The failing tests are test_popen
>and test_cmd_line.
>
>The test_cmd_line failure is shallow: the test fails to properly
>quote sys.executable when passing it to os.popen. I propose to
>fix this in Python 2.5.1; see #1559413
>
>test_popen is more tricky. This code has always failed AFAICT,
>except that the test itself is a recent addition. The test tries
>to pass the following command to os.popen
>
>"c:\Program Files\python25\python.exe" -c "import sys;print sys.version"
>
>For some reason, os.popen invokes doesn't directly start Python as
>a new process, but install invokes
>
>cmd.exe /c "c:\Program Files\python25\python.exe" -c "import sys;print
>sys.version"
>
>Can somebody remember what the reason is to invoke cmd.exe (or COMSPEC)
>in os.popen?

I would guess it was done to force cmd.exe-style argument parsing in the
subprocess, which is optional on Win32.

>
>In any case, cmd.exe fails to execute this, claiming that c:\Program
>is not a valid executable. It would run
>
>cmd.exe /c "c:\Program Files\python25\python.exe"
>
>just fine, so apparently, the problem is with argument that have
>multiple pairs of quotes. I found, through experimentation, that it
>*will* accept
>
>cmd.exe /c ""c:\Program Files\python25\python.exe" -c "import sys;print
>sys.version""
>
>(i.e. doubling the quotes at the beginning and the end). I'm not quite
>sure what algorithm cmd.exe uses for parsing, but it appears that
>adding a pair of quotes works in all cases (at least those I could think
>of). See # 1559298

You can find the quoting/dequoting rules used by cmd.exe documented on msdn:

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vclang/html/_pluslang_Parsing_C.2b2b_.Command.2d.Line_Arguments.asp

Interpreting them is something of a challenge (my favorite part is how the
examples imply that the final argument is automatically uppercased ;)

Here is an attempted implementation of the quoting rules:

http://twistedmatrix.com/trac/browser/trunk/twisted/python/win32.py#L41

Whether or not it is correct is probably a matter of discussion.  If you find
a more generally correct solution, I would certainly like to know about it.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Twisted-Python] Newbie question

[Jean-Paul Calderone]

Sorry, brainfart.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Twisted-Python] Newbie question

[Jean-Paul Calderone]

On Thu, 7 Sep 2006 11:41:48 -0400, Timothy Fitz <[EMAIL PROTECTED]> wrote:
>On 9/5/06, Jean-Paul Calderone <[EMAIL PROTECTED]> wrote:
>>You cannot stop the reactor and then start it again.
>
>Why don't the reactors throw if this happens? This question comes up
>almost once a month.
>

One could just as easily ask why no one bothers to read mailing list
archives to see if their question has been answered before.

No one will ever know, it is just one of the mysteries of the universe.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com