date:20090401


On Wed, 1 Apr 2009 at 05:37, [email protected] wrote:

This case arises rarely, but it does turn up every now and again.  If you


For some definition of "rarely".  I don't handle CVS files generated by
Windows very often, but I've run into it a least a couple times.  That
says to me that it isn't all that rare in the wild.  (One out of
fifty?  But I'm sure it depends on your data sources; some people
will run into it often, others almost never.)

Of course, on unix it doesn't help much having those newlines preserved,
since there are few tools on unix other than the CSV module that even
attempt to deal with newlines inside quoted strings being data, but on
Windows it makes a difference.

It would actually be nice if the CSV module had an option for turning
those quoted newlines into spaces, but that's a feature request and
is out of scope for this discussion :)


   Antoine> The documentation is, IMO, wrong even in 2.x. Just yesterday I
   Antoine> had to open a CSV file in 'rU' mode because it had Windows line
   Antoine> endings and I'm under Linux


That sounds like a bug, IMO.  From the source code it looks like the
2.6 _csv module should be handling that, and certainly intended to
handle it.

--David
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Python 2.6 64-bit Mac Release

2009-04-01 Thread Peck, Jon

Apparently the Mac Python 2.6.1 Installer image does not include 64-bit 
binaries.  Is this going to change?  Is there some technical reason why these 
are not included?  We are hoping to support that in our next 64-bit release.

 

Thanks for your help.

 

Jon K. Peck

SPSS Inc.

[email protected]

(ip) phone 312-651-3435

 

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] All Hail the FLUFL

2009-04-01 Thread The Extruder

On behalf of the entire Python community and as CTO of Cheez Whiz Global
Conglomerates, Inc. I would like to extend My Thanks to our BDEVIL's many
Selfless and Dedicated years of service.  I can say without remorse that the
State of the Art in the Gas Propelled Cheezical Sciences would be 11 years
behind schedule if it weren't for the BDEVIL and Python.

However, as we at CWGC have been Contemplating an upgrade to our Pythonic
Orange Oscillation Process for many years - we're still on Python 1.5.1 and
due to our Centuries-Old corporate culture, only upgrade to multiples of two -
we have been dismayed by recent Unfortunate Decisions negatively impacting our
hope of world-wide long term acceptance of Python 3.0.2.  Thus I applaud you
for your chosen New Path, wish you a Speedy Shirpa Assisted Ascent, and I
welcome our new FLUFL's similarly Lofty Ascent to Pythonic Overlordhood.

You may not realize that the technology behind our groundbreaking Subsonic
Plasticine Extrusion of Coagulated Flavor Oil utilize inequalities to a vast
degree.  In fact, our many Published Papers confirm our mathematical
leadership in the Algebraic States of Less Than and More Than Simultaneity.
Were it not for the Diamond Operator, billions, nay! trillions of crackers
would have languished Unadorned, Unenjoyed, and Unloved.

For this reason, the sole choice of the Evil Hash-Equal was enough to force us
to Seriously Investigate a switch to Stenographic Non-deterministic SQL (a
promising new scripting language somewhat similar to Ruby).  It is with an
Overboiling Pot of Joy that we fully support Official FLUFL Act 2.  Trust me
when I say that with this single reversion, the world of High Velocity
Extremely High Pressure Milkyish Orange Goop Delivery Devices will never be
the same.

I am also Ecstatic at the reversal of DVCS decision.  It is with no Small
Irony that I admit the mere utterance of any derivative of the root word
"Mercury" is a firing offence in my Establishment.

In honor of our new FLUFL, I am directing our CFO Timmy "The Larch" Lomax (no
direct relation to myself) to donate the sum of USA $23,250 to the PSU in
furtherance of their mission.  If there is a PyCon sponsorship level Above
Diamond (may we suggest "Orange"?) we would be honored to claim that Pinnacle
for 2010.  Atlanta is located very near our Secret Manufacturing Facility and
I would be remiss if I did not direct additional PyCon branded delivery of
5000 cans of our Premium Velvet Brand Cheez Whiz Lunchables with Detachable
Shooters.  I think the 2010 conference attendees will appreciate the
diversion and hope this will entice people to join our Sprint next year.

foolish-ly y'rs,
frank
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] issue5578 - explanation


On Wed, 1 Apr 2009 at 13:12, Chris Withers wrote:

Guido van Rossum wrote:

 Well hold on for a minute, I remember we used to have an exec
 statement in a class body in the standard library, to define some file
 methods in socket.py IIRC. 


But why an exec?! Surely there must be some other way to do this than an 
exec?


Maybe, but this sure is gnarly code:

_s = ("def %s(self, *args): return self._sock.%s(*args)\n\n"
  "%s.__doc__ = _realsocket.%s.__doc__\n")
for _m in _socketmethods:
exec _s % (_m, _m, _m, _m)
del _m, _s

Guido's memory is good, that's from the _socketobject class in
socket.py.

--David
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] issue5578 - explanation

2009-04-01 Thread Jeremy Hylton

I posted in the bug report, but repeating here:  I don't remember why
exec in a nested function changed either.  It would help if someone
could summarize why we made the change.  (I hope I didn't do it <0.2
wink>.)

Jeremy

On Tue, Mar 31, 2009 at 11:36 PM, Maciej Fijalkowski  wrote:
> Because classes have now it's own local scope (according to Martin)
>
> It's not about exec in class, it's about exec in class in nested function.
>
> On Wed, Apr 1, 2009 at 5:25 AM, Guido van Rossum  wrote:
>> Well hold on for a minute, I remember we used to have an exec
>> statement in a class body in the standard library, to define some file
>> methods in socket.py IIRC.  It's a totally different case than exec in
>> a nested function, and I don't believe it should be turned into a
>> syntax error at all. An exec in a class body is probably meant to
>> define some methods or other class attributes. I actually think the
>> 2.5 behavior is correct, and I don't know why it changed in 2.6.
>>
>> --Guido
>>
>> On Tue, Mar 31, 2009 at 8:15 PM, Maciej Fijalkowski  wrote:
>>> So. The issue was closed and I suppose it was closed by not entirely
>>> understanding
>>> the problem (or I didn't get it completely).
>>>
>>> The question is - what the following code should do?
>>>
>>> def f():
>>>  a = 2
>>>  class C:
>>>    exec 'a = 42'
>>>    abc = a
>>>  return C
>>>
>>> print f().abc
>>>
>>> (quick answer - on python2.5 it return 42, on python 2.6 and up it
>>> returns 2, the patch changes
>>> it to syntax error).
>>>
>>> I would say that returning 2 is the less obvious thing to do. The
>>> reason why IMO this should
>>> be a syntax error is this code:
>>>
>>> def f():
>>>  a = 2
>>>  def g():
>>>    exec 'a = 42'
>>>    abc = a
>>>
>>> which throws syntax error.
>>>
>>> Cheers,
>>> fijal
>>> ___
>>> Python-Dev mailing list
>>> [email protected]
>>> http://mail.python.org/mailman/listinfo/python-dev
>>> Unsubscribe: 
>>> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>>>
>>
>>
>>
>> --
>> --Guido van Rossum (home page: http://www.python.org/~guido/)
>>
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/jeremy%40alum.mit.edu
>
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Let's update CObject API so it is safe and regular!

2009-04-01 Thread Brett Cannon

On Tue, Mar 31, 2009 at 20:34, Guido van Rossum  wrote:

> Can you get Jim Fulton's feedback? ISTR he originated this.
>

I thought Neal started this idea?

-Brett


>
> On Tue, Mar 31, 2009 at 12:14 PM, Larry Hastings 
> wrote:
> >
> > The CObject API has two flaws.
> >
> > First, there is no usable type safety mechanism.  You can store a void
> > *object, and a void *description.  There is no established schema for
> > the description; it could be an integer cast to a pointer, or it could
> > point to memory of any configuration, or it could be NULL.  Thus users
> > of the CObject API generally ignore it--thus working without any type
> > safety whatsoever.  A programmer could crash the interpreter from pure
> > Python by mixing and matching CObjects from different modules (e.g. give
> > "curses" a CObject from "_ctypes").
> >
> > Second, the destructor callback is defined as taking *either* one *or*
> > two parameters, depending on whether the "descr" pointer is non-NULL. One
> > can debate the finer points of what is and isn't defined behavior in
> > C, but at its heart this is a sloppy API.
> >
> > MvL and I discussed this last night and decided to float a revision of
> > the API.  I wrote the patch, though, so don't blame Martin if you don't
> > like my specific approach.
> >
> > The color of this particular bike shed is:
> > * The PyCObject is now a private data structure; you must use accessors.
> >  I added accessors for all the members.
> > * The constructors and the main accessor (PyCObject_AsVoidPtr) now all
> >  *require* a "const char *type" parameter, which must be a non-NULL C
> >  string of non-zero length.  If you call that accessor and the "type"
> >  is invalid *or doesn't match*, it fails.
> > * The destructor now takes the PyObject *, not the PyCObject *.  You
> >  must use accessors to get your hands on the data inside to free it.
> >
> > Yes, you can easily skip around the "matching type" restriction by
> > calling PyCObject_AsVoidPtr(cobj, PyCObject_GetType(cobj)).  The point
> > of my API changes is to *encourage* correct use.
> >
> > I've posted a patch implementing this change in the 3.1 trunk to the
> > bug tracker:
> >
> >   http://bugs.python.org/issue5630
> >
> > I look forward to your comments!
> >
> >
> > /larry/
> >
> > ___
> > Python-Dev mailing list
> > [email protected]
> > http://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > http://mail.python.org/mailman/options/python-dev/guido%40python.org
> >
>
>
>
> --
> --Guido van Rossum (home page: 
> http://www.python.org/~guido/
> )
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Let's update CObject API so it is safe and regular!

2009-04-01 Thread Kristján Valur Jónsson

What are the semantics of the "type" argument for PyCObject_FromVoidPtr()?
-Does it do a strdup, or is the type required to be valid while the object 
exists (e.g. a static string)?
-How is the type match determined, strcmp, or pointer comparison?

-Original Message-
From: [email protected] 
[mailto:[email protected]] On Behalf Of Larry 
Hastings
Sent: 31. mars 2009 19:15
To: [email protected]
Subject: [Python-Dev] Let's update CObject API so it is safe and regular!

* The constructors and the main accessor (PyCObject_AsVoidPtr) now all
  *require* a "const char *type" parameter, which must be a non-NULL C
  string of non-zero length.  If you call that accessor and the "type"
  is invalid *or doesn't match*, it fails.

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.6 64-bit Mac Release

2009-04-01 Thread Ronald Oussoren



On 1 Apr, 2009, at 8:45, Peck, Jon wrote:

Apparently the Mac Python 2.6.1 Installer image does not include 64- 
bit binaries.  Is this going to change?  Is there some technical  
reason why these are not included?  We are hoping to support that in  
our next 64-bit release.


The 2.6 installer image does not include 64-bit binaries.  As of this  
week the script that creates the installer can build an installer that  
does support 64-bit code as well, but that only works on Leopard  
systems.


I'm thinking about how to distribute binaries that support 64-bit code  
without unduly complicating the world. The easiest option for me would  
be to have two installers: one 32-bit only that supports OSX 10.3.9  
and later and a 4-way universal one that supports OSX Leopard and  
later.  It might be possible to have a single installer that supports  
64-bit code on Leopard but is usable on 10.3.9 as well, but I haven't  
checked yet how much that would complicate the build.


Ronald



smime.p7s
Description: S/MIME cryptographic signature
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.6 64-bit Mac Release

2009-04-01 Thread Michael Foord


Ronald Oussoren wrote:


On 1 Apr, 2009, at 8:45, Peck, Jon wrote:

Apparently the Mac Python 2.6.1 Installer image does not include 
64-bit binaries.  Is this going to change?  Is there some technical 
reason why these are not included?  We are hoping to support that in 
our next 64-bit release.


The 2.6 installer image does not include 64-bit binaries.  As of this 
week the script that creates the installer can build an installer that 
does support 64-bit code as well, but that only works on Leopard systems.


I'm thinking about how to distribute binaries that support 64-bit code 
without unduly complicating the world. The easiest option for me would 
be to have two installers: one 32-bit only that supports OSX 10.3.9 
and later and a 4-way universal one that supports OSX Leopard and 
later.  It might be possible to have a single installer that supports 
64-bit code on Leopard but is usable on 10.3.9 as well, but I haven't 
checked yet how much that would complicate the build.




Two installers sounds OK to me, particularly if it simplifies the build 
process but allows us to still support 64bit.


Michael


Ronald



___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
  



--
http://www.ironpythoninaction.com/

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] issue5578 - explanation

2009-04-01 Thread Jeremy Hylton

Eeek, I think it was me.  Part of the AST changes involved raising a
SyntaxError when exec was used in a scope that had a free variable,
since the behavior is pretty much undefined.  If the compiler decides
a variable is free, then it can't be assigned to in the function body.
 The compiled exec code can't know whether the variable is local or
free in the exec context, only that it should generate a STORE_NAME
opcode.  The STORE_NAME can't possibly work.

It looks like I did a bad job of documenting the change, though.  I
had forgotton about it ,because it was three or four years ago.

It looks like the same exception should be raised for the class statement.

Jeremy

On Wed, Apr 1, 2009 at 11:00 AM, R. David Murray  wrote:
> On Wed, 1 Apr 2009 at 13:12, Chris Withers wrote:
>>
>> Guido van Rossum wrote:
>>>
>>>  Well hold on for a minute, I remember we used to have an exec
>>>  statement in a class body in the standard library, to define some file
>>>  methods in socket.py IIRC.
>>
>> But why an exec?! Surely there must be some other way to do this than an
>> exec?
>
> Maybe, but this sure is gnarly code:
>
>    _s = ("def %s(self, *args): return self._sock.%s(*args)\n\n"
>          "%s.__doc__ = _realsocket.%s.__doc__\n")
>    for _m in _socketmethods:
>        exec _s % (_m, _m, _m, _m)
>    del _m, _s
>
> Guido's memory is good, that's from the _socketobject class in
> socket.py.
>
> --David
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/jeremy%40alum.mit.edu
>
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 3to2 Project

2009-04-01 Thread Ron DuPlain

On Mon, Mar 30, 2009 at 9:29 PM, Benjamin Peterson  wrote:
> 2009/3/30 Collin Winter :
>> If anyone is interested in working on this during the PyCon sprints or
>> otherwise, here are some easy, concrete starter projects that would
>> really help move this along:
>> - The core refactoring engine needs to be broken out from 2to3. In
>> particular, the tests/ and fixes/ need to get pulled up a directory,
>> out of lib2to3/.
>> - Once that's done, lib2to3 should then be renamed to something like
>> librefactor or something else that indicates its more general nature.
>> This will allow both 2to3 and 3to2 to more easily share the core
>> components.
>
> FWIW, I think it is unfortunately too late to make this change. We've
> already released it as lib2to3 in the standard library and I have
> actually seen it used in other projects. (PythonScope, for example.)
>

Paul Kippes and I have been sprinting on this.  We put lib2to3 into a
refactor package and kept a shell lib2to3 to support the old
interface.

We are able to run 2to3, 3to2, lib2to3 tests, and refactor tests.  We
only have a few simple 3to2 fixes now, but they should be easy to add.
 We kept the old lib2to3 tests to make sure we didn't break anything.
As things settle down, I'd like to verify that our new lib2to3 is
backward-compatible (since right now it points to the new refactor
lib) with one of the external projects.

We've been using hg to push changesets between each other, but we'll
be committing to the svn sandbox before the week is out.  I'm heading
out today, but Paul is sticking around another day.

It's a start,

Ron

>
> --
> Regards,
> Benjamin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Wing IDE and python.wpr

2009-04-01 Thread Michael Foord


Hello all,

How many are using the Wing IDE to work on core Python?

It would be nice to have a 'python.wpr' checked in to trunk, as I have 
to recreate the project file every time I do a new checkout. Would this 
be useful for anyone else? Where is a good place for it to live? 
Littering the top level directory seems like a bad idea but I can't see 
anywhere else immediately *obvious* (no reason it has to live at the top 
level).


Wing can be configured to use two files for the project - one file for 
the basic configuration (which would be checked in) and one for your 
personal settings (which files you have open, how many windows you are 
using etc) and would be svn-ignored.


Michael Foord

--
http://www.ironpythoninaction.com/

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Let's update CObject API so it is safe and regular!



Brett Cannon wrote:
On Tue, Mar 31, 2009 at 20:34, Guido van Rossum > wrote:


Can you get Jim Fulton's feedback? ISTR he originated this.


I thought Neal started this idea?


The earliest revision spotted in "svn blame cobject.[ch]" is 5782:

   svn log -r 5782
   
   r5782 | guido | 1996-01-11 16:44:03 -0800 (Thu, 11 Jan 1996) | 2 lines

   opaque C object a la Jim Fulton

I'll email Jim Fulton and inquire.


/larry/
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Wing IDE and python.wpr

2009-04-01 Thread Michael Foord


Michael Foord wrote:

Hello all,

How many are using the Wing IDE to work on core Python?

It would be nice to have a 'python.wpr' checked in to trunk, as I have 
to recreate the project file every time I do a new checkout. Would 
this be useful for anyone else? Where is a good place for it to live? 
Littering the top level directory seems like a bad idea but I can't 
see anywhere else immediately *obvious* (no reason it has to live at 
the top level).


Wing can be configured to use two files for the project - one file for 
the basic configuration (which would be checked in) and one for your 
personal settings (which files you have open, how many windows you are 
using etc) and would be svn-ignored.

The Wing project file is now checked in. It is Misc/python-wing.wpr

The project is configured with SVN integration enabled, with two file 
configuration and the wpu file SVN ignored plus the main project 
directory added.


The wpr file is text so changes are diff friendly.

There is an issue with the way the project is displayed - the Misc 
directory is the top-level with '..' showing as another directory in the 
project. This issue will be resolved in the next version of Wing.


There are various other feature-requests now with Wing to better support 
using it for developing Python. Currently the debugger doesn't work with 
a newly built version of Python and the executable name / location is 
platform dependent and so setting a custom executable would only work on 
one platform.


It would be easy to add custom tools to (for example) integrate regrtest 
or do the configure / make dance on a fresh checkout.


All the best,

Michael



Michael Foord




--
http://www.ironpythoninaction.com/

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Let's update CObject API so it is safe and regular!


Kristján Valur Jónsson wrote:

What are the semantics of the "type" argument for PyCObject_FromVoidPtr()?
  


From the patch, from the documentation comment above the prototype for 
PyCObject_FromVoidPtr() in Include/cobject.h:


   The "type" string must point to a legal C string of non-zero length,



-Does it do a strdup, or is the type required to be valid while the object 
exists (e.g. a static string)?
  


From the patch, continuing on from where we just left off:

   and this string must outlive the CObject.



-How is the type match determined, strcmp, or pointer comparison?


From the patch, observing the code in the static function 
_is_legal_cobject_and_type() in Objects/cobject.c:


   if (!type || !*type) {
   PyErr_SetString(PyExc_TypeError, invalidType);
   return 0;
   }
   if (strcmp(type, self->type)) {
   PyErr_SetString(PyExc_TypeError, incorrectType);
   return 0;
   }


A method for answering further such questions suggests itself,


/larry//
/
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Let's update CObject API so it is safe and regular!

2009-04-01 Thread Jim Fulton

On Mar 31, 2009, at 3:14 PM, Larry Hastings wrote:

(Thanks for calling my attention to this. :)

The CObject API has two flaws.

First, there is no usable type safety mechanism. You can store a void
*object, and a void *description. There is no established schema for
the description; it could be an integer cast to a pointer, or it could
point to memory of any configuration, or it could be NULL. Thus users
of the CObject API generally ignore it--thus working without any type
safety whatsoever. A programmer could crash the interpreter from pure
Python by mixing and matching CObjects from different modules (e.g.
give

"curses" a CObject from "_ctypes").

The description field wasn't in the original CObject implementation
that I was involved with many years ago. Looking at it now, I don't
think it is intended as a type-safety mechanism at all, but as a way
to pass data to the destructor. I don't know what motivated this. (I
don't know why it it's called "description". This name seems to be
very confusing.)

The only type-safety mechanism for a CObject is it's identity. If you
want to make sure you're using the foomodule api, make sure the
address of the CObject is the same as the address of the api object
exported by the module.

The exporting module should automate use of the C API by providing an
appropriate header file, as described in http://docs.python.org/extending/extending.html#providing-a-c-api-for-an-extension-module
.

Second, the destructor callback is defined as taking *either* one *or*
two parameters, depending on whether the "descr" pointer is non-
NULL. One can debate the finer points of what is and isn't defined
behavior in

C, but at its heart this is a sloppy API.

It was necessary for backward compatibility. I don't know what
motivated this, so I don't know if the benefit was worth the ugliness.

MvL and I discussed this last night and decided to float a revision of
the API. I wrote the patch, though, so don't blame Martin if you
don't

like my specific approach.

The color of this particular bike shed is:
* The PyCObject is now a private data structure; you must use
accessors.

I added accessors for all the members.

The original implementation didn't expose the structure. I don't know
why it was exposed. It would be backward incompatible to hide it again
now.

* The constructors and the main accessor (PyCObject_AsVoidPtr) now all
*require* a "const char *type" parameter, which must be a non-NULL C
string of non-zero length. If you call that accessor and the "type"
is invalid *or doesn't match*, it fails.

That would break backward compatibility. Are you proposing this for
Python 3?

What would be the gain in this? The CObject is already a type
identifier for itself. In any case, client code generally doesn't
mess with CObjects directly anyway.

* The destructor now takes the PyObject *, not the PyCObject *. You
must use accessors to get your hands on the data inside to free it.

It currently isn't passed the CObject, but the C pointer that it
holds. In any case, changing the API isn't practical, at least not
for Python 2.

Yes, you can easily skip around the "matching type" restriction by
calling PyCObject_AsVoidPtr(cobj, PyCObject_GetType(cobj)). The point
of my API changes is to *encourage* correct use.

I've posted a patch implementing this change in the 3.1 trunk to the
bug tracker:

http://bugs.python.org/issue5630

I look forward to your comments!

-1

I don't see that this gains anything.

1. All you're adding, afaict is a name for the API and the (address of
the) CObject itself already provides this.

2. Only code provided by the module provider should be accessing the
CObject exported by the module.

Jim

--
Jim Fulton
Zope Corporation

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] bdb.py trace C implementation?

2009-04-01 Thread David Christian

Hi all,
I've recently written a C version of the trace function used in
figleaf (the coverage tool written by Titus).  After a few rewrites to
add in caching, etc, it gives users a significant speedup.  One person
stated that switching to the C version caused coverage to decrease
from a 442% slowdown to only a 56% slowdown.

You can see my C implementation at:
 
http://github.com/ctb/figleaf/blob/e077155956c288b68704b09889ebcd675ba02240/figleaf/_coverage.c

(Specific comments about the implementation welcome off-list).

I'd like to attempt something similar for bdb.py (only for the trace
function).  A naive C trace function which duplicated the current
python function should speed up bdb significantly.  I would initially
write the smallest part of the C implementation that I could.
Basically the tracing function would call back out to python at any
point where a line requires action.

I'd be willing to maintain the C implementation.  I would be willing
to write those tests that are possible as well.

Is this something that would be likely to be accepted?

Thanks,
David Christian
Senior Software Engineer
rPath, Inc.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] CSV, bytes and encodings


On Wed, 1 Apr 2009 at 10:53, Antoine Pitrou wrote:

Perhaps. But without using 'rU' the file couldn't be read at all.
(I'm not sure it was Windows line endings by the way; perhaps Macintosh ones;
anyway, it didn't work using 'rb')


I just tested it in 2.6.  It must have been old-mac (\r), which indeed
gave me the error message you mentioned.  Windows lineneds worked fine
for me reading in binary mode on linux.


I have to add that if individual fields really can contain newlines, then the
CSV module ought to be smarter when /saving/ those fields. I've inadvertently
tried to produce a CSV file with such fields and it ended up wrong when opened
as a spreadsheet (text after the newlines was ignored in Gnumeric and in
OpenOffice, while Excel displayed a spurious additional row containing only the
text after the newline).


I just added some tests to trunk that seem to indicate this case is
handled correctly in terms of preserving the data.  Maybe you didn't
write the file such that the fields with the newlines were quoted?
And of course how non-Excel applications handle that data on import
can be different from how Excel handles it.

--David
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bdb.py trace C implementation?

2009-04-01 Thread Benjamin Peterson

2009/4/1 David Christian :
> Hi all,
> I've recently written a C version of the trace function used in
> figleaf (the coverage tool written by Titus).  After a few rewrites to
> add in caching, etc, it gives users a significant speedup.  One person
> stated that switching to the C version caused coverage to decrease
> from a 442% slowdown to only a 56% slowdown.
>
> You can see my C implementation at:
>  http://github.com/ctb/figleaf/blob/e077155956c288b68704b09889ebcd675ba02240/figleaf/_coverage.c
>
> (Specific comments about the implementation welcome off-list).
>
> I'd like to attempt something similar for bdb.py (only for the trace
> function).  A naive C trace function which duplicated the current
> python function should speed up bdb significantly.  I would initially
> write the smallest part of the C implementation that I could.
> Basically the tracing function would call back out to python at any
> point where a line requires action.
>
> I'd be willing to maintain the C implementation.  I would be willing
> to write those tests that are possible as well.
>
> Is this something that would be likely to be accepted?

Generally debugging doesn't require good performance, so this is
definitely low priority. However, if you can contribute it, I don't
have a problem with it.



-- 
Regards,
Benjamin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] CSV, bytes and encodings


OK, Antoine, having merged my newline tests to py3k and having
them work when lineend is set to '', as you suggested on the
ticket, I'm inclined to agree with you that this is a doc bug.

Skip?

--David
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bdb.py trace C implementation?

On Wed, Apr 1, 2009 at 3:25 PM, Benjamin Peterson  wrote:
> 2009/4/1 David Christian :
>> Hi all,
>> I've recently written a C version of the trace function used in
>> figleaf (the coverage tool written by Titus).  After a few rewrites to
>> add in caching, etc, it gives users a significant speedup.  One person
>> stated that switching to the C version caused coverage to decrease
>> from a 442% slowdown to only a 56% slowdown.
>>
>> You can see my C implementation at:
>>  http://github.com/ctb/figleaf/blob/e077155956c288b68704b09889ebcd675ba02240/figleaf/_coverage.c
>>
>> (Specific comments about the implementation welcome off-list).
>>
>> I'd like to attempt something similar for bdb.py (only for the trace
>> function).  A naive C trace function which duplicated the current
>> python function should speed up bdb significantly.  I would initially
>> write the smallest part of the C implementation that I could.
>> Basically the tracing function would call back out to python at any
>> point where a line requires action.
>>
>> I'd be willing to maintain the C implementation.  I would be willing
>> to write those tests that are possible as well.
>>
>> Is this something that would be likely to be accepted?
>
> Generally debugging doesn't require good performance, so this is
> definitely low priority. However, if you can contribute it, I don't
> have a problem with it.

Tracing has other uses besides debugging though. In particular,
coverage, which usually wants per-line data. Also, sometimes if you
set a breakpoint in a function it turns on tracing for the entire
function. This can sometimes be annoyingly slow. So, personally, I am
more positive than that, and hope it will make it in.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bdb.py trace C implementation?

2009-04-01 Thread Benjamin Peterson

2009/4/1 Guido van Rossum :
> Tracing has other uses besides debugging though.

The OP said he wished to implement a C trace function for bdb.
Wouldn't that make it only applicable to debugging?



-- 
Regards,
Benjamin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bdb.py trace C implementation?

2009-04-01 Thread Robert Kern


On 2009-04-01 17:53, Benjamin Peterson wrote:

2009/4/1 Guido van Rossum:

Tracing has other uses besides debugging though.


The OP said he wished to implement a C trace function for bdb.
Wouldn't that make it only applicable to debugging?


Once you are at the breakpoint and stepping through the code manually, the 
performance is not all that important. However, up until that breakpoint, you 
are running a lot of code "in bulk". It would be useful to have a performant 
trace function that interferes with that code the least.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bdb.py trace C implementation?

2009-04-01 Thread David Christian

On Wed, Apr 1, 2009 at 6:53 PM, Benjamin Peterson  wrote:
> 2009/4/1 Guido van Rossum :
>> Tracing has other uses besides debugging though.
>
> The OP said he wished to implement a C trace function for bdb.
> Wouldn't that make it only applicable to debugging?
>
> Benjamin
>
I was suggesting a speedup for debugging.  However, I could certainly
also contribute my figleaf work that I referenced earlier, with a few
tweaks, as a tracing replacement for the tracing function in trace.py.

My concern with moving the coverage tracing code in particular to the
standard library is that it tries to extract the maximum speed by
being clever*, and certainly has not been out in the wild for long
enough.  I would write something much more conservative as a starting
point for bdb.py.  I expect that any C implementation that was
thinking about performance at all would be much better than the status
quo.

* figleaf checks a regular expression to determine whether or not we
wish to trace a particular file.  If the file is not being traced, I
switch to the profiler instead of the line tracer, which means that
the trace function only gets called twice per function instead of once
per line.  This can give a large speedup when you are skipping the
entire standard library, at some measurable cost per function call,
and a cost in code complexity.

---
David Christian
Senior Software Engineer
rPath, Inc
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bdb.py trace C implementation?

On Wed, Apr 1, 2009 at 3:53 PM, Benjamin Peterson  wrote:
> 2009/4/1 Guido van Rossum :
>> Tracing has other uses besides debugging though.
>
> The OP said he wished to implement a C trace function for bdb.
> Wouldn't that make it only applicable to debugging?

I honestly don't recall, but I believe pretty much everyone who uses
tracing does so via bdb.py. And yes, when debugging sometimes you have
to silently skip 1000 iterations until a condition becomes true, and
the tracking speed matters.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Let's update CObject API so it is safe and regular!



Jim Fulton wrote:
The only type-safety mechanism for a CObject is it's identity.  If you 
want to make sure you're using the foomodule api, make sure the 
address of the CObject is the same as the address of the api object 
exported by the module.
That doesn't help.  Here's a program that crashes the interpreter, 
something I shouldn't be able to do from pure Python:


   import _socket
   import cStringIO
   cStringIO.cStringIO_CAPI = _socket.CAPI

   import cPickle
   s = cPickle.dumps([1, 2, 3])

How can cPickle determine that cStringIO.cStringIO_CAPI is legitimate?

That would break backward compatibility. Are you proposing this for 
Python 3?


I'm proposing this for Python 3.1.  My understanding is that breaking 
backwards compatibility is still on the table, which is why I wrote the 
patch the way I did.  If we have to preserve the existing API, I still 
think we should add new APIs and deprecate the old ones.


It's worth noting that there's been demand for this for a long time.  
Check out this comment from Include/datetime.h:


   #define PyDateTime_IMPORT \
   PyDateTimeAPI = (PyDateTime_CAPI*)
   PyCObject_Import("datetime", \
  
   "datetime_CAPI")


   /* This macro would be used if PyCObject_ImportEx() was created.
   #define PyDateTime_IMPORT \
   PyDateTimeAPI = (PyDateTime_CAPI*)
   PyCObject_ImportEx("datetime", \
  
   "datetime_CAPI", \
  
   DATETIME_API_MAGIC)

   */

That was checked in by Tim Peters on 2004-06-20, r36214.  (At least, in 
the py3k/trunk branch; I'd hope it would be the same revision number in 
other branches.)



/larry/
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PyDict_SetItem hook

2009-04-01 Thread John Ehresman

I've written a proof of concept patch to add a hook to PyDict_SetItem at 
 http://bugs.python.org/issue5654  My motivation is to enable 
watchpoints in a python debugger that are called when an attribute or 
global changes.  I know that this won't cover function locals and 
objects with slots (as Martin pointed out).


We talked about this at the sprints and a few issues came up:

* Is this worth it for debugger watchpoint support?  This is a feature 
that probably wouldn't be used regularly but is extremely useful in some 
situations.


* Would it be better to create a namespace dict subclass of dict, use it 
for modules, classes, & instances, and only allow watches of the 
subclass instances?


* To what extent should non-debugger code use the hook?  At one end of 
the spectrum, the hook could be made readily available for non-debug use 
and at the other end, it could be documented as being debug only, 
disabled in python -O, & not exposed in the stdlib to python code.


John
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Get the standard library to declare the versions it provides!

2009-04-01 Thread Chris Withers


Fred Drake wrote:
Even simple cases present issues with regard to this.  For example, I 
work on a project that relies on the uuid module, so we declare a 
dependency on Ka-Ping Ye's uuid module (since we're using Python 2.4).  
How should we write that in a version-agnostic way if we want to use the 
standard library version of that module with newer Pythons?


Well, that could be done be getting standard library modules to:

- declare what version they are
- be overridable why installed packages

That way, the fact that the standard library's development moves at the 
speed of frozen tar wouldn't stop packages in it being developed and 
released seperately for people who want to use newer versions of them 
and aren't in a situation where they need "batteries included"...


cheers,

Chris

--
Simplistix - Content Management, Zope & Python Consulting
   - http://www.simplistix.co.uk
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Let's update CObject API so it is safe and regular!

2009/4/1 Larry Hastings :
>
> Jim Fulton wrote:
>
> The only type-safety mechanism for a CObject is it's identity.  If you want
> to make sure you're using the foomodule api, make sure the address of the
> CObject is the same as the address of the api object exported by the module.
>
> That doesn't help.  Here's a program that crashes the interpreter, something
> I shouldn't be able to do from pure Python:
>
> import _socket
> import cStringIO
> cStringIO.cStringIO_CAPI = _socket.CAPI
>
> import cPickle
> s = cPickle.dumps([1, 2, 3])
>
> How can cPickle determine that cStringIO.cStringIO_CAPI is legitimate?

This is a bug in cPickle. It calls the PycString_IMPORT macro at the
very end of its init_stuff() function without checking for success.
This macro calls PyCObject_Import("cStringIO", "cStringIO_CAPI") which
in turn calls PyCObject_AsVoidPtr() on the object that it finds as
cStringIO.cStringIO_CAPI, and this function *does* do a type check and
sets an exception if the object isn't a PyCObject instance. However
cPickle's initialization doesn't check for errors immediately and
apparently some later code overrides the exception.

The fix should be simple: insert

  if (PyErr_Occurred()) return -1;

immediately after the line

  PycString_IMPORT;

in init_stuff() in cPickle.c. This will cause the import of cPickle to
fail with an exception and all should be well.

I have to say, I haven't understood this whole thread, but I'm
skeptical about a redesign. But perhaps you can come up with an
example that doesn't rely on this cPickle bug?

--Guido

> That would break backward compatibility. Are you proposing this for Python
> 3?
>
> I'm proposing this for Python 3.1.  My understanding is that breaking
> backwards compatibility is still on the table, which is why I wrote the
> patch the way I did.  If we have to preserve the existing API, I still think
> we should add new APIs and deprecate the old ones.
>
> It's worth noting that there's been demand for this for a long time.  Check
> out this comment from Include/datetime.h:
>
> #define PyDateTime_IMPORT \
>     PyDateTimeAPI = (PyDateTime_CAPI*) PyCObject_Import("datetime", \
>     "datetime_CAPI")
>
> /* This macro would be used if PyCObject_ImportEx() was created.
> #define PyDateTime_IMPORT \
>     PyDateTimeAPI = (PyDateTime_CAPI*) PyCObject_ImportEx("datetime", \
>     "datetime_CAPI",
> \
>
> DATETIME_API_MAGIC)
> */
>
> That was checked in by Tim Peters on 2004-06-20, r36214.  (At least, in the
> py3k/trunk branch; I'd hope it would be the same revision number in other
> branches.)
>
>
> /larry/
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>



-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PyDict_SetItem hook

2009-04-01 Thread Collin Winter

On Wed, Apr 1, 2009 at 4:29 PM, John Ehresman  wrote:
> I've written a proof of concept patch to add a hook to PyDict_SetItem at
>  http://bugs.python.org/issue5654  My motivation is to enable watchpoints in
> a python debugger that are called when an attribute or global changes.  I
> know that this won't cover function locals and objects with slots (as Martin
> pointed out).
>
> We talked about this at the sprints and a few issues came up:
>
> * Is this worth it for debugger watchpoint support?  This is a feature that
> probably wouldn't be used regularly but is extremely useful in some
> situations.
>
> * Would it be better to create a namespace dict subclass of dict, use it for
> modules, classes, & instances, and only allow watches of the subclass
> instances?
>
> * To what extent should non-debugger code use the hook?  At one end of the
> spectrum, the hook could be made readily available for non-debug use and at
> the other end, it could be documented as being debug only, disabled in
> python -O, & not exposed in the stdlib to python code.

Have you measured the impact on performance?

Collin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Let's update CObject API so it is safe and regular!

Guido van Rossum wrote:

This is a bug in cPickle. It calls the PycString_IMPORT macro at the
very end of its init_stuff() function without checking for success.

The bug you cite is a genuine bug, but that's not what I'm exploiting.

% python
>>> import _socket
>>> _socket.CAPI

The PyCObject_Import() call in PycString_IMPORT doesn't return 
failure--it returns a valid CObject.  I stuck the *wrong* CObject in 
cStringIO on purpose.  With the current API there's no way for cPickle 
to tell that it's using the wrong one.

For what it's worth, the previous example was for Python 2.x.  (Python 3 
doesn't have "cStringIO" or "cPickle".)  Here's an example that crashes 
python in my py3k/trunk (sync'd Monday morning).  And this one's only 
three lines:

   import unicodedata
   import _multibytecodec
   _multibytecodec.__create_codec(unicodedata.ucnhash_CAPI)

/larry/
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 3.1a2

2009-04-01 Thread Hirokazu Yamamoto



Hirokazu Yamamoto wrote:


I added #5499 to release blocker because it needs specification 
decision. (It's too strong?)


Thank you for fixing this. I also added

#5391: mmap: read_byte/write_byte and object type
#5410: msvcrt bytes cleanup

which depend on this issue. These are also API spec issue.
#5410 is easy, but #5391 still needs decision which of getarg("c") or 
getarg("b") read_byte/write_byte should use.

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 3.1a2

2009-04-01 Thread Benjamin Peterson

2009/4/1 Hirokazu Yamamoto :
>
> Hirokazu Yamamoto wrote:
>>
>> I added #5499 to release blocker because it needs specification decision.
>> (It's too strong?)
>
> Thank you for fixing this. I also added
>
> #5391: mmap: read_byte/write_byte and object type
> #5410: msvcrt bytes cleanup
>
> which depend on this issue. These are also API spec issue.
> #5410 is easy, but #5391 still needs decision which of getarg("c") or
> getarg("b") read_byte/write_byte should use.

I'm afraid neither of these bugs are anywhere near my areas of
expertise, so I'll leave resolution of them to the experts. :)



-- 
Regards,
Benjamin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PyDict_SetItem hook

2009-04-01 Thread Christian Heimes

John Ehresman wrote:
> * To what extent should non-debugger code use the hook?  At one end of
> the spectrum, the hook could be made readily available for non-debug use
> and at the other end, it could be documented as being debug only,
> disabled in python -O, & not exposed in the stdlib to python code.

To explain Collin's mail:
Python's dict implementation is crucial to the performance of any Python
program. Modules, types, instances all rely on the speed of Python's
dict type because most of them use a dict to store their name space.
Even the smallest change to the C code may lead to a severe performance
penalty. This is especially true for set and get operations.

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PyDict_SetItem hook

2009-04-01 Thread Raymond Hettinger




John Ehresman wrote:

* To what extent should non-debugger code use the hook?  At one end of
the spectrum, the hook could be made readily available for non-debug use
and at the other end, it could be documented as being debug only,
disabled in python -O, & not exposed in the stdlib to python code.


To explain Collin's mail:
Python's dict implementation is crucial to the performance of any Python
program. Modules, types, instances all rely on the speed of Python's
dict type because most of them use a dict to store their name space.
Even the smallest change to the C code may lead to a severe performance
penalty. This is especially true for set and get operations.



See my comments in http://bugs.python.org/issue5654


Raymond
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Let's update CObject API so it is safe and regular!

On Wed, Apr 1, 2009 at 5:39 PM, Larry Hastings  wrote:
>
> Guido van Rossum wrote:
>
> This is a bug in cPickle. It calls the PycString_IMPORT macro at the
> very end of its init_stuff() function without checking for success.
>
>
> The bug you cite is a genuine bug, but that's not what I'm exploiting.
>
> % python
 import _socket
 _socket.CAPI
> 
>
> The PyCObject_Import() call in PycString_IMPORT doesn't return failure--it
> returns a valid CObject.  I stuck the *wrong* CObject in cStringIO on
> purpose.  With the current API there's no way for cPickle to tell that it's
> using the wrong one.

Ouch. So true.

> For what it's worth, the previous example was for Python 2.x.  (Python 3
> doesn't have "cStringIO" or "cPickle".)  Here's an example that crashes
> python in my py3k/trunk (sync'd Monday morning).  And this one's only three
> lines:
>
> import unicodedata
> import _multibytecodec
> _multibytecodec.__create_codec(unicodedata.ucnhash_CAPI)

Yeah, any two CAPI objects can be used to play this trick, as long as
you have some place that calls them. :-(

So what's your solution? If it was me I'd change the API to put the
full module name and variable name of the object inside the object and
have the IMPORT call check that. Then you can only have crashes if
some extension module cheats, and surely there are many other ways
that C extensions can cheat, so that doesn't bother me. :)

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PyDict_SetItem hook

2009-04-01 Thread John Ehresman


Collin Winter wrote:

Have you measured the impact on performance?


I've tried to test using pystone, but am seeing more differences between 
runs than there is between python w/ the patch and w/o when there is no 
hook installed.  The highest pystone is actually from the binary w/ the 
patch, which I don't really believe unless it's some low level code 
generation affect.  The cost is one test of a global variable and then a 
switch to the branch that doesn't call the hooks.


I'd be happy to try to come up with better numbers next week after I get 
home from pycon.


John
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Let's update CObject API so it is safe and regular!



Guido van Rossum wrote:

Yeah, any two CAPI objects can be used to play this trick, as long as
you have some place that calls them. :-(


FWIW, I can't take credit for this observation.  Neal Norwitz threw me 
at this class of problem at the Py3k sprints in August 2007 at Google 
Mountain View, specifically with curses, though the approach he 
suggested then was removing the CObjects.  Then, Monday night MvL and I 
re-established the problem based on my dim memories.



So what's your solution? If it was me I'd change the API to put the
full module name and variable name of the object inside the object and
have the IMPORT call check that. Then you can only have crashes if
some extension module cheats, and surely there are many other ways
that C extensions can cheat, so that doesn't bother me. :)


My proposed API requires that the creator of the CObject pass in a 
"type" string, which must be of nonzero length, and the caller must pass 
in a matching string.  I figured that was easy to get right and 
sufficient for "consenting adults".  Note also this cheap 
exported-vtable hack isn't the only use of CObjects; for example _ctypes 
uses them to wrap plenty of one-off objects which are never set as 
attributes of the _ctypes module.  We'd like a solution that enforces 
some safety for those too, without creating spurious module attributes.



/larry//
/
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Let's update CObject API so it is safe and regular!

2009-04-01 Thread Lisandro Dalcin

On Wed, Apr 1, 2009 at 11:58 PM, Larry Hastings  wrote:
>
> Guido van Rossum wrote:
>>
>> Yeah, any two CAPI objects can be used to play this trick, as long as
>> you have some place that calls them. :-(
>
> FWIW, I can't take credit for this observation.  Neal Norwitz threw me at
> this class of problem at the Py3k sprints in August 2007 at Google Mountain
> View, specifically with curses, though the approach he suggested then was
> removing the CObjects.
>

IMHO, removing them would be a really bad idea... PyCObject's are the
documented recommended way to make ext modules export its API's, and
that works pretty well in practice, and more well now with your
approach.

>
>> So what's your solution? If it was me I'd change the API to put the
>> full module name and variable name of the object inside the object and
>> have the IMPORT call check that. Then you can only have crashes if
>> some extension module cheats, and surely there are many other ways
>> that C extensions can cheat, so that doesn't bother me. :)
>
> My proposed API requires that the creator of the CObject pass in a "type"
> string, which must be of nonzero length, and the caller must pass in a
> matching string.  I figured that was easy to get right and sufficient for
> "consenting adults".

Just for reference, I'll comment how Cython uses this. First, Cython
exports API in a function-by-function basis (instead of a single
pointer to a C struct with function pointers, as e.g. cStringIO, or an
array of func pointers, as e.g. NumPy). All these are cached in a
"private" module global (a dict) named "__pyx_api__". See the link
below, for example:

http://mpi4py.scipy.org/docs/api/mpi4py.MPI-module.html#__pyx_capi__

So the dict keys are the exported function names. Moreover, the
PyCObject's "desc" are a C string with the function signature. Cython
retrieves a function by name from the dict and checks that the
expected signature match. BTW, now I believe Cython should also use
the function name for the "descr" :-)

The only issue with this approach for Cython is that PyCObject
currently stores "void*" (i.e., pointers to data), but does not have
room for "void(*)(void)" (i.e. pointers to functions, aka code).
Recently I had to write some hackery using type-punning with unions to
avoid the illegal conversion problem between pointers to data and
functions.

Larry, I did not understand your comments in the tracker about this.
Why do you see the above approach a miss-use of the API? All this
works extremely well in practice... A Cython-implement extension
module can export its API, and next you can consume it from Cython,
and moreover from hand-written C extension (and then you can easily
write SWIG typemaps).  And as the function are exported one by one,
you can even add stuff to some module API, and the consumers will not
notice the thing (API tables implemented with pointer to C struct or
array of function pointers, you need to be more careful for API
exporting being backward)

-- 
Lisandro Dalcín
---
Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC)
Instituto de Desarrollo Tecnológico para la Industria Química (INTEC)
Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET)
PTLC - Güemes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Let's update CObject API so it is safe and regular!

On Wed, Apr 1, 2009 at 7:58 PM, Larry Hastings  wrote:
> Guido van Rossum wrote:
>> Yeah, any two CAPI objects can be used to play this trick, as long as
>> you have some place that calls them. :-(
>
> FWIW, I can't take credit for this observation.  Neal Norwitz threw me at
> this class of problem at the Py3k sprints in August 2007 at Google Mountain
> View, specifically with curses, though the approach he suggested then was
> removing the CObjects.  Then, Monday night MvL and I re-established the
> problem based on my dim memories.
>
>> So what's your solution? If it was me I'd change the API to put the
>> full module name and variable name of the object inside the object and
>> have the IMPORT call check that. Then you can only have crashes if
>> some extension module cheats, and surely there are many other ways
>> that C extensions can cheat, so that doesn't bother me. :)
>
> My proposed API requires that the creator of the CObject pass in a "type"
> string, which must be of nonzero length, and the caller must pass in a
> matching string.  I figured that was easy to get right and sufficient for
> "consenting adults".

OK, my proposal would be to agree on the value of this string too:
"module.variable".

> Note also this cheap exported-vtable hack isn't the
> only use of CObjects; for example _ctypes uses them to wrap plenty of
> one-off objects which are never set as attributes of the _ctypes module.
>  We'd like a solution that enforces some safety for those too, without
> creating spurious module attributes.

Why would you care about safety for ctypes? It's about as unsafe as it
gets anyway. Coredump emptor I say.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] CSV, bytes and encodings

2009-04-01 Thread Aahz

On Wed, Apr 01, 2009, [email protected] wrote:
> 
> Antoine> Perhaps. But without using 'rU' the file couldn't be read at
> Antoine> all.  (I'm not sure it was Windows line endings by the way;
> Antoine> perhaps Macintosh ones; anyway, it didn't work using 'rb')
> 
> Please file a bug report and assign to me.  Does it work in 2.x?  What was
> the source of the file?

Perhaps there have been changes, but in my last job, I was running into
this problem with Python 2.3, and I also needed to open with 'rU' under
Linux.
-- 
Aahz ([email protected])   <*> http://www.pythoncraft.com/

"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are, by
definition, not smart enough to debug it."  --Brian W. Kernighan
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] CSV, bytes and encodings

2009-04-01 Thread Antoine Pitrou

Le mercredi 01 avril 2009 à 18:22 -0400, R. David Murray a écrit :
> I just added some tests to trunk that seem to indicate this case is
> handled correctly in terms of preserving the data.  Maybe you didn't
> write the file such that the fields with the newlines were quoted?

I used the default csv.writer into a StringIO, and the whole was then
returned as the response of an HTTP request (with the proper
Content-Type and Content-Disposition headers). I assume quoting is
enabled by default?

> And of course how non-Excel applications handle that data on import
> can be different from how Excel handles it.

Of course, but when three major spreadsheet software (including Excel
itself) choke on the embedded newline, there might be a problem (or
not :)).
(please note that as for Excel I couldn't test myself, a client of mine
did)

Regards

Antoine.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] CSV, bytes and encodings