subject:"Re\: \[Python\-Dev\] cffi in stdlib"

On Wed, Feb 27, 2013 at 3:39 AM, Terry Reedy tjre...@udel.edu wrote:
 On 2/26/2013 10:13 AM, Maciej Fijalkowski wrote:

 I would like to discuss on the language summit a potential inclusion
 of cffi[1] into stdlib.


 How does it compare in terms of speed. One reason ctypes has not replaces
 hand-tuned swig is that it apparently is much slower. I know that someone,
 for instance, once did a pygame fork using ctypes to wrap SDL, and got it to
 run, but gave it up for comparative slowness reasons.

I never did the comparison with hand-tuned swig. It's faster than
ctypes for calls and manipulation, C extension beat it to some degree.
It's uber fast on PyPy (a bit over the native call, but we'll get it
to the native call level).
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

On Wed, Feb 27, 2013 at 6:53 AM, Eli Bendersky eli...@gmail.com wrote:

 * Work either at the level of the ABI (Application Binary Interface)

 or the API (Application Programming Interface). Usually, C libraries
 have a specified C API but often not an ABI (e.g. they may document a
 “struct” as having at least these fields, but maybe more). (ctypes
 works at the ABI level, whereas Cython and native C extensions work at
 the API level.)



 I read the cffi docs once again and went through some of the examples. I
 want to divide this to two topics.

 One is what you call the ABI level. IMHO, it's hands down superior to
 ctypes. Your readdir demo demonstrates this very nicely. I would definitely
 want to see this in the stdlib as an alternative way to interface to C
 shared objects  DLLs.

 Two is what you call the API level, which is where my opinion becomes
 mixed. Some things just don't feel right to me:

 1. Tying in a C compiler into the flow of a program. I'm not sure whether we
 have precedents for it in the stdlib. Does this work on Windows where
 libraries and DLLs are usually built with MSVC?


Yes. Precedent in the stdlib is really the C API. All the same rules
apply (including build and ship a dll).

 2. Using a function called verify to create stuff. This may sound like a
 naming bikeshed, but it's not. It ties in to the question - why is this
 needed?

We welcome a better opinion of name (indeed verify is not that great).
This elevates ABI to API so either invokes the C compiler or reads
stuff from the cache.


 3. The partial type specifications in C with ellipsis. What is the point?
 You have the C declarations somewhere anyhow, so why introduce this? The
 ABI level boasts having just C and Python to write, but those partial
 ellipsis-ridden declarations are hardly C.

No, you don't. Some libraries contain macros for example (like
OpenSSL) where you just can't use ABI because it makes no sense. It's
less common on windows where binary compatibility is important,
however looking on linux, multiple stdlib declaration would use
ellipsis in the man page. I can't seem to find one right now, but it's
something like:

struct X {
   int public_field;
   ...
}

which is impossible to do correctly with ctypes without exposing some
sort of platform dependency that might change without warning.

Another usages are #define SQLITE_OK ... which you don't know at the
time of writing (people assume those won't change and the do change).


 Please see this as constructive criticism, and I fully admit that I may not
 have went deep enough to see the big benefits of the API level that would
 justify all that complexity.

Sure. It might sound strange, but on linux (or posix in general)
binary compatibility is not a thing, so you need to do stuff to
account for that.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

On Wed, Feb 27, 2013 at 9:29 AM, Ronald Oussoren ronaldousso...@mac.com wrote:

 On 26 Feb, 2013, at 16:13, Maciej Fijalkowski fij...@gmail.com wrote:

 Hello.

 I would like to discuss on the language summit a potential inclusion
 of cffi[1] into stdlib.

 The API in general looks nice, but I do have some concens w.r.t. including 
 cffi in the stdlib.

 1. Why is cffi completely separate from ctypes, instead of layered on top of 
 it? That is, add a utility module to ctypes that can parse C declarations and 
 generate the right ctypes definitions.

Because ctypes API is a mess and magic. We needed a cleaner (and much
smaller) model.


 2. Cffi has a dependencies on pycparser and that module and its dependencies 
 would therefore also be added to the stdlib (even if they'd be hidden in the 
 cffi package)

Yes. pycparser and ply.


 3. Cffi basicly contains a (limited) C parser, and those are notoriously hard 
 to get exactly right. Luckily cffi only needs to interpret declarations and 
 not the full language, but even so this can be a risk of subtle bugs.

It seems to work.


 4. And finally a technical concern: how well does cffi work with fat binaries 
 on OSX? In particular, will the distutils support generate cached data for 
 all architectures supported by a fat binary?

no idea.


 Also, after playing around with it for 5 minutes I don't quite understand how 
 to use it. Let's say I want to wrap a function CGPoint CGPointMake(CGFloat 
 x, CGFloat y). Is is possible to avoid mentioning the exact typedef for 
 CGFloat somewhere? I tried using:

ffi.cdef(typedef ... CGFloat; typedef struct { CGFloat x; CGFloat y; } 
 CGPoint; CGPoint CGPointMake(CGFloat x, CGFloat y);)

 But that results in an error when calling verify:

TypeError: field 'struct $CGPoint.x' has ctype 'struct $CGFloat' of 
 unknown size

 From a first glance this doesn't seem to buy me that much w.r.t. ctypes, I 
 still have to declare the actual type of CGFloat which is documented as some 
 floating point type.

 Ronald

typedef ... is assumed to be a struct. Copy pasting from
documentation, we might want to have a better support for that, but
what you do right now is to do:

ffi.cdef(const int mysize;)
lib = ffi.verify(const int mysize = sizeof(THE_TYPE);)
print lib.mysize

which will tell you the size (and then you can pick up which float you
want). I agree that some generic support for that would be cool.

Cheers,
fijal
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

2013-02-27 Thread Ronald Oussoren


On 27 Feb, 2013, at 10:06, Maciej Fijalkowski fij...@gmail.com wrote:

 On Wed, Feb 27, 2013 at 9:29 AM, Ronald Oussoren ronaldousso...@mac.com 
 wrote:
 
 On 26 Feb, 2013, at 16:13, Maciej Fijalkowski fij...@gmail.com wrote:
 
 Hello.
 
 I would like to discuss on the language summit a potential inclusion
 of cffi[1] into stdlib.
 
 The API in general looks nice, but I do have some concens w.r.t. including 
 cffi in the stdlib.
 
 1. Why is cffi completely separate from ctypes, instead of layered on top of 
 it? That is, add a utility module to ctypes that can parse C declarations 
 and generate the right ctypes definitions.
 
 Because ctypes API is a mess and magic. We needed a cleaner (and much
 smaller) model.

The major advantages of starting over is probably that you can hide the 
complexity and that opens opportunities for optimizations. That said, I'm not 
convinced that ctypes is unnecessarily complex.
 
 
 2. Cffi has a dependencies on pycparser and that module and its dependencies 
 would therefore also be added to the stdlib (even if they'd be hidden in the 
 cffi package)
 
 Yes. pycparser and ply.

Which aren't part of the stdlib right now.

 
 
 3. Cffi basicly contains a (limited) C parser, and those are notoriously 
 hard to get exactly right. Luckily cffi only needs to interpret declarations 
 and not the full language, but even so this can be a risk of subtle bugs.
 
 It seems to work.

That's not a confidency inspiring comment :-).  That said,  I use a hacked up 
fork of pycparser to parse Apple's Cocoa headers for PyObjC and it appears to 
work fine for that.

 
 
 4. And finally a technical concern: how well does cffi work with fat 
 binaries on OSX? In particular, will the distutils support generate cached 
 data for all architectures supported by a fat binary?
 
 no idea.

That's somehting that will have to be resolved before cffi can be included in 
the stdlib, fat binaries are supported by CPython and are used the binary 
installers.

Ronald
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

On Wed, Feb 27, 2013 at 11:18 AM, Ronald Oussoren
ronaldousso...@mac.com wrote:

 On 27 Feb, 2013, at 10:06, Maciej Fijalkowski fij...@gmail.com wrote:

 On Wed, Feb 27, 2013 at 9:29 AM, Ronald Oussoren ronaldousso...@mac.com 
 wrote:

 On 26 Feb, 2013, at 16:13, Maciej Fijalkowski fij...@gmail.com wrote:

 Hello.

 I would like to discuss on the language summit a potential inclusion
 of cffi[1] into stdlib.

 The API in general looks nice, but I do have some concens w.r.t. including 
 cffi in the stdlib.

 1. Why is cffi completely separate from ctypes, instead of layered on top 
 of it? That is, add a utility module to ctypes that can parse C 
 declarations and generate the right ctypes definitions.

 Because ctypes API is a mess and magic. We needed a cleaner (and much
 smaller) model.

 The major advantages of starting over is probably that you can hide the 
 complexity and that opens opportunities for optimizations. That said, I'm not 
 convinced that ctypes is unnecessarily complex.

I implemented ctypes. It is.

 4. And finally a technical concern: how well does cffi work with fat 
 binaries on OSX? In particular, will the distutils support generate cached 
 data for all architectures supported by a fat binary?

 no idea.

 That's somehting that will have to be resolved before cffi can be included in 
 the stdlib, fat binaries are supported by CPython and are used the binary 
 installers.

 Ronald

if cpython supports it and you can load it using dlopen, it does work
then (it really is just building a C extension on the API level).
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

2013-02-27 Thread Richard Oudkerk


On 27/02/2013 9:21am, Maciej Fijalkowski wrote:

That's somehting that will have to be resolved before cffi can be included in 
the stdlib, fat binaries are supported by CPython and are used the binary 
installers.

Ronald

if cpython supports it and you can load it using dlopen, it does work
then (it really is just building a C extension on the API level).


On Windows ctypes has CDLL for libraries using the cdecl calling 
convention and WINDLL for libraries with the stdcall calling convention.


I can't see any similar distinction in cffi's documentation.  Can cffi 
magically work out which calling convention to use, or are you 
restricted to the cdecl calling convention?


--
Richard

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

On Wed, Feb 27, 2013 at 1:45 PM, Richard Oudkerk shibt...@gmail.com wrote:
 On 27/02/2013 9:21am, Maciej Fijalkowski wrote:

 That's somehting that will have to be resolved before cffi can be
  included in the stdlib, fat binaries are supported by CPython and are used
  the binary installers.
 
 Ronald

 if cpython supports it and you can load it using dlopen, it does work
 then (it really is just building a C extension on the API level).


 On Windows ctypes has CDLL for libraries using the cdecl calling convention
 and WINDLL for libraries with the stdcall calling convention.

 I can't see any similar distinction in cffi's documentation.  Can cffi
 magically work out which calling convention to use, or are you restricted to
 the cdecl calling convention?

copy pasting from docs:

Windows: you can’t yet specify the calling convention of callbacks.
(For regular calls, the correct calling convention should be
automatically inferred by the C backend.) Use an indirection, like in
the example just above.

I think it means you can't use the ABI version and specify the calling
convention. It's a reasonable bug report (the calling convention on
API version works though)


 --
 Richard


 ___
 Python-Dev mailing list
 Python-Dev@python.org
 http://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe:
 http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

2013-02-27 Thread Nick Coghlan

On Wed, Feb 27, 2013 at 7:06 PM, Maciej Fijalkowski fij...@gmail.com wrote:
 3. Cffi basicly contains a (limited) C parser, and those are notoriously 
 hard to get exactly right. Luckily cffi only needs to interpret declarations 
 and not the full language, but even so this can be a risk of subtle bugs.

 It seems to work.

C itself shouldn't be *too* bad - it's trying to parse C++ that is
truly insane (which is why Dave Beazley explicitly warned Maciej and
Armin away from that based on his experience with SWIG).

Looking at pycparsing more closely though, there may be a problem with
the C preprocessing step on Windows. While on Linux it uses the system
cpp, for Windows it currently bundles a copy of the LCC
preprocessor, which may pose licensing challenges
(https://github.com/drh/lcc/blob/master/CPYRIGHT). However, this
concern may not be applicable to cffi: given cffi's requirement for a
full external C compiler to actually *create* the C extension modules
during development, and the ability to retrieve everything necessary
directly from the created C extension when no C compiler is available,
it seems there would be no reason to ever need the bundled C
preprocessor. So we simply don't ship it, and the LCC license becomes
irrelevant.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

On Wed, Feb 27, 2013 at 2:39 PM, Nick Coghlan ncogh...@gmail.com wrote:
 On Wed, Feb 27, 2013 at 7:06 PM, Maciej Fijalkowski fij...@gmail.com wrote:
 3. Cffi basicly contains a (limited) C parser, and those are notoriously 
 hard to get exactly right. Luckily cffi only needs to interpret 
 declarations and not the full language, but even so this can be a risk of 
 subtle bugs.

 It seems to work.

 C itself shouldn't be *too* bad - it's trying to parse C++ that is
 truly insane (which is why Dave Beazley explicitly warned Maciej and
 Armin away from that based on his experience with SWIG).

 Looking at pycparsing more closely though, there may be a problem with
 the C preprocessing step on Windows. While on Linux it uses the system
 cpp, for Windows it currently bundles a copy of the LCC
 preprocessor, which may pose licensing challenges
 (https://github.com/drh/lcc/blob/master/CPYRIGHT). However, this
 concern may not be applicable to cffi: given cffi's requirement for a
 full external C compiler to actually *create* the C extension modules
 during development, and the ability to retrieve everything necessary
 directly from the created C extension when no C compiler is available,
 it seems there would be no reason to ever need the bundled C
 preprocessor. So we simply don't ship it, and the LCC license becomes
 irrelevant.

 Cheers,
 Nick.

 --
 Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia

cffi does not use preprocessor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

2013-02-27 Thread Nick Coghlan

On Wed, Feb 27, 2013 at 7:00 PM, Maciej Fijalkowski fij...@gmail.com wrote:
 On Wed, Feb 27, 2013 at 6:53 AM, Eli Bendersky eli...@gmail.com wrote:
 2. Using a function called verify to create stuff. This may sound like a
 naming bikeshed, but it's not. It ties in to the question - why is this
 needed?

 We welcome a better opinion of name (indeed verify is not that great).
 This elevates ABI to API so either invokes the C compiler or reads
 stuff from the cache.

Have you considered the name create_api? After all, you're
essentially creating a Python API given a C header declaration and a
specified extension module to cache the result.

The details on extension module caching, and the fact that it won't
even look for a C compiler if the extension module is already
available should also be in the main section of the docs on the
verification step rather than only being down in the internal
documentation for cffi.verifier.Verifier

Cheers,
Nick.

P.S. I created both of those as tracker issues:
- https://bitbucket.org/cffi/cffi/issue/59/missing-content-in-cffiverify
- 
https://bitbucket.org/cffi/cffi/issue/60/proposed-rename-for-verify-and-verifier



-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

2013-02-27 Thread Antoine Pitrou

Le Wed, 27 Feb 2013 12:15:05 +1300,
Greg Ewing greg.ew...@canterbury.ac.nz a écrit :
 Antoine Pitrou wrote:
  Or we'll go straight to 5.
  (or switch to date-based numbering :-))
 
 We could go the Apple route and start naming them after
 species of snake.

We have to find sufficiently silly species of snakes, though.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

2013-02-27 Thread Xavier Morel

On 2013-02-27, at 14:31 , Antoine Pitrou wrote:

 Le Wed, 27 Feb 2013 12:15:05 +1300,
 Greg Ewing greg.ew...@canterbury.ac.nz a écrit :
 Antoine Pitrou wrote:
 Or we'll go straight to 5.
 (or switch to date-based numbering :-))
 
 We could go the Apple route and start naming them after
 species of snake.
 
 We have to find sufficiently silly species of snakes, though.

With about 3000 extant snake species, that should be manageable.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

2013-02-27 Thread Joao S. O. Bueno

On 27 February 2013 10:31, Antoine Pitrou solip...@pitrou.net wrote:

 Le Wed, 27 Feb 2013 12:15:05 +1300,
 Greg Ewing greg.ew...@canterbury.ac.nz a écrit :
  Antoine Pitrou wrote:
   Or we'll go straight to 5.
   (or switch to date-based numbering :-))
 
  We could go the Apple route and start naming them after
  species of snake.

 We have to find sufficiently silly species of snakes, though.


Monty Python feature movies.
There are less than snake species, but imagine porting
Django to be able to run on Life of Brian!


 Regards

 Antoine.


 ___
 Python-Dev mailing list
 Python-Dev@python.org
 http://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe:
 http://mail.python.org/mailman/options/python-dev/jsbueno%40python.org.br

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

2013-02-27 Thread Armin Rigo

Hi Guido,

On Tue, Feb 26, 2013 at 8:24 PM, Guido van Rossum gu...@python.org wrote:
 From a software engineering perspective, 10 years is indistinguishable
 from infinity, so I don't care what happens 10 years from now -- as
 long as you don't blame me. :-)

I can't resist: around today it is the 10th anniversary of PyPy.  :-)


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

2013-02-27 Thread Eli Bendersky

On Tue, Feb 26, 2013 at 11:29 PM, Ronald Oussoren ronaldousso...@mac.comwrote:

On 26 Feb, 2013, at 16:13, Maciej Fijalkowski fij...@gmail.com wrote:

Hello.

I would like to discuss on the language summit a potential inclusion
of cffi[1] into stdlib.

The API in general looks nice, but I do have some concens w.r.t. including
cffi in the stdlib.

1. Why is cffi completely separate from ctypes, instead of layered on top
of it? That is, add a utility module to ctypes that can parse C
declarations and generate the right ctypes definitions.

2. Cffi has a dependencies on pycparser and that module and its
dependencies would therefore also be added to the stdlib (even if they'd be
hidden in the cffi package)

3. Cffi basicly contains a (limited) C parser, and those are notoriously
hard to get exactly right. Luckily cffi only needs to interpret
declarations and not the full language, but even so this can be a risk of
subtle bugs.

pycparser has been around for more than 4 years and is pretty stable. I
know that it's been used to parse Windows headers, Linux system headers and
other scary things for a C parser to handle. It's only known problem has
to do with over-creative abuses of C's context sensitivity:
http://eli.thegreenplace.net/2011/05/02/the-context-sensitivity-of-c%E2%80%99s-grammar-revisited/

In practice it doesn't come up often, and almost never in declarations,
which is the subset cffi uses pycparser for.

Eli
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

2013-02-27 Thread Eli Bendersky

  I read the cffi docs once again and went through some of the examples. I
  want to divide this to two topics.
 
  One is what you call the ABI level. IMHO, it's hands down superior to
  ctypes. Your readdir demo demonstrates this very nicely. I would
 definitely
  want to see this in the stdlib as an alternative way to interface to C
  shared objects  DLLs.
 
  Two is what you call the API level, which is where my opinion becomes
  mixed. Some things just don't feel right to me:
 
  1. Tying in a C compiler into the flow of a program. I'm not sure
 whether we
  have precedents for it in the stdlib. Does this work on Windows where
  libraries and DLLs are usually built with MSVC?
 

 Yes. Precedent in the stdlib is really the C API. All the same rules
 apply (including build and ship a dll).


So would you say that the main use of the API level is provide an
alternative for writing C API code to interface to C libraries. IOW, it's
in competition with Swig?



  2. Using a function called verify to create stuff. This may sound like
 a
  naming bikeshed, but it's not. It ties in to the question - why is this
  needed?

 We welcome a better opinion of name (indeed verify is not that great).
 This elevates ABI to API so either invokes the C compiler or reads
 stuff from the cache.


Can you elaborate on what elevates ABI to API means here?


  3. The partial type specifications in C with ellipsis. What is the point?
  You have the C declarations somewhere anyhow, so why introduce this? The
  ABI level boasts having just C and Python to write, but those partial
  ellipsis-ridden declarations are hardly C.

 No, you don't. Some libraries contain macros for example (like
 OpenSSL) where you just can't use ABI because it makes no sense. It's
 less common on windows where binary compatibility is important,
 however looking on linux, multiple stdlib declaration would use
 ellipsis in the man page.


It would be useful to find an actual example and discuss it concretely.


 I can't seem to find one right now, but it's
 something like:

 struct X {
int public_field;
...
 }

 which is impossible to do correctly with ctypes without exposing some
 sort of platform dependency that might change without warning.

 Another usages are #define SQLITE_OK ... which you don't know at the
 time of writing (people assume those won't change and the do change).


Do you mean that the value of SQLITE_OK changed over time (now it's 0, but
used to be different?)

If so, then in a realistic use case, how would the API level help solve
this?

Eli
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

On Wed, Feb 27, 2013 at 5:57 PM, Eli Bendersky eli...@gmail.com wrote:

  I read the cffi docs once again and went through some of the examples. I
  want to divide this to two topics.
 
  One is what you call the ABI level. IMHO, it's hands down superior to
  ctypes. Your readdir demo demonstrates this very nicely. I would
  definitely
  want to see this in the stdlib as an alternative way to interface to C
  shared objects  DLLs.
 
  Two is what you call the API level, which is where my opinion becomes
  mixed. Some things just don't feel right to me:
 
  1. Tying in a C compiler into the flow of a program. I'm not sure
  whether we
  have precedents for it in the stdlib. Does this work on Windows where
  libraries and DLLs are usually built with MSVC?
 

 Yes. Precedent in the stdlib is really the C API. All the same rules
 apply (including build and ship a dll).


 So would you say that the main use of the API level is provide an
 alternative for writing C API code to interface to C libraries. IOW, it's in
 competition with Swig?

the general goal is to provide alternative for writing C API by hand
(so SWIG, ctypes, cython, whatever).




  2. Using a function called verify to create stuff. This may sound like
  a
  naming bikeshed, but it's not. It ties in to the question - why is this
  needed?

 We welcome a better opinion of name (indeed verify is not that great).
 This elevates ABI to API so either invokes the C compiler or reads
 stuff from the cache.


 Can you elaborate on what elevates ABI to API means here?


  3. The partial type specifications in C with ellipsis. What is the
  point?
  You have the C declarations somewhere anyhow, so why introduce this? The
  ABI level boasts having just C and Python to write, but those partial
  ellipsis-ridden declarations are hardly C.

 No, you don't. Some libraries contain macros for example (like
 OpenSSL) where you just can't use ABI because it makes no sense. It's
 less common on windows where binary compatibility is important,
 however looking on linux, multiple stdlib declaration would use
 ellipsis in the man page.


 It would be useful to find an actual example and discuss it concretely.

#define Py_INCREF(x) (x - refcnt++)

the API here is Py_INCREF, which is a macro and does not exist in the
DLL. copy-pasting the implementation is not using the API, it's hoping
that stuff won't change in the future (and say makes it not working
when PyPy implements Py_INCREF as a function). Some POSIX things are
macros in one version of glibc and functions in other and there is no
change in API so your code will silently stop working.



 I can't seem to find one right now, but it's
 something like:

 struct X {
int public_field;
...
 }

 which is impossible to do correctly with ctypes without exposing some
 sort of platform dependency that might change without warning.

 Another usages are #define SQLITE_OK ... which you don't know at the
 time of writing (people assume those won't change and the do change).


 Do you mean that the value of SQLITE_OK changed over time (now it's 0, but
 used to be different?)

 If so, then in a realistic use case, how would the API level help solve
 this?

 Eli

no the API is SQLITE_OK. The actual numeric value of 0 is *not* part
of the API. it might be 0 it might be not, you never know. In cffi you
write:

#define SQLITE_OK ...

and let the compiler figure out. in ctypes you just hardcode 0. I
ended up once with a scary mess of ifs because BSD Linux and OS X were
declaring the same numeric constants with different values.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

On 27 February 2013 11:53, Maciej Fijalkowski fij...@gmail.com wrote:
 I think it means you can't use the ABI version and specify the calling
 convention. It's a reasonable bug report (the calling convention on
 API version works though)

That would be a deal-breaker for my use case of quick scripts working
with the Windows API on a machine with no C compiler. I'd have to use
ctypes in that case until cffi had this feature.
Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

 27 February 2013 18:24, Paul Moore p.f.mo...@gmail.com wrote:
 On 27 February 2013 11:53, Maciej Fijalkowski fij...@gmail.com wrote:
 I think it means you can't use the ABI version and specify the calling
 convention. It's a reasonable bug report (the calling convention on
 API version works though)

 That would be a deal-breaker for my use case of quick scripts working
 with the Windows API on a machine with no C compiler. I'd have to use
 ctypes in that case until cffi had this feature.

One other use case for the ABI level over the API level - the ABI
level (no C extension) can be used across Python versions, where the
API level needs a separate compiled extension per Python version. This
can be a big deal on Windows at least, where users (not developers)
with no compilers on their systems are common.

I'm not trying to argue against cffi, just pointing out some cases
that should be considered.
Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

2013-02-27 Thread Brett Cannon

On Wed, Feb 27, 2013 at 1:28 PM, Paul Moore p.f.mo...@gmail.com wrote:

  27 February 2013 18:24, Paul Moore p.f.mo...@gmail.com wrote:
  On 27 February 2013 11:53, Maciej Fijalkowski fij...@gmail.com wrote:
  I think it means you can't use the ABI version and specify the calling
  convention. It's a reasonable bug report (the calling convention on
  API version works though)
 
  That would be a deal-breaker for my use case of quick scripts working
  with the Windows API on a machine with no C compiler. I'd have to use
  ctypes in that case until cffi had this feature.

 One other use case for the ABI level over the API level - the ABI
 level (no C extension) can be used across Python versions, where the
 API level needs a separate compiled extension per Python version. This
 can be a big deal on Windows at least, where users (not developers)
 with no compilers on their systems are common.


Is that still true for Windows even with the stable ABI and the dropping of
narrow/wide Unicode builds?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

2013-02-27 Thread Armin Rigo

Hi Paul,

On Wed, Feb 27, 2013 at 7:24 PM, Paul Moore p.f.mo...@gmail.com wrote:
 On 27 February 2013 11:53, Maciej Fijalkowski fij...@gmail.com wrote:
 I think it means you can't use the ABI version and specify the calling
 convention. It's a reasonable bug report (the calling convention on
 API version works though)

 That would be a deal-breaker for my use case of quick scripts working
 with the Windows API on a machine with no C compiler. I'd have to use
 ctypes in that case until cffi had this feature.

That's not correct: you can't indeed give the calling convention, but
it is not needed for the common case.  What is not supported is only
Python-defined callbacks using the Windows-specific convention --- as
documented, there is a workaround for that case.

And, in case you wonder, this automatic detection comes from ctypes.
I copied the hacked-up version of libffi for Windows from ctypes to
cffi, and the logic to detect the actual calling convention and
support both is there.  The difference is only that in ctypes, after
it did the call (always successfully), it checks the convention that
was found to be used by the C function, and if it differs from the one
specified by the user, then it complains.  I basically just removed
the complaining part.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

On 27 February 2013 19:08, Armin Rigo ar...@tunes.org wrote:
 That's not correct: you can't indeed give the calling convention, but
 it is not needed for the common case.  What is not supported is only
 Python-defined callbacks using the Windows-specific convention --- as
 documented, there is a workaround for that case.

OK, that's cool. As I said, I really need to actually experiment with
cffi - this thread has certainly made me want to do so.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

2013-02-27 Thread Daniel Holth

On Wed, Feb 27, 2013 at 4:18 AM, Ronald Oussoren ronaldousso...@mac.com wrote:

 On 27 Feb, 2013, at 10:06, Maciej Fijalkowski fij...@gmail.com wrote:

 On Wed, Feb 27, 2013 at 9:29 AM, Ronald Oussoren ronaldousso...@mac.com 
 wrote:

 On 26 Feb, 2013, at 16:13, Maciej Fijalkowski fij...@gmail.com wrote:

 Hello.

 I would like to discuss on the language summit a potential inclusion
 of cffi[1] into stdlib.

 The API in general looks nice, but I do have some concens w.r.t. including 
 cffi in the stdlib.

 1. Why is cffi completely separate from ctypes, instead of layered on top 
 of it? That is, add a utility module to ctypes that can parse C 
 declarations and generate the right ctypes definitions.

 Because ctypes API is a mess and magic. We needed a cleaner (and much
 smaller) model.

 The major advantages of starting over is probably that you can hide the 
 complexity and that opens opportunities for optimizations. That said, I'm not 
 convinced that ctypes is unnecessarily complex.

cffi actually does have a ctypes backend in addition to the ffi and
compile a CPython extension backends. But the ctypes backend is
guaranteed to be slow and messy because it is ctypes.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

On 27 February 2013 19:26, Paul Moore p.f.mo...@gmail.com wrote:
 On 27 February 2013 19:08, Armin Rigo ar...@tunes.org wrote:
 That's not correct: you can't indeed give the calling convention, but
 it is not needed for the common case.  What is not supported is only
 Python-defined callbacks using the Windows-specific convention --- as
 documented, there is a workaround for that case.

 OK, that's cool. As I said, I really need to actually experiment with
 cffi - this thread has certainly made me want to do so.

Actually, looking at the cffi documentation, I can't see how I'd
translate the following very simple example of something I do quite a
lot in ctypes:

from ctypes import windll
MessageBox = windll.User32.MessageBoxW
MessageBox(0, Hello, world!, Title, 0)

Note - I wrote this from memory, I honestly don't know without looking
it up the precise argument types for MessageBoxW - and even if I did,
I suspect they would all be macros from windows.h. I don't want to
invoke the C compiler by using verify, and I don't see in the docs how
I'd get the macro definitions any other way.

If anyone could show me a cffi equivalent, I'd be very grateful.
Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

2013-02-27 Thread Greg Ewing


Antoine Pitrou wrote:

We have to find sufficiently silly species of snakes, though.


Glancing through http://en.wikipedia.org/wiki/List_of_snakes,
we have:

Wart snakes
Java wart snakes
Pipe snakes
Stiletto snakes
Rubber boas
Dog-faced water snakes
Shovel-nosed snakes
Hook-nosed snakes
Leaf-nosed snakes
Lyre snakes
Cat-eyed snakes
Worm snakes
North American hog-nosed snakes
Moccasins
Humpnose vipers
Puff adders
Carpet vipers
False horned vipers
Sunbeam snakes

And then there's this:

http://www.nairaland.com/1009227/new-species-snake-discovered-brazil

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

2013-02-27 Thread Terry Reedy


On 2/27/2013 3:31 PM, Paul Moore wrote:

from ctypes import windll
MessageBox = windll.User32.MessageBoxW
MessageBox(0, Hello, world!, Title, 0)

 Note - I wrote this from memory,

Gee, that is pretty easy. Worked perfectly

--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

2013-02-27 Thread Armin Rigo

Hi Paul,

On Wed, Feb 27, 2013 at 9:31 PM, Paul Moore p.f.mo...@gmail.com wrote:
 from ctypes import windll
 MessageBox = windll.User32.MessageBoxW
 MessageBox(0, Hello, world!, Title, 0)

You are right that it's a bit cumbersome in cffi up to and including
0.5, but in the cffi trunk all standard Windows types are included.
So the general answer to your question is: we google MessageBox and
copy that line from the microsoft site, and manually remove the
unnecessary WINAPI and _In_opt declarations:

from cffi import FFI
ffi = FFI()
ffi.cdef(
int MessageBox(HWND hWnd, LPCTSTR lpText, LPCTSTR lpCaption, UINT uType);
)
lib = ffi.dlopen(USER32.DLL)
lib.MessageBox(ffi.NULL, Hello, world!, Title, 0)

That's a slightly unfair example, because in this case it happens to
work with ctypes without specifying the argtypes and the restype.  I
would argue that this feature of ctypes is not a good thing: it's
mostly the same as saying you only need to declare argtypes and
restype if you get nonsense results or segfaults.

Note for completeness that the version with verify() simply replaces
lib = ffi.dlopen(USER32.DLL) with lib = ffi.verify() (no
#include necessary here).  Then you cannot misdeclare anything without
getting clear compile-time errors at the verify().  The compilation
occurs only once (it's cached) and by writing two lines in your
setup.py you can distribute binary installations, just like you do
with hand-written C extension modules; so the extra burden of
accessing the API level is in my opinion very small compared to its
segfault-avoiding gain.  But I know that either level of access can
make sense in different situations.  Typically, on Windows, the
ABI-level works fine; as argued elsewhere, on other platforms and/or
with some libraries, the API-level is definitely more suited.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

2013-02-27 Thread Neil Hodgson

Armin Rigo:

 So the general answer to your question is: we google MessageBox and
 copy that line from the microsoft site, and manually remove the
 unnecessary WINAPI and _In_opt declarations:

   Wouldn't it be better to understand the SAL annotations like _In_opt so that 
spurious NULLs (for example) produce a good exception from cffi instead of 
failing inside the system call?

   Neil

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

On Tue, Feb 26, 2013 at 5:42 PM, Nick Coghlan ncogh...@gmail.com wrote:
 On Wed, Feb 27, 2013 at 1:13 AM, Maciej Fijalkowski fij...@gmail.com wrote:
 Hello.

 I would like to discuss on the language summit a potential inclusion
 of cffi[1] into stdlib.

 I think cffi is well worth considering as a possible inclusion for
 Python 3.4. (In particular, I'm a fan of the fact it just uses C
 syntax to declare what you're trying to talk to)

 If I'm reading the dependencies correctly, we would also need to bring
 Eli Bendersky's pycparser into the stdlib, correct? (not an objection,
 just an observation, although we'd obviously need Eli's explicit
 endorsement for that).

Yes, although, depending on the circumstances it's possible to hide it
somewhere in cffi and not make it an official stdlib API (then several
rules does not apply). It also pulls in some version of ply (or a
generated parser).


 There would also be the matter of converting the cffi test suite from
 py.test to stdlib unittest.

I volunteer to do that, if necessary.


 Cheers,
 Nick.

 --
 Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

2013-02-26 Thread Dirkjan Ochtman

On Tue, Feb 26, 2013 at 4:13 PM, Maciej Fijalkowski fij...@gmail.com wrote:
 I would like to discuss on the language summit a potential inclusion
 of cffi[1] into stdlib. This is a project Armin Rigo has been working
 for a while, with some input from other developers. It seems that the
 main reason why people would prefer ctypes over cffi these days is
 because it's included in stdlib, which is not generally the reason I
 would like to hear. Our calls to not use C extensions and to use an
 FFI instead has seen very limited success with ctypes and quite a lot
 more since cffi got released. The API is fairly stable right now with
 minor changes going in and it'll definitely stablize until Python 3.4
 release.

I think this would be great to have in the stdlib.

I think it's useful to note that PyPy is planning to include this in
their stdlib for the next release, too, right?

Cheers,

Dirkjan
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

2013-02-26 Thread Antoine Pitrou

Le Tue, 26 Feb 2013 17:13:44 +0200,
Maciej Fijalkowski fij...@gmail.com a écrit :
 Hello.
 
 I would like to discuss on the language summit a potential inclusion
 of cffi[1] into stdlib. This is a project Armin Rigo has been working
 for a while, with some input from other developers. It seems that the
 main reason why people would prefer ctypes over cffi these days is
 because it's included in stdlib, which is not generally the reason I
 would like to hear. Our calls to not use C extensions and to use an
 FFI instead has seen very limited success with ctypes and quite a lot
 more since cffi got released. The API is fairly stable right now with
 minor changes going in and it'll definitely stablize until Python 3.4
 release.

Do you intend to actually maintain it inside the CPython repository?

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

On Tue, Feb 26, 2013 at 7:42 AM, Nick Coghlan ncogh...@gmail.com wrote:

 On Wed, Feb 27, 2013 at 1:13 AM, Maciej Fijalkowski fij...@gmail.com
 wrote:
  Hello.
 
  I would like to discuss on the language summit a potential inclusion
  of cffi[1] into stdlib.

 I think cffi is well worth considering as a possible inclusion for
 Python 3.4. (In particular, I'm a fan of the fact it just uses C
 syntax to declare what you're trying to talk to)


I'm cautiously +0.5 because I'd really like to see a strong comparison case
being made vs. ctypes. I've used ctypes many times and it was easy and
effortless (well, except the segfaults when wrong argument types are
declared :-). I'll be really interesting in seeing concrete examples that
demonstrate how CFFI is superior.



 If I'm reading the dependencies correctly, we would also need to bring
 Eli Bendersky's pycparser into the stdlib, correct? (not an objection,
 just an observation, although we'd obviously need Eli's explicit
 endorsement for that).


Of course I have no problem with this. I obviously signed the contributor
agreement by this point ;-) pycparser depends on David Beazley's PLY
though, so that will have to be pulled in as well.

Eli
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

On Tue, Feb 26, 2013 at 8:26 AM, Maciej Fijalkowski fij...@gmail.comwrote:

 On Tue, Feb 26, 2013 at 5:42 PM, Nick Coghlan ncogh...@gmail.com wrote:
  On Wed, Feb 27, 2013 at 1:13 AM, Maciej Fijalkowski fij...@gmail.com
 wrote:
  Hello.
 
  I would like to discuss on the language summit a potential inclusion
  of cffi[1] into stdlib.
 
  I think cffi is well worth considering as a possible inclusion for
  Python 3.4. (In particular, I'm a fan of the fact it just uses C
  syntax to declare what you're trying to talk to)
 
  If I'm reading the dependencies correctly, we would also need to bring
  Eli Bendersky's pycparser into the stdlib, correct? (not an objection,
  just an observation, although we'd obviously need Eli's explicit
  endorsement for that).

 Yes, although, depending on the circumstances it's possible to hide it
 somewhere in cffi and not make it an official stdlib API (then several
 rules does not apply). It also pulls in some version of ply (or a
 generated parser).


I'll be the first one to admit that pycparser is almost certainly not
generally useful enough to be exposed in the stdlib. So just using it as an
implementation detail is absolutely fine. PLY is a more interesting
question, however, since PLY is somewhat more generally useful. That said,
I see all this as implementation details that shouldn't distract us from
the main point of whether cffi should be added.

Eli
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

On Tue, Feb 26, 2013 at 6:34 PM, Eli Bendersky eli...@gmail.com wrote:

 On Tue, Feb 26, 2013 at 7:42 AM, Nick Coghlan ncogh...@gmail.com wrote:

 On Wed, Feb 27, 2013 at 1:13 AM, Maciej Fijalkowski fij...@gmail.com
 wrote:
  Hello.
 
  I would like to discuss on the language summit a potential inclusion
  of cffi[1] into stdlib.

 I think cffi is well worth considering as a possible inclusion for
 Python 3.4. (In particular, I'm a fan of the fact it just uses C
 syntax to declare what you're trying to talk to)


 I'm cautiously +0.5 because I'd really like to see a strong comparison case
 being made vs. ctypes. I've used ctypes many times and it was easy and
 effortless (well, except the segfaults when wrong argument types are
 declared :-). I'll be really interesting in seeing concrete examples that
 demonstrate how CFFI is superior.

My main issue with ctypes, other than confusing API, which is up to
taste, is that you just cannot wrap some libraries, like OpenSSL
because of API vs ABI. OpenSSL uses macros extensively. Another point
is that even C POSIX stdlib gives you incomplete structs and you have
to guess where to put what fields.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

On Tue, Feb 26, 2013 at 6:32 PM, Antoine Pitrou solip...@pitrou.net wrote:
 Le Tue, 26 Feb 2013 17:13:44 +0200,
 Maciej Fijalkowski fij...@gmail.com a écrit :
 Hello.

 I would like to discuss on the language summit a potential inclusion
 of cffi[1] into stdlib. This is a project Armin Rigo has been working
 for a while, with some input from other developers. It seems that the
 main reason why people would prefer ctypes over cffi these days is
 because it's included in stdlib, which is not generally the reason I
 would like to hear. Our calls to not use C extensions and to use an
 FFI instead has seen very limited success with ctypes and quite a lot
 more since cffi got released. The API is fairly stable right now with
 minor changes going in and it'll definitely stablize until Python 3.4
 release.

 Do you intend to actually maintain it inside the CPython repository?

 Regards

 Antoine.

Once we put it in, yes, of course. Me Armin and Alex.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

2013-02-26 Thread Armin Rigo

Hi,

On Tue, Feb 26, 2013 at 5:38 PM, Maciej Fijalkowski fij...@gmail.com wrote:
 Do you intend to actually maintain it inside the CPython repository?

 Once we put it in, yes, of course. Me Armin and Alex.

Yes, I confirm. :-)

Armin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

2013-02-26 Thread Nick Coghlan

On Wed, Feb 27, 2013 at 2:39 AM, Maciej Fijalkowski fij...@gmail.com wrote:
 On Tue, Feb 26, 2013 at 6:34 PM, Eli Bendersky eli...@gmail.com wrote:

 On Tue, Feb 26, 2013 at 7:42 AM, Nick Coghlan ncogh...@gmail.com wrote:

 On Wed, Feb 27, 2013 at 1:13 AM, Maciej Fijalkowski fij...@gmail.com
 wrote:
  Hello.
 
  I would like to discuss on the language summit a potential inclusion
  of cffi[1] into stdlib.

 I think cffi is well worth considering as a possible inclusion for
 Python 3.4. (In particular, I'm a fan of the fact it just uses C
 syntax to declare what you're trying to talk to)


 I'm cautiously +0.5 because I'd really like to see a strong comparison case
 being made vs. ctypes. I've used ctypes many times and it was easy and
 effortless (well, except the segfaults when wrong argument types are
 declared :-). I'll be really interesting in seeing concrete examples that
 demonstrate how CFFI is superior.

 My main issue with ctypes, other than confusing API, which is up to
 taste, is that you just cannot wrap some libraries, like OpenSSL
 because of API vs ABI. OpenSSL uses macros extensively. Another point
 is that even C POSIX stdlib gives you incomplete structs and you have
 to guess where to put what fields.

For me, it's mostly the fact that ctypes is *less* typesafe than C
itself, because you can't easily consume the header files directly,
which leads directly to the segfaults caused by wrong declarations.
While at the time of inclusion there was a solid practicality beats
purity case for adding ctypes (and it's taken us quite a long way),
being less typesafe than C is still quite an achievement :)

As an experienced C programmer, not having to learn a new hybrid
language for declarations is also a plus, but the significantly less
likely to segfault aspect is the big one.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

On Tue, Feb 26, 2013 at 8:39 AM, Maciej Fijalkowski fij...@gmail.comwrote:

 On Tue, Feb 26, 2013 at 6:34 PM, Eli Bendersky eli...@gmail.com wrote:
 
  On Tue, Feb 26, 2013 at 7:42 AM, Nick Coghlan ncogh...@gmail.com
 wrote:
 
  On Wed, Feb 27, 2013 at 1:13 AM, Maciej Fijalkowski fij...@gmail.com
  wrote:
   Hello.
  
   I would like to discuss on the language summit a potential inclusion
   of cffi[1] into stdlib.
 
  I think cffi is well worth considering as a possible inclusion for
  Python 3.4. (In particular, I'm a fan of the fact it just uses C
  syntax to declare what you're trying to talk to)
 
 
  I'm cautiously +0.5 because I'd really like to see a strong comparison
 case
  being made vs. ctypes. I've used ctypes many times and it was easy and
  effortless (well, except the segfaults when wrong argument types are
  declared :-). I'll be really interesting in seeing concrete examples that
  demonstrate how CFFI is superior.

 My main issue with ctypes, other than confusing API, which is up to
 taste, is that you just cannot wrap some libraries, like OpenSSL
 because of API vs ABI. OpenSSL uses macros extensively. Another point
 is that even C POSIX stdlib gives you incomplete structs and you have
 to guess where to put what fields.


Yep, I can think of some reasons too. But just mentioning it so you
remember explicitly listing the advantages when it comes to writing a PEP
or some other sort of formal proposal. An FWIW, I think there's already
enough positive feedback to at least start drafting a PEP. It can always be
rejected later ;-)

Eli
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

On Tue, Feb 26, 2013 at 7:07 PM, Eli Bendersky eli...@gmail.com wrote:



 On Tue, Feb 26, 2013 at 8:39 AM, Maciej Fijalkowski fij...@gmail.com
 wrote:

 On Tue, Feb 26, 2013 at 6:34 PM, Eli Bendersky eli...@gmail.com wrote:
 
  On Tue, Feb 26, 2013 at 7:42 AM, Nick Coghlan ncogh...@gmail.com
  wrote:
 
  On Wed, Feb 27, 2013 at 1:13 AM, Maciej Fijalkowski fij...@gmail.com
  wrote:
   Hello.
  
   I would like to discuss on the language summit a potential inclusion
   of cffi[1] into stdlib.
 
  I think cffi is well worth considering as a possible inclusion for
  Python 3.4. (In particular, I'm a fan of the fact it just uses C
  syntax to declare what you're trying to talk to)
 
 
  I'm cautiously +0.5 because I'd really like to see a strong comparison
  case
  being made vs. ctypes. I've used ctypes many times and it was easy and
  effortless (well, except the segfaults when wrong argument types are
  declared :-). I'll be really interesting in seeing concrete examples
  that
  demonstrate how CFFI is superior.

 My main issue with ctypes, other than confusing API, which is up to
 taste, is that you just cannot wrap some libraries, like OpenSSL
 because of API vs ABI. OpenSSL uses macros extensively. Another point
 is that even C POSIX stdlib gives you incomplete structs and you have
 to guess where to put what fields.


 Yep, I can think of some reasons too. But just mentioning it so you remember
 explicitly listing the advantages when it comes to writing a PEP or some
 other sort of formal proposal. An FWIW, I think there's already enough
 positive feedback to at least start drafting a PEP. It can always be
 rejected later ;-)

 Eli


It's copy-pasted in the original mail and it's in the docs. Anyway,
this is just to familizarize people with the library. We're going to
discuss it on the language summit at pycon.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

2013-02-26 Thread Daniel Holth

+1


On Tue, Feb 26, 2013 at 12:14 PM, Maciej Fijalkowski fij...@gmail.comwrote:

 On Tue, Feb 26, 2013 at 7:07 PM, Eli Bendersky eli...@gmail.com wrote:
 
 
 
  On Tue, Feb 26, 2013 at 8:39 AM, Maciej Fijalkowski fij...@gmail.com
  wrote:
 
  On Tue, Feb 26, 2013 at 6:34 PM, Eli Bendersky eli...@gmail.com
 wrote:
  
   On Tue, Feb 26, 2013 at 7:42 AM, Nick Coghlan ncogh...@gmail.com
   wrote:
  
   On Wed, Feb 27, 2013 at 1:13 AM, Maciej Fijalkowski 
 fij...@gmail.com
   wrote:
Hello.
   
I would like to discuss on the language summit a potential
 inclusion
of cffi[1] into stdlib.
  
   I think cffi is well worth considering as a possible inclusion for
   Python 3.4. (In particular, I'm a fan of the fact it just uses C
   syntax to declare what you're trying to talk to)
  
  
   I'm cautiously +0.5 because I'd really like to see a strong comparison
   case
   being made vs. ctypes. I've used ctypes many times and it was easy and
   effortless (well, except the segfaults when wrong argument types are
   declared :-). I'll be really interesting in seeing concrete examples
   that
   demonstrate how CFFI is superior.
 
  My main issue with ctypes, other than confusing API, which is up to
  taste, is that you just cannot wrap some libraries, like OpenSSL
  because of API vs ABI. OpenSSL uses macros extensively. Another point
  is that even C POSIX stdlib gives you incomplete structs and you have
  to guess where to put what fields.
 
 
  Yep, I can think of some reasons too. But just mentioning it so you
 remember
  explicitly listing the advantages when it comes to writing a PEP or some
  other sort of formal proposal. An FWIW, I think there's already enough
  positive feedback to at least start drafting a PEP. It can always be
  rejected later ;-)
 
  Eli
 

 It's copy-pasted in the original mail and it's in the docs. Anyway,
 this is just to familizarize people with the library. We're going to
 discuss it on the language summit at pycon.
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 http://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe:
 http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

2013-02-26 Thread Paul Moore

On 26 February 2013 16:34, Eli Bendersky eli...@gmail.com wrote:
 I'm cautiously +0.5 because I'd really like to see a strong comparison case
 being made vs. ctypes. I've used ctypes many times and it was easy and
 effortless (well, except the segfaults when wrong argument types are
 declared :-). I'll be really interesting in seeing concrete examples that
 demonstrate how CFFI is superior.

I'm probably the same, mainly because I've successfully used ctypes in
the past, but I've never used cffi. That's something I need to
rectify.

One point which I *think* is correct, but which I don't see noted
anywhere. Am I right that cffi needs a C compiler involved in the
process, at least somewhere? If that's the case, then it is not a
suitable option for at least one use case that I have, writing quick
hacks involving the Windows API on a machine that doesn't have a C
compiler installed. Another possible case would be writing zip-safe
code - if cffi involves a compiled C extension, it won't work when
loaded from a zipfile.

In general, a proper, unbiased comparison between cffi and ctypes
would be useful.

BTW, I assume that the intention is that both cffi and ctypes remain
available indefinitely? Nobody's looking to deprecate ctypes?

Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

2013-02-26 Thread Paul Moore

On 26 February 2013 18:34, Maciej Fijalkowski fij...@gmail.com wrote:
 One point which I *think* is correct, but which I don't see noted
 anywhere. Am I right that cffi needs a C compiler involved in the
 process, at least somewhere? If that's the case, then it is not a
 suitable option for at least one use case that I have, writing quick
 hacks involving the Windows API on a machine that doesn't have a C
 compiler installed. Another possible case would be writing zip-safe
 code - if cffi involves a compiled C extension, it won't work when
 loaded from a zipfile.

 cffi does require a C compiler during either runtime or installation,
 if you use the API. You can still use the ABI level, which is like
 ctypes, only if you need the extra stuff, you need a C compiler.

Cool, thanks. So if I use it like ctypes, cffi doesn't need a C
compiler? That's nice, I never knew that.

I definitely need to try it out. (So many projects, so little time... :-))

Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

On Tue, Feb 26, 2013 at 8:14 PM, Paul Moore p.f.mo...@gmail.com wrote:
 On 26 February 2013 16:34, Eli Bendersky eli...@gmail.com wrote:
 I'm cautiously +0.5 because I'd really like to see a strong comparison case
 being made vs. ctypes. I've used ctypes many times and it was easy and
 effortless (well, except the segfaults when wrong argument types are
 declared :-). I'll be really interesting in seeing concrete examples that
 demonstrate how CFFI is superior.

 I'm probably the same, mainly because I've successfully used ctypes in
 the past, but I've never used cffi. That's something I need to
 rectify.

 One point which I *think* is correct, but which I don't see noted
 anywhere. Am I right that cffi needs a C compiler involved in the
 process, at least somewhere? If that's the case, then it is not a
 suitable option for at least one use case that I have, writing quick
 hacks involving the Windows API on a machine that doesn't have a C
 compiler installed. Another possible case would be writing zip-safe
 code - if cffi involves a compiled C extension, it won't work when
 loaded from a zipfile.

cffi does require a C compiler during either runtime or installation,
if you use the API. You can still use the ABI level, which is like
ctypes, only if you need the extra stuff, you need a C compiler.


 In general, a proper, unbiased comparison between cffi and ctypes
 would be useful.

that obviously can't come from me.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

2013-02-26 Thread R. David Murray

On Tue, 26 Feb 2013 18:14:26 +, Paul Moore p.f.mo...@gmail.com wrote:
 BTW, I assume that the intention is that both cffi and ctypes remain
 available indefinitely? Nobody's looking to deprecate ctypes?

I would expect that ctypes would be deprecated eventually simply because
there aren't very many people interested in maintaining it, and probably
fewer if cffi is accepted.  That said, I would not expect it to leave
the stdlib until either the bit rot was so bad it wouldn't be worth
shipping it, or (more likely) we reach Python4 and decide at that time
that it is time for it to go.

Of course, this is just me talking, we only have a *very* vague sense of
the house for what Python4 means at this point :)

--David
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

2013-02-26 Thread Guido van Rossum

Generally speaking, deferring something to Python 4 means never.

On Tue, Feb 26, 2013 at 11:06 AM, R. David Murray rdmur...@bitdance.com wrote:
 On Tue, 26 Feb 2013 18:14:26 +, Paul Moore p.f.mo...@gmail.com wrote:
 BTW, I assume that the intention is that both cffi and ctypes remain
 available indefinitely? Nobody's looking to deprecate ctypes?

 I would expect that ctypes would be deprecated eventually simply because
 there aren't very many people interested in maintaining it, and probably
 fewer if cffi is accepted.  That said, I would not expect it to leave
 the stdlib until either the bit rot was so bad it wouldn't be worth
 shipping it, or (more likely) we reach Python4 and decide at that time
 that it is time for it to go.

 Of course, this is just me talking, we only have a *very* vague sense of
 the house for what Python4 means at this point :)

 --David
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 http://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe: 
 http://mail.python.org/mailman/options/python-dev/guido%40python.org



--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

2013-02-26 Thread Brett Cannon

On Tue, Feb 26, 2013 at 2:12 PM, Guido van Rossum gu...@python.org wrote:

 Generally speaking, deferring something to Python 4 means never.


Does that mean your aversion to double digit version numbers (i.e. 3.10) is
gone or you expect to freeze Python in carbonite by then?

-Brett



 On Tue, Feb 26, 2013 at 11:06 AM, R. David Murray rdmur...@bitdance.com
 wrote:
  On Tue, 26 Feb 2013 18:14:26 +, Paul Moore p.f.mo...@gmail.com
 wrote:
  BTW, I assume that the intention is that both cffi and ctypes remain
  available indefinitely? Nobody's looking to deprecate ctypes?
 
  I would expect that ctypes would be deprecated eventually simply because
  there aren't very many people interested in maintaining it, and probably
  fewer if cffi is accepted.  That said, I would not expect it to leave
  the stdlib until either the bit rot was so bad it wouldn't be worth
  shipping it, or (more likely) we reach Python4 and decide at that time
  that it is time for it to go.
 
  Of course, this is just me talking, we only have a *very* vague sense of
  the house for what Python4 means at this point :)
 
  --David
  ___
  Python-Dev mailing list
  Python-Dev@python.org
  http://mail.python.org/mailman/listinfo/python-dev
  Unsubscribe:
 http://mail.python.org/mailman/options/python-dev/guido%40python.org



 --
 --Guido van Rossum (python.org/~guido)
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 http://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe:
 http://mail.python.org/mailman/options/python-dev/brett%40python.org

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

2013-02-26 Thread Guido van Rossum

With 1.5 years per release, it'd be 10 years before we'd hit 3.10.
From a software engineering perspective, 10 years is indistinguishable
from infinity, so I don't care what happens 10 years from now -- as
long as you don't blame me. :-)

On Tue, Feb 26, 2013 at 11:21 AM, Brett Cannon br...@python.org wrote:



 On Tue, Feb 26, 2013 at 2:12 PM, Guido van Rossum gu...@python.org wrote:

 Generally speaking, deferring something to Python 4 means never.


 Does that mean your aversion to double digit version numbers (i.e. 3.10) is
 gone or you expect to freeze Python in carbonite by then?

 -Brett



 On Tue, Feb 26, 2013 at 11:06 AM, R. David Murray rdmur...@bitdance.com
 wrote:
  On Tue, 26 Feb 2013 18:14:26 +, Paul Moore p.f.mo...@gmail.com
  wrote:
  BTW, I assume that the intention is that both cffi and ctypes remain
  available indefinitely? Nobody's looking to deprecate ctypes?
 
  I would expect that ctypes would be deprecated eventually simply because
  there aren't very many people interested in maintaining it, and probably
  fewer if cffi is accepted.  That said, I would not expect it to leave
  the stdlib until either the bit rot was so bad it wouldn't be worth
  shipping it, or (more likely) we reach Python4 and decide at that time
  that it is time for it to go.
 
  Of course, this is just me talking, we only have a *very* vague sense
  of
  the house for what Python4 means at this point :)
 
  --David
  ___
  Python-Dev mailing list
  Python-Dev@python.org
  http://mail.python.org/mailman/listinfo/python-dev
  Unsubscribe:
  http://mail.python.org/mailman/options/python-dev/guido%40python.org



 --
 --Guido van Rossum (python.org/~guido)
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 http://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe:
 http://mail.python.org/mailman/options/python-dev/brett%40python.org





--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

2013-02-26 Thread Antoine Pitrou

On Tue, 26 Feb 2013 14:21:03 -0500
Brett Cannon br...@python.org wrote:
 On Tue, Feb 26, 2013 at 2:12 PM, Guido van Rossum gu...@python.org wrote:
 
  Generally speaking, deferring something to Python 4 means never.
 
 
 Does that mean your aversion to double digit version numbers (i.e. 3.10) is
 gone or you expect to freeze Python in carbonite by then?

Or we'll go straight to 5.
(or switch to date-based numbering :-))

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

2013-02-26 Thread Greg Ewing


Antoine Pitrou wrote:

Or we'll go straight to 5.
(or switch to date-based numbering :-))


We could go the Apple route and start naming them after
species of snake.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

2013-02-26 Thread Terry Reedy


On 2/26/2013 10:13 AM, Maciej Fijalkowski wrote:


I would like to discuss on the language summit a potential inclusion
of cffi[1] into stdlib.


How does it compare in terms of speed. One reason ctypes has not 
replaces hand-tuned swig is that it apparently is much slower. I know 
that someone, for instance, once did a pygame fork using ctypes to wrap 
SDL, and got it to run, but gave it up for comparative slowness reasons.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib