[Cython] History of SWIG and applicability to Cython
Hi, here's an interesting history wrap-up of SWIG, by its original author. One thing that stroke me when I read this (or rather when I was half-way through) was that it might be possible to use SWIG as a glue code generator for .pxd files and trivial wrapper code. Not sure how hard that would be - it does sound like such a complex system could also be somewhat tricky to extend ... Stefan Original-Message Subject: SWIG (was Re: Ctypes and the stdlib) Date: Mon, 29 Aug 2011 07:41:23 -0500 From: David Beazley To: python-dev...@python.org CC: David Beazley On Mon, Aug 29, 2011 at 12:27 PM, Guido van Rossum wrote: I wonder if for this particular purpose SWIG isn't the better match. (If SWIG weren't universally hated, even by its original author. :-) Hate is probably a strong word, but as the author of Swig, let me chime in here ;-). I think there are probably some lessons to be learned from Swig. As Nick noted, Swig is best suited when you have control over both sides (C/C++ and Python) of whatever code you're working with. In fact, the original motivation for Swig was to give application programmers (scientists in my case), a means for automatically generating the Python bindings to their code. However, there was one other important assumption--and that was the fact that all of your "real code" was going to be written in C/C++ and that the Python scripting interface was just an optional add-on (perhaps even just a throw-away thing). Keep in mind, Swig was first created in 1995 and at that time, the use of Python (or any similar language) was a pretty radical idea in the sciences. Moreover, there was a lot of legacy code that people just weren't going to abandon. Thus, I alwa ys viewed Swig as a kind of transitional vehicle for getting people to use Python who might otherwise not even consider it. Getting back to Nick's point though, to really use Swig effectiv ely, it was always known that you might have to reorganize or refactor your C/C++ code to make it more Python friendly. However, due to the automatic wrapper generation, you didn't have to do it all at once. Basically your code could organically evolve and Swig would just keep up with whatever you were doing. In my projects, we'd usually just tuck Swig away in some Makefile somewhere and forget about it. One of the major complexities of Swig is the fact that it attempts to parse C/C++ header files. This very notion is actually a dangerous trap waiting for anyone who wants to wander into it. You might look at a header file and say, well how hard could it be to just grab a few definitions out of there? I'll just write a few regexs or come up with some simple hack for recognizing function definitions or something. Yes, you can do that, but you're immediately going to find that whatever approach you take starts to break down into horrible corner cases. Swig started out like this and quickly turned into a quagmire of esoteric bug reports. All sorts of problems with preprocessor macros, typedefs, missing headers, and other things. For awhile, I would get these bug reports that would g o something like "I had this C++ class inside a namespace with an abstract method taking a typedef'd const reference to this smart pointer . and Swig broke." Hell, I can't even underst and the bug report let alone know how to fix it. Almost all of these bugs were due to the fact that Swig started out as a hack and didn't really have any kind of solid conceptual foundation for how it should be put together. If you flash forward a bit, from about 2001-2004 there was a very serious push to fix these kinds of issues. Although it was not a complete rewrite of Swig, there were a huge number of changes to how it worked during this time. Swig grew a fully compatible C++ preprocessor that fully supported macros A complete C++ type system was implemented including support for namespaces, templates, and even such things as template partial specialization. Swig evolved into a multi-pass compiler that was doing all sorts of global analysis of the interface. Just to give you an idea, Swig would do things such as automatically detect/wrap C++ smart pointers. It could wrap overloaded C++ methods/function. Also, if you had a C++ class with virtual methods, it would only make one Python wrapper funct ion and then reuse across all wrapped subclasses. Under the covers of all of this, the implementation basically evolved into a sophisticated macro preprocessor coupled with a pattern matching engine built on top of the C++ type system. For example, you could write patterns that matched specific C++ types (the much hated "typemap" feature) and you could write patterns that matched entire C++ declarations. This whole pattern matching approach had a huge power if you knew what you were doing. For example, I had a graduate student working on adding "contracts" to Sw
Re: [Cython] History of SWIG and applicability to Cython
Stefan Behnel, 29.08.2011 16:33: here's an interesting history wrap-up of SWIG, by its original author. [...] And an interesting reply: http://thread.gmane.org/gmane.comp.python.devel/126425/focus=126440 I didn't know clang even had Python bindings for its parser. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] History of SWIG and applicability to Cython
On 29 August 2011 19:25, Stefan Behnel wrote: > Stefan Behnel, 29.08.2011 16:33: >> >> here's an interesting history wrap-up of SWIG, by its original author. >> [...] > > And an interesting reply: > > http://thread.gmane.org/gmane.comp.python.devel/126425/focus=126440 > > I didn't know clang even had Python bindings for its parser. > > Stefan > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > Last time I checked I think they didn't support the preprocessor yet, and I do think you want access to macros. Macro constants are easy if you have access to the preprocessor, I don't know about macro functions... perhaps those should remain the user's burden. I would certainly welcome a good pxd generator. There is one such generator out there, can't seem to find it right now. I think it uses gccxml (which parses C++ and generates xml). There's also ctypesgen, which is also based on gccxml. I think clang or gccxml would be easier and less of a hack than swig, although I haven't looked at swig's implementation. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] History of SWIG and applicability to Cython
On Mon, Aug 29, 2011 at 10:31 PM, mark florisson wrote: > On 29 August 2011 19:25, Stefan Behnel wrote: >> Stefan Behnel, 29.08.2011 16:33: >>> >>> here's an interesting history wrap-up of SWIG, by its original author. >>> [...] >> >> And an interesting reply: >> >> http://thread.gmane.org/gmane.comp.python.devel/126425/focus=126440 >> >> I didn't know clang even had Python bindings for its parser. >> >> Stefan >> ___ >> cython-devel mailing list >> cython-devel@python.org >> http://mail.python.org/mailman/listinfo/cython-devel >> > > Last time I checked I think they didn't support the preprocessor yet, > and I do think you want access to macros. Macro constants are easy if > you have access to the preprocessor, I don't know about macro > functions... perhaps those should remain the user's burden. I would > certainly welcome a good pxd generator. > > There is one such generator out there, can't seem to find it right > now. I did one such thing, but it is more of an hack than a real solution: https://github.com/cournape/cython-codegen Using gccxml is a PITA: - you cannot force C mode (always assume C++) - building gccxml is not pleasant clang is much better for this kind of things, cheers, David ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] History of SWIG and applicability to Cython
I've been working on CWrap in my spare time. https://github.com/enthought/cwrap The goal is to be SWIG for Cython. It currently outputs pxd files for the headers of a C library. It's not mature, but it works. (It can output pxd files for the whole of Intel's IPP and MKL libraries). I hope to eventually output pyx implementation files as well. It currently uses gccxml, but I hope to make the switch to Clang. We've been talking amongst ourselves @ enthought to lobby you guys eventually to merge it with Cython main, but I just haven't had the bandwidth to get it up to that point. Chris On Mon, Aug 29, 2011 at 3:42 PM, David Cournapeau wrote: > On Mon, Aug 29, 2011 at 10:31 PM, mark florisson > wrote: > > On 29 August 2011 19:25, Stefan Behnel wrote: > >> Stefan Behnel, 29.08.2011 16:33: > >>> > >>> here's an interesting history wrap-up of SWIG, by its original author. > >>> [...] > >> > >> And an interesting reply: > >> > >> http://thread.gmane.org/gmane.comp.python.devel/126425/focus=126440 > >> > >> I didn't know clang even had Python bindings for its parser. > >> > >> Stefan > >> ___ > >> cython-devel mailing list > >> cython-devel@python.org > >> http://mail.python.org/mailman/listinfo/cython-devel > >> > > > > Last time I checked I think they didn't support the preprocessor yet, > > and I do think you want access to macros. Macro constants are easy if > > you have access to the preprocessor, I don't know about macro > > functions... perhaps those should remain the user's burden. I would > > certainly welcome a good pxd generator. > > > > There is one such generator out there, can't seem to find it right > > now. > > I did one such thing, but it is more of an hack than a real solution: > > https://github.com/cournape/cython-codegen > > Using gccxml is a PITA: > - you cannot force C mode (always assume C++) > - building gccxml is not pleasant > > clang is much better for this kind of things, > > cheers, > > David > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] History of SWIG and applicability to Cython
On Mon, Aug 29, 2011 at 7:33 AM, Stefan Behnel wrote: > Hi, > > here's an interesting history wrap-up of SWIG, by its original author. Very interesting thread. > One thing that stroke me when I read this (or rather when I was half-way > through) was that it might be possible to use SWIG as a glue code generator > for .pxd files and trivial wrapper code. Not sure how hard that would be - > it does sound like such a complex system could also be somewhat tricky to > extend ... Yeah, I've thought about this before too. It's probably a project worth pursuing (especially given all the swig wrapper and wrapped code out there) but haven't ever looked seriously into it. Clang seems like a more solid option long-term, and what I would pick if I were to write a wrapper-generator from scratch (especially for C++). Of course there's still all the messiness of pointer memory management, type translation etc. The idea of automatically parsing header files has been around for a *long* time, and lots of people have come up with partial solutions (e.g. "good enough" for their tasks) but solving the generic problem can be quite hard. So they don't get lost, I've created a page at http://wiki.cython.org/AutoPxd . Eventually it'd be nice if one such solution became good enough to be the "standard" solution that everyone put their weight behind. As for inclusion in Cython, I think it's still to early to tell, but the a standalone tool to do the .h -> .pxd/.pyx translation would be quite useful even if it remains a separate project. (And the two could know about/interface with each other, even if they weren't bundled together.) - Robert > Original-Message > Subject: SWIG (was Re: Ctypes and the stdlib) > Date: Mon, 29 Aug 2011 07:41:23 -0500 > From: David Beazley > To: python-dev...@python.org > CC: David Beazley > > On Mon, Aug 29, 2011 at 12:27 PM, Guido van Rossum > wrote: > >> I wonder if for >> this particular purpose SWIG isn't the better match. (If SWIG weren't >> universally hated, even by its original author. :-) > > Hate is probably a strong word, but as the author of Swig, let me chime in > here ;-). I think there are probably some lessons to be learned from Swig. > > As Nick noted, Swig is best suited when you have control over both sides > (C/C++ and Python) of whatever code you're working with. In fact, the > original motivation for Swig was to give application programmers > (scientists in my case), a means for automatically generating the Python > bindings to their code. However, there was one other important > assumption--and that was the fact that all of your "real code" was going to > be written in C/C++ and that the Python scripting interface was just an > optional add-on (perhaps even just a throw-away thing). Keep in mind, Swig > was first created in 1995 and at that time, the use of Python (or any > similar language) was a pretty radical idea in the sciences. Moreover, > there was a lot of legacy code that people just weren't going to abandon. > Thus, I alwa > ys viewed Swig as a kind of transitional vehicle for getting people to use > Python who might otherwise not even consider it. Getting back to Nick's > point though, to really use Swig effectiv > ely, it was always known that you might have to reorganize or refactor your > C/C++ code to make it more Python friendly. However, due to the automatic > wrapper generation, you didn't have to do it all at once. Basically your > code could organically evolve and Swig would just keep up with whatever you > were doing. In my projects, we'd usually just tuck Swig away in some > Makefile somewhere and forget about it. > > One of the major complexities of Swig is the fact that it attempts to parse > C/C++ header files. This very notion is actually a dangerous trap waiting > for anyone who wants to wander into it. You might look at a header file and > say, well how hard could it be to just grab a few definitions out of there? > I'll just write a few regexs or come up with some simple hack for > recognizing function definitions or something. Yes, you can do that, but > you're immediately going to find that whatever approach you take starts to > break down into horrible corner cases. Swig started out like this and > quickly turned into a quagmire of esoteric bug reports. All sorts of > problems with preprocessor macros, typedefs, missing headers, and other > things. For awhile, I would get these bug reports that would g > o something like "I had this C++ class inside a namespace with an abstract > method taking a typedef'd const reference to this smart pointer . and > Swig broke." Hell, I can't even underst > and the bug report let alone know how to fix it. Almost all of these bugs > were due to the fact that Swig started out as a hack and didn't really have > any kind of solid conceptual foundation for how it should be put together. > > If you flash forward a bit, from about 2001-2004 there was a very seri
Re: [Cython] non-virtual methods
On Sun, Aug 28, 2011 at 4:00 AM, Vitja Makarov wrote:
> 2011/8/27 Vitja Makarov :
>> 2011/8/27 Vitja Makarov :
>>> 2011/8/26 Vitja Makarov :
2011/8/25 Vitja Makarov :
> 2011/8/25 Vitja Makarov :
>> 2011/8/25 Stefan Behnel :
>>> Vitja Makarov, 25.08.2011 20:32:
2011/8/25 Stefan Behnel:
>
> Vitja Makarov, 25.08.2011 18:11:
>>
>> 2011/8/24 Stefan Behnel:
>>>
>>> Vitja Makarov, 24.08.2011 21:17:
I tried final classes:
2. In this example foo call is done through virtual table
cimport cython
@cython.final
cdef class Foo:
cdef foo(self):
print 'haha'
def test():
cdef Foo a = Foo()
a.foo()
__pyx_t_1 = ((struct __pyx_vtabstruct_3yyy_Foo
*)__pyx_v_a->__pyx_vtab)->foo(__pyx_v_a); if (unlikely(!__pyx_t_1))
{__pyx_filename = __pyx_f[0]; __pyx_lineno = 12; __pyx_clineno =
__LINE__; goto __pyx_L1_error;}
>>>
>>> Right, this is not implemented yet. Feel free to do so. Also see
>>>
>>> http://trac.cython.org/cython_trac/ticket/474
>>>
>>>
3. I can't use final decorator for methods (error reported)
>>>
>>> http://trac.cython.org/cython_trac/ticket/586
>>
>> What is the prefered syntax keyword inline or final decorator?
>
> "final" is the right option here.
>
> They are orthogonal concepts. Only because you declare a method
> "final"
> does
> not mean you want to inline it, and just because you declare it
> "inline"
> does not (necessarily) mean that you cannot override it. Admittedly,
> the
> semantics of an overridable inline method may turn out to be somewhat
> obscure and error prone, so I think it's a good idea to let "inline"
> imply
> "final". But not the other way round.
But both inline and final methods should bypass vtab, right?
>>>
>>> Yes. But in the "final" case, it's always clear which method
>>> implementation
>>> to use - it's not overridable, so there is only one choice. In the
>>> "inline"
>>> case, it could still be overridable and we may have a subtype of the
>>> declared type in our hands at runtime, thus choosing the wrong method at
>>> compile time. That's why only the "final" case is safe.
>>>
>>> Note that I'm only talking about the semantics of the qualifier
>>> themselves
>>> here. If we allow "inline" methods, I think we should force them to be
>>> "final" as well. But that's a practical choice, not a semantic
>>> implication.
>>>
>>>
Also I'm not sure about C inline qualifier here.
>>>
>>> That's what "inline" requests.
>>>
>>>
I see three options:
- non-virtual: bypass vtab
- final: non-virtual, non-overridable
>>>
>>> How would you want to bypass the vtable in the "non-virtual" case if the
>>> method is overridable?
>>>
>>>
- inline: non-virtual, C inline qualifier is used
>>>
>>> Correct.
>>>
>>
>> Ok.
>>
>> I think it's better to implement final method then user could choose
>> to use inline qualifier or not.
>>
>
> I tried it here:
> https://github.com/vitek/cython/commit/ddf80a80dc75aced2cd92dc32afa77a7bcf2de02
>
> There is one problem: vtab bypassing should be enabled if final method
> is defined in the same module.
> I don't know how to check that final method comes from cimport (it's
> okay with pxd, the problem is pyx)
>
https://github.com/vitek/cython/commit/6e91fc257a683018ba6be340d384f9a7c34ef425
Here is update version. I've add tree asserts and final method's
prototypes.
--
vitja.
>>>
>>> I've created ticket for compiler crash when cython.final is used
>>> inside pxd file:
>>>
>>> http://trac.cython.org/cython_trac/ticket/722
>>>
>>> Also I've updated final methods test case (added tree path assertions)
>>> https://github.com/vitek/cython/commit/92edb09419c9b77a792f7c43e6ddd760b00c4e74
>>>
>>>
>>> About declaration origin detection may be it's a good idea to have a
>>> flag at scope level something like is_pxd_scope or
>>> is_declaration_scope?
>>>
>>
>> It seems to me that I found a way to fix pxd/pyx cimport problem. I've
>> created pull request:
>>
>> https://github.com/cython/cython/pull/59
>>
>
> I've add support for inline methods, now you can declare inline method
> in pxd file:
>
> cdef class Foo:
> cdef inline foo(self):
> return 1
Cool.
I suppose for cross-module calls, non-inline methods vtables are best
one can do
Re: [Cython] non-virtual methods
2011/8/30 Robert Bradshaw :
> On Sun, Aug 28, 2011 at 4:00 AM, Vitja Makarov
> wrote:
>> 2011/8/27 Vitja Makarov :
>>> 2011/8/27 Vitja Makarov :
2011/8/26 Vitja Makarov :
> 2011/8/25 Vitja Makarov :
>> 2011/8/25 Vitja Makarov :
>>> 2011/8/25 Stefan Behnel :
Vitja Makarov, 25.08.2011 20:32:
>
> 2011/8/25 Stefan Behnel:
>>
>> Vitja Makarov, 25.08.2011 18:11:
>>>
>>> 2011/8/24 Stefan Behnel:
Vitja Makarov, 24.08.2011 21:17:
>
> I tried final classes:
> 2. In this example foo call is done through virtual table
>
> cimport cython
>
> @cython.final
> cdef class Foo:
> cdef foo(self):
> print 'haha'
>
> def test():
> cdef Foo a = Foo()
> a.foo()
>
> __pyx_t_1 = ((struct __pyx_vtabstruct_3yyy_Foo
> *)__pyx_v_a->__pyx_vtab)->foo(__pyx_v_a); if
> (unlikely(!__pyx_t_1))
> {__pyx_filename = __pyx_f[0]; __pyx_lineno = 12; __pyx_clineno =
> __LINE__; goto __pyx_L1_error;}
Right, this is not implemented yet. Feel free to do so. Also see
http://trac.cython.org/cython_trac/ticket/474
> 3. I can't use final decorator for methods (error reported)
http://trac.cython.org/cython_trac/ticket/586
>>>
>>> What is the prefered syntax keyword inline or final decorator?
>>
>> "final" is the right option here.
>>
>> They are orthogonal concepts. Only because you declare a method
>> "final"
>> does
>> not mean you want to inline it, and just because you declare it
>> "inline"
>> does not (necessarily) mean that you cannot override it. Admittedly,
>> the
>> semantics of an overridable inline method may turn out to be somewhat
>> obscure and error prone, so I think it's a good idea to let "inline"
>> imply
>> "final". But not the other way round.
>
> But both inline and final methods should bypass vtab, right?
Yes. But in the "final" case, it's always clear which method
implementation
to use - it's not overridable, so there is only one choice. In the
"inline"
case, it could still be overridable and we may have a subtype of the
declared type in our hands at runtime, thus choosing the wrong method
at
compile time. That's why only the "final" case is safe.
Note that I'm only talking about the semantics of the qualifier
themselves
here. If we allow "inline" methods, I think we should force them to be
"final" as well. But that's a practical choice, not a semantic
implication.
> Also I'm not sure about C inline qualifier here.
That's what "inline" requests.
> I see three options:
>
> - non-virtual: bypass vtab
> - final: non-virtual, non-overridable
How would you want to bypass the vtable in the "non-virtual" case if
the
method is overridable?
> - inline: non-virtual, C inline qualifier is used
Correct.
>>>
>>> Ok.
>>>
>>> I think it's better to implement final method then user could choose
>>> to use inline qualifier or not.
>>>
>>
>> I tried it here:
>> https://github.com/vitek/cython/commit/ddf80a80dc75aced2cd92dc32afa77a7bcf2de02
>>
>> There is one problem: vtab bypassing should be enabled if final method
>> is defined in the same module.
>> I don't know how to check that final method comes from cimport (it's
>> okay with pxd, the problem is pyx)
>>
>
>
> https://github.com/vitek/cython/commit/6e91fc257a683018ba6be340d384f9a7c34ef425
>
> Here is update version. I've add tree asserts and final method's
> prototypes.
>
> --
> vitja.
>
I've created ticket for compiler crash when cython.final is used
inside pxd file:
http://trac.cython.org/cython_trac/ticket/722
Also I've updated final methods test case (added tree path assertions)
https://github.com/vitek/cython/commit/92edb09419c9b77a792f7c43e6ddd760b00c4e74
About declaration origin detection may be it's a good idea to have a
flag at scope level something like is_pxd_scope or
is_declaration_scope?
>>>
>>> It seems to me that I found a way to fix pxd/pyx cimport problem. I've
>>> created pull request:
>>>
>>> https://github.com/cython/cython/pull/59
>>>
>>
>> I've add support for inline m
