Re: [Python-Dev] libffi embedded in CPython
On 25 March 2015 at 08:05, Maciej Fijalkowski fij...@gmail.com wrote: Linux crashes. The mechanism for detecting the number of arguments is only available on windows (note that this is a band-aid anyway, since if your arguments are of the wrong kind you segfault anyway). We do have two copies of libffi one for windows one for unix anyway, don't we? Yes, this is in relation to looking at whether we can just use the upstream libffi directly on Windows. It looks like we guarantee a ValueError for the wrong number of arguments on Windows and upstream libffi doesn't give a way to detect that, so there may be a backward compatibility issue in doing so. Thanks for checking for me. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Tue, Mar 24, 2015 at 11:31 PM, Paul Moore p.f.mo...@gmail.com wrote: On 12 March 2015 at 17:44, Paul Moore p.f.mo...@gmail.com wrote: On 12 March 2015 at 17:26, Brett Cannon br...@python.org wrote: I'm all for ditching our 'libffi_msvc' in favor of adding libffi as another 'external' for the Windows build. I have managed to get _ctypes to build on Windows using vanilla libffi sources, prepared using their configure script from within Git Bash and built with our usual Windows build system (properly patched). Unfortunately, making things usable will take some work on ctypes itself, which I'm not qualified to do. I'm happy to pass on my procedure and patches for getting to the point of successful compilation to anyone who feels up to fixing the things that are broken. So it seems possible to use upstream libffi but will require some work. I'd be willing to contemplate helping out on the Windows side of things, if nobody else steps up (with the proviso that I have little free time, and I'm saying this without much idea of what's involved :-)) If Zachary can give a bit more detail on what the work on ctypes is, and/or put what he has somewhere that I could have a look at, that might help. One thing that seems to be an issue. On Windows, ctypes detects if the FFI call used the wrong number of arguments off the stack, and raises a ValueError if it does. The tests rely on that behaviour. But it's based on ffi_call() returning a value, which upstream libffi doesn't do. As far as I can tell (not that the libffi docs are exactly comprehensive...) there's no way of getting that information from upstream libffi. What does Unix ctypes do when faced with a call being made with the wrong number of arguments? On Windows, using upstream libffi and omitting the existing check, it seems to crash the Python process, which obviously isn't good. But the test that fails is Windows-specific, and short of going through all the tests looking for one that checks passing the wrong number of arguments and isn't platform-specific, I don't know how Unix handles this. Can anyone on Unix tell me if a ctypes call with the wrong number of arguments returns ValueError on Unix? Something like strcmp() (with no args) should do as a test, I guess... If there's a way Unix handles this, I can see about replicating it on Windows. But if there isn't, I fear we could always need a patched libffi to maintain the interface we currently have... Thanks, Paul Linux crashes. The mechanism for detecting the number of arguments is only available on windows (note that this is a band-aid anyway, since if your arguments are of the wrong kind you segfault anyway). We do have two copies of libffi one for windows one for unix anyway, don't we? ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Wed, 25 Mar 2015 08:15:18 + Paul Moore p.f.mo...@gmail.com wrote: On 25 March 2015 at 08:05, Maciej Fijalkowski fij...@gmail.com wrote: Linux crashes. The mechanism for detecting the number of arguments is only available on windows (note that this is a band-aid anyway, since if your arguments are of the wrong kind you segfault anyway). We do have two copies of libffi one for windows one for unix anyway, don't we? Yes, this is in relation to looking at whether we can just use the upstream libffi directly on Windows. It looks like we guarantee a ValueError for the wrong number of arguments on Windows and upstream libffi doesn't give a way to detect that, so there may be a backward compatibility issue in doing so. I'm not sure we guarantee anything. In any case, it's only a small proportion of the kind of crashes you can get by messing the signature. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Wed, 25 Mar 2015 09:22:01 + Paul Moore p.f.mo...@gmail.com wrote: On 25 March 2015 at 09:09, Antoine Pitrou solip...@pitrou.net wrote: I'm not sure we guarantee anything. In any case, it's only a small proportion of the kind of crashes you can get by messing the signature. Fair point. I guess what I'm asking is, would it be OK to remove the code that checks for a stack size discrepancy and raises ValueError, and the tests that verify this behaviour, as part of switching to using upstream libffi directly? IMHO yes. Of course it's still a behaviour change, but relying on it sounds seriously broken. On a related note, is there any information available on how the externals repo is maintained? In particular, should things in there be exact copies of upstream, or is it OK to include extra data (in this case, the results of running configure for the Windows build)? It works for me either way, it's just a matter of how the build process would be structured and maintained. Zachary or Steve could probably answer you on that one. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On 25 March 2015 at 09:09, Antoine Pitrou solip...@pitrou.net wrote: I'm not sure we guarantee anything. In any case, it's only a small proportion of the kind of crashes you can get by messing the signature. Fair point. I guess what I'm asking is, would it be OK to remove the code that checks for a stack size discrepancy and raises ValueError, and the tests that verify this behaviour, as part of switching to using upstream libffi directly? On a related note, is there any information available on how the externals repo is maintained? In particular, should things in there be exact copies of upstream, or is it OK to include extra data (in this case, the results of running configure for the Windows build)? It works for me either way, it's just a matter of how the build process would be structured and maintained. Thanks, Paul Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Mar 25, 2015 4:22 AM, Paul Moore p.f.mo...@gmail.com wrote: On 25 March 2015 at 09:09, Antoine Pitrou solip...@pitrou.net wrote: I'm not sure we guarantee anything. In any case, it's only a small proportion of the kind of crashes you can get by messing the signature. Fair point. I guess what I'm asking is, would it be OK to remove the code that checks for a stack size discrepancy and raises ValueError, and the tests that verify this behaviour, as part of switching to using upstream libffi directly? On a related note, is there any information available on how the externals repo is maintained? In particular, should things in there be exact copies of upstream, or is it OK to include extra data (in this case, the results of running configure for the Windows build)? It works for me either way, it's just a matter of how the build process would be structured and maintained. Its not extremely clear how it's supposed to be done; look at the differences between how we handle OpenSSL and Tcl/Tk, for example. One way or the other, though, we will store the configure output so that our build doesn't depend on any more than it absolutely has to. -- Zach On a phone ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Mar 25, 2015 9:28 PM, Nick Coghlan ncogh...@gmail.com wrote: On 26 March 2015 at 01:57, Steve Dower steve.do...@microsoft.com wrote: Zachary Ware wrote: On Mar 25, 2015 4:22 AM, Paul Moore p.f.mo...@gmail.com wrote: On a related note, is there any information available on how the externals repo is maintained? In particular, should things in there be exact copies of upstream, or is it OK to include extra data (in this case, the results of running configure for the Windows build)? It works for me either way, it's just a matter of how the build process would be structured and maintained. Its not extremely clear how it's supposed to be done; look at the differences between how we handle OpenSSL and Tcl/Tk, for example. One way or the other, though, we will store the configure output so that our build doesn't depend on any more than it absolutely has to. It would be nice for at least one other person to know how to do it. I can't see the SVN logs, but I feel like you're the only person who's touched it in a while :) As well as the configure output, we sometimes include very light patching (for example, Tcl/Tk currently has a patch to their makefile to work around a VC14 bug). I'm not sure whether that's considered good practice or not, but I prefer only needing to do it once vs. building it into the build system. It likely makes sense to try to at least write down the status quo regarding the externals repo as a new top level section in the Developer's Guide. That way you'll at least have a shared understanding of the starting point for any further improvements. I'll try to get something thrown together at least by the time I leave PyCon. Unfortunately I can't guarantee anything sooner than that. -- Zach On a phone ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On 26 March 2015 at 12:40, Zachary Ware zachary.ware+py...@gmail.com wrote: On Mar 25, 2015 9:28 PM, Nick Coghlan ncogh...@gmail.com wrote: On 26 March 2015 at 01:57, Steve Dower steve.do...@microsoft.com wrote: Zachary Ware wrote: On Mar 25, 2015 4:22 AM, Paul Moore p.f.mo...@gmail.com wrote: On a related note, is there any information available on how the externals repo is maintained? In particular, should things in there be exact copies of upstream, or is it OK to include extra data (in this case, the results of running configure for the Windows build)? It works for me either way, it's just a matter of how the build process would be structured and maintained. Its not extremely clear how it's supposed to be done; look at the differences between how we handle OpenSSL and Tcl/Tk, for example. One way or the other, though, we will store the configure output so that our build doesn't depend on any more than it absolutely has to. It would be nice for at least one other person to know how to do it. I can't see the SVN logs, but I feel like you're the only person who's touched it in a while :) As well as the configure output, we sometimes include very light patching (for example, Tcl/Tk currently has a patch to their makefile to work around a VC14 bug). I'm not sure whether that's considered good practice or not, but I prefer only needing to do it once vs. building it into the build system. It likely makes sense to try to at least write down the status quo regarding the externals repo as a new top level section in the Developer's Guide. That way you'll at least have a shared understanding of the starting point for any further improvements. I'll try to get something thrown together at least by the time I leave PyCon. Unfortunately I can't guarantee anything sooner than that. That sounds plenty soon enough to me - my own list of I should really do something about that some day items stretches back years, as the issue tracker occasionally reminds me :) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On 03/11/2015 06:27 PM, Brett Cannon wrote: On Mon, Dec 22, 2014 at 4:49 PM Jim J. Jewett jimjjew...@gmail.com wrote: On Thu, Dec 18, 2014, at 14:13, Maciej Fijalkowski wrote: ... http://bugs.python.org/issue23085 ... is there any reason any more for libffi being included in CPython? Paul Moore wrote: Probably the easiest way of moving this forward would be for someone to identify the CPython-specific patches in the current version ... Christian Heimes wrote: That's easy. All patches are tracked in the diff file https://hg.python.org/cpython/file/3de678cd184d/Modules/_ ctypes/libffi.diff That (200+ lines) doesn't seem to have all the C changes, such as the win64 sizeof changes from issue 11835. Besides http://bugs.python.org/issue23085, there is at least http://bugs.python.org/issue22733 http://bugs.python.org/issue20160 http://bugs.python.org/issue11835 which sort of drives home the point that making sure we have a good merge isn't trivial, and this isn't an area where we should just assume that tests will catch everything. I don't think it is just a quicky waiting on permission. I've no doubt that upstream libffi is better in many ways, but those are ways people have already learned to live with. That said, I haven't seen any objections in principle, except perhaps from Steve Dower in the issues. (I *think* he was just saying not worth the time to me, but it was ambiguous.) I do believe that Christian or Maciej *could* sort things out well enough; I have no insight into whether they have (or someone else has) the time to actually do so. Did anyone ever step forward to do this? I'm a bit worried about the long-term viability of ctypes if we don't have a maintainer or at least someone making sure we are staying up-to-date with upstream libffi. my plan was to merge the next libffi release, something along the lines which is released with GCC 5. But this release isn't yet done. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Wed, Mar 11, 2015 at 8:20 PM Guido van Rossum gu...@python.org wrote: Can we please decouple the ctypes deprecation discussion from efforts to upgrade cffi? They can coexist just fine, and they don't even really solve the same problem. I mostly proposed deprecating ctypes because we were not keeping up with libffi upstream. If we solve the latter I'm not bothered enough to personally pursue the former. -Brett On Wed, Mar 11, 2015 at 3:20 PM, Brett Cannon br...@python.org wrote: On Wed, Mar 11, 2015 at 6:03 PM Paul Moore p.f.mo...@gmail.com wrote: On 11 March 2015 at 21:45, Maciej Fijalkowski fij...@gmail.com wrote: Is it possible to use cffi without a C compiler/headers as easily than ctypes? yes, it has two modes, one that does that and the other that does extra safety at the cost of a C compiler So if someone were to propose a practical approach to including cffi into the stdlib, *and* assisting the many Windows projects using ctypes for access to the Windows API [1], then there may be a reasonable argument for deprecating ctypes. But nobody seems to be doing that, rather the suggestion appears to be just to deprecate a widely used part of the stdlib offering no migration path :-( You're ignoring that it's not maintained, which is the entire reason I brought this up. No one seems to want to touch the code. Who knows what improvements, bugfixes, etc. exist upstream in libffi that we lack because no one wants to go through and figure it out. If someone would come forward and help maintain it then I have no issue with it sticking around. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
I started this message about 3 months ago; at this point I'm just getting it posted so it stops rotting in my Drafts folder. On Mon, Dec 22, 2014 at 3:49 PM, Jim J. Jewett jimjjew...@gmail.com wrote: On Thu, Dec 18, 2014, at 14:13, Maciej Fijalkowski wrote: ... http://bugs.python.org/issue23085 ... is there any reason any more for libffi being included in CPython? Paul Moore wrote: Probably the easiest way of moving this forward would be for someone to identify the CPython-specific patches in the current version ... Christian Heimes wrote: That's easy. All patches are tracked in the diff file https://hg.python.org/cpython/file/3de678cd184d/Modules/_ctypes/libffi.diff That (200+ lines) doesn't seem to have all the C changes, such as the win64 sizeof changes from issue 11835. I took a little bit of time to look at this a while ago. Something that quickly became clear that I'm not sure everybody is on the same page about (I, for one, had no idea previously) is that libffi for Windows and libffi for everything else are entirely separate beasts: the Windows build pulls all of its libffi source from Modules/_ctypes/libffi_msvc and doesn't even touch Modules/_ctypes/libffi (which is libffi for nearly everything else). Most of the files in libffi_msvc haven't been touched in about 4 years, except for a couple fixes by Steve Dower recently. README.ctypes in that folder states that the files were taken from the libffi CVS tree in 2004 and there have been some custom modifications since then, so there is really very little resemblance between libffi_msvc and current libffi. There's also 'libffi_arm_wince' and 'libffi_osx'. I would be fairly amazed if Python itself built on any version of Windows CE; personally I would be for ripping out any supposed support that we have for it (though if somebody is actually using it and it works, it's not hurting anything and should stay). That's a completely different issue, though. I'm all for ditching our 'libffi_msvc' in favor of adding libffi as another 'external' for the Windows build. I have managed to get _ctypes to build on Windows using vanilla libffi sources, prepared using their configure script from within Git Bash and built with our usual Windows build system (properly patched). Unfortunately, making things usable will take some work on ctypes itself, which I'm not qualified to do. I'm happy to pass on my procedure and patches for getting to the point of successful compilation to anyone who feels up to fixing the things that are broken. As for the lightly-patched version of libffi that we keep around for Linux et. al., looking at the diff file I don't see that there's a whole lot of reason to keep it around. -- Zach ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On 12 March 2015 at 17:26, Brett Cannon br...@python.org wrote: I'm all for ditching our 'libffi_msvc' in favor of adding libffi as another 'external' for the Windows build. I have managed to get _ctypes to build on Windows using vanilla libffi sources, prepared using their configure script from within Git Bash and built with our usual Windows build system (properly patched). Unfortunately, making things usable will take some work on ctypes itself, which I'm not qualified to do. I'm happy to pass on my procedure and patches for getting to the point of successful compilation to anyone who feels up to fixing the things that are broken. So it seems possible to use upstream libffi but will require some work. I'd be willing to contemplate helping out on the Windows side of things, if nobody else steps up (with the proviso that I have little free time, and I'm saying this without much idea of what's involved :-)) If Zachary can give a bit more detail on what the work on ctypes is, and/or put what he has somewhere that I could have a look at, that might help. Paul. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Thu, Mar 12, 2015 at 10:39 AM Zachary Ware zachary.ware+py...@gmail.com wrote: I started this message about 3 months ago; at this point I'm just getting it posted so it stops rotting in my Drafts folder. Thanks for looking into this! On Mon, Dec 22, 2014 at 3:49 PM, Jim J. Jewett jimjjew...@gmail.com wrote: On Thu, Dec 18, 2014, at 14:13, Maciej Fijalkowski wrote: ... http://bugs.python.org/issue23085 ... is there any reason any more for libffi being included in CPython? Paul Moore wrote: Probably the easiest way of moving this forward would be for someone to identify the CPython-specific patches in the current version ... Christian Heimes wrote: That's easy. All patches are tracked in the diff file https://hg.python.org/cpython/file/3de678cd184d/Modules/_ ctypes/libffi.diff That (200+ lines) doesn't seem to have all the C changes, such as the win64 sizeof changes from issue 11835. I took a little bit of time to look at this a while ago. Something that quickly became clear that I'm not sure everybody is on the same page about (I, for one, had no idea previously) is that libffi for Windows and libffi for everything else are entirely separate beasts: the Windows build pulls all of its libffi source from Modules/_ctypes/libffi_msvc and doesn't even touch Modules/_ctypes/libffi (which is libffi for nearly everything else). Most of the files in libffi_msvc haven't been touched in about 4 years, except for a couple fixes by Steve Dower recently. README.ctypes in that folder states that the files were taken from the libffi CVS tree in 2004 and there have been some custom modifications since then, so there is really very little resemblance between libffi_msvc and current libffi. That's what I was afraid of. There's also 'libffi_arm_wince' and 'libffi_osx'. I would be fairly amazed if Python itself built on any version of Windows CE; personally I would be for ripping out any supposed support that we have for it (though if somebody is actually using it and it works, it's not hurting anything and should stay). That's a completely different issue, though. I'm all for ditching our 'libffi_msvc' in favor of adding libffi as another 'external' for the Windows build. I have managed to get _ctypes to build on Windows using vanilla libffi sources, prepared using their configure script from within Git Bash and built with our usual Windows build system (properly patched). Unfortunately, making things usable will take some work on ctypes itself, which I'm not qualified to do. I'm happy to pass on my procedure and patches for getting to the point of successful compilation to anyone who feels up to fixing the things that are broken. So it seems possible to use upstream libffi but will require some work. As for the lightly-patched version of libffi that we keep around for Linux et. al., looking at the diff file I don't see that there's a whole lot of reason to keep it around. For UNIX OSs we could probably rely on the system libffi then. What's the situation on OS X? Anyone know if it has libffi, or would be need to be pulled in to be used like on Windows? ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Thu, Mar 12, 2015 at 8:35 PM, Ned Deily n...@acm.org wrote: In article CAP1=2w7cx5jpqv_pr61rqs1ubusjf5f6kg0cd-qcwr2+9ij...@mail.gmail.com, For UNIX OSs we could probably rely on the system libffi then. What's the situation on OS X? Anyone know if it has libffi, or would be need to be pulled in to be used like on Windows? Ronald (in http://bugs.python.org/issue23534): On OSX the internal copy of libffi that's used is based on the one in PyObjC, which in turn is based on the version of libffi on opensource.apple.com (IIRC with some small patches that fix minor issues found by the PyObjC testsuite). -- Ned Deily, n...@acm.org From pypy experience, libffi installed on OS X tends to just work (we never had any issues with those) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Thu, Mar 12, 2015 at 6:26 PM, Brett Cannon br...@python.org wrote: For UNIX OSs we could probably rely on the system libffi then. Yes, it's possible to use the system libffi -- that's what most linux distros already do -- but only if you use dynamic linking. If you want to statically link libffi (useful when embedding, making installations that are a little more portable across systems, or when you're a company like Google that wants to statically link all the things) it's a little more confusing. (The reason Brett started this thread is that I mentioned these issues in an internal Google meeting that he happened to be in ;-P) _ctypes currently contains a bunch of duplicate definitions of the ffi_type_* structs (in Modules/_ctypes/cfield.c.) It also contains a separate implementation of ffi_closure_alloc and ffi_closure_free (in Modules/_ctypes/malloc_closure.c) which is only used on OSX. I guess those redefinitions are to cope with (much) older libffis, but that's just a guess. The redefinitions aren't a problem when using dynamic linking, as long as the new definitions aren't incompatible, but if libffi were to change the ABI of the ffi_type structs, yuck. It can be a problem when statically linking, depending on how you link -- and the only reason it works with the bundled libffi is that ctypes *does not build libffi the normal way*. Instead it runs libffi's configure and then selectively builds a few source files from it, using distutils. It would be a really, really good idea to clean up that mess if we're not going to bundle libffi anymore :-P Screaming-and-stabbing-at-setup.py'ly y'rs, -- Thomas Wouters tho...@python.org Hi! I'm an email virus! Think twice before sending your email to help me spread! ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
In article CAP1=2w7cx5jpqv_pr61rqs1ubusjf5f6kg0cd-qcwr2+9ij...@mail.gmail.com, For UNIX OSs we could probably rely on the system libffi then. What's the situation on OS X? Anyone know if it has libffi, or would be need to be pulled in to be used like on Windows? Ronald (in http://bugs.python.org/issue23534): On OSX the internal copy of libffi that's used is based on the one in PyObjC, which in turn is based on the version of libffi on opensource.apple.com (IIRC with some small patches that fix minor issues found by the PyObjC testsuite). -- Ned Deily, n...@acm.org ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Thu, Mar 12, 2015 at 12:44 PM, Paul Moore p.f.mo...@gmail.com wrote: I'd be willing to contemplate helping out on the Windows side of things, if nobody else steps up (with the proviso that I have little free time, and I'm saying this without much idea of what's involved :-)) If Zachary can give a bit more detail on what the work on ctypes is, and/or put what he has somewhere that I could have a look at, that might help. I'll send you everything I've got next chance I have, but I don't know when that will be. -- Zach ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Wed, Mar 11, 2015 at 11:34 PM, Victor Stinner victor.stin...@gmail.com wrote: Le 11 mars 2015 18:29, Brett Cannon br...@python.org a écrit : I'm going to propose a somewhat controversial idea: let's deprecate the ctypes module. In the past I tried to deprecate many functions or modules because they are rarely or never used. Many developers prefered to keep them. By the way, I still want to remove plat-xxx modules like IN or CDROM :-) Getopt was deprecated when optparse was added to the stdlib. Then optparse was deprecated when argparse was added to the stdlib. Cython and cffi are not part of the stdlib and can be hard to install on some platforms. Ctypes is cool because it doesn't require C headers nor a C compiler. Is it possible to use cffi without a C compiler/headers as easily than ctypes? yes, it has two modes, one that does that and the other that does extra safety at the cost of a C compiler ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
Le 11 mars 2015 18:29, Brett Cannon br...@python.org a écrit : I'm going to propose a somewhat controversial idea: let's deprecate the ctypes module. In the past I tried to deprecate many functions or modules because they are rarely or never used. Many developers prefered to keep them. By the way, I still want to remove plat-xxx modules like IN or CDROM :-) Getopt was deprecated when optparse was added to the stdlib. Then optparse was deprecated when argparse was added to the stdlib. Cython and cffi are not part of the stdlib and can be hard to install on some platforms. Ctypes is cool because it doesn't require C headers nor a C compiler. Is it possible to use cffi without a C compiler/headers as easily than ctypes? If you want to move forward, you should help to integrate cffi into the stdlib. What's the status? Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Wed, Mar 11, 2015 at 8:31 PM, Wes Turner wes.tur...@gmail.com wrote: On Mar 11, 2015 12:55 PM, Maciej Fijalkowski fij...@gmail.com wrote: On Wed, Mar 11, 2015 at 7:50 PM, Antoine Pitrou solip...@pitrou.net wrote: On Wed, 11 Mar 2015 17:27:58 + Brett Cannon br...@python.org wrote: Did anyone ever step forward to do this? I'm a bit worried about the long-term viability of ctypes if we don't have a maintainer or at least someone making sure we are staying up-to-date with upstream libffi. The ctypes module is a dangerous thing, so having a chunk of C code that isn't being properly maintained seems to me to make it even more dangerous. Depends what you call dangerous. C code doesn't rot quicker than pure Python code :-) Also, libffi really offers a wrapper around platform ABIs, which rarely change. And yet, lesser known ABIs in libffi contain bugs (as we discovered trying to work there with anything else than x86 really). Also there *are* ABI differencies that change slowly over time (e.g. requiring stack to be 16 byte aligned) Are there tests for this? What do you mean? The usual failure mode is will segfault every now and again if the moon is in the right position (e.g. the stack alignment thing only happens if the underlaying function uses certain SSE instructions that compilers emit these days in certain circumstances) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Wed, Mar 11, 2015 at 8:05 PM, Antoine Pitrou solip...@pitrou.net wrote: On Wed, 11 Mar 2015 19:54:58 +0200 Maciej Fijalkowski fij...@gmail.com wrote: Depends what you call dangerous. C code doesn't rot quicker than pure Python code :-) Also, libffi really offers a wrapper around platform ABIs, which rarely change. And yet, lesser known ABIs in libffi contain bugs (as we discovered trying to work there with anything else than x86 really). Also there *are* ABI differencies that change slowly over time (e.g. requiring stack to be 16 byte aligned) Well, sure. The point is, such bugs are unlikely to appear at a fast rate... Also, I don't understand why libffi issues would affect cffi any less than it affects ctypes, at least in the compiler-less mode of operation. My point here was only about shipping own libffi vs using the system one (and it does affect cffi equally with or without compiler) We now have things like cffi and Cython for people who need to interface with C code. Both of those projects are maintained. And they are not overly difficult to work with. But they are not ctypes. For example, cffi wouldn't be obvious to use for interfacing with non-C code, since it requires you to write C-like declarations. You mean like Fortran? Or what precisely? Any toolchain that can generate native code. It can be Fortran, but it can also be code generated at runtime without there being any external declaration. Having to generate C declarations for such code would be a distraction. Of course, if cffi gains the same ability as ctypes (namely to lookup a function and declare its signature without going through the FFI.cdef() interface), that issue disappears. As a side note, ctypes has a large number of users, so even if it were deprecated that wouldn't be a good reason to stop maintaining it. And calling cffi simple while it relies on a parser of the C language (which would then have to be bundled with Python) is a bit misleading IMO. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Wed, 11 Mar 2015 19:54:58 +0200 Maciej Fijalkowski fij...@gmail.com wrote: Depends what you call dangerous. C code doesn't rot quicker than pure Python code :-) Also, libffi really offers a wrapper around platform ABIs, which rarely change. And yet, lesser known ABIs in libffi contain bugs (as we discovered trying to work there with anything else than x86 really). Also there *are* ABI differencies that change slowly over time (e.g. requiring stack to be 16 byte aligned) Well, sure. The point is, such bugs are unlikely to appear at a fast rate... Also, I don't understand why libffi issues would affect cffi any less than it affects ctypes, at least in the compiler-less mode of operation. We now have things like cffi and Cython for people who need to interface with C code. Both of those projects are maintained. And they are not overly difficult to work with. But they are not ctypes. For example, cffi wouldn't be obvious to use for interfacing with non-C code, since it requires you to write C-like declarations. You mean like Fortran? Or what precisely? Any toolchain that can generate native code. It can be Fortran, but it can also be code generated at runtime without there being any external declaration. Having to generate C declarations for such code would be a distraction. Of course, if cffi gains the same ability as ctypes (namely to lookup a function and declare its signature without going through the FFI.cdef() interface), that issue disappears. As a side note, ctypes has a large number of users, so even if it were deprecated that wouldn't be a good reason to stop maintaining it. And calling cffi simple while it relies on a parser of the C language (which would then have to be bundled with Python) is a bit misleading IMO. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On 11 March 2015 at 17:27, Brett Cannon br...@python.org wrote: I'm going to propose a somewhat controversial idea: let's deprecate the ctypes module. We now have things like cffi and Cython for people who need to interface with C code. Both of those projects are maintained. And they are not overly difficult to work with. All of this seems to match the reasons we added ctypes in the first place while also being safer to use than ctypes. -1. On Windows a huge amount of code uses ctypes to interface with the Windows API. Many of those projects transitioned from pywin32 because that is a big dependency, and a binary build. Not having in-core access to the Windows API would be a huge problem for many projects. Who would rewrite projects like pyreadline or colorama, for example? How would pip (which has to vendor everything and can't include binary dependencies because of its nature) locate the correct windows config file folders without ctypes in core? Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Wed, Mar 11, 2015 at 7:50 PM, Antoine Pitrou solip...@pitrou.net wrote: On Wed, 11 Mar 2015 17:27:58 + Brett Cannon br...@python.org wrote: Did anyone ever step forward to do this? I'm a bit worried about the long-term viability of ctypes if we don't have a maintainer or at least someone making sure we are staying up-to-date with upstream libffi. The ctypes module is a dangerous thing, so having a chunk of C code that isn't being properly maintained seems to me to make it even more dangerous. Depends what you call dangerous. C code doesn't rot quicker than pure Python code :-) Also, libffi really offers a wrapper around platform ABIs, which rarely change. And yet, lesser known ABIs in libffi contain bugs (as we discovered trying to work there with anything else than x86 really). Also there *are* ABI differencies that change slowly over time (e.g. requiring stack to be 16 byte aligned) I'm going to propose a somewhat controversial idea: let's deprecate the ctypes module. This is gratuitous. I'm +1 on deprecating ctypes We now have things like cffi and Cython for people who need to interface with C code. Both of those projects are maintained. And they are not overly difficult to work with. But they are not ctypes. For example, cffi wouldn't be obvious to use for interfacing with non-C code, since it requires you to write C-like declarations. You mean like Fortran? Or what precisely? I don't understand why cffi would be safer than ctypes. At least not in the operation mode where it doesn't need to invoke a C compiler. Cython is a completely different beast, it requires a separate compilation pass which makes it useless in some situations. Our main motivation for safer is comes with less magic and less gotchas, which also means does less stuff. It's also smaller. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Mar 11, 2015 12:55 PM, Maciej Fijalkowski fij...@gmail.com wrote: On Wed, Mar 11, 2015 at 7:50 PM, Antoine Pitrou solip...@pitrou.net wrote: On Wed, 11 Mar 2015 17:27:58 + Brett Cannon br...@python.org wrote: Did anyone ever step forward to do this? I'm a bit worried about the long-term viability of ctypes if we don't have a maintainer or at least someone making sure we are staying up-to-date with upstream libffi. The ctypes module is a dangerous thing, so having a chunk of C code that isn't being properly maintained seems to me to make it even more dangerous. Depends what you call dangerous. C code doesn't rot quicker than pure Python code :-) Also, libffi really offers a wrapper around platform ABIs, which rarely change. And yet, lesser known ABIs in libffi contain bugs (as we discovered trying to work there with anything else than x86 really). Also there *are* ABI differencies that change slowly over time (e.g. requiring stack to be 16 byte aligned) Are there tests for this? I'm going to propose a somewhat controversial idea: let's deprecate the ctypes module. This is gratuitous. I'm +1 on deprecating ctypes -1. These docs are helpful for understanding the pros and cons of different CPython - C interfaces. https://scipy-lectures.github.io/advanced/interfacing_with_c/interfacing_with_c.html (https://github.com/scipy-lectures/scipy-lecture-notes/issues/131 discusses documenting CFFI here as well) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Wed, 11 Mar 2015 17:27:58 + Brett Cannon br...@python.org wrote: Did anyone ever step forward to do this? I'm a bit worried about the long-term viability of ctypes if we don't have a maintainer or at least someone making sure we are staying up-to-date with upstream libffi. The ctypes module is a dangerous thing, so having a chunk of C code that isn't being properly maintained seems to me to make it even more dangerous. Depends what you call dangerous. C code doesn't rot quicker than pure Python code :-) Also, libffi really offers a wrapper around platform ABIs, which rarely change. I'm going to propose a somewhat controversial idea: let's deprecate the ctypes module. This is gratuitous. We now have things like cffi and Cython for people who need to interface with C code. Both of those projects are maintained. And they are not overly difficult to work with. But they are not ctypes. For example, cffi wouldn't be obvious to use for interfacing with non-C code, since it requires you to write C-like declarations. All of this seems to match the reasons we added ctypes in the first place while also being safer to use than ctypes. I don't understand why cffi would be safer than ctypes. At least not in the operation mode where it doesn't need to invoke a C compiler. Cython is a completely different beast, it requires a separate compilation pass which makes it useless in some situations. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Wed, Mar 11, 2015 at 6:03 PM Paul Moore p.f.mo...@gmail.com wrote: On 11 March 2015 at 21:45, Maciej Fijalkowski fij...@gmail.com wrote: Is it possible to use cffi without a C compiler/headers as easily than ctypes? yes, it has two modes, one that does that and the other that does extra safety at the cost of a C compiler So if someone were to propose a practical approach to including cffi into the stdlib, *and* assisting the many Windows projects using ctypes for access to the Windows API [1], then there may be a reasonable argument for deprecating ctypes. But nobody seems to be doing that, rather the suggestion appears to be just to deprecate a widely used part of the stdlib offering no migration path :-( You're ignoring that it's not maintained, which is the entire reason I brought this up. No one seems to want to touch the code. Who knows what improvements, bugfixes, etc. exist upstream in libffi that we lack because no one wants to go through and figure it out. If someone would come forward and help maintain it then I have no issue with it sticking around. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Wed, 11 Mar 2015 22:20:58 + Brett Cannon br...@python.org wrote: You're ignoring that it's not maintained, According to which metric/criterion? changeset: 94932:86c9ef950288 user:Steve Dower steve.do...@microsoft.com date:Tue Mar 10 09:56:38 2015 -0700 summary: Issue #23606: Disable ctypes.util.find_library(c) on Windows so tests are skipped while we figure out how best to approach the CRT change changeset: 94756:7c6e3358221a user:Serhiy Storchaka storch...@gmail.com date:Thu Feb 26 15:27:57 2015 +0200 summary: Silenced minor GCC warnings. changeset: 94653:a84ae2ccd220 user:Serhiy Storchaka storch...@gmail.com date:Mon Feb 16 20:52:17 2015 +0200 summary: Issue #23450: Fixed possible integer overflows. [etc.] Who knows what improvements, bugfixes, etc. exist upstream in libffi that we lack because no one wants to go through and figure it out. Well, who knows? How about you enlighten us about that? And why do you think cffi, which *also* requires libffi, would be better served in that regard? Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Thu, Mar 12, 2015 at 12:20 AM, Brett Cannon br...@python.org wrote: On Wed, Mar 11, 2015 at 6:03 PM Paul Moore p.f.mo...@gmail.com wrote: On 11 March 2015 at 21:45, Maciej Fijalkowski fij...@gmail.com wrote: Is it possible to use cffi without a C compiler/headers as easily than ctypes? yes, it has two modes, one that does that and the other that does extra safety at the cost of a C compiler So if someone were to propose a practical approach to including cffi into the stdlib, *and* assisting the many Windows projects using ctypes for access to the Windows API [1], then there may be a reasonable argument for deprecating ctypes. But nobody seems to be doing that, rather the suggestion appears to be just to deprecate a widely used part of the stdlib offering no migration path :-( You're ignoring that it's not maintained, which is the entire reason I brought this up. No one seems to want to touch the code. Who knows what improvements, bugfixes, etc. exist upstream in libffi that we lack because no one wants to go through and figure it out. If someone would come forward and help maintain it then I have no issue with it sticking around. It's a bit worse than that. Each time someone wants to touch the code (e.g. push back the upstream libffi fixes), there is we need to review it, but there is noone to do it, noone knows how it works, don't touch it kind of feedback, which leads to disincentives to potential maintainers. I would be likely willing to rip off the libffi from CPython as it is for example (and just use the upstream one) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Thu, Mar 12, 2015 at 12:31 AM, Antoine Pitrou solip...@pitrou.net wrote: On Wed, 11 Mar 2015 23:10:14 +0200 Maciej Fijalkowski fij...@gmail.com wrote: Well, sure. The point is, such bugs are unlikely to appear at a fast rate... Also, I don't understand why libffi issues would affect cffi any less than it affects ctypes, at least in the compiler-less mode of operation. My point here was only about shipping own libffi vs using the system one (and it does affect cffi equally with or without compiler) So what? If ctypes used the system libffi as cffi does, it would by construction be at least portable as cffi is. The only reason the bundled libffi was patched at some point was to be *more* portable than vanilla libffi is. So, really, I don't see how switching from ctypes to cffi solves any of this. You're missing my point. Ripping off the libffi from CPython is a good idea to start with. Maybe deprecating ctypes is *also* a good idea, but it's a separate discussion point. It certainly does not solve the libffi problem. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Wed, 11 Mar 2015 22:55:33 + Paul Moore p.f.mo...@gmail.com wrote: On 11 March 2015 at 22:33, Maciej Fijalkowski fij...@gmail.com wrote: You're missing my point. Ripping off the libffi from CPython is a good idea to start with. Maybe deprecating ctypes is *also* a good idea, but it's a separate discussion point. It certainly does not solve the libffi problem. OK, so let's focus on the libffi side of things and ignore deprecating or replacing ctypes. I guess I don't see a problem with a proof-of-concept patch to upgrade the libffi (obviously it's not possible to rely on a system libffi on Windows, but treating it as an external might work). If it passes all the tests on all platforms, maybe it could be considered. Agreed. We have enough tests and enough buildbots that it can be attempted. (besides, the only more-or-less officially supported platforms are Linux, Windows, OS X; we also try to not to be too broken on the BSDs, but that's it) Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On 11 March 2015 at 22:33, Maciej Fijalkowski fij...@gmail.com wrote: You're missing my point. Ripping off the libffi from CPython is a good idea to start with. Maybe deprecating ctypes is *also* a good idea, but it's a separate discussion point. It certainly does not solve the libffi problem. OK, so let's focus on the libffi side of things and ignore deprecating or replacing ctypes. I guess I don't see a problem with a proof-of-concept patch to upgrade the libffi (obviously it's not possible to rely on a system libffi on Windows, but treating it as an external might work). If it passes all the tests on all platforms, maybe it could be considered. I don't see how anyone can say yes, we'll do it without seeing evidence that it'll work. But equally, I don't think there's any reason to say it absolutely wouldn't be accepted regardless of evidence. So why not prepare a patch for 3.6 (I assume it's too late to make such a big change for 3.5) and we'll see what happens. Or even better, prepare a test build of 3.5 or even 3.4 that switches libffi - people can replace an existing install (I'd be willing to) and test it in live situations. But unless someone provides a patch, the status quo will win. At least until an actual bug that affects live code forces the issue. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Wed, 11 Mar 2015 23:10:14 +0200 Maciej Fijalkowski fij...@gmail.com wrote: Well, sure. The point is, such bugs are unlikely to appear at a fast rate... Also, I don't understand why libffi issues would affect cffi any less than it affects ctypes, at least in the compiler-less mode of operation. My point here was only about shipping own libffi vs using the system one (and it does affect cffi equally with or without compiler) So what? If ctypes used the system libffi as cffi does, it would by construction be at least portable as cffi is. The only reason the bundled libffi was patched at some point was to be *more* portable than vanilla libffi is. So, really, I don't see how switching from ctypes to cffi solves any of this. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Thu, Dec 18, 2014, at 14:13, Maciej Fijalkowski wrote: ... http://bugs.python.org/issue23085 ... is there any reason any more for libffi being included in CPython? Paul Moore wrote: Probably the easiest way of moving this forward would be for someone to identify the CPython-specific patches in the current version ... Christian Heimes wrote: That's easy. All patches are tracked in the diff file https://hg.python.org/cpython/file/3de678cd184d/Modules/_ctypes/libffi.diff That (200+ lines) doesn't seem to have all the C changes, such as the win64 sizeof changes from issue 11835. Besides http://bugs.python.org/issue23085, there is at least http://bugs.python.org/issue22733 http://bugs.python.org/issue20160 http://bugs.python.org/issue11835 which sort of drives home the point that making sure we have a good merge isn't trivial, and this isn't an area where we should just assume that tests will catch everything. I don't think it is just a quicky waiting on permission. I've no doubt that upstream libffi is better in many ways, but those are ways people have already learned to live with. That said, I haven't seen any objections in principle, except perhaps from Steve Dower in the issues. (I *think* he was just saying not worth the time to me, but it was ambiguous.) I do believe that Christian or Maciej *could* sort things out well enough; I have no insight into whether they have (or someone else has) the time to actually do so. -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Thu, Dec 18, 2014 at 10:36 PM, Jim J. Jewett jimjjew...@gmail.com wrote: On Thu, Dec 18, 2014, at 14:13, Maciej Fijalkowski wrote: ... http://bugs.python.org/issue23085 ... is there any reason any more for libffi being included in CPython? [And why a fork, instead of just treating it as an external dependency] Benjamin Peterson responded: It has some sort of Windows related patches. No one seems to know whether they're still needed for newer libffi. Unfortunately, ctypes doesn't currently have a maintainer. Are any of the following false? (1) Ideally, we would treat it as an external dependency. (2) At one point, it was intentionally forked to get in needed patches, including at least some for 64 bit windows with MSVC. (3) Upstream libffi maintenance has picked back up. (4) Alas, that means the switch merge would not be trivial. (5) In theory, we could now switch to the external version. [In particular, does libffi have a release policy such that we could assume the newest released version is safe, so long as our integration doesn't break?] (6) By its very nature, libffi changes are risky and undertested. At the moment, that is also true of its primary user, ctypes. (7) So a switch is OK in theory, but someone has to do the non-trivial testing and merging, and agree to support both libffi and and ctypes in the future. Otherwise, stable wins. (8) The need for future support makes this a bad candidate for patches wanted/bug bounty/GSoC. -jJ I would like to add that not doing anything is not a good strategy either, because you accumulate bugs that get fixed upstream (I'm pretty sure all the problems from cpython got fixed in upstream libffi, but not all libffi fixes made it to cpython). ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On 19 December 2014 at 08:26, Maciej Fijalkowski fij...@gmail.com wrote: I would like to add that not doing anything is not a good strategy either, because you accumulate bugs that get fixed upstream (I'm pretty sure all the problems from cpython got fixed in upstream libffi, but not all libffi fixes made it to cpython). Probably the easiest way of moving this forward would be for someone to identify the CPython-specific patches in the current version, and check if they are addressed in the latest libffi version. They haven't been applied as they are, I gather, but maybe equivalent fixes have been made. I've no idea how easy that would be (presumably not trivial, or someone would already have done it). If the patches aren't needed any more, upgrading becomes a lot more plausible. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On 19.12.2014 10:52, Paul Moore wrote: Probably the easiest way of moving this forward would be for someone to identify the CPython-specific patches in the current version, and check if they are addressed in the latest libffi version. They haven't been applied as they are, I gather, but maybe equivalent fixes have been made. I've no idea how easy that would be (presumably not trivial, or someone would already have done it). If the patches aren't needed any more, upgrading becomes a lot more plausible. That's easy. All patches are tracked in the diff file https://hg.python.org/cpython/file/3de678cd184d/Modules/_ctypes/libffi.diff . The file *should* be up to date. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Fri, 19 Dec 2014 09:52:26 + Paul Moore p.f.mo...@gmail.com wrote: On 19 December 2014 at 08:26, Maciej Fijalkowski fij...@gmail.com wrote: I would like to add that not doing anything is not a good strategy either, because you accumulate bugs that get fixed upstream (I'm pretty sure all the problems from cpython got fixed in upstream libffi, but not all libffi fixes made it to cpython). Probably the easiest way of moving this forward would be for someone to identify the CPython-specific patches in the current version, and check if they are addressed in the latest libffi version. They haven't been applied as they are, I gather, but maybe equivalent fixes have been made. I've no idea how easy that would be (presumably not trivial, or someone would already have done it). If the patches aren't needed any more, upgrading becomes a lot more plausible. Another strategy is to dump our private fork, link with upstream instead, and see what breaks. Presumably, our test suite should be able to catch some (most?) of that breakage. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On 19 December 2014 at 23:01, Antoine Pitrou solip...@pitrou.net wrote: On Fri, 19 Dec 2014 09:52:26 + Paul Moore p.f.mo...@gmail.com wrote: On 19 December 2014 at 08:26, Maciej Fijalkowski fij...@gmail.com wrote: I would like to add that not doing anything is not a good strategy either, because you accumulate bugs that get fixed upstream (I'm pretty sure all the problems from cpython got fixed in upstream libffi, but not all libffi fixes made it to cpython). Probably the easiest way of moving this forward would be for someone to identify the CPython-specific patches in the current version, and check if they are addressed in the latest libffi version. They haven't been applied as they are, I gather, but maybe equivalent fixes have been made. I've no idea how easy that would be (presumably not trivial, or someone would already have done it). If the patches aren't needed any more, upgrading becomes a lot more plausible. Another strategy is to dump our private fork, link with upstream instead, and see what breaks. Presumably, our test suite should be able to catch some (most?) of that breakage. And if we're going to do something like that for 3.5, now's the time, since we still have a lot of lead time on the 3.5 release. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] libffi embedded in CPython
After reading this http://bugs.python.org/issue23085 and remembering struggling having our own patches into cpython's libffi (but not into libffi itself), I wonder, is there any reason any more for libffi being included in CPython? Cheers, fijal ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
Maciej Fijalkowski wrote: After reading this http://bugs.python.org/issue23085 and remembering struggling having our own patches into cpython's libffi (but not into libffi itself), I wonder, is there any reason any more for libffi being included in CPython? We use it for ctypes, so there's certainly still a need. Are you asking whether we need a fork of it as opposed to treating it like an external (like OpenSSL)? Cheers, fijal ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Thu, Dec 18, 2014 at 9:17 PM, Steve Dower steve.do...@microsoft.com wrote: Maciej Fijalkowski wrote: After reading this http://bugs.python.org/issue23085 and remembering struggling having our own patches into cpython's libffi (but not into libffi itself), I wonder, is there any reason any more for libffi being included in CPython? We use it for ctypes, so there's certainly still a need. Are you asking whether we need a fork of it as opposed to treating it like an external (like OpenSSL)? yes (why is there a copy of libffi in the cpython source). And I'm asking not why it landed there, but why it is still there ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Thu, Dec 18, 2014, at 14:13, Maciej Fijalkowski wrote: After reading this http://bugs.python.org/issue23085 and remembering struggling having our own patches into cpython's libffi (but not into libffi itself), I wonder, is there any reason any more for libffi being included in CPython? It has some sort of Windows related patches. No one seems to know whether they're still needed for newer libffi. Unfortunately, ctypes doesn't currently have a maintainer. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
well, the problem is essentially that libffi gets patched (e.g. for ARM) and it does not make it's way to CPython quickly. This is unlikely to be a security issue (for a variety of reasons, including ctypes), but it's still an issue I think. Segfaults related to e.g. stack alignment are hard to debug On Thu, Dec 18, 2014 at 9:30 PM, Benjamin Peterson benja...@python.org wrote: On Thu, Dec 18, 2014, at 14:13, Maciej Fijalkowski wrote: After reading this http://bugs.python.org/issue23085 and remembering struggling having our own patches into cpython's libffi (but not into libffi itself), I wonder, is there any reason any more for libffi being included in CPython? It has some sort of Windows related patches. No one seems to know whether they're still needed for newer libffi. Unfortunately, ctypes doesn't currently have a maintainer. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Thu, Dec 18, 2014, at 14:50, Maciej Fijalkowski wrote: well, the problem is essentially that libffi gets patched (e.g. for ARM) and it does not make it's way to CPython quickly. This is unlikely to be a security issue (for a variety of reasons, including ctypes), but it's still an issue I think. Segfaults related to e.g. stack alignment are hard to debug Certainly it's a suboptimal situation, but resolving it requires someone to figure out whether we still need/want whatever patches are in there. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Thu, Dec 18, 2014, at 14:50, Maciej Fijalkowski wrote: well, the problem is essentially that libffi gets patched (e.g. for ARM) and it does not make it's way to CPython quickly. This is unlikely to be a security issue (for a variety of reasons, including ctypes), but it's still an issue I think. Segfaults related to e.g. stack alignment are hard to debug Certainly it's a suboptimal situation, but resolving it requires someone to figure out whether we still need/want whatever patches are in there. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Thu, Dec 18, 2014, at 14:13, Maciej Fijalkowski wrote: ... http://bugs.python.org/issue23085 ... is there any reason any more for libffi being included in CPython? [And why a fork, instead of just treating it as an external dependency] Benjamin Peterson responded: It has some sort of Windows related patches. No one seems to know whether they're still needed for newer libffi. Unfortunately, ctypes doesn't currently have a maintainer. Are any of the following false? (1) Ideally, we would treat it as an external dependency. (2) At one point, it was intentionally forked to get in needed patches, including at least some for 64 bit windows with MSVC. (3) Upstream libffi maintenance has picked back up. (4) Alas, that means the switch merge would not be trivial. (5) In theory, we could now switch to the external version. [In particular, does libffi have a release policy such that we could assume the newest released version is safe, so long as our integration doesn't break?] (6) By its very nature, libffi changes are risky and undertested. At the moment, that is also true of its primary user, ctypes. (7) So a switch is OK in theory, but someone has to do the non-trivial testing and merging, and agree to support both libffi and and ctypes in the future. Otherwise, stable wins. (8) The need for future support makes this a bad candidate for patches wanted/bug bounty/GSoC. -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Thu, Dec 18, 2014, at 15:36, Jim J. Jewett wrote: On Thu, Dec 18, 2014, at 14:13, Maciej Fijalkowski wrote: ... http://bugs.python.org/issue23085 ... is there any reason any more for libffi being included in CPython? [And why a fork, instead of just treating it as an external dependency] Benjamin Peterson responded: It has some sort of Windows related patches. No one seems to know whether they're still needed for newer libffi. Unfortunately, ctypes doesn't currently have a maintainer. Are any of the following false? (1) Ideally, we would treat it as an external dependency. (2) At one point, it was intentionally forked to get in needed patches, including at least some for 64 bit windows with MSVC. (3) Upstream libffi maintenance has picked back up. (4) Alas, that means the switch merge would not be trivial. (5) In theory, we could now switch to the external version. [In particular, does libffi have a release policy such that we could assume the newest released version is safe, so long as our integration doesn't break?] (6) By its very nature, libffi changes are risky and undertested. At the moment, that is also true of its primary user, ctypes. (7) So a switch is OK in theory, but someone has to do the non-trivial testing and merging, and agree to support both libffi and and ctypes in the future. Otherwise, stable wins. (8) The need for future support makes this a bad candidate for patches wanted/bug bounty/GSoC. Sounds about right. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
