Re: LuaJIT - an alternative for current Python C bindings
On 11/15/2012 07:44 PM, Alek Paunov wrote: LuaJIT is currently not available for our secondary architectures, so this would need fixing first. (not available meaning exactly that, there is no fallback to a slow C interpreter mode or something like that.) Oh, I completely forgot the secondary architectures (besides ARM). Florian, you follow LuaJIT - what other architectures (except the clearly missing s390) should be covered ?. ppc64 isn't supported, either. I am commenting on the possibility for deploying LuaJIT FFI as a partial replacement (problematic + future) of the hand written or SWIG Python bindings in the first place, not switching the system language from Python to Lua. In a system tool/daemon written in Python: * Python loads libluajit.so (import lupa; from lupa import LuaRuntime; ...) And this will reduce memory overhead? I doubt it. Every C object will be wrapped in a high-level Python object, a low-level Python object (provided by lupa), a Lua registry slot (used by lupa), and an FFI wrapper (provided by LuaJIT). Correctness requires that the GIL (or some other lock) is held while C functions are called through the FFI because LuaJIT is not reentrant. There are some open issues, too. LuaJIT hasn't yet found a way to handle C preprocessor defines (which are often used to provide constants). There is not attempt at all to deal with conflicting declarations. If two modules need definitions for time_t or struct timespec, one of them will fail with an attempt to redefine error. (Don't get me wrong, LuaJIT is a great piece of work.) -- Florian Weimer / Red Hat Product Security Team -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: LuaJIT - an alternative for current Python C bindings
On 11/15/2012 01:49 AM, Alek Paunov wrote: So, to me it seems natural joining all above together to start thinking for replacing the classic python C bindings with thin textual or bytecode(*) LuaJIT/FFI shims in benefit of things with great set of dependencies like Anaconda (thanks to lupa, Lua objects behaves like Python ones in Python). The result will be order of magnitude easier for maintenance (towards the APIs evolution) and faster code. LuaJIT is currently not available for our secondary architectures, so this would need fixing first. (not available meaning exactly that, there is no fallback to a slow C interpreter mode or something like that.) It's also not clear how significant the actual memory savings will be because some of the Lua OO frameworks have quite a bit of overhead. And the eventually arriving abrt plugin will bring some weight with it, too. This is nothing against Lua or LuaJIT. I just don't think that switching programming languages will magically reduce resource consumption. This has to be an explicit design goal, or else it is not likely to happen. I'm sure it's possible to write resource-conserving Python code, too. It's just the baseline overhead from the run-time system which is unavoidable, but that's fairly small for Python (apparently less than 2 MB of unshared RSS per process). -- Florian Weimer / Red Hat Product Security Team -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: LuaJIT - an alternative for current Python C bindings
On Thu, Nov 15, 2012 at 12:18 PM, Florian Weimer fwei...@redhat.com wrote: On 11/15/2012 01:49 AM, Alek Paunov wrote: So, to me it seems natural joining all above together to start thinking for replacing the classic python C bindings with thin textual or bytecode(*) LuaJIT/FFI shims in benefit of things with great set of dependencies like Anaconda (thanks to lupa, Lua objects behaves like Python ones in Python). The result will be order of magnitude easier for maintenance (towards the APIs evolution) and faster code. LuaJIT is currently not available for our secondary architectures, so this would need fixing first. (not available meaning exactly that, there is no fallback to a slow C interpreter mode or something like that.) It's also not clear how significant the actual memory savings will be because some of the Lua OO frameworks have quite a bit of overhead. And the eventually arriving abrt plugin will bring some weight with it, too. This is nothing against Lua or LuaJIT. I just don't think that switching programming languages will magically reduce resource consumption. This has to be an explicit design goal, or else it is not likely to happen. I'm sure it's possible to write resource-conserving Python code, too. It's just the baseline overhead from the run-time system which is unavoidable, but that's fairly small for Python (apparently less than 2 MB of unshared RSS per process). Porting yum to python3 might be an easier task ... from the python 3.3 changelog [1]: The storage of Unicode strings now depends on the highest codepoint in the string: pure ASCII and Latin1 strings (U+-U+00FF) use 1 byte per codepoint; BMP strings (U+-U+) use 2 bytes per codepoint; non-BMP strings (U+1-U+10) use 4 bytes per codepoint. The net effect is that for most applications, memory usage of string storage should decrease significantly - especially compared to former wide unicode builds - as, in many cases, strings will be pure ASCII even in international contexts (because many strings store non-human language data, such as XML fragments, HTTP headers, JSON-encoded data, etc.). We also hope that it will, for the same reasons, increase CPU cache efficiency on non-trivial applications. The memory usage of Python 3.3 is two to three times smaller than Python 3.2, and a little bit better than Python 2.7, on a Django benchmark (see the PEP for details). [1] http://docs.python.org/3/whatsnew/3.3.html -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: LuaJIT - an alternative for current Python C bindings
On Thu, Nov 15, 2012 at 2:18 PM, drago01 drag...@gmail.com wrote: On Thu, Nov 15, 2012 at 12:18 PM, Florian Weimer fwei...@redhat.com wrote: On 11/15/2012 01:49 AM, Alek Paunov wrote: So, to me it seems natural joining all above together to start thinking for replacing the classic python C bindings with thin textual or bytecode(*) LuaJIT/FFI shims in benefit of things with great set of dependencies like Anaconda (thanks to lupa, Lua objects behaves like Python ones in Python). The result will be order of magnitude easier for maintenance (towards the APIs evolution) and faster code. LuaJIT is currently not available for our secondary architectures, so this would need fixing first. (not available meaning exactly that, there is no fallback to a slow C interpreter mode or something like that.) It's also not clear how significant the actual memory savings will be because some of the Lua OO frameworks have quite a bit of overhead. And the eventually arriving abrt plugin will bring some weight with it, too. This is nothing against Lua or LuaJIT. I just don't think that switching programming languages will magically reduce resource consumption. This has to be an explicit design goal, or else it is not likely to happen. I'm sure it's possible to write resource-conserving Python code, too. It's just the baseline overhead from the run-time system which is unavoidable, but that's fairly small for Python (apparently less than 2 MB of unshared RSS per process). Porting yum to python3 might be an easier task ... from the python 3.3 changelog [1]: The storage of Unicode strings now depends on the highest codepoint in the string: pure ASCII and Latin1 strings (U+-U+00FF) use 1 byte per codepoint; BMP strings (U+-U+) use 2 bytes per codepoint; non-BMP strings (U+1-U+10) use 4 bytes per codepoint. The net effect is that for most applications, memory usage of string storage should decrease significantly - especially compared to former wide unicode builds - as, in many cases, strings will be pure ASCII even in international contexts (because many strings store non-human language data, such as XML fragments, HTTP headers, JSON-encoded data, etc.). We also hope that it will, for the same reasons, increase CPU cache efficiency on non-trivial applications. The memory usage of Python 3.3 is two to three times smaller than Python 3.2, and a little bit better than Python 2.7, on a Django benchmark (see the PEP for details). [1] http://docs.python.org/3/whatsnew/3.3.html Oh seems like the large savings are just compared to python 3.2 ... so would not really help. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: LuaJIT - an alternative for current Python C bindings
Hi Florian, On 15.11.2012 13:18, Florian Weimer wrote: On 11/15/2012 01:49 AM, Alek Paunov wrote: So, to me it seems natural joining all above together to start thinking for replacing the classic python C bindings with thin textual or bytecode(*) LuaJIT/FFI shims in benefit of things with great set of dependencies like Anaconda (thanks to lupa, Lua objects behaves like Python ones in Python). The result will be order of magnitude easier for maintenance (towards the APIs evolution) and faster code. LuaJIT is currently not available for our secondary architectures, so this would need fixing first. (not available meaning exactly that, there is no fallback to a slow C interpreter mode or something like that.) Oh, I completely forgot the secondary architectures (besides ARM). Florian, you follow LuaJIT - what other architectures (except the clearly missing s390) should be covered ?. It's also not clear how significant the actual memory savings will be because some of the Lua OO frameworks have quite a bit of overhead. And the eventually arriving abrt plugin will bring some weight with it, too. This is nothing against Lua or LuaJIT. I just don't think that switching programming languages will magically reduce resource consumption. This has to be an explicit design goal, or else it is not likely to happen. I'm sure it's possible to write resource-conserving Python code, too. It's just the baseline overhead from the run-time system which is unavoidable, but that's fairly small for Python (apparently less than 2 MB of unshared RSS per process). Apparently, my pseudo-English is not enough for a normal communication :-(. I am commenting on the possibility for deploying LuaJIT FFI as a partial replacement (problematic + future) of the hand written or SWIG Python bindings in the first place, not switching the system language from Python to Lua. In a system tool/daemon written in Python: * Python loads libluajit.so (import lupa; from lupa import LuaRuntime; ...) * The C libraries (the targets of the binding) are described as LuaJIT FFI style bindings in simplified C [my primary point for that is correctness (avoiding the memory leaks and reference counting problems) and lowering the burden of the APIs evolution maintenance, not lowering the general memory footprint] * The current OO style Python wrappers (the higher level part of the classical C bindings) are implemented with the Lua metamethods over plain C objects (without any Lua OO frameworks) * Python calls the LuaJIT C lib object wrappers (as close as possible to the Python APIs exposed by the current bindings on top of the Python C API). Kind Regards, Alek -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
LuaJIT - an alternative for current Python C bindings (was: Re: raising warning flag on firewalld-default feature)
On 12.11.2012 21:34, Steve Grubb wrote: But the problem I see is a lot of libraries are wrapped by swig, which leaks memory like a sieve. If swig didn't generate such leaky code, Python based daemons wouldn't be as scary. IMHO, Python is one of the best ways to express management logic. As have stated elsewhere in the thread, from the footprint perspective, python-libs can be decomposed to something including much smaller python-libs-core. The remaining problem is the quality of part of the C bindings. Otherwise, it is great advantage for Fedora that huge part of the system tools are already polished in Python (my own experience is that the C code in many tools/libs shipped with Fedora is of relative poor quality, as already Miloslav explained). David Malcolm already has some real results addressing the Python bindings issues trough static analysis based on his great gcc-python-plugin/libcpychecker work. In the future, this machinery can be used for bug free bindings generation too. Another option for addressing this is ... to just get rid of part of them in their current form. LuaJIT 2.0 is a reimplementation of Lua scripting language, which has build the reputation of one of the fastest compilers (comparable only with gcc) in the course of his 3 years! beta cycle (2.0 final release was announced early this week) The size of luajit.so (interpreter + trace compiler + C parser for the FFI definitions) is 452088 bytes (on x86_64, for comparison sqlite3.so, which is a python requirement is 677672) Unlike other FFI implementations LuaJIT FFI works compiling the functions calls/C structs access to native code. Another difference is that the foreign C definitions are presented in native C syntax: http://luajit.org/ext_ffi_tutorial.html#zlib On top of this kind of FFI, LuaJIT already have bunch of library bindings - as instance the kernel ABI wrapper: https://github.com/justincormack/ljsyscall For a long time already Fedora ships lunatic-python (loading and bridging the Lua interpreter as python module) More recent and feature complete bridge is: https://github.com/scoder/lupa from the author of lxml module. So, to me it seems natural joining all above together to start thinking for replacing the classic python C bindings with thin textual or bytecode(*) LuaJIT/FFI shims in benefit of things with great set of dependencies like Anaconda (thanks to lupa, Lua objects behaves like Python ones in Python). The result will be order of magnitude easier for maintenance (towards the APIs evolution) and faster code. Furthermore, we can get LuaJIT for free, because it is full API/ABI compatible with current Fedora (PUC Rio) Lua 5.1, which is pulled by RPM anyway - we can just replace Lua with LuaJIT as RPM provider for F20. Kind Regards, Alek (*) Unlike original Lua, LuaJIT bytecode is portable across architectures. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: LuaJIT - an alternative for current Python C bindings (was: Re: raising warning flag on firewalld-default feature)
How much Python code are you proposing someone ports to Lua? ;-) On Wed, Nov 14, 2012 at 4:49 PM, Alek Paunov a...@declera.com wrote: On 12.11.2012 21:34, Steve Grubb wrote: But the problem I see is a lot of libraries are wrapped by swig, which leaks memory like a sieve. If swig didn't generate such leaky code, Python based daemons wouldn't be as scary. IMHO, Python is one of the best ways to express management logic. As have stated elsewhere in the thread, from the footprint perspective, python-libs can be decomposed to something including much smaller python-libs-core. The remaining problem is the quality of part of the C bindings. Otherwise, it is great advantage for Fedora that huge part of the system tools are already polished in Python (my own experience is that the C code in many tools/libs shipped with Fedora is of relative poor quality, as already Miloslav explained). David Malcolm already has some real results addressing the Python bindings issues trough static analysis based on his great gcc-python-plugin/libcpychecker work. In the future, this machinery can be used for bug free bindings generation too. Another option for addressing this is ... to just get rid of part of them in their current form. LuaJIT 2.0 is a reimplementation of Lua scripting language, which has build the reputation of one of the fastest compilers (comparable only with gcc) in the course of his 3 years! beta cycle (2.0 final release was announced early this week) The size of luajit.so (interpreter + trace compiler + C parser for the FFI definitions) is 452088 bytes (on x86_64, for comparison sqlite3.so, which is a python requirement is 677672) Unlike other FFI implementations LuaJIT FFI works compiling the functions calls/C structs access to native code. Another difference is that the foreign C definitions are presented in native C syntax: http://luajit.org/ext_ffi_tutorial.html#zlib On top of this kind of FFI, LuaJIT already have bunch of library bindings - as instance the kernel ABI wrapper: https://github.com/justincormack/ljsyscall For a long time already Fedora ships lunatic-python (loading and bridging the Lua interpreter as python module) More recent and feature complete bridge is: https://github.com/scoder/lupa from the author of lxml module. So, to me it seems natural joining all above together to start thinking for replacing the classic python C bindings with thin textual or bytecode(*) LuaJIT/FFI shims in benefit of things with great set of dependencies like Anaconda (thanks to lupa, Lua objects behaves like Python ones in Python). The result will be order of magnitude easier for maintenance (towards the APIs evolution) and faster code. Furthermore, we can get LuaJIT for free, because it is full API/ABI compatible with current Fedora (PUC Rio) Lua 5.1, which is pulled by RPM anyway - we can just replace Lua with LuaJIT as RPM provider for F20. Kind Regards, Alek (*) Unlike original Lua, LuaJIT bytecode is portable across architectures. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel -- Twitter: http://twitter.com/znmeb; Computational Journalism Publishers Workbench: http://znmeb.github.com/Computational-Journalism-Publishers-Workbench/ How the Hell can the lion sleep with all those people singing A weem oh way! at the top of their lungs? -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: LuaJIT - an alternative for current Python C bindings
On 15.11.2012 04:51, M. Edward (Ed) Borasky wrote: How much Python code are you proposing someone ports to Lua? ;-) I am proposing mostly porting C code to Lua/FFI (the code of the problematic Python C bindings) -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
