Hi David, On Wed, Jan 27, 2010 at 8:37 AM, David Malcolm <dmalc...@redhat.com> wrote: [snip] > As a downstream distributor of Python, a major pain point for me is when > Python embeds a copy of a library's source code, rather than linking > against a system library (zlib, libffi and expat spring to mind): if > bugs (e.g. security issues) arise in a library, I have to go chasing > down all of the embedded copies of the library, rather than having > dynamic linking deal with it for me. > > So I have some concerns about having a copy of LLVM embedded in Python's > source tree, which I believe other distributors of Python would echo; > our rough preference ordering is: > > dynamic linking > static linking > source code copy > > I would like CPython to be faster, and if it means dynamically linking > against the system LLVM, that's probably OK (though I have some C++ > concerns already discussed elsewhere in this thread). If it means > statically linking, or worse, having a separate copy of the LLVM source > as an implementation detail of CPython, that would be painful.
We absolutely do not want CPython to include a copy of LLVM in its source tree. Unladen Swallow has done this to make it easier to pick up changes to LLVM's codebase as we make them, but this is not a viable model for CPython's long-term development. As mentioned in http://www.python.org/dev/peps/pep-3146/#managing-llvm-releases-c-api-changes, one of our full-time engineers is tasked with fixing all critical issues in LLVM before LLVM's 2.7 release so that CPython can simply use that release. We are currently statically linking against LLVM because that's what LLVM best supports, but that's not set in stone. We can make LLVM better support shared linking; it's one of the open issues as to whether this is important to python-dev (http://www.python.org/dev/peps/pep-3146/#open-issues), and it sounds like the answer is "Yes". Our priorities will adjust accordingly. To answer the individual bullet points: > I see that the u-s developers have been run into issues in LLVM itself, > and fixed them (bravo!), and seem to have done a good job of sending > those fixes back to LLVM for inclusion. [1] > > Some questions for the U-S devs: > - will it be possible to dynamically link against the system LLVM? > (the PEP currently seems to speak of statically linking against it) We currently link statically, but we will fix LLVM to better support shared linking. > - does the PEP anticipate that the Python source tree will start > embedding a copy of the LLVM source tree? No, that would be a terrible idea, and we do not endorse it. > - to what extent do you anticipate further changes needed in LLVM for > U-S? (given the work you've already put in, I expect the answer is > "probably a lot, but we can't know what those will be yet") We have fixed all the critical issues that our testing has exposed and have now moved on to "nice to have" items that will aid future optimizations. LLVM 2.7 will probably be released in May, so we still have time to fix LLVM as needed. > - do you anticipate all of these changes being accepted by the > upstream LLVM maintainers? Three of the Unladen Swallow committers are also LLVM committers. Our patches have historically been accepted enthusiastically by the LLVM maintainers, and we believe this warm relationship will continue. > - to what extent would these changes be likely to break API and/or ABI > compat with other users of LLVM (i.e. would a downstream distributor of > CPython be able to simply apply the necessary patches to the system LLVM > in order to track? if they did so, would it require a recompilation of > all of the other users of the system LLVM?) As mentioned in http://www.python.org/dev/peps/pep-3146/#managing-llvm-releases-c-api-changes, every LLVM release introduces incompatibilities to the C++ API. Our experience has been that these API changes are easily remedied. We recommend that CPython depend on a single version of LLVM and not try to track LLVM trunk. > - if Python needed to make a dot-release of LLVM, would LLVM allow > Python to increment the SONAME version identifying the ABI within the > DSO (".so") files, and guarantee not to reuse that SONAME version? (so > that automated ABI dependency tracking in e.g. RPM can identify the ABI > incompatibilities without being stomped on by a future upstream LLVM > release) Nick Lewycky is better-positioned to answer that. > - is your aim to minimize the difference between upstream LLVM and the > U-S copy of LLVM? ("yes", probably) > - will you publish/track the differences between upstream LLVM and the > U-S copy of LLVM somewhere? I see that you currently do this here: [2], > though it strikes me that the canonical representation of the LLVM > you're using is in the embedded copy in Utils/llvm, rather than e.g. a > recipe like "grab upstream release foo and apply this short list of > patches" (a distributed SCM might help here, or a tool like jhbuild [3]) We generally use LLVM trunk verbatim, but will sometimes cherrypick upstream patches that we want (since doing the full vendor merge can take a while). Right now, we're using an unmodified snapshot of LLVM. I've added language to the PEP to clarify some of these points: - No in-tree copies of LLVM/Clang: http://codereview.appspot.com/186247/diff2/5004:5006/5007 - Shared linking of LLVM: http://codereview.appspot.com/186247/diff2/5006:6007/5008 I've filed http://code.google.com/p/unladen-swallow/issues/detail?id=130 so that we have our own issue to track this, in addition to the upstream LLVM bug (http://llvm.org/PR3201). Thanks, Collin Winter _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
