Paul Moore <p.f.mo...@gmail.com> added the comment:
On Sun, 5 Aug 2018 at 06:51, Stefan Behnel <rep...@bugs.python.org> wrote: > This whole idea looks backwards and complicated. As Brett noted, .pyc files > were moved out of the source tree, because they are build time artifacts and > not sources. With that analogy, it's the .so files that would have to move, > not the .pyx or .py source files, which are sources for the compiled .so > files. I disagree. In *Python* terms: * .py files are the executable units of Python code * .so/.pyd files are the executable units of extension modules * .pyc/.pyo files are runtime artifacts that cache the results of the compilation step of loading .py code. There's no equivalent for .so/.pyd files as there's no compilation step needed when loading them. * .c, .h., .pyx, ... files are the source code for .so/.pyd files. There's no equivalent for .py files, because Python code is executable from source. Executable units of code go on sys.path. Cached runtime artifacts go in __pycache__. There's no defined location for source code. Whether you agree with the above or not, please accept that it's a consistent view of things. Sourceless distributions of Python code are an oddball corner case, but they do *not* demonstrate that Python is a compiled language and the .pyc/.pyo is the "executable" and the .py file is the "source". At least not in my opinion. It's entirely reasonable that we want to ensure that an exception object, whether raised from a Python file or a compiled extension module, includes a reference to the source of the issue, and that the traceback mechanism can, if at all possible, locate that source and display it in the exception traceback. That's basically what the loader "get_source" method is for - isn't the problem here simply that get_source for .pyd/.so files returns None, which in turn is because there's no obvious way to get to the source file for a general binary (and so the current implementation takes the easy way out and gives up[1])? Note that Cython is *not* unique here. Any tool that generates .so/.pyd files has the same problem, and we should be looking for a general solution. C code is the obvious example, and while I doubt anyone is going to ship C sources with a Python extension, it's still a valid case. Doesn't CFFI have a mode where it compiles a .pyd at wheel build time? Shouldn't that be able to locate the source? As a separate example, consider Jython, which may want to be able to locate the Java (or Groovy, or Scala, ...) source for a .class file on sys.path, and would have the same problem with get_source that CPython does with .so/.pyd files. So basically what's needed here is a modification to ExtensionFileLoader to return a more useful value from get_source. How it *does* that is up for debate, and brings in questions around what standards the packaging community want to establish for shipping extension sources, etc. And because the resulting decisions will end up implemented as stdlib code, they need to be properly thought through, as the timescales to fix design mistakes are quite long. "Bung stuff on sys.path" isn't really a well thought through solution in that sense, at least IMO - regardless of the fact that it's in effect what linecache supported prior to Python 3.3. So as a way forward, how about this: 1. Agree that what we actually need a better get_source method on ExtensionFileLoader 2. Adjourn to distutils-sig to agree a standard for where tools that generate .pyd/.so files should put their sources, if they should choose to ship their sources in the wheel with the final built extension. 3. Come back here with the results of that discussion to produce a PR that implements that change to ExtensionFileLoader The one possible fly in the ointment is if there are use cases that we need to support where a single .so/.pyd file is built from *multiple* source files, as get_source doesn't allow for that. Then it's back to the drawing board... Paul [1] I remember working on the original design of PEP 302 and "take the easy way out and give up" is *precisely* the thinking behind this :-) ---------- _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32797> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com