[issue32797] Tracebacks from Cython modules no longer work

Petr Viktorin added the comment: New changeset 057f4078b044325dae4af55c4c64b684edaca315 by Petr Viktorin (jdemeyer) in branch 'master': bpo-32797: improve documentation of linecache.getline (GH-9540) https://github.com/python/cpython/commit/057f4078b044325dae4af55c4c64b684edaca315

[issue32797] Tracebacks from Cython modules no longer work

Jeroen Demeyer added the comment: I pushed a documentation-only patch on PR 9540 to better document status quo. Can somebody please review either PR 6653 or PR 9540? -- ___ Python tracker

[issue32797] Tracebacks from Cython modules no longer work

Change by Jeroen Demeyer : -- pull_requests: +8942 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue32797] Tracebacks from Cython modules no longer work

Jeroen Demeyer added the comment: > Or, define a new "get_sourcemap()" method that could return additional > metadata, e.g. a line number mapping. What's your use case for that? I am asking because it might make more sense for get_source() (or whatever its replacement is) to return the

[issue32797] Tracebacks from Cython modules no longer work

Jeroen Demeyer added the comment: > I would love, for example, to be able to get stack traces within extension > modules integrated into Python tracebacks if they are compiled with some > appropriate debug flags. Awesome idea, I want to work on that :-) This should be possible using the

[issue32797] Tracebacks from Cython modules no longer work

Stefan Behnel added the comment: Or, define a new "get_sourcemap()" method that could return additional metadata, e.g. a line number mapping. -- ___ Python tracker ___

[issue32797] Tracebacks from Cython modules no longer work

Erik Bray added the comment: > To do that, I believe it can be made to work in much the same way it did in > Python 2 if SageMath were to do the following: > > 1. Define a subclass of ExtensionModuleLoader [1] that overrides get_source() > to also look for a ".pyx" file adjacent to the

[issue32797] Tracebacks from Cython modules no longer work

Erik Bray added the comment: To add, while an enhancement just to linecache would make sense in its own right, I don't see the problem with also extending the Loader.get_source() API to be more useful as well. Its current behavior is to just return a string (or None), but it seems to me

[issue32797] Tracebacks from Cython modules no longer work

Erik Bray added the comment: Brett: > As Nick said, we have no generalized concept of source maps and I think coming up with that is what will be required to solve this as i personally don't view Loader.get_source() is not meant to be a generalized concept of some form of source code but

[issue32797] Tracebacks from Cython modules no longer work

Jeroen Demeyer added the comment: I think I could make linecache use the ResourceReader API to find the source file instead of looking in the sys.path entries. However, that's orthogonal to PR 6653: we still need PR 6653 to continue looking for the source file in the first place.

[issue32797] Tracebacks from Cython modules no longer work

Stefan Behnel added the comment: > SageMath is the only project that I know which actually installs .pyx sources. Ah, right. I wrongly remembered that they are automatically included in binary packages, but that only applies to .py source of Cython compiled Python modules (which are

[issue32797] Tracebacks from Cython modules no longer work

Jeroen Demeyer added the comment: I just realized the existence of https://docs.python.org/3/library/importlib.html#importlib.abc.ResourceReader Isn't this *exactly* what we need here? -- ___ Python tracker

[issue32797] Tracebacks from Cython modules no longer work

Jeroen Demeyer added the comment: > If you're OK with that, I don't see the problem. I recall that we already agreed several months ago, when I submitted https://github.com/python/cpython/pull/6653 -- ___ Python tracker

[issue32797] Tracebacks from Cython modules no longer work

Paul Moore added the comment: If you're OK with that, I don't see the problem. My objection was with the claims that ExtensionLoader.get_source shouldn't return None or shouldn't exist. PEP 302 mandates that loaders have a get_source, and that get_source should return None if the loader

[issue32797] Tracebacks from Cython modules no longer work

Jeroen Demeyer added the comment: > One other thing I will say is that the PEP 302 APIs were seemingly designed > for working with Python source and were not necessarily oriented towards > alternative code representations that were executed That's also my impression. I'm perfectly fine with

[issue32797] Tracebacks from Cython modules no longer work

Jeroen Demeyer added the comment: > Now, though, we need a way for SageMath to get it working on releases up to > and including 3.7, *without* any help from Cython or CPython That's not really what I need. We already have a work-around in SageMath: try: from importlib.machinery import

[issue32797] Tracebacks from Cython modules no longer work

Jeroen Demeyer added the comment: > So, where is the filename coming from? Python 3.7.0 (default, Jul 23 2018, 10:07:21) [GCC 5.4.0 20160609] on linux Type "help", "copyright", "credits" or "license" for more information. >>> from sage.all import Integer >>> try: ... Integer(1)/Integer(0)

[issue32797] Tracebacks from Cython modules no longer work

Paul Moore added the comment: On Sun, 5 Aug 2018 at 18:10, Brett Cannon wrote: > One other thing I will say is that the PEP 302 APIs were seemingly designed > for working with Python source and were not necessarily oriented towards > alternative code representations that were executed, e.g.

[issue32797] Tracebacks from Cython modules no longer work

Brett Cannon added the comment: On Sun, Aug 5, 2018, 08:55 Nick Coghlan, wrote: > > Nick Coghlan added the comment: > > While I'd be inclined to agree with Paul's analysis if CPython were a > greenfield project, I also think if SageMath had already been ported to > Python 3.2, we would have

[issue32797] Tracebacks from Cython modules no longer work

Nick Coghlan added the comment: While I'd be inclined to agree with Paul's analysis if CPython were a greenfield project, I also think if SageMath had already been ported to Python 3.2, we would have considered this change a behavioural regression between 3.2 and 3.3 resulting from the

[issue32797] Tracebacks from Cython modules no longer work

Paul Moore added the comment: So, where is the filename coming from? Looking at exception and frame objects, I don't see a "source filename" attribute anywhere - have I missed something here? -- ___ Python tracker

[issue32797] Tracebacks from Cython modules no longer work

Jeroen Demeyer added the comment: > The one possible fly in the ointment is if there are use cases that we need to support where a single .so/.pyd file is built from *multiple* source files, as get_source doesn't allow for that. Yes, we must support that. A cython module may have multiple

[issue32797] Tracebacks from Cython modules no longer work

Paul Moore added the comment: On Sun, 5 Aug 2018 at 06:51, Stefan Behnel wrote: > This whole idea looks backwards and complicated. As Brett noted, .pyc files > were moved out of the source tree, because they are build time artifacts and > not sources. With that analogy, it's the .so files

[issue32797] Tracebacks from Cython modules no longer work

Jeroen Demeyer added the comment: > Keeping two identical package structures in different places, one for .py > files and generated .so files, and one for Cython source files (.pyx, .pxd, > .pxi), is not desirable from a user perspective, and would require namespace > packages for the

[issue32797] Tracebacks from Cython modules no longer work

Jeroen Demeyer added the comment: Nick: do you agree that this "source-only metapath" should by default contain an entry to look in sys.path for source files? -- ___ Python tracker

[issue32797] Tracebacks from Cython modules no longer work

Jeroen Demeyer added the comment: > In a subdirectory similar to __pycache__. I thought about that, but I don't like the idea because then the directory structure of the actual sources would be different from the directory structure of the installed sources. So for example, how should "pip

[issue32797] Tracebacks from Cython modules no longer work

Stefan Behnel added the comment: FWIW, I can see that Cython is a special case because it can control the source line mapping and reporting through the C-API. Other code generators might not be able to do that, e.g. for a DSL or template language that gets translated to Python code, for

[issue32797] Tracebacks from Cython modules no longer work

Nick Coghlan added the comment: Regarding PEP 302 compliance: until Python 3.3, the builtin import system wasn't present on sys.metapath - it was its own implicit legacy thing. The change in Python 3.3 was to fix that, and make it work based on a handful of default sys.metapath entries. Not

[issue32797] Tracebacks from Cython modules no longer work

Brett Cannon added the comment: On Sat, Aug 4, 2018, 16:07 Jeroen Demeyer, wrote: > > Jeroen Demeyer added the comment: > > To everybody who mentioned that Cython sources don't belong on sys.path: > where *do* they belong then? > In a subdirectory similar to __pycache__. > One issue which

[issue32797] Tracebacks from Cython modules no longer work

Jeroen Demeyer added the comment: To everybody who mentioned that Cython sources don't belong on sys.path: where *do* they belong then? One issue which hasn't been mentioned before is packaging. By installing Cython sources along with the generated .so files, we can re-use all the existing

[issue32797] Tracebacks from Cython modules no longer work

Brett Cannon added the comment: On Sat, Aug 4, 2018, 11:52 Jeroen Demeyer, wrote: > > Jeroen Demeyer added the comment: > > > In my view (and that of the documentation for sys.path), sys.path is > where you put things that the Python interpreter can load - .so files, .pyc > files and .py

[issue32797] Tracebacks from Cython modules no longer work

Jeroen Demeyer added the comment: > In my view (and that of the documentation for sys.path), sys.path is where > you put things that the Python interpreter can load - .so files, .pyc files > and .py files. It's quite typical for packages to install stuff in site-packages which the

[issue32797] Tracebacks from Cython modules no longer work

Paul Moore added the comment: > What's wrong with that? Installing the .pyx sources together with the .so > compiled modules makes a lot of sense to me: it is very analogous to > installing the .py sources together with the .pyc byte-compiled files. In >

[issue32797] Tracebacks from Cython modules no longer work

Jeroen Demeyer added the comment: > Then instead of adding the source directory to sys.path What's wrong with that? Installing the .pyx sources together with the .so compiled modules makes a lot of sense to me: it is very analogous to installing the .py sources together with the .pyc

[issue32797] Tracebacks from Cython modules no longer work

Nick Coghlan added the comment: As a completely minimal strawman design that's the closest fit to what SafeMath is already doing: what if there was a "linecache.fallback_source_path" attribute that linecache searched with importlib whenever a loader returned None to indicate that the module

[issue32797] Tracebacks from Cython modules no longer work

Nick Coghlan added the comment: This problem isn't unique to Cython extension modules - it exists for pyc-only distribution of pure Python files as well. The underlying problem is that we don't currently have a mechanism comparable to JavaScript source maps, whereby a preprocessed runtime

[issue32797] Tracebacks from Cython modules no longer work

Change by Brett Cannon : -- nosy: +brett.cannon, eric.snow, ncoghlan, petr.viktorin ___ Python tracker ___ ___ Python-bugs-list

[issue32797] Tracebacks from Cython modules no longer work

Change by Stefan Behnel : -- type: -> behavior versions: +Python 3.5, Python 3.6, Python 3.7, Python 3.8 ___ Python tracker ___

[issue32797] Tracebacks from Cython modules no longer work

Stephan Hohe added the comment: > Do you mean the "python" command-line program? Yes, that's what I used. > That uses a different algorithm: I see, it only works for top-level modules. You're right, that's not a real solution. --

[issue32797] Tracebacks from Cython modules no longer work

Jeroen Demeyer added the comment: > How does CPython display the source for tracebacks in Cython modules? Do you mean the "python" command-line program? That uses a different algorithm: to find a file FOO, it searches [os.path.join(p, os.path.basename(FOO)) for p in

[issue32797] Tracebacks from Cython modules no longer work

Paul Moore added the comment: That's interesting. It sounds like linecache might be working differently then. -- ___ Python tracker

[issue32797] Tracebacks from Cython modules no longer work

Stephan Hohe added the comment: Yes, I tried the Python 3.5 that comes with my system as well as the latest checkout from github. Both show source code lines in tracebacks for me. I used a rather simple test setup, just two directories with .so and .pyx which I added to

[issue32797] Tracebacks from Cython modules no longer work

Paul Moore added the comment: > How does CPython display the source for tracebacks in Cython modules? It > seems to work there as long as the Cython .pyx files are somewhere in the > import path. Is that in Python 3.x? The issue we're discussing is only in Python 3.3+

[issue32797] Tracebacks from Cython modules no longer work

Stephan Hohe added the comment: How does CPython display the source for tracebacks in Cython modules? It seems to work there as long as the Cython .pyx files are somewhere in the import path. -- nosy: +sth ___ Python tracker

[issue32797] Tracebacks from Cython modules no longer work

Jeroen Demeyer added the comment: > But the standard library has no need to ever find source for extension modules > So there's no need for the stdlib to be involved The standard library is not a closed system. It's not meant to support only itself, it's supposed to be an

[issue32797] Tracebacks from Cython modules no longer work

Jeroen Demeyer added the comment: Since you are asking various "meta" questions, let me explain the wider context first. This bug report is coming from SageMath. Currently, SageMath only supports Python 2.7 but we are slowly and steadily porting it to Python 3. This bug

[issue32797] Tracebacks from Cython modules no longer work

Paul Moore added the comment: >> IMO, debating whether a None return means there's absolutely no chance of >> there being source is silly - the best the loader can say is that it can't >> provide source. > > I don't think it is silly: if "None" means that the loader cannot

[issue32797] Tracebacks from Cython modules no longer work

Jeroen Demeyer added the comment: > IMO, debating whether a None return means there's absolutely no chance of > there being source is silly - the best the loader can say is that it can't > provide source. I don't think it is silly: if "None" means that the loader cannot

[issue32797] Tracebacks from Cython modules no longer work

Paul Moore added the comment: As I noted on python-ideas, continuing to search along sys.path if the loader returns None seems to match what the linecache docs say. If the issue had been phrased as "the implementation of linecache doesn't follow the docs" then I'd say

[issue32797] Tracebacks from Cython modules no longer work

Change by Jeroen Demeyer : -- keywords: +patch pull_requests: +6349 stage: -> patch review ___ Python tracker ___

[issue32797] Tracebacks from Cython modules no longer work

Erik Bray added the comment: +1, that seems obvious to me like better behavior. -- ___ Python tracker ___

[issue32797] Tracebacks from Cython modules no longer work

Jeroen Demeyer added the comment: Alternatively, instead of not implementing `loader.get_source()`, we could define a new semantic: if it returns `NotImplemented`, assume that the loader doesn't know how to get the sources. In that case, linecache would fall back to the

[issue32797] Tracebacks from Cython modules no longer work

Jeroen Demeyer added the comment: > I don't think there's a bug in Python here, and that this is a problem that > needs to be solved on the Cython end. I'm not necessarily disagreeing here. It all depends on how ExtensionFileLoader is meant to be used. Should it try

[issue32797] Tracebacks from Cython modules no longer work

Jeroen Demeyer added the comment: > Why? What would that help with? PEP 302 says get_source() can return None > [if] no sources are found. Returning None implies that it's absolutely impossible that there are sources to be found. But in certain cases (in the case of

[issue32797] Tracebacks from Cython modules no longer work

Erik Bray added the comment: On Feb 8, 2018 12:55, "Jeroen Demeyer" wrote: New submission from Jeroen Demeyer : Displaying the source code in tracebacks for Cython-compiled extension modules in IPython no longer works due

[issue32797] Tracebacks from Cython modules no longer work

New submission from Jeroen Demeyer : Displaying the source code in tracebacks for Cython-compiled extension modules in IPython no longer works due to PEP 302. Various fixes are possible, the two most obvious ones are: 1. linecache should continue searching for the