[issue37166] inspect.findsource doesn't handle shortened files gracefully

[Irit Katriel]

Irit Katriel  added the comment:

Fixed under issue17735.

--
nosy: +iritkatriel
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37166] inspect.findsource doesn't handle shortened files gracefully

[Irit Katriel]

Change by Irit Katriel :


--
resolution:  -> duplicate
superseder:  -> inspect.findsource raises IndexError

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37166] inspect.findsource doesn't handle shortened files gracefully

[Michael Bejda]

Michael Bejda  added the comment:

I don't think linecache.checkcache helps here. In your example, it would detect 
the modification, but I don't see how it could persist data if the compilation 
and execution are different processes.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37166] inspect.findsource doesn't handle shortened files gracefully

[Michael Bejda]

Michael Bejda  added the comment:

I do not believe it is specific to the way we package things:

$ cat a.py:
import inspect








def foo(): pass

inspect.getsource(foo)
$ python3 -m compileall -b a.py
$ black a.py  # to remove the empty lines
$ python3 a.pyc  # Index error
Traceback (most recent call last):
  File "a.py", line 12, in 
  File "/usr/local/fbcode/gcc-5-glibc-2.23/lib/python3.6/inspect.py", line 968, 
in getsource
lines, lnum = getsourcelines(object)
  File "/usr/local/fbcode/gcc-5-glibc-2.23/lib/python3.6/inspect.py", line 955, 
in getsourcelines
lines, lnum = findsource(object)
  File "/usr/local/fbcode/gcc-5-glibc-2.23/lib/python3.6/inspect.py", line 828, 
in findsource
if pat.match(lines[lnum]): break
IndexError: list index out of range


You are correct, the heuristic may not return the correct function. However, if 
we are, for any reason, unable to detect the source mismatch, it would be 
better to return 0 line (for the beginning of the module) or raise a 
SystemError instead of an IndexError (which the user does not expect).

--
nosy: +mib

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37166] inspect.findsource doesn't handle shortened files gracefully

[STINNER Victor]

STINNER Victor  added the comment:

If the file content changed, I'm not sure that the folloing heuristic will 
always find the expected function:

lnum = object.co_firstlineno - 1
pat = 
re.compile(r'^(\s*def\s)|(\s*async\s+def\s)|(.*(? 0:
if pat.match(lines[lnum]): break
lnum = lnum - 1
return lines, lnum

The regex doesn't search for the function name.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37166] inspect.findsource doesn't handle shortened files gracefully

[STINNER Victor]

STINNER Victor  added the comment:

inspect.getsource() starts with linecache.checkcache(file) which if the file 
modification time changed or if the file size changed. Using regular files on 
disk, I don't see how this bug is possible:


$ cat x.py 
import inspect
import os
import linecache






def foo(): pass

filename = __file__
linecache.getlines(filename)

if 1:
stat = os.stat(filename)
atime = stat.st_atime
mtime = stat.st_mtime
with open(filename, "w+") as fp:
lines = [line for line in fp if line.strip()]
fp.seek(0)
fp.writelines(lines)
os.utime(filename, (atime, mtime))

print(inspect.getsource(foo))

vstinner@apu$ /bin/cp x.py y.py; ./python y.py
Traceback (most recent call last):
  File "y.py", line 25, in 
  File "/home/vstinner/prog/python/master/Lib/inspect.py", line 985, in 
getsource
lines, lnum = getsourcelines(object)
  File "/home/vstinner/prog/python/master/Lib/inspect.py", line 967, in 
getsourcelines
lines, lnum = findsource(object)
  File "/home/vstinner/prog/python/master/Lib/inspect.py", line 798, in 
findsource
raise OSError('could not get source code')
OSError: could not get source code


Does this bug only affect Facebook who use "compile the application into an 
executable package (.par)"?
https://github.com/python/cpython/pull/13850#pullrequestreview-249815779

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37166] inspect.findsource doesn't handle shortened files gracefully

[Michael Bejda]

Change by Michael Bejda :


--
keywords: +patch
pull_requests: +13727
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/13850

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37166] inspect.findsource doesn't handle shortened files gracefully

[SilentGhost]

SilentGhost  added the comment:

3.6 is in security-only mode, this doesn't look like a security issue to me

--
nosy: +SilentGhost, yselivanov
versions: +Python 3.9 -Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37166] inspect.findsource doesn't handle shortened files gracefully

[Tim Hatch]

Change by Tim Hatch :


--
type:  -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37166] inspect.findsource doesn't handle shortened files gracefully

[Tim Hatch]

New submission from Tim Hatch :

inspect.findsource() can trigger IndexError when co_firstlineno is larger than 
len(linecache.getlines()).

If you have a situation where the file that linecache finds doesn't match the 
imported module, then you're not guaranteed that co_firstlineno on the code 
objects makes any sense.  We hit this in a special kind of par file, but it 
could be triggered by shortening the file, like doing a git checkout of an 
older version while it's running.

a.py (3 lines):
# line 1
# line 2
def foo(): pass  # co_firstlineno=3

a.py.v2 (1 line):
def foo(): pass

repro:
import a
# modification happens, cp a.py.v2 a.py
inspect.getsource(a.foo)


Although linecache.getline() does bounds checking, `inspect.findsource()` 
(which is used by various other functions, including `inspect.stack()`) grabs 
all the lines and loops through them.  The bug is in this section of 
`inspect.py`:

if iscode(object):
if not hasattr(object, 'co_firstlineno'):
raise OSError('could not find function definition')
lnum = object.co_firstlineno - 1
pat = 
re.compile(r'^(\s*def\s)|(\s*async\s+def\s)|(.*(? 0:
if pat.match(lines[lnum]): break
lnum = lnum - 1
return lines, lnum
raise OSError('could not find code object')

Found through future.utils.raise_from which actually doesn't need to be using 
`inspect.stack()`, it can switch to `sys._getframe(2)` or so.  We should have a 
PR ready shortly, but wondering if this can be backported to at least 3.6?

--
components: Library (Lib)
messages: 344761
nosy: lisroach, thatch, vstinner
priority: normal
severity: normal
status: open
title: inspect.findsource doesn't handle shortened files gracefully
versions: Python 3.6, Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com