Paul Sokolovsky <[email protected]> added the comment:
> the idea was proposed purely to "close a gap"
That pinpoints it well. I was just writing a tutorial on implementing custom
import hooks, with the idea to show people how easy it to do it in Python. As
first step, I explained that it's bad idea to do any transformations on surface
representation of a program. At the very least, it should be converted to token
stream. But then I found that I need to explain that we need to convert it
back, which sounds pretty weird and undermines the idea:
def xform(token_stream):
for t in token_stream:
if t[0] == tokenize.NAME and t[1] == "function":
yield (tokenize.NAME, "lambda") + t[2:]
else:
yield t
with open(filename, "rb") as f:
# Fairly speaking, tokenizing just to convert back to string form
# isn't too efficient, but CPython doesn't offer us a way to parse
# token stream so far, so we have no choice.
source = tokenize.untokenize(xform(tokenize.tokenize(f.readline)))
mod = type(imphook)("")
exec(source, vars(mod))
return mod
Having written that comment, I thought I could as well just make one more step
and monkey-patch "ast" for parse_tokens() function - I'll need to explain that,
but the explanation probably wouldn't sound worse than the explanation above.
And then it was just one more step to actually submit patch for
ast.parse_tokens(), and that's how this ticket was created!
----------
_______________________________________
Python tracker <[email protected]>
<https://bugs.python.org/issue42729>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
