On Sun, Feb 23, 2020 at 5:17 PM Andrew Barnert <abarn...@yahoo.com> wrote:
> On Feb 23, 2020, at 05:44, André Roberge <andre.robe...@gmail.com> wrote: > > > In an attempt to make it easier to create import hooks to "play" with > non-standard syntax, I have created a library that is simply called "ideas" > [2]. > It includes quite a few toy examples meant to illustrate what can be done. > It also includes a slightly less minimal and hopefully less clumsy way of > filtering the token stream. A reasonably complete documentation is > available [3]. > > > This is very cool. It’s something I keep wanting to do but never get > around to it. I even wrote [a blog post about how to hack on Python without > hacking on CPython][1] > Yes, I was familiar with it; see the bottom of https://aroberge.github.io/ideas/docs/html/motivation.html > > > I’ve only skimmed your intro so far, but a couple of early comments. > > First, one of the parts that always blocked me was coming up with a good > way to write token-level transformers. It looks like you solved that; maybe > it’s worth pulling the tokenizer tools out into a separate library to post > on PyPI separately? > I've thought of doing that ... but decided to wait to get more feedback on this entire project and have both the import hook and the tokenizer tools "mature and grow together" a bit. > > Second, I always get sidetracked with the idea that it would be great if > we could feed a user-generated token stream into the compiler (the same way > we can feed a user-generated AST into the next stage) instead of needing to > monkey around with untokenizing. This isn’t too hard to do, but it is a big > enough change that it was always hard to convince anyone it was worth it. > Your module might change that. > > Finally, the other tricky bit is making hacks that span levels. For > example, a text or token hook transformer that tags things in a way that > can be used by an AST or bytecode transformer—or even by an actual CPython > patch to the codegen or ceval stages (so you can write the new semantics of > your proposed idea, but not have to write the new syntax part). I had a > hard time explaining this in my blog post, but maybe you can do better. :) > I doubt it very much as understanding the non-Python code in CPython is well beyond my comfort level. > > Also, if you haven’t looked at MacroPy, you definitely should. > I did look at it in the past based on some recommendations (perhaps yours) on Python-ideas. Time to revisit it... And maybe reference it in the docs. Sometimes it’s so much easier to write > something as a MacroPy macro that, even if you have to make the syntax a > little more clunky than you wanted, it’s worth doing that way. > [1]: > http://stupidpythonideas.blogspot.com/2015/06/hacking-python-without-hacking-python.html > <http://stupidpythonideas.blogspot.com/2015/06/hacking-python-without-hacking-python.html?m=1> > > >
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/SWKVA7PD4OSL2LHMI6TC6GZ22JB2IFJ2/ Code of Conduct: http://python.org/psf/codeofconduct/
