Hello André, On Sat, 24 Oct 2020 13:32:30 -0300 André Roberge <andre.robe...@gmail.com> wrote:
> # Experimental Syntax Proposal > > I would like to propose that Python adopts a modified process > before introducing significant changes of its syntax. [] > ## New suggested process > > Assuming one of the options mentioned above is adopted, > before changes to the syntax are introduced in a Python version, > they would be made available to be used either as an encoding variant > or an import hook giving enough time for interested pythonistas to > experiment with the new syntax, writing actual code and possibly > trying out various alternatives. Thanks for posting this proposal. It should be pretty clear that's the best process which should be followed. It's also should be pretty clear that (almost) nobody follows it. If anything, lack of responses by the substance of the matter proposed, i.e. the *process*, not technical means to achieve it, is very telling. So again, I fully agree that trying to implement any Python changes in the Python itself (instead of rushing to hack a particular implementation in C) should the best practice. Now let's talk why it's not that way. The main reasons would be lack of technical best practices to achieve that, and perceived complexity of existing ways to do that. Even from your proposal lack of the "best" solution is clear: > * By using an import hook. This is the most powerful approach as it > allows one to do changes either at the source level > (prior to parsing), or at the AST level, or both. > > * By using a custom encoding. This only allows transformations at the > source level. So, "custom encoding" way is essentially a hack. As it operates on the surface "stream of characters" representation of a program, only trivial, or imprecise transformations can be implemented. Alternatively, a program can be parsed, modified and dumped again as a stream of characters, just to immediately be parsed by the Python interpreter again. In either case, it's a hack, and should be rejected as a viable approach to a problem domain. Now you say that the second choice is "import hooks". But import hooks in Python is its own kingdom of "wonderland". Import hooks aren't intended to "allow to experiment with syntax", they are intended to do anything what's possible with module loading (and many things that aren't possible, I would add). So, they have rather complex, non-intuitive API, thru which anyone would need to wade, neck-deep, before implementing something useful. In other words, both choices you list aren't really viable to experiment with syntax/semantics. That may be a good explanation why nobody rushes to. I'd formulate choices to experiment with syntax/semantics differently. Let's remember the compilation pipeline: the source is tokenized, then it's parsed into AST, then AST is compiled into bytecode. 1. The central part of the pipeline is the AST. Many interesting features can be implemented on AST level. Others can be at least prototyped, using existing syntactic elements (by assigning to them new semantics). Of course, arbitrary syntax changes can't be implemented this way. The good news that for simple experiments and demonstrations you don't need import hooks in any way (just run "python3 -m my_python_dialect source.py"). 2. For when AST-level transformation is not enough, it should be possible to "fork" or "subclass" the entire compilation pipeline (in Python of course). Second choice is fully and ultimately flexible, but also not attainable with the standard Python3 alone. It's even more sad that it was with Python2. So, let's see: there's "tokenize" (https://docs.python.org/3/library/tokenize.html) module implemented in the stdlib in Python, which you can fork and modify as you like. But that's dead end, because to-AST module is implemented in C, and accepts raw characters anyway. Then AST-to-bytecode compiler is also implemented in C. You can't easily "subclass" any of them to implement your changes, as you may imagine. The story was different with Python2, which had "compiler" package: https://docs.python.org/2/library/compiler.html . Some time ago, I ported this compiler to CPython3.5: https://github.com/pfalcon/python-compiler/ . Specifically, I ported AST-to-bytecode compiler part, as the most interesting. AST parser needs to be sourced from yet another 3rd-party module (but choices definitely exist). Note that import hooks are still orthogonal to this approach either. Where they kick in is when you want to use your changes not just for "experimenting", but kind of "for real". Again, import hooks on the base level aren't ideal choice for that - they "mud waters" too much. What's needed is higher-level API specifically for the usecase of letting Python source of modules to run thru custom tokenizer/parser/bytecode compiler. The best known (non-adhoc) approach to that is PEP511: https://www.python.org/dev/peps/pep-0511/. But that PEP is the vivid example of Python core developers self-policing, and policing the community: "This PEP was seen as blessing new Python-like programming languages which are close but incompatible with the regular Python language. It was decided to not promote syntaxes incompatible with Python." So, well, the best action is that somebody implements it anyway, and maintain as a 3rd-party module. Unless there's there's a clear vision for even simpler API yet fully general API, and then that should be implemented and promoted. -------- This turned out to be a long intro. As I said, I fully agree with you, that changes to Python should be prototyped in Python. And there's no better way to "agree" than actually dogfood this approach to oneself. As I lately argued here on the list that implementing block-level scoping for Python is "not rocket science at all" and "a notch above trivial", I decided to do just that - and subject myself to coding it up, using the very python-compiler project I mentioned above. The result is a branch on that repo: https://github.com/pfalcon/python-compiler/tree/for-block-scope (I'll post a separate message with details.) -- Best regards, Paul mailto:pmis...@gmail.com _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/W2FRELA6ACEPIZKA24RURYJD4XPPJNB7/ Code of Conduct: http://python.org/psf/codeofconduct/
