Guido van Rossum <gu...@python.org> added the comment: I think merging the tokenizers still makes sense. We can then document top-level tokenize.py (in 3.8 and later) as guaranteed to be able to tokenize anything going back to Python 2.7. And since lib2to3/pgen2 it is undocumented I presume removing lib2to3/pgen2/tokenize.py isn't going to break anything -- but if we worry about that it could be made into a trivial wrapper for top-level tokenize.py.
Still, the improvements you're planning to lib2to3 (no matter how compatible) will benefit more people sooner if you extract it into its own PyPI package. Not everybody can upgrade to 3.7 as soon as Instagram. :-) ---------- _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33337> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
