Dustin J. Mitchell added the comment: >From my read of this bug, there are two distinct tasks mentioned:
1. make PyTokenizer_* part of the Python-level API 2. re-implement 'tokenize' in terms of that Python-level API #1 is largely complete in Andrew's latest patch, but that will likely need: * rebasing * hiding struct fields * documentation #2 is, I think, a separate project. There may be good reasons *not* to do this which I'm not aware of, and barring such reasons the rewrite will be difficult and could potentially change behavior like issue2180. So I would suggest filing a new issue for #2 when #1 is complete. And I'll work on #1. ---------- nosy: +djmitche _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue3353> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
