Just be aware that the C tokenizer interface is NOT a public interface and is there only so we can test the C tokenizer itself. This can and will break at any point without previous warning in any way.Pablo Galindo SalgadoOn 3 Nov 2022, at 18:10, David J W wrote:Following up, Pablo spotted my pro
Following up, Pablo spotted my problem with the mixup of NL & NEWLINE
tokens. I was using tokenize.py in cPython's stdlib with a simple python
script to build ridiculously strict unit tests.
My solution to that problem was originally to figure out how to access
cPython's internal c tokenizer but
