[issue16224] tokenize.untokenize() misbehaves when moved to "compatiblity mode"

Terry J. Reedy added the comment: While I am closing this as a duplicate, I will use some of your patch, including one test, and credit you as well. Switching from 5-tuples to 2-tuples, as in one of your test cases, is not currently a supported use case, Compat currently re-iterates the entir

[issue16224] tokenize.untokenize() misbehaves when moved to "compatiblity mode"

Thomas Kluyver added the comment: I think this is a duplicate of #8478. -- nosy: +takluyver ___ Python tracker ___ ___ Python-bugs-lis

[issue16224] tokenize.untokenize() misbehaves when moved to "compatiblity mode"

Eric Snow added the comment: Actually, here's a patch with the first option. It preserves iterators as iterators, rather than dumping them into a list. I've also rolled the tests from issue16221 into this patch. Consequently, if the patch is suitable, that issue can be closed. -- k

[issue16224] tokenize.untokenize() misbehaves when moved to "compatiblity mode"

New submission from Eric Snow: When tokenize.untokenize() encounters a 2-tuple, it moves to compatibility mode, where only the token type and string are used from that point forward. There are two closely related problems: * when the iterable is a sequence, the portion of the sequence prior t