[issue28418] Raise Deprecation warning for tokenize.generate_tokens

Karthikeyan Singaravelan added the comment: Since generate_tokens was made public and documented with issue12486 I am closing this issue as won't fix as part of triaging. Feel free to reopen this issue if needed. Thanks -- resolution: -> wont fix stage: -> resolved status: open

[issue28418] Raise Deprecation warning for tokenize.generate_tokens

Thomas Kluyver added the comment: Yes, I think this can be closed now. -- ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue28418] Raise Deprecation warning for tokenize.generate_tokens

Karthikeyan Singaravelan added the comment: I just stumbled upon this issue while triaging. With the related issue https://bugs.python.org/issue12486 closed making generate_tokens public can this issue be closed since the original issue is not valid? Thanks -- nosy: +xtreak

[issue28418] Raise Deprecation warning for tokenize.generate_tokens

Raymond Hettinger added the comment: Thanks Thomas. I've been a long term user of tokenize.generate_tokens() and would be sad to see it go. The underlying _tokenize() functionality is used else within the module, so there the no benefit to removing the API.

[issue28418] Raise Deprecation warning for tokenize.generate_tokens

Thomas Kluyver added the comment: I've opened a PR moving in the other direction (making this public rather than deprecating it): https://github.com/python/cpython/pull/6957 -- nosy: +takluyver ___ Python tracker

[issue28418] Raise Deprecation warning for tokenize.generate_tokens

Martin Panter added the comment: There is related discussion in Issue 12486 about supporting unencoded text input. The current patch there actually already raises a warning and removes call sites from the Python library, though it does not add a doc string. -- nosy: +martin.panter

[issue28418] Raise Deprecation warning for tokenize.generate_tokens

New submission from Matthias Bussonnier: the Tokenize module has the following code: # An undocumented, backwards compatible, API for all the places in the standard # library that expect to be able to use tokenize with strings def generate_tokens(readline): return _tokenize(readline, None)