New submission from Batuhan Taskaya <isidenti...@gmail.com>:
Since the parser & symbol modules are removed, there is no real use case for ISTERMINAL/ISNONTERMINAL since the only input that contains grammar symbols is the input of the tokenize, which all its outputs are terminals. Which restricts the practicality of token.ISTERMINAL/ISNONTERMINAL. I'd propose we either deprecate them or remove them completely. Search over the most populer PyPI packages (a set of 4k) showed there are only 2 projects that uses these functions, which all cases coupled with the parser module. pyglet-1.5.7/pyglet/text/formats/attributed.py 133: if token.ISNONTERMINAL(node[0]): gdata-2.0.18/tests/coverage.py 328: if token.ISNONTERMINAL(tree[0]): If we are going through a deprecation period, I will also propose to remove ISEOF, since it can be easily replaced and very under used. The only project that uses token.ISEOF in my dataset is asttokens, which I would say all cases there can be easily replaced tok.type == token.EOF asttokens-2.0.4/asttokens/mark_tokens.py 105: not token.ISEOF(t.type)): Also, these functions are originally reflections of C macros used in the past, and since they are not used right now, we can also remove those macros. I have no opinions about the token.NT_OFFSET though, but there are a couple of occurrences so we might keep that? Paste-3.4.4/paste/util/PySourceColor.py 205:ARGS = token.NT_OFFSET + 1 206:DOUBLECOMMENT = token.NT_OFFSET + 2 ipython-7.18.1/IPython/utils/PyColorize.py 54:_KEYWORD = token.NT_OFFSET + 1 55:_TEXT = token.NT_OFFSET + 2 ---------- keywords: 3.4regression messages: 379844 nosy: BTaskaya, gvanrossum, lys.nikolaou, pablogsal priority: normal severity: normal status: open title: Deprecating / removing token.ISTERMINAL/ISNONTERMINAL _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue42187> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
