Hello community, here is the log from the commit of package python-pycparser for openSUSE:Factory checked in at 2020-03-12 22:57:59 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Comparing /work/SRC/openSUSE:Factory/python-pycparser (Old) and /work/SRC/openSUSE:Factory/.python-pycparser.new.3160 (New) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "python-pycparser" Thu Mar 12 22:57:59 2020 rev:11 rq:783100 version:2.20 Changes: -------- --- /work/SRC/openSUSE:Factory/python-pycparser/python-pycparser.changes 2019-01-15 09:14:43.518327513 +0100 +++ /work/SRC/openSUSE:Factory/.python-pycparser.new.3160/python-pycparser.changes 2020-03-12 22:58:14.986989383 +0100 @@ -1,0 +2,16 @@ +Mon Mar 9 20:59:13 UTC 2020 - Dirk Mueller <dmuel...@suse.com> + +- update to 2.20: + - #61: Fix slow backtracking when parsing strings. + - #99: Parser for FuncDecl incorrectly sets declname attribute on return type. + - #310: Fix crash when file starts with a semicolon. + - #313: Fix array type generation. + - #314: Fix failed parsing of unnamed function parameters with array dim + qualifiers. + - #315: Fix pointer type generation. + - #324: Fixes for u/l constant integer suffix. + - #346: Fix error transforming an empty switch. + - #350: Recognize integer multicharacter constants like 'ABCD'. + - #363: Fix incorrect AST when parsing offsetof. + +------------------------------------------------------------------- Old: ---- pycparser-2.19.tar.gz New: ---- pycparser-2.20.tar.gz ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Other differences: ------------------ ++++++ python-pycparser.spec ++++++ --- /var/tmp/diff_new_pack.xpP5SN/_old 2020-03-12 22:58:15.502989589 +0100 +++ /var/tmp/diff_new_pack.xpP5SN/_new 2020-03-12 22:58:15.506989590 +0100 @@ -1,7 +1,7 @@ # # spec file for package python-pycparser # -# Copyright (c) 2018 SUSE LINUX GmbH, Nuernberg, Germany. +# Copyright (c) 2020 SUSE LLC # # All modifications and additions to the file contributed by third parties # remain the property of their copyright owners, unless otherwise agreed @@ -18,12 +18,12 @@ %{?!python_module:%define python_module() python-%{**} python3-%{**}} Name: python-pycparser -Version: 2.19 +Version: 2.20 Release: 0 Summary: C parser in Python License: BSD-3-Clause Group: Development/Languages/Python -Url: https://github.com/eliben/pycparser +URL: https://github.com/eliben/pycparser Source0: https://files.pythonhosted.org/packages/source/p/pycparser/pycparser-%{version}.tar.gz Source99: %{name}-rpmlintrc Patch1: fix-lexer-build.patch ++++++ pycparser-2.19.tar.gz -> pycparser-2.20.tar.gz ++++++ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/pycparser-2.19/CHANGES new/pycparser-2.20/CHANGES --- old/pycparser-2.19/CHANGES 2018-09-19 14:20:30.000000000 +0200 +++ new/pycparser-2.20/CHANGES 2020-03-04 15:07:38.000000000 +0100 @@ -1,9 +1,26 @@ ++ Version 2.20 (2020.03.04) + + - #61: Fix slow backtracking when parsing strings. + - #99: Parser for FuncDecl incorrectly sets declname attribute on return type. + - #310: Fix crash when file starts with a semicolon. + - #313: Fix array type generation. + - #314: Fix failed parsing of unnamed function parameters with array dim + qualifiers. + - #315: Fix pointer type generation. + - #324: Fixes for u/l constant integer suffix. + - #346: Fix error transforming an empty switch. + - #350: Recognize integer multicharacter constants like 'ABCD'. + - #363: Fix incorrect AST when parsing offsetof. + + Version 2.19 (2018.09.19) - PR #277: Fix parsing of floating point literals - PR #254: Add support for parsing empty structs - PR #240: Fix enum formatting in generated C code (also #216) - PR #222: Add support for #pragma in struct declarations + - There are reports that this release doesn't work with Python 2.6 (#281). + Please note that the minimal supported version is 2.7; the required versions + are listed in the README file. + Version 2.18 (2017.07.04) @@ -149,11 +166,11 @@ + Version 2.05 (2011.10.16) - Added support for the C99 ``_Bool`` type and ``stdbool.h`` header file - - Expanded ``examples/explore_ast.py`` with more details on working with the + - Expanded ``examples/explore_ast.py`` with more details on working with the AST - Relaxed the rules on parsing unnamed struct members (helps parse ``windows.h``) - Bug fixes: - + * Fixed spacing issue for some type declarations * Issue 47: display empty statements (lone ';') correctly after parsing @@ -161,34 +178,34 @@ - License changed from LGPL to BSD - Bug fixes: - + * Issue 31: constraining the scope of typedef definitions * Issues 33, 35: fixes for the c-to-c.py example - + - Added C99 integer types to fake headers - Added unit tests for the c-to-c.py example + Version 2.03 (2011.03.06) - Bug fixes: - + * Issue 17: empty file-level declarations * Issue 18: empty statements and declarations in functions * Issue 19: anonymous structs & union fields * Issue 23: fix coordinates of Cast nodes - + - New example added (``examples/c-to-c.py``) for translating ASTs generated by ``pycparser`` back into C code. - ``pycparser`` is now on PyPI (Python Package Index) - Created `FAQ <http://code.google.com/p/pycparser/wiki/FAQ>`_ on - the ``pycparser`` project page + the ``pycparser`` project page - Removed support for Python 2.5. ``pycparser`` supports Python 2 from 2.6 and on, and Python 3. + Version 2.02 (2010.12.10) - * The name of a ``NamedInitializer`` node was turned into a sequence of nodes - instead of an attribute, to make it discoverable by the AST node visitor. + * The name of a ``NamedInitializer`` node was turned into a sequence of nodes + instead of an attribute, to make it discoverable by the AST node visitor. * Documentation updates + Version 2.01 (04.12.2010) @@ -218,7 +235,7 @@ + Version 1.06 (2010.04.10) - * Bug fixes: + * Bug fixes: + coord not propagated to FuncCall nodes + lexing of the ^= token (XOREQUALS) @@ -246,7 +263,7 @@ + Version 1.02 (2009.01.16) * Fixed problem of parsing struct/enum/union names that were named similarly - to previously defined ``typedef`` types. + to previously defined ``typedef`` types. + Version 1.01 (2009.01.09) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/pycparser-2.19/PKG-INFO new/pycparser-2.20/PKG-INFO --- old/pycparser-2.19/PKG-INFO 2018-09-19 14:32:55.000000000 +0200 +++ new/pycparser-2.20/PKG-INFO 2020-03-04 15:09:51.000000000 +0100 @@ -1,10 +1,11 @@ Metadata-Version: 1.2 Name: pycparser -Version: 2.19 +Version: 2.20 Summary: C parser in Python Home-page: https://github.com/eliben/pycparser Author: Eli Bendersky Author-email: eli...@gmail.com +Maintainer: Eli Bendersky License: BSD Description: pycparser is a complete parser of the C language, written in diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/pycparser-2.19/README.rst new/pycparser-2.20/README.rst --- old/pycparser-2.19/README.rst 2018-09-19 14:21:17.000000000 +0200 +++ new/pycparser-2.20/README.rst 2020-03-04 15:07:56.000000000 +0100 @@ -1,5 +1,5 @@ =============== -pycparser v2.19 +pycparser v2.20 =============== :Author: `Eli Bendersky <https://eli.thegreenplace.net/>`_ @@ -161,6 +161,9 @@ <https://eli.thegreenplace.net/2015/on-parsing-c-type-declarations-and-fake-headers>`_ for more details. +Note that the fake headers are not included in the ``pip`` package nor installed +via ``setup.py`` (`#224 <https://github.com/eliben/pycparser/issues/224>`_). + Basic usage ----------- diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/pycparser-2.19/examples/dump_ast.py new/pycparser-2.20/examples/dump_ast.py --- old/pycparser-2.19/examples/dump_ast.py 2018-07-25 14:51:15.000000000 +0200 +++ new/pycparser-2.20/examples/dump_ast.py 2019-06-27 14:47:27.000000000 +0200 @@ -19,7 +19,9 @@ if __name__ == "__main__": argparser = argparse.ArgumentParser('Dump AST') argparser.add_argument('filename', help='name of file to parse') + argparser.add_argument('--coord', help='show coordinates in the dump', + action='store_true') args = argparser.parse_args() ast = parse_file(args.filename, use_cpp=False) - ast.show() + ast.show(showcoord=args.coord) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/pycparser-2.19/examples/func_calls.py new/pycparser-2.20/examples/func_calls.py --- old/pycparser-2.19/examples/func_calls.py 2018-07-25 14:51:15.000000000 +0200 +++ new/pycparser-2.20/examples/func_calls.py 2019-06-27 14:47:27.000000000 +0200 @@ -17,9 +17,7 @@ from pycparser import c_parser, c_ast, parse_file -# A visitor with some state information (the funcname it's -# looking for) -# +# A visitor with some state information (the funcname it's looking for) class FuncCallVisitor(c_ast.NodeVisitor): def __init__(self, funcname): self.funcname = funcname @@ -27,6 +25,9 @@ def visit_FuncCall(self, node): if node.name.name == self.funcname: print('%s called at %s' % (self.funcname, node.name.coord)) + # Visit args in case they contain more func calls. + if node.args: + self.visit(node.args) def show_func_calls(filename, funcname): diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/pycparser-2.19/pycparser/__init__.py new/pycparser-2.20/pycparser/__init__.py --- old/pycparser-2.19/pycparser/__init__.py 2018-09-19 14:21:08.000000000 +0200 +++ new/pycparser-2.20/pycparser/__init__.py 2020-03-04 15:07:48.000000000 +0100 @@ -8,7 +8,7 @@ # License: BSD #----------------------------------------------------------------- __all__ = ['c_lexer', 'c_parser', 'c_ast'] -__version__ = '2.19' +__version__ = '2.20' import io from subprocess import check_output diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/pycparser-2.19/pycparser/_build_tables.py new/pycparser-2.20/pycparser/_build_tables.py --- old/pycparser-2.19/pycparser/_build_tables.py 2018-07-25 14:51:15.000000000 +0200 +++ new/pycparser-2.20/pycparser/_build_tables.py 2019-06-27 14:47:27.000000000 +0200 @@ -10,13 +10,17 @@ # License: BSD #----------------------------------------------------------------- +# Insert '.' and '..' as first entries to the search path for modules. +# Restricted environments like embeddable python do not include the +# current working directory on startup. +import sys +sys.path[0:0] = ['.', '..'] + # Generate c_ast.py from _ast_gen import ASTCodeGenerator ast_gen = ASTCodeGenerator('_c_ast.cfg') ast_gen.generate(open('c_ast.py', 'w')) -import sys -sys.path[0:0] = ['.', '..'] from pycparser import c_parser # Generates the tables diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/pycparser-2.19/pycparser/ast_transforms.py new/pycparser-2.20/pycparser/ast_transforms.py --- old/pycparser-2.19/pycparser/ast_transforms.py 2018-07-25 14:51:15.000000000 +0200 +++ new/pycparser-2.20/pycparser/ast_transforms.py 2019-08-24 14:56:59.000000000 +0200 @@ -74,7 +74,8 @@ # Goes over the children of the Compound below the Switch, adding them # either directly below new_compound or below the last Case as appropriate - for child in switch_node.stmt.block_items: + # (for `switch(cond) {}`, block_items would have been None) + for child in (switch_node.stmt.block_items or []): if isinstance(child, (c_ast.Case, c_ast.Default)): # If it's a Case/Default: # 1. Add it to the Compound and mark as "last case" diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/pycparser-2.19/pycparser/c_generator.py new/pycparser-2.20/pycparser/c_generator.py --- old/pycparser-2.19/pycparser/c_generator.py 2018-07-25 14:51:15.000000000 +0200 +++ new/pycparser-2.20/pycparser/c_generator.py 2019-06-27 14:47:27.000000000 +0200 @@ -119,7 +119,7 @@ return s def visit_Cast(self, n): - s = '(' + self._generate_type(n.to_type) + ')' + s = '(' + self._generate_type(n.to_type, emit_declname=False) + ')' return s + ' ' + self._parenthesize_unless_simple(n.expr) def visit_ExprList(self, n): @@ -291,6 +291,15 @@ def visit_FuncDecl(self, n): return self._generate_type(n) + def visit_ArrayDecl(self, n): + return self._generate_type(n, emit_declname=False) + + def visit_TypeDecl(self, n): + return self._generate_type(n, emit_declname=False) + + def visit_PtrDecl(self, n): + return self._generate_type(n, emit_declname=False) + def _generate_struct_union_enum(self, n, name): """ Generates code for structs, unions, and enums. name should be 'struct', 'union', or 'enum'. @@ -359,7 +368,7 @@ s += self._generate_type(n.type) return s - def _generate_type(self, n, modifiers=[]): + def _generate_type(self, n, modifiers=[], emit_declname = True): """ Recursive generation from a type node. n is the type node. modifiers collects the PtrDecl, ArrayDecl and FuncDecl modifiers encountered on the way down to a TypeDecl, to allow proper @@ -373,23 +382,29 @@ if n.quals: s += ' '.join(n.quals) + ' ' s += self.visit(n.type) - nstr = n.declname if n.declname else '' + nstr = n.declname if n.declname and emit_declname else '' # Resolve modifiers. # Wrap in parens to distinguish pointer to array and pointer to # function syntax. # for i, modifier in enumerate(modifiers): if isinstance(modifier, c_ast.ArrayDecl): - if (i != 0 and isinstance(modifiers[i - 1], c_ast.PtrDecl)): - nstr = '(' + nstr + ')' - nstr += '[' + self.visit(modifier.dim) + ']' + if (i != 0 and + isinstance(modifiers[i - 1], c_ast.PtrDecl)): + nstr = '(' + nstr + ')' + nstr += '[' + if modifier.dim_quals: + nstr += ' '.join(modifier.dim_quals) + ' ' + nstr += self.visit(modifier.dim) + ']' elif isinstance(modifier, c_ast.FuncDecl): - if (i != 0 and isinstance(modifiers[i - 1], c_ast.PtrDecl)): - nstr = '(' + nstr + ')' + if (i != 0 and + isinstance(modifiers[i - 1], c_ast.PtrDecl)): + nstr = '(' + nstr + ')' nstr += '(' + self.visit(modifier.args) + ')' elif isinstance(modifier, c_ast.PtrDecl): if modifier.quals: - nstr = '* %s %s' % (' '.join(modifier.quals), nstr) + nstr = '* %s%s' % (' '.join(modifier.quals), + ' ' + nstr if nstr else '') else: nstr = '*' + nstr if nstr: s += ' ' + nstr @@ -397,11 +412,12 @@ elif typ == c_ast.Decl: return self._generate_decl(n.type) elif typ == c_ast.Typename: - return self._generate_type(n.type) + return self._generate_type(n.type, emit_declname = emit_declname) elif typ == c_ast.IdentifierType: return ' '.join(n.names) + ' ' elif typ in (c_ast.ArrayDecl, c_ast.PtrDecl, c_ast.FuncDecl): - return self._generate_type(n.type, modifiers + [n]) + return self._generate_type(n.type, modifiers + [n], + emit_declname = emit_declname) else: return self.visit(n) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/pycparser-2.19/pycparser/c_lexer.py new/pycparser-2.20/pycparser/c_lexer.py --- old/pycparser-2.19/pycparser/c_lexer.py 2018-07-25 14:51:15.000000000 +0200 +++ new/pycparser-2.20/pycparser/c_lexer.py 2019-12-21 15:41:30.000000000 +0100 @@ -19,7 +19,7 @@ tokens. The public attribute filename can be set to an initial - filaneme, but the lexer will update it upon #line + filename, but the lexer will update it upon #line directives. """ def __init__(self, error_func, on_lbrace_func, on_rbrace_func, @@ -130,7 +130,7 @@ 'TYPEID', # constants - 'INT_CONST_DEC', 'INT_CONST_OCT', 'INT_CONST_HEX', 'INT_CONST_BIN', + 'INT_CONST_DEC', 'INT_CONST_OCT', 'INT_CONST_HEX', 'INT_CONST_BIN', 'INT_CONST_CHAR', 'FLOAT_CONST', 'HEX_FLOAT_CONST', 'CHAR_CONST', 'WCHAR_CONST', @@ -205,23 +205,49 @@ # parse all correct code, even if it means to sometimes parse incorrect # code. # - simple_escape = r"""([a-zA-Z._~!=&\^\-\\?'"])""" - decimal_escape = r"""(\d+)""" - hex_escape = r"""(x[0-9a-fA-F]+)""" - bad_escape = r"""([\\][^a-zA-Z._~^!=&\^\-\\?'"x0-7])""" + # The original regexes were taken verbatim from the C syntax definition, + # and were later modified to avoid worst-case exponential running time. + # + # simple_escape = r"""([a-zA-Z._~!=&\^\-\\?'"])""" + # decimal_escape = r"""(\d+)""" + # hex_escape = r"""(x[0-9a-fA-F]+)""" + # bad_escape = r"""([\\][^a-zA-Z._~^!=&\^\-\\?'"x0-7])""" + # + # The following modifications were made to avoid the ambiguity that allowed backtracking: + # (https://github.com/eliben/pycparser/issues/61) + # + # - \x was removed from simple_escape, unless it was not followed by a hex digit, to avoid ambiguity with hex_escape. + # - hex_escape allows one or more hex characters, but requires that the next character(if any) is not hex + # - decimal_escape allows one or more decimal characters, but requires that the next character(if any) is not a decimal + # - bad_escape does not allow any decimals (8-9), to avoid conflicting with the permissive decimal_escape. + # + # Without this change, python's `re` module would recursively try parsing each ambiguous escape sequence in multiple ways. + # e.g. `\123` could be parsed as `\1`+`23`, `\12`+`3`, and `\123`. + + simple_escape = r"""([a-wyzA-Z._~!=&\^\-\\?'"]|x(?![0-9a-fA-F]))""" + decimal_escape = r"""(\d+)(?!\d)""" + hex_escape = r"""(x[0-9a-fA-F]+)(?![0-9a-fA-F])""" + bad_escape = r"""([\\][^a-zA-Z._~^!=&\^\-\\?'"x0-9])""" escape_sequence = r"""(\\("""+simple_escape+'|'+decimal_escape+'|'+hex_escape+'))' + + # This complicated regex with lookahead might be slow for strings, so because all of the valid escapes (including \x) allowed + # 0 or more non-escaped characters after the first character, simple_escape+decimal_escape+hex_escape got simplified to + + escape_sequence_start_in_string = r"""(\\[0-9a-zA-Z._~!=&\^\-\\?'"])""" + cconst_char = r"""([^'\\\n]|"""+escape_sequence+')' char_const = "'"+cconst_char+"'" wchar_const = 'L'+char_const + multicharacter_constant = "'"+cconst_char+"{2,4}'" unmatched_quote = "('"+cconst_char+"*\\n)|('"+cconst_char+"*$)" bad_char_const = r"""('"""+cconst_char+"""[^'\n]+')|('')|('"""+bad_escape+r"""[^'\n]*')""" # string literals (K&R2: A.2.6) - string_char = r"""([^"\\\n]|"""+escape_sequence+')' + string_char = r"""([^"\\\n]|"""+escape_sequence_start_in_string+')' string_literal = '"'+string_char+'*"' wstring_literal = 'L'+string_literal - bad_string_literal = '"'+string_char+'*?'+bad_escape+string_char+'*"' + bad_string_literal = '"'+string_char+'*'+bad_escape+string_char+'*"' # floating constants (K&R2: A.2.5.3) exponent_part = r"""([eE][-+]?[0-9]+)""" @@ -443,6 +469,10 @@ # Must come before bad_char_const, to prevent it from # catching valid char constants as invalid # + @TOKEN(multicharacter_constant) + def t_INT_CONST_CHAR(self, t): + return t + @TOKEN(char_const) def t_CHAR_CONST(self, t): return t diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/pycparser-2.19/pycparser/c_parser.py new/pycparser-2.20/pycparser/c_parser.py --- old/pycparser-2.19/pycparser/c_parser.py 2018-08-31 15:00:24.000000000 +0200 +++ new/pycparser-2.20/pycparser/c_parser.py 2020-03-03 15:33:36.000000000 +0100 @@ -529,8 +529,7 @@ def p_translation_unit_2(self, p): """ translation_unit : translation_unit external_declaration """ - if p[2] is not None: - p[1].extend(p[2]) + p[1].extend(p[2]) p[0] = p[1] # Declarations always come as lists (because they can be @@ -557,7 +556,7 @@ def p_external_declaration_4(self, p): """ external_declaration : SEMI """ - p[0] = None + p[0] = [] def p_pp_directive(self, p): """ pp_directive : PPHASH @@ -1411,12 +1410,13 @@ p[0] = self._type_modify_decl(decl=p[1], modifier=arr) def p_direct_abstract_declarator_3(self, p): - """ direct_abstract_declarator : LBRACKET assignment_expression_opt RBRACKET + """ direct_abstract_declarator : LBRACKET type_qualifier_list_opt assignment_expression_opt RBRACKET """ + quals = (p[2] if len(p) > 4 else []) or [] p[0] = c_ast.ArrayDecl( type=c_ast.TypeDecl(None, None, None), - dim=p[2], - dim_quals=[], + dim=p[3] if len(p) > 4 else p[2], + dim_quals=quals, coord=self._token_coord(p, 1)) def p_direct_abstract_declarator_4(self, p): @@ -1740,8 +1740,7 @@ if len(p) == 2: p[0] = p[1] elif len(p) == 4: - field = c_ast.ID(p[3], self._token_coord(p, 3)) - p[0] = c_ast.StructRef(p[1], p[2], field, p[1].coord) + p[0] = c_ast.StructRef(p[1], p[2], p[3], p[1].coord) elif len(p) == 5: p[0] = c_ast.ArrayRef(p[1], p[3], p[1].coord) else: @@ -1766,9 +1765,23 @@ | INT_CONST_OCT | INT_CONST_HEX | INT_CONST_BIN + | INT_CONST_CHAR """ + uCount = 0 + lCount = 0 + for x in p[1][-3:]: + if x in ('l', 'L'): + lCount += 1 + elif x in ('u', 'U'): + uCount += 1 + t = '' + if uCount > 1: + raise ValueError('Constant cannot have more than one u/U suffix.') + elif lCount > 2: + raise ValueError('Constant cannot have more than two l/L suffix.') + prefix = 'unsigned ' * uCount + 'long ' * lCount p[0] = c_ast.Constant( - 'int', p[1], self._token_coord(p, 1)) + prefix + 'int', p[1], self._token_coord(p, 1)) def p_constant_2(self, p): """ constant : FLOAT_CONST diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/pycparser-2.19/pycparser.egg-info/PKG-INFO new/pycparser-2.20/pycparser.egg-info/PKG-INFO --- old/pycparser-2.19/pycparser.egg-info/PKG-INFO 2018-09-19 14:32:55.000000000 +0200 +++ new/pycparser-2.20/pycparser.egg-info/PKG-INFO 2020-03-04 15:09:51.000000000 +0100 @@ -1,10 +1,11 @@ Metadata-Version: 1.2 Name: pycparser -Version: 2.19 +Version: 2.20 Summary: C parser in Python Home-page: https://github.com/eliben/pycparser Author: Eli Bendersky Author-email: eli...@gmail.com +Maintainer: Eli Bendersky License: BSD Description: pycparser is a complete parser of the C language, written in diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/pycparser-2.19/pycparser.egg-info/SOURCES.txt new/pycparser-2.20/pycparser.egg-info/SOURCES.txt --- old/pycparser-2.19/pycparser.egg-info/SOURCES.txt 2018-09-19 14:32:55.000000000 +0200 +++ new/pycparser-2.20/pycparser.egg-info/SOURCES.txt 2020-03-04 15:09:51.000000000 +0100 @@ -71,6 +71,7 @@ utils/fake_libc_include/ctype.h utils/fake_libc_include/dirent.h utils/fake_libc_include/dlfcn.h +utils/fake_libc_include/emmintrin.h utils/fake_libc_include/endian.h utils/fake_libc_include/envz.h utils/fake_libc_include/errno.h @@ -87,6 +88,7 @@ utils/fake_libc_include/grp.h utils/fake_libc_include/iconv.h utils/fake_libc_include/ieeefp.h +utils/fake_libc_include/immintrin.h utils/fake_libc_include/inttypes.h utils/fake_libc_include/iso646.h utils/fake_libc_include/langinfo.h @@ -115,6 +117,7 @@ utils/fake_libc_include/semaphore.h utils/fake_libc_include/setjmp.h utils/fake_libc_include/signal.h +utils/fake_libc_include/smmintrin.h utils/fake_libc_include/spawn.h utils/fake_libc_include/stdarg.h utils/fake_libc_include/stdbool.h diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/pycparser-2.19/setup.cfg new/pycparser-2.20/setup.cfg --- old/pycparser-2.19/setup.cfg 2018-09-19 14:32:55.000000000 +0200 +++ new/pycparser-2.20/setup.cfg 2020-03-04 15:09:51.000000000 +0100 @@ -7,5 +7,4 @@ [egg_info] tag_build = tag_date = 0 -tag_svn_revision = 0 diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/pycparser-2.19/setup.py new/pycparser-2.20/setup.py --- old/pycparser-2.19/setup.py 2018-09-19 14:20:49.000000000 +0200 +++ new/pycparser-2.20/setup.py 2020-03-04 15:07:45.000000000 +0100 @@ -43,7 +43,7 @@ C compilers or analysis tools. """, license='BSD', - version='2.19', + version='2.20', author='Eli Bendersky', maintainer='Eli Bendersky', author_email='eli...@gmail.com', diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/pycparser-2.19/tests/test_c_generator.py new/pycparser-2.20/tests/test_c_generator.py --- old/pycparser-2.19/tests/test_c_generator.py 2018-05-21 15:18:15.000000000 +0200 +++ new/pycparser-2.20/tests/test_c_generator.py 2019-06-27 14:47:27.000000000 +0200 @@ -1,11 +1,12 @@ +import os +import platform import sys -import textwrap import unittest # Run from the root dir sys.path.insert(0, '.') -from pycparser import c_parser, c_generator, c_ast +from pycparser import c_parser, c_generator, c_ast, parse_file _c_parser = c_parser.CParser( lex_optimize=False, @@ -332,6 +333,62 @@ name='', ) + def test_array_decl(self): + self._assert_ctoc_correct('int g(const int a[const 20]){}') + ast = parse_to_ast('const int a[const 20];') + generator = c_generator.CGenerator() + self.assertEqual(generator.visit(ast.ext[0].type), + 'const int [const 20]') + self.assertEqual(generator.visit(ast.ext[0].type.type), + 'const int') + + def test_ptr_decl(self): + src = 'const int ** const x;' + self._assert_ctoc_correct(src) + ast = parse_to_ast(src) + generator = c_generator.CGenerator() + self.assertEqual(generator.visit(ast.ext[0].type), + 'const int ** const') + self.assertEqual(generator.visit(ast.ext[0].type.type), + 'const int *') + self.assertEqual(generator.visit(ast.ext[0].type.type.type), + 'const int') + + +class TestCasttoC(unittest.TestCase): + def _find_file(self, name): + test_dir = os.path.dirname(__file__) + name = os.path.join(test_dir, 'c_files', name) + assert os.path.exists(name) + return name + + def test_to_type(self): + src = 'int *x;' + generator = c_generator.CGenerator() + test_fun = c_ast.FuncCall(c_ast.ID('test_fun'), c_ast.ExprList([])) + + ast1 = parse_to_ast(src) + int_ptr_type = ast1.ext[0].type + int_type = int_ptr_type.type + self.assertEqual(generator.visit(c_ast.Cast(int_ptr_type, test_fun)), + '(int *) test_fun()') + self.assertEqual(generator.visit(c_ast.Cast(int_type, test_fun)), + '(int) test_fun()') + + @unittest.skipUnless(platform.system() == 'Linux', + 'cpp only works on Linux') + def test_to_type_with_cpp(self): + generator = c_generator.CGenerator() + test_fun = c_ast.FuncCall(c_ast.ID('test_fun'), c_ast.ExprList([])) + memmgr_path = self._find_file('memmgr.h') + + ast2 = parse_file(memmgr_path, use_cpp=True) + void_ptr_type = ast2.ext[-3].type.type + void_type = void_ptr_type.type + self.assertEqual(generator.visit(c_ast.Cast(void_ptr_type, test_fun)), + '(void *) test_fun()') + self.assertEqual(generator.visit(c_ast.Cast(void_type, test_fun)), + '(void) test_fun()') if __name__ == "__main__": unittest.main() diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/pycparser-2.19/tests/test_c_lexer.py new/pycparser-2.20/tests/test_c_lexer.py --- old/pycparser-2.19/tests/test_c_lexer.py 2017-03-10 15:02:29.000000000 +0100 +++ new/pycparser-2.20/tests/test_c_lexer.py 2019-12-21 15:41:30.000000000 +0100 @@ -77,6 +77,10 @@ self.assertTokensTypes('0xf7', ['INT_CONST_HEX']) self.assertTokensTypes('0b110', ['INT_CONST_BIN']) self.assertTokensTypes('0x01202AAbbf7Ul', ['INT_CONST_HEX']) + self.assertTokensTypes("'12'", ['INT_CONST_CHAR']) + self.assertTokensTypes("'123'", ['INT_CONST_CHAR']) + self.assertTokensTypes("'1AB4'", ['INT_CONST_CHAR']) + self.assertTokensTypes(r"'1A\n4'", ['INT_CONST_CHAR']) # no 0 before x, so ID catches it self.assertTokensTypes('xf7', ['ID']) @@ -116,6 +120,7 @@ self.assertTokensTypes(r"""'\t'""", ['CHAR_CONST']) self.assertTokensTypes(r"""'\''""", ['CHAR_CONST']) self.assertTokensTypes(r"""'\?'""", ['CHAR_CONST']) + self.assertTokensTypes(r"""'\0'""", ['CHAR_CONST']) self.assertTokensTypes(r"""'\012'""", ['CHAR_CONST']) self.assertTokensTypes(r"""'\x2f'""", ['CHAR_CONST']) self.assertTokensTypes(r"""'\x2f12'""", ['CHAR_CONST']) @@ -149,6 +154,24 @@ self.assertTokensTypes( '"\123\123\123\123\123\123\123\123\123\123\123\123\123\123\123\123"', ['STRING_LITERAL']) + # Note: a-zA-Z and '.-~^_!=&;,' are allowed as escape chars to support #line + # directives with Windows paths as filenames (..\..\dir\file) + self.assertTokensTypes( + r'"\x"', + ['STRING_LITERAL']) + self.assertTokensTypes( + r'"\a\b\c\d\e\f\g\h\i\j\k\l\m\n\o\p\q\r\s\t\u\v\w\x\y\z\A\B\C\D\E\F\G\H\I\J\K\L\M\N\O\P\Q\R\S\T\U\V\W\X\Y\Z"', + ['STRING_LITERAL']) + self.assertTokensTypes( + r'"C:\x\fa\x1e\xited"', + ['STRING_LITERAL']) + # The lexer is permissive and allows decimal escapes (not just octal) + self.assertTokensTypes( + '"jx\9"', + ['STRING_LITERAL']) + self.assertTokensTypes( + '"fo\9999999"', + ['STRING_LITERAL']) def test_mess(self): self.assertTokensTypes( @@ -428,14 +451,24 @@ def test_char_constants(self): self.assertLexerError("'", ERR_UNMATCHED_QUOTE) self.assertLexerError("'b\n", ERR_UNMATCHED_QUOTE) - - self.assertLexerError("'jx'", ERR_INVALID_CCONST) + self.assertLexerError("'\\xaa\n'", ERR_UNMATCHED_QUOTE) + + self.assertLexerError(r"'123\12a'", ERR_INVALID_CCONST) + self.assertLexerError(r"'123\xabg'", ERR_INVALID_CCONST) + self.assertLexerError("''", ERR_INVALID_CCONST) + self.assertLexerError("'abcjx'", ERR_INVALID_CCONST) self.assertLexerError(r"'\*'", ERR_INVALID_CCONST) def test_string_literals(self): - self.assertLexerError(r'"jx\9"', ERR_STRING_ESCAPE) + self.assertLexerError(r'"jx\`"', ERR_STRING_ESCAPE) self.assertLexerError(r'"hekllo\* on ix"', ERR_STRING_ESCAPE) self.assertLexerError(r'L"hekllo\* on ix"', ERR_STRING_ESCAPE) + # Should not suffer from slow backtracking + self.assertLexerError(r'"\123\123\123\123\123\123\123\123\123\123\123\123\123\123\123\123\123\123\123\`\123\123\123\123\123\123\123\123\123\123\123\123\123\123\123\123\123\123\123\123"', ERR_STRING_ESCAPE) + self.assertLexerError(r'"\xf1\x23\xf1\x23\xf1\x23\xf1\x23\xf1\x23\xf1\x23\xf1\x23\xf1\x23\xf1\x23\x23\`\xf1\x23\xf1\x23\xf1\x23\xf1\x23\xf1\x23\xf1\x23\xf1\x23\xf1\x23\xf1\x23\xf1\x23"', ERR_STRING_ESCAPE) + # Should not suffer from slow backtracking when there's no end quote + self.assertLexerError(r'"\123\123\123\123\123\123\123\123\123\123\123\123\123\123\123\`\123\123\123\123\123\123\123\123\123\123\123\123\123\123\123\123\123\123\12\123456', ERR_ILLEGAL_CHAR) + self.assertLexerError(r'"\x23\x23\x23\x23\x23\x23\x23\x23\x23\x23\x23\x23\x23\x23\x23\`\x23\x23\x23\x23\x23\x23\x23\x23\x23\x23\x23\x23\x23\x23\x23\x23\x23\x23\x2\x23456', ERR_ILLEGAL_CHAR) def test_preprocessor(self): self.assertLexerError('#line "ka"', ERR_FILENAME_BEFORE_LINE) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/pycparser-2.19/tests/test_c_parser.py new/pycparser-2.20/tests/test_c_parser.py --- old/pycparser-2.19/tests/test_c_parser.py 2018-08-31 15:01:56.000000000 +0200 +++ new/pycparser-2.20/tests/test_c_parser.py 2020-03-03 15:33:36.000000000 +0100 @@ -136,6 +136,15 @@ ['Decl', 'foo', ['TypeDecl', ['IdentifierType', ['int']]]]) + def test_initial_semi(self): + t = self.parse(';') + self.assertEqual(len(t.ext), 0) + t = self.parse(';int foo;') + self.assertEqual(len(t.ext), 1) + self.assertEqual(expand_decl(t.ext[0]), + ['Decl', 'foo', + ['TypeDecl', ['IdentifierType', ['int']]]]) + def test_coords(self): """ Tests the "coordinates" of parsed elements - file name, line and column numbers, with modification @@ -413,6 +422,7 @@ ['TypeDecl', ['IdentifierType', ['int']]]]]]) def test_func_decls_with_array_dim_qualifiers(self): + # named function parameter self.assertEqual(self.get_decl('int zz(int p[static 10]);'), ['Decl', 'zz', ['FuncDecl', @@ -443,6 +453,30 @@ ['TypeDecl', ['IdentifierType', ['int']]]]]]], ['TypeDecl', ['IdentifierType', ['int']]]]]) + # unnamed function parameter + self.assertEqual(self.get_decl('int zz(int [const 10]);'), + ['Decl', 'zz', + ['FuncDecl', + [['Typename', ['ArrayDecl', '10', ['const'], + ['TypeDecl', ['IdentifierType', ['int']]]]]], + ['TypeDecl', ['IdentifierType', ['int']]]]]) + + self.assertEqual(self.get_decl('int zz(int [restrict][5]);'), + ['Decl', 'zz', + ['FuncDecl', + [['Typename', ['ArrayDecl', '', ['restrict'], + ['ArrayDecl', '5', [], + ['TypeDecl', ['IdentifierType', ['int']]]]]]], + ['TypeDecl', ['IdentifierType', ['int']]]]]) + + self.assertEqual(self.get_decl('int zz(int [const restrict volatile 10][5]);'), + ['Decl', 'zz', + ['FuncDecl', + [['Typename', ['ArrayDecl', '10', ['const', 'restrict', 'volatile'], + ['ArrayDecl', '5', [], + ['TypeDecl', ['IdentifierType', ['int']]]]]]], + ['TypeDecl', ['IdentifierType', ['int']]]]]) + def test_qualifiers_storage_specifiers(self): def assert_qs(txt, index, quals, storage): d = self.parse(txt).ext[index] @@ -495,6 +529,18 @@ ['IdentifierType', ['int']]]]]]) def test_offsetof(self): + def expand_ref(n): + if isinstance(n, StructRef): + return ['StructRef', expand_ref(n.name), expand_ref(n.field)] + elif isinstance(n, ArrayRef): + return ['ArrayRef', expand_ref(n.name), expand_ref(n.subscript)] + elif isinstance(n, ID): + return ['ID', n.name] + elif isinstance(n, Constant): + return ['Constant', n.type, n.value] + else: + raise TypeError("Unexpected type " + n.__class__.__name__) + e = """ void foo() { int a = offsetof(struct S, p); @@ -512,8 +558,20 @@ self.assertIsInstance(s1.args.exprs[1], ID) s3 = compound.block_items[2].init self.assertIsInstance(s3.args.exprs[1], StructRef) + self.assertEqual(expand_ref(s3.args.exprs[1]), + ['StructRef', + ['StructRef', ['ID', 'p'], ['ID', 'q']], + ['ID', 'r']]) s4 = compound.block_items[3].init self.assertIsInstance(s4.args.exprs[1], ArrayRef) + self.assertEqual(expand_ref(s4.args.exprs[1]), + ['ArrayRef', + ['ArrayRef', + ['StructRef', + ['ArrayRef', ['ID', 'p'], ['Constant', 'int', '5']], + ['ID', 'q']], + ['Constant', 'int', '4']], + ['Constant', 'int', '5']]) def test_compound_statement(self): e = """ @@ -828,6 +886,19 @@ ['Decl', 'd', ['TypeDecl', ['IdentifierType', ['char']]]]]]]]) + def test_struct_with_initial_semi(self): + s1 = """ + struct { + ;int a; + } foo; + """ + s1_ast = self.parse(s1) + self.assertEqual(expand_decl(s1_ast.ext[0]), + ['Decl', 'foo', + ['TypeDecl', ['Struct', None, + [['Decl', 'a', + ['TypeDecl', ['IdentifierType', ['int']]]]]]]]) + def test_anonymous_struct_union(self): s1 = """ union @@ -1240,6 +1311,22 @@ self.assertEqual(self.get_decl_init(d55), ['Constant', 'float', '0xDE.38p0']) + d6 = 'int i = 1;' + self.assertEqual(self.get_decl_init(d6), + ['Constant', 'int', '1']) + + d61 = 'long int li = 1l;' + self.assertEqual(self.get_decl_init(d61), + ['Constant', 'long int', '1l']) + + d62 = 'unsigned int ui = 1u;' + self.assertEqual(self.get_decl_init(d62), + ['Constant', 'unsigned int', '1u']) + + d63 = 'unsigned long long int ulli = 1LLU;' + self.assertEqual(self.get_decl_init(d63), + ['Constant', 'unsigned long long int', '1LLU']) + def test_decl_named_inits(self): d1 = 'int a = {.k = 16};' self.assertEqual(self.get_decl_init(d1), @@ -1407,6 +1494,10 @@ void main() { #pragma foo for(;;) {} + #pragma baz + { + int i = 0; + } #pragma } struct s { @@ -1423,12 +1514,16 @@ self.assertEqual(s1_ast.ext[1].body.block_items[0].coord.line, 4) self.assertIsInstance(s1_ast.ext[1].body.block_items[2], Pragma) - self.assertEqual(s1_ast.ext[1].body.block_items[2].string, '') + self.assertEqual(s1_ast.ext[1].body.block_items[2].string, 'baz') self.assertEqual(s1_ast.ext[1].body.block_items[2].coord.line, 6) + self.assertIsInstance(s1_ast.ext[1].body.block_items[4], Pragma) + self.assertEqual(s1_ast.ext[1].body.block_items[4].string, '') + self.assertEqual(s1_ast.ext[1].body.block_items[4].coord.line, 10) + self.assertIsInstance(s1_ast.ext[2].type.type.decls[0], Pragma) self.assertEqual(s1_ast.ext[2].type.type.decls[0].string, 'baz') - self.assertEqual(s1_ast.ext[2].type.type.decls[0].coord.line, 9) + self.assertEqual(s1_ast.ext[2].type.type.decls[0].coord.line, 13) def test_pragmacomp_or_statement(self): s1 = r''' @@ -1475,8 +1570,10 @@ self.assertIsInstance(s1_ast.ext[0].body.block_items[4].iftrue.block_items[1], Assignment) self.assertIsInstance(s1_ast.ext[0].body.block_items[5], Switch) self.assertIsInstance(s1_ast.ext[0].body.block_items[5].stmt.stmts[0], Compound) - self.assertIsInstance(s1_ast.ext[0].body.block_items[5].stmt.stmts[0].block_items[0], Pragma) - self.assertIsInstance(s1_ast.ext[0].body.block_items[5].stmt.stmts[0].block_items[1], Assignment) + self.assertIsInstance(s1_ast.ext[0].body.block_items[5].stmt.stmts[0].block_items[0], + Pragma) + self.assertIsInstance(s1_ast.ext[0].body.block_items[5].stmt.stmts[0].block_items[1], + Assignment) class TestCParser_whole_code(TestCParser_base): @@ -1719,6 +1816,7 @@ switch = ps1.ext[0].body.block_items[0] block = switch.stmt.block_items + self.assertEqual(len(block), 4) assert_case_node(block[0], '10') self.assertEqual(len(block[0].stmts), 3) assert_case_node(block[1], '20') @@ -1746,6 +1844,7 @@ switch = ps2.ext[0].body.block_items[0] block = switch.stmt.block_items + self.assertEqual(len(block), 5) assert_default_node(block[0]) self.assertEqual(len(block[0].stmts), 2) assert_case_node(block[1], '10') @@ -1757,6 +1856,18 @@ assert_case_node(block[4], '40') self.assertEqual(len(block[4].stmts), 1) + s3 = r''' + int foo(void) { + switch (myvar) { + } + return 0; + } + ''' + ps3 = self.parse(s3) + switch = ps3.ext[0].body.block_items[0] + + self.assertEqual(switch.stmt.block_items, []) + def test_for_statement(self): s2 = r''' void x(void) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/pycparser-2.19/utils/fake_libc_include/_fake_defines.h new/pycparser-2.20/utils/fake_libc_include/_fake_defines.h --- old/pycparser-2.19/utils/fake_libc_include/_fake_defines.h 2018-04-13 05:18:12.000000000 +0200 +++ new/pycparser-2.20/utils/fake_libc_include/_fake_defines.h 2019-12-21 15:41:30.000000000 +0100 @@ -199,3 +199,23 @@ #define va_end(_list) #endif + +/* Vectors */ +#define __m128 int +#define __m128_u int +#define __m128d int +#define __m128d_u int +#define __m128i int +#define __m128i_u int +#define __m256 int +#define __m256_u int +#define __m256d int +#define __m256d_u int +#define __m256i int +#define __m256i_u int +#define __m512 int +#define __m512_u int +#define __m512d int +#define __m512d_u int +#define __m512i int +#define __m512i_u int diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/pycparser-2.19/utils/fake_libc_include/emmintrin.h new/pycparser-2.20/utils/fake_libc_include/emmintrin.h --- old/pycparser-2.19/utils/fake_libc_include/emmintrin.h 1970-01-01 01:00:00.000000000 +0100 +++ new/pycparser-2.20/utils/fake_libc_include/emmintrin.h 2019-08-24 14:56:59.000000000 +0200 @@ -0,0 +1,2 @@ +#include "_fake_defines.h" +#include "_fake_typedefs.h" diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/pycparser-2.19/utils/fake_libc_include/immintrin.h new/pycparser-2.20/utils/fake_libc_include/immintrin.h --- old/pycparser-2.19/utils/fake_libc_include/immintrin.h 1970-01-01 01:00:00.000000000 +0100 +++ new/pycparser-2.20/utils/fake_libc_include/immintrin.h 2019-08-24 14:56:59.000000000 +0200 @@ -0,0 +1,2 @@ +#include "_fake_defines.h" +#include "_fake_typedefs.h" diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/pycparser-2.19/utils/fake_libc_include/smmintrin.h new/pycparser-2.20/utils/fake_libc_include/smmintrin.h --- old/pycparser-2.19/utils/fake_libc_include/smmintrin.h 1970-01-01 01:00:00.000000000 +0100 +++ new/pycparser-2.20/utils/fake_libc_include/smmintrin.h 2019-08-24 14:56:59.000000000 +0200 @@ -0,0 +1,2 @@ +#include "_fake_defines.h" +#include "_fake_typedefs.h"