Richard Cooper has submitted this change. (
https://gem5-review.googlesource.com/c/public/gem5/+/70237?usp=email )
Change subject: scons: Fix gem5 Python3.11 build.
......................................................................
scons: Fix gem5 Python3.11 build.
The code generation in gem5's build system requires the use of Regular
Expression flags when defining the regular expressions used for
tokenization. However, the Python Lex-Yacc (PLY) [1] library used by
gem5 does not allow the user sufficient control of the flags for RE
compilation.
Previously, gem5 used inline RE flags to control RE compilation.
However, from Python 3.11, inline RE flags must be at the start of the
RE string. Because PLY wraps the user supplied RE strings before
compilation, there is no way for the user to supply a RE string with
the inline flag at the start. This makes gem5 incompatible with Python
3.11 when using PLY.
This change modifies gem5's build files to patch `re.compile` with a
wrapped version that can handle embedded flags anywhere in the RE
string, for all current versions of Python. The patched version
re-formats the user supplied RE string to convert inline RE flags to
explicit RE flags.
This patch is intended as a temporary stop-gap until PLY can be fixed
upstream.
See the gem5 Issue Tracker [2] for more details.
[1] https://github.com/dabeaz/ply
[2] https://gem5.atlassian.net/browse/GEM5-1321
Change-Id: I3ab371f2e5cf267c0a89caaf8a2bacfed78545ef
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/70237
Maintainer: Bobby Bruce <bbr...@ucdavis.edu>
Maintainer: Jason Lowe-Power <power...@gmail.com>
Reviewed-by: Bobby Bruce <bbr...@ucdavis.edu>
Reviewed-by: Boris Shingarov <shinga...@labware.com>
Tested-by: kokoro <noreply+kok...@google.com>
---
M SConstruct
M site_scons/gem5_scons/__init__.py
2 files changed, 49 insertions(+), 2 deletions(-)
Approvals:
Boris Shingarov: Looks good to me, approved
Bobby Bruce: Looks good to me, approved; Looks good to me, approved
kokoro: Regressions pass
Jason Lowe-Power: Looks good to me, approved
diff --git a/SConstruct b/SConstruct
index b784a04..e09f0d5 100755
--- a/SConstruct
+++ b/SConstruct
@@ -1,6 +1,6 @@
# -*- mode:python -*-
-# Copyright (c) 2013, 2015-2020 ARM Limited
+# Copyright (c) 2013, 2015-2020, 2023 ARM Limited
# All rights reserved.
#
# The license below extends only to copyright in the software and shall
@@ -171,6 +171,10 @@
Export('MakeAction')
+# Patch re.compile to support inline flags anywhere within a RE
+# string. Required to use PLY with Python 3.11+.
+gem5_scons.patch_re_compile_for_inline_flags()
+
########################################################################
#
# Set up the main build environment.
diff --git a/site_scons/gem5_scons/__init__.py
b/site_scons/gem5_scons/__init__.py
index 7214876..6d6226c 100644
--- a/site_scons/gem5_scons/__init__.py
+++ b/site_scons/gem5_scons/__init__.py
@@ -1,4 +1,4 @@
-# Copyright (c) 2013, 2015-2017 ARM Limited
+# Copyright (c) 2013, 2015-2017, 2023 ARM Limited
# All rights reserved.
#
# The license below extends only to copyright in the software and shall
@@ -302,6 +302,48 @@
return pickle.loads(node.read())
+def patch_re_compile_for_inline_flags():
+ """Patch `re.compile` with a version that can handle RE strings with
+ inline flags anywhere within them. This is required to use PLY
+ with Python 3.11+.
+
+ """
+
+ import re
+ from functools import partial
+
+ def _inline_flag_aware_re_compile(re_compile, re_str, flags=0x0):
+ """Provide an alternative implementation of `re.compile` that
allows
+ inline flags that are not at the start of the regular
+ expression string.
+
+ From Python 3.11, the `re` module only supports inline flags
+ at the start of the RE string. This makes it impossible to add
+ flags to the Lexer strings when using PLY, because PLY embeds
+ the user supplied token REs, and does not provide sufficient
+ control of the `flags` argument.
+
+ """
+ _flags_map = {
+ ("(?a)", b"(?a)"): re.ASCII,
+ ("(?i)", b"(?i)"): re.IGNORECASE,
+ ("(?L)", b"(?L)"): re.LOCALE,
+ ("(?m)", b"(?m)"): re.MULTILINE,
+ ("(?s)", b"(?s)"): re.DOTALL,
+ ("(?x)", b"(?x)"): re.VERBOSE,
+ }
+ for (pattern_s, pattern_b), flag in _flags_map.items():
+ pattern = pattern_b if isinstance(re_str, bytes) else pattern_s
+ replacement = b"" if isinstance(re_str, bytes) else ""
+ if pattern in re_str:
+ flags |= flag
+ re_str = re_str.replace(pattern, replacement)
+ return re_compile(re_str, flags)
+
+ # Patch the default `re.compile`
+ re.compile = partial(_inline_flag_aware_re_compile, re.compile)
+
+
__all__ = [
"Configure",
"EnvDefaults",
@@ -312,4 +354,5 @@
"MakeActionTool",
"ToValue",
"FromValue",
+ "patch_re_compile_for_inline_flags",
]
--
To view, visit
https://gem5-review.googlesource.com/c/public/gem5/+/70237?usp=email
To unsubscribe, or for help writing mail filters, visit
https://gem5-review.googlesource.com/settings
Gerrit-MessageType: merged
Gerrit-Project: public/gem5
Gerrit-Branch: develop
Gerrit-Change-Id: I3ab371f2e5cf267c0a89caaf8a2bacfed78545ef
Gerrit-Change-Number: 70237
Gerrit-PatchSet: 3
Gerrit-Owner: Richard Cooper <richard.coo...@arm.com>
Gerrit-Reviewer: Bobby Bruce <bbr...@ucdavis.edu>
Gerrit-Reviewer: Boris Shingarov <shinga...@labware.com>
Gerrit-Reviewer: Gabe Black <gabe.bl...@gmail.com>
Gerrit-Reviewer: Jason Lowe-Power <ja...@lowepower.com>
Gerrit-Reviewer: Jason Lowe-Power <power...@gmail.com>
Gerrit-Reviewer: Richard Cooper <richard.coo...@arm.com>
Gerrit-Reviewer: kokoro <noreply+kok...@google.com>
_______________________________________________
gem5-dev mailing list -- gem5-dev@gem5.org
To unsubscribe send an email to gem5-dev-le...@gem5.org
