Bobby Bruce has submitted this change. (
https://gem5-review.googlesource.com/c/public/gem5/+/63432?usp=email )
Change subject: python,stdlib: Add multiprocessing module
......................................................................
python,stdlib: Add multiprocessing module
This changeset replicates some of the multiprocessing module
implementation from the python standard library in gem5. The goal of
this and following changesets is to enable users to use a *single* set
of python scripts to run and analyze a suite of gem5 simulations.
We must reimplement some of the multiprocessing module becaue it is not
flexible enough to allow for customized command line parameter to the
"python" executable (gem5 in our case). To get around this, I extended
the Process and context objects to be gem5 specific.
The next steps is to wrap the Process and Pool types with gem5-specific
versions that will improve their usability for our needs. With this
changeset, these objects are usable, but it will require significant
user effort to reach the goal of running/analyzing many different gem5
simulations.
There are some limitation:
- The pool will only work if the max tasks per child is 1
- The functions that are executed must come from another module
As an example, the following code should work after applying this
change.
test.py:
```python
from gem5.utils.multiprocessing import Process, Pool
from sim import info, run_sim
if __name__ == '__m5_main__' or __name__ == '__main__':
info('main line')
p1 = Process(target=run_sim, args=('bob',))
p2 = Process(target=run_sim, args=('jane',))
p1.start()
p2.start()
p2.join()
p1.join()
with Pool(processes=4, maxtasksperchild=1) as pool:
pool.map(run_sim, range(10))
```
sim.py:
```
import os
def info(title):
print(title)
print('module name:', __name__)
print('parent process:', os.getppid())
print('process id:', os.getpid())
def run_sim(name):
info('function g')
from gem5.prebuilt.demo.x86_demo_board import X86DemoBoard
from gem5.resources.resource import Resource
from gem5.simulate.simulator import Simulator
board = X86DemoBoard()
board.set_kernel_disk_workload(
kernel=Resource("x86-linux-kernel-5.4.49"),
disk_image=Resource("x86-ubuntu-18.04-img"),
)
simulator = Simulator(board=board)
simulator.run(max_ticks=10000000)
```
Change-Id: I4348ebaa75d006949ec96e732f5dc2a5173c6048
Signed-off-by: Jason Lowe-Power <[email protected]>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/63432
Tested-by: kokoro <[email protected]>
Maintainer: Jason Lowe-Power <[email protected]>
Reviewed-by: Bobby Bruce <[email protected]>
Reviewed-by: Richard Cooper <[email protected]>
---
M src/python/SConscript
A src/python/gem5/utils/multiprocessing/README.md
A src/python/gem5/utils/multiprocessing/__init__.py
A src/python/gem5/utils/multiprocessing/_command_line.py
A src/python/gem5/utils/multiprocessing/context.py
A src/python/gem5/utils/multiprocessing/popen_spawn_gem5.py
6 files changed, 459 insertions(+), 0 deletions(-)
Approvals:
Jason Lowe-Power: Looks good to me, approved
Bobby Bruce: Looks good to me, approved
kokoro: Regressions pass
Richard Cooper: Looks good to me, but someone else must approve
diff --git a/src/python/SConscript b/src/python/SConscript
index cbf88c9..6e0f6d7 100644
--- a/src/python/SConscript
+++ b/src/python/SConscript
@@ -241,6 +241,14 @@
PySource('gem5.utils', 'gem5/utils/filelock.py')
PySource('gem5.utils', 'gem5/utils/override.py')
PySource('gem5.utils', 'gem5/utils/requires.py')
+PySource('gem5.utils.multiprocessing',
+ 'gem5/utils/multiprocessing/__init__.py')
+PySource('gem5.utils.multiprocessing',
+ 'gem5/utils/multiprocessing/_command_line.py')
+PySource('gem5.utils.multiprocessing',
+ 'gem5/utils/multiprocessing/context.py')
+PySource('gem5.utils.multiprocessing',
+ 'gem5/utils/multiprocessing/popen_spawn_gem5.py')
PySource('', 'importer.py')
PySource('m5', 'm5/__init__.py')
diff --git a/src/python/gem5/utils/multiprocessing/README.md
b/src/python/gem5/utils/multiprocessing/README.md
new file mode 100644
index 0000000..da2116c
--- /dev/null
+++ b/src/python/gem5/utils/multiprocessing/README.md
@@ -0,0 +1,71 @@
+# gem5's wrapper around python multiprocessing
+
+This module wraps python's multiprocessing module so that it works with
gem5.
+The multiprocessing module creates new python processes, but there is no
way to customize the way these processes are created.
+This wrapper extends the python multiprocessing to support passing new
arguments to the python (or gem5 in this case) executable when a new
process is created.
+
+This code replicates some of the multiprocessing module implementation
from the python standard library in gem5.
+The goal of this code is to enable users to use a *single* set of python
scripts to run and analyze a suite of gem5 simulations.
+
+We must reimplement some of the multiprocessing module because it is not
flexible enough to allow for customized command line parameter to
the "python" executable (gem5 in our case).
+To get around this, I extended the Process and context objects to be gem5
specific.
+
+The next steps is to wrap the Process and Pool types with gem5-specific
versions that will improve their usability for our needs.
+With this changeset, these objects are usable, but it will require
significant user effort to reach the goal of running/analyzing many
different gem5 simulations.
+
+## Example use
+
+test.py:
+
+```python
+from gem5.utils.multiprocessing import Process, Pool
+from sim import info, run_sim
+if __name__ == '__m5_main__' or __name__ == '__main__':
+ info('main line')
+ p1 = Process(target=run_sim, args=('bob',))
+ p2 = Process(target=run_sim, args=('jane',))
+ p1.start()
+ p2.start()
+ p2.join()
+ p1.join()
+ with Pool(processes=4, maxtasksperchild=1) as pool:
+ pool.map(run_sim, range(10))
+```
+
+sim.py:
+
+```python
+import os
+def info(title):
+ print(title)
+ print('module name:', __name__)
+ print('parent process:', os.getppid())
+ print('process id:', os.getpid())
+def run_sim(name):
+ info('function g')
+ from gem5.prebuilt.demo.x86_demo_board import X86DemoBoard
+ from gem5.resources.resource import Resource
+ from gem5.simulate.simulator import Simulator
+ board = X86DemoBoard()
+ board.set_kernel_disk_workload(
+ kernel=Resource("x86-linux-kernel-5.4.49"),
+ disk_image=Resource("x86-ubuntu-18.04-img"),
+ )
+ simulator = Simulator(board=board)
+ simulator.run(max_ticks=10000000)
+```
+
+Then, you can run `gem5 test.py`.
+This will execute `run_sim` 12 times.
+The first two will run in parallel, then the last 10 will run in parallel
with up to 4 running at once.
+
+## Limitations
+
+- This only supports the spawn context. This is important because we need
a fresh gem5 process for every subprocess.
+- When using `Pool`, the `maxtasksperchild` must be 1.
+- Process synchronization (queues, pipes, etc.) hasn't been tested
+- Functions that are used to execute in the subprocess must be imported
from another module. In other words, we cannot pickle functions in the
main/runner module.
+
+## Implementation notes
+
+- The `_start_method` must be `None` for the `Spawn_gem5Process` class.
Otherwise, in `_bootstrap` in the `BaseProcess` it will try to force the
`_start_method` to be gem5-specific, which the `multiprocessing` module
doesn't understand.
diff --git a/src/python/gem5/utils/multiprocessing/__init__.py
b/src/python/gem5/utils/multiprocessing/__init__.py
new file mode 100644
index 0000000..680aeac
--- /dev/null
+++ b/src/python/gem5/utils/multiprocessing/__init__.py
@@ -0,0 +1,33 @@
+# Copyright (c) 2022 The Regents of The University of California
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met: redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer;
+# redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution;
+# neither the name of the copyright holders nor the names of its
+# contributors may be used to endorse or promote products derived from
+# this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+from .context import Process
+
+from .context import gem5Context
+
+Pool = gem5Context().Pool
+
+__all__ = ["Process", "Pool"]
diff --git a/src/python/gem5/utils/multiprocessing/_command_line.py
b/src/python/gem5/utils/multiprocessing/_command_line.py
new file mode 100644
index 0000000..f682775
--- /dev/null
+++ b/src/python/gem5/utils/multiprocessing/_command_line.py
@@ -0,0 +1,101 @@
+# Copyright (c) 2022 The Regents of The University of California
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met: redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer;
+# redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution;
+# neither the name of the copyright holders nor the names of its
+# contributors may be used to endorse or promote products derived from
+# this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+"""
+This file contains extensions of the multiprocessing module to be used
with gem5.
+Specifically, it contains the code to produce the command line for spawned
processes.
+Some code inspired by the Python standard library implementation of the
+multiprocessing module (i.e., cpython/Lib/multiprocessing/).
+"""
+
+import sys
+from multiprocessing import spawn, util
+
+
+def _gem5_args_for_multiprocessing(name):
+ from m5 import options
+
+ # Options that are disallowed with multiprocessing
+ disallowed = [
+ options.build_info,
+ options.copyright,
+ options.readme,
+ options.interactive,
+ options.pdb,
+ options.verbose,
+ options.debug_break,
+ options.debug_help,
+ options.debug_flags,
+ options.debug_start,
+ options.debug_end,
+ options.debug_ignore,
+ options.list_sim_objects,
+ ]
+ if any(disallowed):
+ raise Exception(
+ f"Disallowed option for multiprocessing. "
+ f"See {__file__} for details."
+ )
+
+ # Options not forwarded:
+ # --allow-remote-connections, --listener-mode, --dump-config,
--json-config
+ # --dot-config, --dot-dvfs-config, --debug-file, --remote-gdb-port, -c
+
+ arguments = [
+ f"--outdir={options.outdir}/{name}",
+ f"--stdout-file={options.stdout_file}",
+ f"--stderr-file={options.stderr_file}",
+ f"--stats-file={options.stats_file}",
+ ]
+ if options.redirect_stdout:
+ arguments.append("--redirect-stdout")
+ if options.redirect_stderr:
+ arguments.append("--redirect-stderr")
+ if options.silent_redirect:
+ arguments.append("--silent-redirect")
+ if options.path:
+ arguments.append(f"--path={':'.join(options.path)}")
+ if options.quiet:
+ arguments.append("--quiet")
+
+ return arguments
+
+
+def get_command_line(name, **kwds):
+ """
+ Returns prefix of command line used for spawning a child process
+ """
+ if getattr(sys, "frozen", False):
+ return [sys.executable, "--multiprocessing-fork"] + [
+ "%s=%r" % item for item in kwds.items()
+ ]
+ else:
+ prog = "from multiprocessing.spawn import spawn_main;
spawn_main(%s)"
+ prog %= ", ".join("%s=%r" % item for item in kwds.items())
+ opts = util._args_from_interpreter_flags()
+ opts.extend(_gem5_args_for_multiprocessing(name))
+ exe = spawn.get_executable()
+ return [exe] + opts + ["-c", prog, "--multiprocessing-fork"]
diff --git a/src/python/gem5/utils/multiprocessing/context.py
b/src/python/gem5/utils/multiprocessing/context.py
new file mode 100644
index 0000000..2108bc6
--- /dev/null
+++ b/src/python/gem5/utils/multiprocessing/context.py
@@ -0,0 +1,75 @@
+# Copyright (c) 2022 The Regents of The University of California
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met: redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer;
+# redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution;
+# neither the name of the copyright holders nor the names of its
+# contributors may be used to endorse or promote products derived from
+# this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+"""
+This file contains extensions of the multiprocessing module to be used
with gem5
+Some code inspired by the Python standard library implementation of the
+multiprocessing module (i.e., cpython/Lib/multiprocessing/).
+"""
+
+from multiprocessing import context, process
+from multiprocessing.context import DefaultContext
+
+# The `_start_method` must be `None` for the `Spawn_gem5Process` class.
+# Otherwise, in `_bootstrap` in the `BaseProcess` it will try to force the
+# `_start_method` to be gem5-specific, which the `multiprocessing` module
+# doesn't understand.
+class Spawn_gem5Process(process.BaseProcess):
+ _start_method = None
+
+ @staticmethod
+ def _Popen(process_obj):
+ from .popen_spawn_gem5 import Popen
+
+ return Popen(process_obj)
+
+
+class Process(process.BaseProcess):
+ _start_method = None
+
+ @staticmethod
+ def _Popen(process_obj):
+ return _default_context.get_context().Process._Popen(process_obj)
+
+
+class gem5Context(context.BaseContext):
+ _name = "spawn_gem5"
+ Process = Spawn_gem5Process
+
+ def get_context(self, method=None):
+ if method is None:
+ return self
+ try:
+ ctx = _concrete_contexts[method]
+ except KeyError:
+ raise ValueError("cannot find context for %r" % method) from
None
+ ctx._check_available()
+ return ctx
+
+
+_concrete_contexts = {"spawn_gem5": gem5Context()}
+
+_default_context = DefaultContext(_concrete_contexts["spawn_gem5"])
diff --git a/src/python/gem5/utils/multiprocessing/popen_spawn_gem5.py
b/src/python/gem5/utils/multiprocessing/popen_spawn_gem5.py
new file mode 100644
index 0000000..13fb336
--- /dev/null
+++ b/src/python/gem5/utils/multiprocessing/popen_spawn_gem5.py
@@ -0,0 +1,95 @@
+# Copyright (c) 2022 The Regents of The University of California
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met: redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer;
+# redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution;
+# neither the name of the copyright holders nor the names of its
+# contributors may be used to endorse or promote products derived from
+# this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+"""
+This file contains extensions of the multiprocessing module to be used
with gem5.
+Specifically, it contains the code to spawn a new gem5 process with Popen.
+Some code is from the Python standard library implementation of the
+multiprocessing module (i.e., cpython/Lib/multiprocessing/).
+"""
+
+import io
+import os
+
+from multiprocessing.context import reduction, set_spawning_popen
+from multiprocessing import popen_spawn_posix
+from multiprocessing import spawn
+from multiprocessing import util
+
+from ._command_line import get_command_line
+
+__all__ = ["Popen"]
+
+
+class Popen(popen_spawn_posix.Popen):
+ method = "spawn_gem5"
+
+ def __init__(self, process_obj):
+ super().__init__(process_obj)
+
+ # Copyright (c) 2001-2022 Python Software Foundation; All Rights
Reserved
+ # from cpython/Lib/multiprocessing/popen_spawn_posix.py
+ def _launch(self, process_obj):
+ from multiprocessing import resource_tracker
+
+ tracker_fd = resource_tracker.getfd()
+ self._fds.append(tracker_fd)
+ prep_data = spawn.get_preparation_data(process_obj._name)
+ fp = io.BytesIO()
+ set_spawning_popen(self)
+ try:
+ reduction.dump(prep_data, fp)
+ reduction.dump(process_obj, fp)
+ finally:
+ set_spawning_popen(None)
+
+ parent_r = child_w = child_r = parent_w = None
+ try:
+ parent_r, child_w = os.pipe()
+ child_r, parent_w = os.pipe()
+ # Note: This next line is the only modification
+ cmd = get_command_line(
+ tracker_fd=tracker_fd,
+ pipe_handle=child_r,
+ name=process_obj.name,
+ )
+ self._fds.extend([child_r, child_w])
+ self.pid = util.spawnv_passfds(
+ spawn.get_executable(), cmd, self._fds
+ )
+ self.sentinel = parent_r
+ with open(parent_w, "wb", closefd=False) as f:
+ f.write(fp.getbuffer())
+ finally:
+ fds_to_close = []
+ for fd in (parent_r, parent_w):
+ if fd is not None:
+ fds_to_close.append(fd)
+ self.finalizer = util.Finalize(self, util.close_fds,
fds_to_close)
+
+ for fd in (child_r, child_w):
+ if fd is not None:
+ os.close(fd)
--
To view, visit
https://gem5-review.googlesource.com/c/public/gem5/+/63432?usp=email
To unsubscribe, or for help writing mail filters, visit
https://gem5-review.googlesource.com/settings
Gerrit-Project: public/gem5
Gerrit-Branch: develop
Gerrit-Change-Id: I4348ebaa75d006949ec96e732f5dc2a5173c6048
Gerrit-Change-Number: 63432
Gerrit-PatchSet: 4
Gerrit-Owner: Jason Lowe-Power <[email protected]>
Gerrit-Reviewer: Andreas Sandberg <[email protected]>
Gerrit-Reviewer: Bobby Bruce <[email protected]>
Gerrit-Reviewer: Jason Lowe-Power <[email protected]>
Gerrit-Reviewer: Jason Lowe-Power <[email protected]>
Gerrit-Reviewer: Richard Cooper <[email protected]>
Gerrit-Reviewer: kokoro <[email protected]>
Gerrit-MessageType: merged
_______________________________________________
gem5-dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
