Hey Gabe, Just so we're on the same page, I'd like to take a moment to outline how I understand this problem/solution. Please let me know if I've understood correctly:
We essentially have three components of the project which interpret gem5 Python code: The gem5 binary, the Marshal binary, and the Scons build system. The gem5 and Marshal binaries both link to the Python library, so we have no problems there; they should interpret things identically. The problematic entity is Scons which will use the host python executable when processing Simobjects et al. at compile time. Your solution is to offload the relevant Scons gem5 Python interpretations/auto-generations to the Marshal binary where they'll use the Python library. Is this correct? If so, I support this move as it makes things considerably easier to work with. While there is the small `--without-python` drawback, I don't personally see this being a big problem for the vast majority of users. I do have a few of questions (I'm not sure if you an answer them, but may be something to think about): 1) Will this make it easier to embed configuration scripts (such as those found in the `configs` directory) into the gem5 binary? 2) Will this change the way we add new SimObjects or will it be more-or-less identical as before? 3) One of our goals moving forward is to enforce more Python type annotations so we can use tools such as MyPy. We've had some problems running this with this due to embedding (i.e., we can't just point MyPy at a SimObject Python file). Do you have any idea if this could be made easier? Kind regards, Bobby -- Dr. Bobby R. Bruce Room 3050, Kemper Hall, UC Davis Davis, CA, 95616 web: https://www.bobbybruce.net On Fri, Jul 30, 2021 at 11:05 PM Gabe Black via gem5-dev <gem5-dev@gem5.org> wrote: > Hi folks, particularly Bobby who made a change related to this fairly > recently. Please offer some feedback, or I'll be forced to assume your > silence is tacit agreement. > > Gabe > > On Wed, Jul 28, 2021 at 8:30 PM Gabe Black <gabe.bl...@gmail.com> wrote: > >> Bump... >> >> On Sat, Jul 24, 2021 at 1:16 AM Gabe Black <gabe.bl...@gmail.com> wrote: >> >>> Hi folks. >>> >>> I think often when we think of how python is used by gem5, many people >>> (myself included) tend to think of it as a monolithic entity, or in other >>> words that there is just *the* python that gets used by everything. The >>> real situation is a little more complicated than that. >>> >>> On the one hand, we have "the" version of the python interpreter itself, >>> which is used to directly run python scripts. This is again an >>> approximation because we could, for instance, have one version of python >>> running SCons itself, and another version running scripts gem5 provides, >>> but for simplicity let's assume that's not the case, or at least that it >>> doesn't matter. >>> >>> Then, we have the python libraries which get built into gem5 and provide >>> it's embedded interpreter. This may not exist at all on the target system, >>> or it could exist but be a different version than the one which you'd get >>> by running the command line "python foo.py". >>> >>> >>> **Marshal binary** >>> >>> The first place this has caused complication is where gem5's python code >>> is packaged up by the build system so that it can be embedded into gem5 in >>> either library form or executable form. Through digging around in the >>> repository, I've found that a long, long time ago, gem5 was only a binary, >>> and the python code was embedded into it by tacking a zip archive onto the >>> end where gem5 could find it. Apparently this doesn't work for libraries, >>> and so the python code needed to be bundled up and more formally injected >>> into the binary as data. Today done by turning the python code into a >>> binary representation, compressing it into a zip file, and then building >>> the resulting file into a byte array declared in c++. >>> >>> There are, to the first order, two main ways to serialize python "stuff" >>> into a string/byte array, pickling or marshalling. Both have their >>> advantages and disadvantages, partially covered here: >>> >>> https://docs.python.org/3/library/pickle.html#module-pickle >>> >>> The main deal breaker disadvantage of the pickle module is that it does >>> *not* handle code, and so all the code inside our python modules would be >>> lost, leaving only the data. The main disadvantage of marshal is that there >>> is *no* guarantee that it will be compatible between different versions of >>> python, since it's intended as a mostly internal representation, for, for >>> instance ".pyc" files. >>> >>> To solve this problem, Andreas added a "marshal" binary which is written >>> in c++, and includes the python interpreter like gem5 itself would. It runs >>> a small embedded script to use the appropriate version of the marshal >>> module to serialize the code we want. Since the code runs under the same >>> version of the interpreter as gem5 itself would use, when gem5 tries to use >>> the data it will be compatible. >>> >>> >>> **Collection of SimObject information** >>> >>> In order to figure out what SimObjects there are, we must actually >>> import all the *.py files which contain them that have been declared to the >>> build system. Once they've all been imported, the SimObject base class will >>> have collected a list of all of the new subclasses which can then be >>> processed by subsequent steps, generating params structs, etc. >>> >>> To do that requires a couple pieces of fairly heavy infrastructure >>> within the build system. First, we must, even before any of gem5's c++ >>> components have been built, build enough python infrastructure so that >>> SimObjects can work more or less like normal and import what they need, >>> etc. To support that, we have a custom and complex custom import handler in >>> the SConscript which pipes around the .py SimObject modules we know about, >>> with a little extra to support m5.defines.buildEnv. Even with this complex >>> infrastructure in place, this only *mostly* covers what SimObjects might >>> try to use. If someone writes a SimObject and puts code in it to import >>> _m5.foo blindly, that *will* blow up, since _m5 is for c++ modules which >>> don't exist and so can't possibly be provided. >>> >>> Second, and perhaps more subtly since we assume certain capabilities in >>> our build system, this means our build system *must* be able to execute >>> python code, because it *must* be able to import these SimObject .py files >>> to know what to build and how to build it. This will make it impossible to >>> ever use a build system which does not allow the execution of arbitrary >>> python as part of it's build flow, without, for instance, some sort of >>> elaborate harness which escapes to an external python interpreter and then >>> passes back complex data from doing the import itself. >>> >>> >>> **--without-python interaction with the gem5 library and binary** >>> >>> There is a fairly fundamental conflict between the gem5 library and the >>> gem5 binary as far as the --without-python flag. At one time, even when the >>> --without-python flag was provided, gem5 would look for the python >>> libraries for the embedded interpreter anyway, and fail if they were >>> absent. This was changed, implying that --without-python means you don't >>> want to use embedded python at all (different from script python, which you >>> need to even run SCons to begin with). Unfortunately, it's currently >>> impossible to build the gem5 binary without python, since it's main >>> function runs as python and it has to run config scripts. That means if you >>> run this command: >>> >>> scons --without-python build/X86/gem5.opt >>> >>> it will fail with an obscure build error and not tell you that that >>> combination is not allowed. >>> >>> When talking specifically about the library, --without-python can make >>> sense, because you're delegating the main function to somebody else, and >>> there is a c++ config *loading* mechanism. >>> >>> Note that I say a c++ config *loading* mechanism, since this mechanism >>> does not create a config from scratch, it simply loads an ini file created >>> earlier in a bootstrapping step. This implies that at least at some point, >>> you were able to build and run a version of gem5 which *did* have the >>> interpreter in it, even if that is not on the current machine. >>> >>> >>> **Incompatibility between script and binary python for SimObjects** >>> >>> Since we have two different pythons, one for scripts and one embedded, >>> it is entirely possible that one version supports some feature which the >>> other doesn't. Even though we require version 3.6 or greater of python, new >>> features are added all the time and may, for instance, have been introduced >>> in version 3.7 or 3.8. It is *not* guaranteed that when importing >>> SimObjects into SCons as part of the build process that we'll get even the >>> same SimObjects period, let alone equivalent definitions of those objects. >>> >>> For example, you could hypothetically check if a certain feature of >>> python was available, and if yes define SimObject A, and if no define >>> SimObject B. >>> >>> >>> **Suggested change** >>> >>> What I think might help solve this problem is to *require* there to be >>> an embeddable python interpreter no matter what, whether you're going to >>> build it into the library, or build the gem5 binary, or not. >>> >>> Then, we would expand the "marshal" program so that instead of being >>> custom made to marshal our python blobs, it would run arbitrary python code >>> using the built in interpreter like gem5 would. We could then use the same >>> import handler as real gem5, removing one extra copy. We could also teach >>> it to import the blobs we've already made with earlier invocations using >>> the wrapper itself, or put code in the script we run inside it which does >>> the importing. >>> >>> Finally, we would require *all* code which is targeted at running inside >>> gem5 to be run by this interpreter wrapper. The wrapper would be able to >>> marshal python modules just like it does today, although with the small >>> script inside it provided from outside. It would also be able to import all >>> the SimObjects as blobs and report what SimObjects exist, and then it could >>> output the C++ code which defines the Param structs, etc. >>> >>> >>> **Drawback** >>> >>> The biggest but only significant drawback I see to this approach is that >>> this will mean having python libraries around for embedded python will no >>> longer be optional, at least during the build process. >>> >>> >>> **Benefits** >>> >>> Guaranteed compatibility for built python code. Simplified build system >>> (not running arbitrary target python in SCons itself, more rules >>> oriented/less sequentially scripted build process). Reduced dependence on >>> running arbitrary python as part of the build process. >>> >>> Gabe >>> >> _______________________________________________ > gem5-dev mailing list -- gem5-dev@gem5.org > To unsubscribe send an email to gem5-dev-le...@gem5.org > %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s
_______________________________________________ gem5-dev mailing list -- gem5-dev@gem5.org To unsubscribe send an email to gem5-dev-le...@gem5.org %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s
