[Python-ideas] Re: Looking for people interested in a Python register virtual machine project
On Mon, Mar 22, 2021 at 5:37 PM Ben Rudiak-Gould wrote: > > On Sun, Mar 21, 2021 at 11:10 PM Chris Angelico wrote: >> >> At what point does the process_objects list cease to be referenced? >> After the last visible use of it, or at the end of the function? > > > In Python as it stands, at the end of the function, as you say. > > Skip Montanaro's PEP suggested that in his register machine, locals would be > dereferenced after their last visible use. I don't think that's intrinsically > a bad idea, but it's not backward compatible. The thing with the process > objects was just an example of currently working code that would break. > > The example has nothing to do with PyQt5 really. I just happen to know that > QProcess objects kill the controlled process when they're collected. I think > it's a bad design, but that's the way it is. > > Another example would be something like > > td = tempfile.TemporaryDirectory() > p = subprocess.Popen([..., td.name, ...], ...) > p.wait() > > where the temporary directory will hang around until the process exits with > current semantics, but not if td is deleted after the second line. Of course > you should use a with statement in this kind of situation, but there's > probably a lot of code that doesn't. > Thanks for the clarification. I think the tempfile example will be a lot easier to explain this with, especially since it requires only the stdlib and isn't implying that there's broken code in a third-party library. I don't like this. In a bracey language (eg C++), you can declare that a variable should expire prior to the end of the function by including it in a set of braces; in Python, you can't do that, and the normal idiom is to reassign the variable or 'del' it. Changing the semantics of when variables cease to be referenced could potentially break a LOT of code. Maybe, if Python were a brand new language today, you could define the semantics that way (and require "with" blocks for anything that has user-visible impact, reserving __del__ for resource disposal ONLY), but as it is, that's a very very sneaky change that will break code in subtle and hard-to-debug ways. (Not sure why this change needs to go alongside the register-based VM, as it seems to my inexpert mind to be quite orthogonal to it; but whatever, I guess there's a good reason.) ChrisA ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/C3CUQYW3TQGJHC7SP5B4QJXFDV2XTEXB/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Looking for people interested in a Python register virtual machine project
On Sun, Mar 21, 2021 at 11:10 PM Chris Angelico wrote: > At what point does the process_objects list cease to be referenced? > After the last visible use of it, or at the end of the function? In Python as it stands, at the end of the function, as you say. Skip Montanaro's PEP suggested that in his register machine, locals would be dereferenced after their last visible use. I don't think that's intrinsically a bad idea, but it's not backward compatible. The thing with the process objects was just an example of currently working code that would break. The example has nothing to do with PyQt5 really. I just happen to know that QProcess objects kill the controlled process when they're collected. I think it's a bad design, but that's the way it is. Another example would be something like td = tempfile.TemporaryDirectory() p = subprocess.Popen([..., td.name, ...], ...) p.wait() where the temporary directory will hang around until the process exits with current semantics, but not if td is deleted after the second line. Of course you should use a with statement in this kind of situation, but there's probably a lot of code that doesn't. ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/EQ372ZPJWQW2GKCLPGXX2A6VKMRZRB36/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Looking for people interested in a Python register virtual machine project
On Mon, Mar 22, 2021 at 3:14 PM Guido van Rossum wrote: > > On Sun, Mar 21, 2021 at 3:35 PM Chris Angelico wrote: >> >> On Mon, Mar 22, 2021 at 7:49 AM Ben Rudiak-Gould wrote: >> > >> > In the "Object Lifetime" section you say "registers should be cleared upon >> > last reference". That isn't safe, since there can be hidden dependencies >> > on side effects of __del__, e.g.: >> > >> > process_objects = create_pipeline() >> > output_process = process_objects[-1] >> > return output_process.wait() >> > >> > If the process class terminates the process in __del__ (PyQt5's QProcess >> > does), then implicitly deleting process_objects after the second line will >> > break the code. >> > >> >> Hang on hang on hang on. After the second line, there are two >> references to the last object, and one to everything else. (If >> create_pipeline returns two objects, one for each end of the pipe, >> then there are two references to the second one, and one to the >> first.) Even if you dispose of process_objects itself on the basis >> that it's not used any more (which I would disagree with, since it's >> very difficult to manage that well), it shouldn't terminate the >> process, because one of the objects is definitely still alive. > > > In the hypothetical scenario, presumably create_pipeline() returns a list of > process objects, where the process class somehow kills the process when it is > finalized. In that case dropping the last reference to process_objects[0] > would kill the first process in the pipeline. I don't know if that's good API > design, but Ben states that PyQt5 does this, and it could stand in for any > number of other APIs that legitimately destroy an external resource when the > last reference is dropped. (E.g., stdlib temporary files.) > The question is really whether process_objects ceases to exist after the last time it's referenced. I may have misinterpreted the thin example here, but let's just focus on process_objects[0] (hereunder "po0" for simplicity), and assume that there's at least two elements in the list. A list begins to exist somewhere inside create_pipeline(), and at the point where that list is returned, it has a reference to po0. That list is returned, and assigned to process_objects, which we assume is a function-local variable. So the function's locals reference process_objects, which references po0. So far, so good. Then we get a new variable output_process, and we lift something unrelated from the list. Then we call a method on an unrelated object, and return from the function. At what point does the process_objects list cease to be referenced? After the last visible use of it, or at the end of the function? My understanding of Python's semantics is that the list object MUST continue to exist all the way up until the function exits, or wording that another way, that the function's call frame has a reference to ALL of its locals, not just the ones that can visibly be seen to be used. Allowing an object to be disposed of early if there are no future uses of it would be quite surprising. It would be different if, before the return statement, "process_objects = None" were inserted. Then the list would cease to be referenced, and po0 would cease to be referenced, and regardless of the exact type of GC being used, it would be legit to ditch it before the wait() call. If *that* version is broken, then there's a problem with the objects in the list depending on each other in a non-Python-visible way, and that's a bug in the library. Can a PyQT user clarify, please? ChrisA ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/FDS3ZZX6F3CB5BSEF6LC7R4UOHKOFBPO/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Looking for people interested in a Python register virtual machine project
On Sun, Mar 21, 2021 at 3:35 PM Chris Angelico wrote:
> On Mon, Mar 22, 2021 at 7:49 AM Ben Rudiak-Gould
> wrote:
> >
> > In the "Object Lifetime" section you say "registers should be cleared
> upon last reference". That isn't safe, since there can be hidden
> dependencies on side effects of __del__, e.g.:
> >
> > process_objects = create_pipeline()
> > output_process = process_objects[-1]
> > return output_process.wait()
> >
> > If the process class terminates the process in __del__ (PyQt5's QProcess
> does), then implicitly deleting process_objects after the second line will
> break the code.
> >
>
> Hang on hang on hang on. After the second line, there are two
> references to the last object, and one to everything else. (If
> create_pipeline returns two objects, one for each end of the pipe,
> then there are two references to the second one, and one to the
> first.) Even if you dispose of process_objects itself on the basis
> that it's not used any more (which I would disagree with, since it's
> very difficult to manage that well), it shouldn't terminate the
> process, because one of the objects is definitely still alive.
>
In the hypothetical scenario, presumably create_pipeline() returns a list
of process objects, where the process class somehow kills the process when
it is finalized. In that case dropping the last reference to
process_objects[0] would kill the first process in the pipeline. I don't
know if that's good API design, but Ben states that PyQt5 does this, and it
could stand in for any number of other APIs that legitimately destroy an
external resource when the last reference is dropped. (E.g., stdlib
temporary files.)
Curiously, this is about the opposite problem that we would have if we were
to replace the current reference counting scheme with some kind of
traditional garbage collection system.
> This is nothing to do with a register-based VM and everything to do
> with standard Python semantics, so this can't change.
>
Exactly. The link with a register-based VM is that if we replaced the value
stack with temporary local variables (as the "register-based" VM scheme
does), we'd have to decide on the lifetimes of those temporary variables.
Finalizing them when the function returns might extend the lifetimes of
some objects compared to the stack-based scheme. But finalizing all local
variables (temporary or not) as soon as they are no longer needed by
subsequent code could *shorten* the lifetimes, as in the above example.
A reasonable solution would be to leave the lifetimes of explicitly named
locals alone, but use the proposed ("as soon as possible") scheme for
temporary variables. This would appear to match how values on the value
stack are treated. Honestly that's how I read the quoted section of Skip's
proto-PEP (since it explicitly mentions registers). A version of the
example that exhibits the same questionable behavior would be this:
return create_pipeline()[-1].wait()
Presumably this would not work correctly with the PyQt5 process class.
--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
[Python-ideas] Re: Installer/target architecture mismatch
Can you file a bug for this on bpo and add Steve Dower to the nosy list? On Fri, Mar 19, 2021 at 17:04 Dany Lisiansky wrote: > An odd suggestion/request here, hope it's the right place to discuss it. > > So I was trying to install python on the Xbox series S (yup..), so far I > got the embedded x86_64 version for windows to work, however, I was unable > to get packages to install properly. > I'm not familiar enough with Windows to know how the installation is > supposed to be laid out, but I figured it might be just a side affect from > using the embedded version. > Now I need to mention that the Xbox is an odd NT based system, it can run > win32/UWP apps, but it comes with a strict requirement of x86_64 and > therefore has no SysWoW64 to support i386. > Naturally, I tried to use the distributed installer, and that's when I > found out that the x86_64 installer itself was in fact compiled for i386. > I can compile a one-off build for myself and go on with my life, but I > also realized it's an oversight because the installer should never assume > the target system has backwards compatibility for older architectures and > stick to the target architecture throughout the installation phase. > > I know It's an extreme edge case, but I guess it's valid and make sense. > Let me know what you think, and how (or if) I should proceed from here. > > DanyL. > ___ > Python-ideas mailing list -- python-ideas@python.org > To unsubscribe send an email to python-ideas-le...@python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at > https://mail.python.org/archives/list/python-ideas@python.org/message/C7UQTGKJCSHBMIXGRMRDSZJOL3LLQF2Q/ > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido (mobile) ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/BZADOG37KZ3N2KFZNSQXLMECMEWQFAKT/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Looking for people interested in a Python register virtual machine project
On Mon, Mar 22, 2021 at 7:49 AM Ben Rudiak-Gould wrote: > > In the "Object Lifetime" section you say "registers should be cleared upon > last reference". That isn't safe, since there can be hidden dependencies on > side effects of __del__, e.g.: > > process_objects = create_pipeline() > output_process = process_objects[-1] > return output_process.wait() > > If the process class terminates the process in __del__ (PyQt5's QProcess > does), then implicitly deleting process_objects after the second line will > break the code. > Hang on hang on hang on. After the second line, there are two references to the last object, and one to everything else. (If create_pipeline returns two objects, one for each end of the pipe, then there are two references to the second one, and one to the first.) Even if you dispose of process_objects itself on the basis that it's not used any more (which I would disagree with, since it's very difficult to manage that well), it shouldn't terminate the process, because one of the objects is definitely still alive. This is nothing to do with a register-based VM and everything to do with standard Python semantics, so this can't change. ChrisA ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/6ZBFVZJK56B2NA42EIWMGDV5T5RDPWGS/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Looking for people interested in a Python register virtual machine project
In the "Object Lifetime" section you say "registers should be cleared upon last reference". That isn't safe, since there can be hidden dependencies on side effects of __del__, e.g.: process_objects = create_pipeline() output_process = process_objects[-1] return output_process.wait() If the process class terminates the process in __del__ (PyQt5's QProcess does), then implicitly deleting process_objects after the second line will break the code. ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/TZRDILZIQXOEZLMK5HOH2HBUG5JY2ZMZ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Looking for people interested in a Python register virtual machine project
Hello, On Sat, 20 Mar 2021 10:54:10 -0500 Skip Montanaro wrote: > Back in the late 90s (!) I worked on a reimagining of the Python > virtual machine as a register-based VM based on 1.5.2. I got part of > the way with that, but never completed it. In the early 2010s, Victor > Stinner got much further using 3.4 as a base. The idea (and dormant > code) has been laying around in my mind (and computers) these past > couple decades, so I took another swing at it starting in late 2019 > after retirement, mostly as a way to keep my head in the game. While I > got a fair bit of the way, it stalled. I've picked it up and put it > down a number of times in the past year, often needing to resolve > conflicts because of churn in the current Python virtual machine. I guess it should be a good idea to answer what's the scope of this project - is it research one or "production" one? If it's research one, why be concerned with the churn of over-modern CPython versions? Wouldn't it be better to just use some scalable, incremental implementation which would allow to forward-port it to a newer version, if it ever comes to that? Otherwise, if it's "production", who's the "customer" and how they "compensate" you for doing work (chasing the moving target) which is clearly of little interest to you and conflicts with the goal of the project? [] > I started on what could only very generously be called a PEP which you > can read here. It includes some of the history of this work as well as > details about what I've managed to do so far: > > https://github.com/smontanaro/cpython/blob/register2/pep-.rst > > If you think any of this is remotely interesting (whether or not you > think you'd like to help), please have a look at the "PEP". Some comments on it: 1. I find it to be rather weak on motivational part. It's starts with a phrase like: > This PEP proposes the addition of register-based instructions to the > existing Python virtual machine, with the intent that they eventually > replace the existing stack-based opcodes. Sorry, what? The purpose of register-based instructions is to just replace stack-based instructions? That's not what's I'd like to hear as the intro phrase. You probably want to replace one with the other because register-based ones offer some benefit, faster execution perhaps? That's what I'd like to hear instead of "deciphering" that between the lines. > They [2 instruction sets] are almost completely distinct. That doesn't correspond to the mental image I would have. In my list, the 2 sets would be exactly the same, except that stack-based encode argument locations implicitly, while register-based - explicitly. Would be interesting to read (in the following "pep" sections) what makes them "almost completely distinct". > Within a single function only one set of opcodes or the other will > be used at any one time. That would be the opposite of "scalable, incremental" development approach mentioned above. Why not allow 2 sets to freely co-exist, and migrate codegeneration/implement code translation gradually? > ## Motivation I'm not sure the content of the section corresponds much to its title. It jumps from background survey of the different Python VM optimizations to (some) implementation details of register VM - leaving "motivation" somewhere "between the lines". > Despite all that effort, opcodes which do nothing more than move data > onto or off of the stack (LOAD_FAST, LOAD_GLOBAL, etc) still account > for nearly half of all opcodes executed. ... And - you intend to change that with a register VM? In which way and how? As an example, LOAD_GLOBAL isn't going anywhere - it loads a variable by *symbolic* name into a register. > Running Pyperformance using a development version of Python 3.9 > showed that the five most frequently executed pure stack opcodes > (LOAD_FAST, STORE_FAST, POP_TOP, DUP_TOP and ROT_TWO) accounted for > 35% of all executed instructions. And you intend to change that with a register VM? How? Quick google search leads to https://www.strchr.com/x86_machine_code_statistics (yeah, that's not VM, it's RM (real machine), stats over different VMs would be definitely welcome): > The most popular instruction is MOV (35% of all instructions). So, is the plan to replace 35% of "five most frequently executed pure stack opcodes" with 35% of register-register move instructions? If not, why it would be different and how would you achieve that? > They are low-cost instructions (compared with CALL_FUNCTION for > example), but still eat up time and space in the virtual machine But that's the problem of any VM - it's slow by definition. There can be less slow and more slow VMs, but VMs can't be fast. So, what's the top-level motivation - is it "making CPython fast" or "making CPython a little bit less slow"? By how much? > Consider the layout of the data section of a Frame object: > All those LOAD_FAST and STORE_FAST instructions just copy pointers > between chunks of RAM wh
