Re: [pypy-dev] Running PyPy on top of CPython

[Matti Picus]

Hi Omer. I think much of what Armin wrote is already in the 
documentation like 
https://doc.pypy.org/en/latest/contributing.html#architecture and 
https://doc.pypy.org/en/latest/contributing.html#test-driven-development. 
Could you see where things can be improved?


Matti


On 11/1/22 4:17 pm, Omer Katz wrote:

Hi Armin,

Would you mind adding the above to the documentation?
I think it'll be useful for newcomers.

Best Regards,
Omer Katz

‫בתאריך יום ב׳, 10 בינו׳ 2022 ב-18:03 מאת ‪Armin Rigo‬‏ 
<‪armin.r...@gmail.com ‬‏>:‬


Hi,

On Mon, 10 Jan 2022 at 15:56, M A mailto:teammember0...@gmail.com>> wrote:
> I think you are confused. I just read in PyPy's documentation
that PyPy could be run from CPython. This sounds like something
that could help save me time by seeing if my changes work. I'm am
not sure why you think I am ignoring the tests. Yes I have tried
them. I am seeing if there is a more efficient way of trying out
my code without having to wait a long time of PyPy to recompile.

Sorry for not explaining myself clearly.  I've been trying all along
to tell you that you don't need to recompile PyPy, ever. As long as
not all the tests I listed are passing, then there is basically no
point.  (except that it feels good to see PyPy half-working instead of
not working at all, of course, but don't try to debug that)

The documentation says "PyPy can be run on top of CPython": that's
almost what all the tests are doing.  They run not the whole PyPy, but
instead some small example RPython code (sometimes written as RPython
in the test, sometimes at a lower level).  The point is that they test
the JIT backend by running it as normal Python code, on top of
CPython.  When working on the JIT backend, you don't want to run the
whole PyPy on top of CPython running with the JIT; while possible in
theory, it is far too slow to be practical.  Instead, run the tests,
which test the same thing but with a small bit of RPython code instead
of the many-thousand-of-lines-of-code of PyPy.

Let me try to explain myself in a different way.
What we're trying to do here is fix a JIT compiler emitting machine
code, at the level of RPython instead of Python.  That means that any
error is likely to cause not just a problem for some specific Python
function which you can find by running Python code; instead, bugs
cause random hard-to-debug crashes that show up at half-random
points, with no obvious one-to-one correspondence to what Python code
is running.  This could be caused for example by the wrong registers
being used or accidentally overwritten in some cases, leading to a
nonsense value propagating further, until the point of crash. Most
tests listed above check all basic cases and problems we encountered
so far.  The last test is producing and running random exemples, in an
attempt to find the random rare-case issues.  It has been the case
that this last test found a remaining bug after running for hours, but
I think that after half a day we can conclude that there is no bug
left to be found.  In the past, there were very rare bugs that went
uncaught by random testing too; these were a real pain to debug---I
once spent more than two weeks straight on a pair of them, running
"gdb" inside a translated PyPy, hacking at the generated C code to
dump more information to try to figure it out.  But these bugs were
more than just in the JIT backend (e.g. some interaction between the
JIT and the GC), and they are fixed now.  I'm telling this story to
explain why I do not recommend going down that path for the simpler
bugs that are already caught by the tests!

So, once more, I recommend working on this by running the tests, and
fixing them, until all tests pass.  Once all tests pass (and not
before) you can try to translate the whole PyPy, and at this point it
will likely work on the first try.

Sorry for not managing to get this message across to you in the past.
I hope I have done so now.


A bientôt,

Armin.
___
pypy-dev mailing list
pypy-dev@python.org 
https://mail.python.org/mailman/listinfo/pypy-dev



___
pypy-dev mailing list
pypy-dev@python.org
https://mail.python.org/mailman/listinfo/pypy-dev

___
pypy-dev mailing list
pypy-dev@python.org
https://mail.python.org/mailman/listinfo/pypy-dev

Re: [pypy-dev] Running PyPy on top of CPython

[Omer Katz]

Hi Armin,

Would you mind adding the above to the documentation?
I think it'll be useful for newcomers.

Best Regards,
Omer Katz

‫בתאריך יום ב׳, 10 בינו׳ 2022 ב-18:03 מאת ‪Armin Rigo‬‏ <‪
armin.r...@gmail.com‬‏>:‬

> Hi,
>
> On Mon, 10 Jan 2022 at 15:56, M A  wrote:
> > I think you are confused. I just read in PyPy's documentation that PyPy
> could be run from CPython. This sounds like something that could help save
> me time by seeing if my changes work. I'm am not sure why you think I am
> ignoring the tests. Yes I have tried them. I am seeing if there is a more
> efficient way of trying out my code without having to wait a long time of
> PyPy to recompile.
>
> Sorry for not explaining myself clearly.  I've been trying all along
> to tell you that you don't need to recompile PyPy, ever. As long as
> not all the tests I listed are passing, then there is basically no
> point.  (except that it feels good to see PyPy half-working instead of
> not working at all, of course, but don't try to debug that)
>
> The documentation says "PyPy can be run on top of CPython": that's
> almost what all the tests are doing.  They run not the whole PyPy, but
> instead some small example RPython code (sometimes written as RPython
> in the test, sometimes at a lower level).  The point is that they test
> the JIT backend by running it as normal Python code, on top of
> CPython.  When working on the JIT backend, you don't want to run the
> whole PyPy on top of CPython running with the JIT; while possible in
> theory, it is far too slow to be practical.  Instead, run the tests,
> which test the same thing but with a small bit of RPython code instead
> of the many-thousand-of-lines-of-code of PyPy.
>
> Let me try to explain myself in a different way.
> What we're trying to do here is fix a JIT compiler emitting machine
> code, at the level of RPython instead of Python.  That means that any
> error is likely to cause not just a problem for some specific Python
> function which you can find by running Python code; instead, bugs
> cause random hard-to-debug crashes that show up at half-random
> points, with no obvious one-to-one correspondence to what Python code
> is running.  This could be caused for example by the wrong registers
> being used or accidentally overwritten in some cases, leading to a
> nonsense value propagating further, until the point of crash.  Most
> tests listed above check all basic cases and problems we encountered
> so far.  The last test is producing and running random exemples, in an
> attempt to find the random rare-case issues.  It has been the case
> that this last test found a remaining bug after running for hours, but
> I think that after half a day we can conclude that there is no bug
> left to be found.  In the past, there were very rare bugs that went
> uncaught by random testing too; these were a real pain to debug---I
> once spent more than two weeks straight on a pair of them, running
> "gdb" inside a translated PyPy, hacking at the generated C code to
> dump more information to try to figure it out.  But these bugs were
> more than just in the JIT backend (e.g. some interaction between the
> JIT and the GC), and they are fixed now.  I'm telling this story to
> explain why I do not recommend going down that path for the simpler
> bugs that are already caught by the tests!
>
> So, once more, I recommend working on this by running the tests, and
> fixing them, until all tests pass.  Once all tests pass (and not
> before) you can try to translate the whole PyPy, and at this point it
> will likely work on the first try.
>
> Sorry for not managing to get this message across to you in the past.
> I hope I have done so now.
>
>
> A bientôt,
>
> Armin.
> ___
> pypy-dev mailing list
> pypy-dev@python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>
___
pypy-dev mailing list
pypy-dev@python.org
https://mail.python.org/mailman/listinfo/pypy-dev