Re: [pypy-dev] Running PyPy on top of CPython

[Matti Picus]

Hi Omer. I think much of what Armin wrote is already in the 
documentation like 
https://doc.pypy.org/en/latest/contributing.html#architecture and 
https://doc.pypy.org/en/latest/contributing.html#test-driven-development. 
Could you see where things can be improved?


Matti


On 11/1/22 4:17 pm, Omer Katz wrote:

Hi Armin,

Would you mind adding the above to the documentation?
I think it'll be useful for newcomers.

Best Regards,
Omer Katz

‫בתאריך יום ב׳, 10 בינו׳ 2022 ב-18:03 מאת ‪Armin Rigo‬‏ 
<‪armin.r...@gmail.com ‬‏>:‬


Hi,

On Mon, 10 Jan 2022 at 15:56, M A mailto:teammember0...@gmail.com>> wrote:
> I think you are confused. I just read in PyPy's documentation
that PyPy could be run from CPython. This sounds like something
that could help save me time by seeing if my changes work. I'm am
not sure why you think I am ignoring the tests. Yes I have tried
them. I am seeing if there is a more efficient way of trying out
my code without having to wait a long time of PyPy to recompile.

Sorry for not explaining myself clearly.  I've been trying all along
to tell you that you don't need to recompile PyPy, ever. As long as
not all the tests I listed are passing, then there is basically no
point.  (except that it feels good to see PyPy half-working instead of
not working at all, of course, but don't try to debug that)

The documentation says "PyPy can be run on top of CPython": that's
almost what all the tests are doing.  They run not the whole PyPy, but
instead some small example RPython code (sometimes written as RPython
in the test, sometimes at a lower level).  The point is that they test
the JIT backend by running it as normal Python code, on top of
CPython.  When working on the JIT backend, you don't want to run the
whole PyPy on top of CPython running with the JIT; while possible in
theory, it is far too slow to be practical.  Instead, run the tests,
which test the same thing but with a small bit of RPython code instead
of the many-thousand-of-lines-of-code of PyPy.

Let me try to explain myself in a different way.
What we're trying to do here is fix a JIT compiler emitting machine
code, at the level of RPython instead of Python.  That means that any
error is likely to cause not just a problem for some specific Python
function which you can find by running Python code; instead, bugs
cause random hard-to-debug crashes that show up at half-random
points, with no obvious one-to-one correspondence to what Python code
is running.  This could be caused for example by the wrong registers
being used or accidentally overwritten in some cases, leading to a
nonsense value propagating further, until the point of crash. Most
tests listed above check all basic cases and problems we encountered
so far.  The last test is producing and running random exemples, in an
attempt to find the random rare-case issues.  It has been the case
that this last test found a remaining bug after running for hours, but
I think that after half a day we can conclude that there is no bug
left to be found.  In the past, there were very rare bugs that went
uncaught by random testing too; these were a real pain to debug---I
once spent more than two weeks straight on a pair of them, running
"gdb" inside a translated PyPy, hacking at the generated C code to
dump more information to try to figure it out.  But these bugs were
more than just in the JIT backend (e.g. some interaction between the
JIT and the GC), and they are fixed now.  I'm telling this story to
explain why I do not recommend going down that path for the simpler
bugs that are already caught by the tests!

So, once more, I recommend working on this by running the tests, and
fixing them, until all tests pass.  Once all tests pass (and not
before) you can try to translate the whole PyPy, and at this point it
will likely work on the first try.

Sorry for not managing to get this message across to you in the past.
I hope I have done so now.


A bientôt,

Armin.
___
pypy-dev mailing list
pypy-dev@python.org 
https://mail.python.org/mailman/listinfo/pypy-dev



___
pypy-dev mailing list
pypy-dev@python.org
https://mail.python.org/mailman/listinfo/pypy-dev

___
pypy-dev mailing list
pypy-dev@python.org
https://mail.python.org/mailman/listinfo/pypy-dev

Re: [pypy-dev] Running PyPy on top of CPython

[Omer Katz]

Hi Armin,

Would you mind adding the above to the documentation?
I think it'll be useful for newcomers.

Best Regards,
Omer Katz

‫בתאריך יום ב׳, 10 בינו׳ 2022 ב-18:03 מאת ‪Armin Rigo‬‏ <‪
armin.r...@gmail.com‬‏>:‬

> Hi,
>
> On Mon, 10 Jan 2022 at 15:56, M A  wrote:
> > I think you are confused. I just read in PyPy's documentation that PyPy
> could be run from CPython. This sounds like something that could help save
> me time by seeing if my changes work. I'm am not sure why you think I am
> ignoring the tests. Yes I have tried them. I am seeing if there is a more
> efficient way of trying out my code without having to wait a long time of
> PyPy to recompile.
>
> Sorry for not explaining myself clearly.  I've been trying all along
> to tell you that you don't need to recompile PyPy, ever. As long as
> not all the tests I listed are passing, then there is basically no
> point.  (except that it feels good to see PyPy half-working instead of
> not working at all, of course, but don't try to debug that)
>
> The documentation says "PyPy can be run on top of CPython": that's
> almost what all the tests are doing.  They run not the whole PyPy, but
> instead some small example RPython code (sometimes written as RPython
> in the test, sometimes at a lower level).  The point is that they test
> the JIT backend by running it as normal Python code, on top of
> CPython.  When working on the JIT backend, you don't want to run the
> whole PyPy on top of CPython running with the JIT; while possible in
> theory, it is far too slow to be practical.  Instead, run the tests,
> which test the same thing but with a small bit of RPython code instead
> of the many-thousand-of-lines-of-code of PyPy.
>
> Let me try to explain myself in a different way.
> What we're trying to do here is fix a JIT compiler emitting machine
> code, at the level of RPython instead of Python.  That means that any
> error is likely to cause not just a problem for some specific Python
> function which you can find by running Python code; instead, bugs
> cause random hard-to-debug crashes that show up at half-random
> points, with no obvious one-to-one correspondence to what Python code
> is running.  This could be caused for example by the wrong registers
> being used or accidentally overwritten in some cases, leading to a
> nonsense value propagating further, until the point of crash.  Most
> tests listed above check all basic cases and problems we encountered
> so far.  The last test is producing and running random exemples, in an
> attempt to find the random rare-case issues.  It has been the case
> that this last test found a remaining bug after running for hours, but
> I think that after half a day we can conclude that there is no bug
> left to be found.  In the past, there were very rare bugs that went
> uncaught by random testing too; these were a real pain to debug---I
> once spent more than two weeks straight on a pair of them, running
> "gdb" inside a translated PyPy, hacking at the generated C code to
> dump more information to try to figure it out.  But these bugs were
> more than just in the JIT backend (e.g. some interaction between the
> JIT and the GC), and they are fixed now.  I'm telling this story to
> explain why I do not recommend going down that path for the simpler
> bugs that are already caught by the tests!
>
> So, once more, I recommend working on this by running the tests, and
> fixing them, until all tests pass.  Once all tests pass (and not
> before) you can try to translate the whole PyPy, and at this point it
> will likely work on the first try.
>
> Sorry for not managing to get this message across to you in the past.
> I hope I have done so now.
>
>
> A bientôt,
>
> Armin.
> ___
> pypy-dev mailing list
> pypy-dev@python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>
___
pypy-dev mailing list
pypy-dev@python.org
https://mail.python.org/mailman/listinfo/pypy-dev

Re: [pypy-dev] Running PyPy on top of CPython

[M A]

> On 9/1/22 9:50 pm, M A wrote:
>> Hello, I read here (https://rpython.readthedocs.io/en/latest/rpython.html) 
>> that PyPy is runnable on top of CPython. How do I start PyPy so it runs in 
>> CPython?
> 
> 
> python2 pypy/bin/pyinteractive.py
> 
> 
> You can get help with
> 
> python2 pypy/bin/pyinteractive.py --help
> 
> 
> It takes ~20 secs to startup. To drop into rpython, hit ctrl-c, to 
> return to interpreted PyPy on top of CPython hit ctrl-d
> 
> 
> In rpython you have access to "space".
> 
> 
> 
> Matti

Thank you very much for this information. It really helped me.
___
pypy-dev mailing list
pypy-dev@python.org
https://mail.python.org/mailman/listinfo/pypy-dev

Re: [pypy-dev] Running PyPy on top of CPython

[Armin Rigo]

Hi,

On Mon, 10 Jan 2022 at 15:56, M A  wrote:
> I think you are confused. I just read in PyPy's documentation that PyPy could 
> be run from CPython. This sounds like something that could help save me time 
> by seeing if my changes work. I'm am not sure why you think I am ignoring the 
> tests. Yes I have tried them. I am seeing if there is a more efficient way of 
> trying out my code without having to wait a long time of PyPy to recompile.

Sorry for not explaining myself clearly.  I've been trying all along
to tell you that you don't need to recompile PyPy, ever. As long as
not all the tests I listed are passing, then there is basically no
point.  (except that it feels good to see PyPy half-working instead of
not working at all, of course, but don't try to debug that)

The documentation says "PyPy can be run on top of CPython": that's
almost what all the tests are doing.  They run not the whole PyPy, but
instead some small example RPython code (sometimes written as RPython
in the test, sometimes at a lower level).  The point is that they test
the JIT backend by running it as normal Python code, on top of
CPython.  When working on the JIT backend, you don't want to run the
whole PyPy on top of CPython running with the JIT; while possible in
theory, it is far too slow to be practical.  Instead, run the tests,
which test the same thing but with a small bit of RPython code instead
of the many-thousand-of-lines-of-code of PyPy.

Let me try to explain myself in a different way.
What we're trying to do here is fix a JIT compiler emitting machine
code, at the level of RPython instead of Python.  That means that any
error is likely to cause not just a problem for some specific Python
function which you can find by running Python code; instead, bugs
cause random hard-to-debug crashes that show up at half-random
points, with no obvious one-to-one correspondence to what Python code
is running.  This could be caused for example by the wrong registers
being used or accidentally overwritten in some cases, leading to a
nonsense value propagating further, until the point of crash.  Most
tests listed above check all basic cases and problems we encountered
so far.  The last test is producing and running random exemples, in an
attempt to find the random rare-case issues.  It has been the case
that this last test found a remaining bug after running for hours, but
I think that after half a day we can conclude that there is no bug
left to be found.  In the past, there were very rare bugs that went
uncaught by random testing too; these were a real pain to debug---I
once spent more than two weeks straight on a pair of them, running
"gdb" inside a translated PyPy, hacking at the generated C code to
dump more information to try to figure it out.  But these bugs were
more than just in the JIT backend (e.g. some interaction between the
JIT and the GC), and they are fixed now.  I'm telling this story to
explain why I do not recommend going down that path for the simpler
bugs that are already caught by the tests!

So, once more, I recommend working on this by running the tests, and
fixing them, until all tests pass.  Once all tests pass (and not
before) you can try to translate the whole PyPy, and at this point it
will likely work on the first try.

Sorry for not managing to get this message across to you in the past.
I hope I have done so now.


A bientôt,

Armin.
___
pypy-dev mailing list
pypy-dev@python.org
https://mail.python.org/mailman/listinfo/pypy-dev

Re: [pypy-dev] Running PyPy on top of CPython

[M A]

> On Jan 10, 2022, at 9:44 AM, Armin Rigo  wrote:
> 
> Hi,
> 
> On Mon, 10 Jan 2022, 3:31 AM M A  wrote:
> Well I am developing for PyPy and was hoping to try my code out without 
> having to build PyPy first.
> 
> I tried twice to tell you how we develop PyPy in the issue: by running tests. 
> You seem to be ignoring that.

I think you are confused. I just read in PyPy's documentation that PyPy could 
be run from CPython. This sounds like something that could help save me time by 
seeing if my changes work. I'm am not sure why you think I am ignoring the 
tests. Yes I have tried them. I am seeing if there is a more efficient way of 
trying out my code without having to wait a long time of PyPy to recompile. 


___
pypy-dev mailing list
pypy-dev@python.org
https://mail.python.org/mailman/listinfo/pypy-dev

Re: [pypy-dev] Running PyPy on top of CPython

[Armin Rigo]

Hi,

On Mon, 10 Jan 2022, 3:31 AM M A  wrote:

> Well I am developing for PyPy and was hoping to try my code out without
> having to build PyPy first.
>

I tried twice to tell you how we develop PyPy in the issue: by running
tests. You seem to be ignoring that. Sorry but I will ignore further
requests from you until you acknowledge you have at least tried what I
recommend first.

Armin

>
___
pypy-dev mailing list
pypy-dev@python.org
https://mail.python.org/mailman/listinfo/pypy-dev

Re: [pypy-dev] Running PyPy on top of CPython

[Matti Picus]

On 10/1/22 4:31 am, M A wrote:

Well I am developing for PyPy and was hoping to try my code out without having 
to build PyPy first.



You asked this question on the issue tracker as well. Running PyPy on 
top of CPython is not a replacement for getting stress tests in the 
issue  comment [0] to pass.



[0] https://foss.heptapod.net/pypy/pypy/-/issues/3566#note_167193

___
pypy-dev mailing list
pypy-dev@python.org
https://mail.python.org/mailman/listinfo/pypy-dev

Re: [pypy-dev] Running PyPy on top of CPython

[M A]

Well I am developing for PyPy and was hoping to try my code out without having 
to build PyPy first.

> On Jan 9, 2022, at 4:59 PM, Dan Stromberg  wrote:
> 
> 
> IINM, Running Pypy overtop of CPython is mostly useful for bootstrapping Pypy 
> - but it's generally easier to download a Pypy binary from the official 
> website.
> 
> If you just want CPython+speed, maybe try 
> https://stromberg.dnsalias.org/~strombrg/speeding-python/ ?
> 
> 
> On Sun, Jan 9, 2022 at 11:51 AM M A  wrote:
> Hello, I read here (https://rpython.readthedocs.io/en/latest/rpython.html) 
> that PyPy is runnable on top of CPython. How do I start PyPy so it runs in 
> CPython?
> ___
> pypy-dev mailing list
> pypy-dev@python.org
> https://mail.python.org/mailman/listinfo/pypy-dev

___
pypy-dev mailing list
pypy-dev@python.org
https://mail.python.org/mailman/listinfo/pypy-dev

Re: [pypy-dev] Running PyPy on top of CPython

[Matti Picus]

On 9/1/22 9:50 pm, M A wrote:

Hello, I read here (https://rpython.readthedocs.io/en/latest/rpython.html) that 
PyPy is runnable on top of CPython. How do I start PyPy so it runs in CPython?



python2 pypy/bin/pyinteractive.py


You can get help with

python2 pypy/bin/pyinteractive.py --help


It takes ~20 secs to startup. To drop into rpython, hit ctrl-c, to 
return to interpreted PyPy on top of CPython hit ctrl-d



In rpython you have access to "space".



Matti


___
pypy-dev mailing list
pypy-dev@python.org
https://mail.python.org/mailman/listinfo/pypy-dev

Re: [pypy-dev] Running PyPy on top of CPython

[Dan Stromberg]

IINM, Running Pypy overtop of CPython is mostly useful for bootstrapping
Pypy - but it's generally easier to download a Pypy binary from the
official website.

If you just want CPython+speed, maybe try
https://stromberg.dnsalias.org/~strombrg/speeding-python/ ?


On Sun, Jan 9, 2022 at 11:51 AM M A  wrote:

> Hello, I read here (https://rpython.readthedocs.io/en/latest/rpython.html)
> that PyPy is runnable on top of CPython. How do I start PyPy so it runs in
> CPython?
> ___
> pypy-dev mailing list
> pypy-dev@python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>
___
pypy-dev mailing list
pypy-dev@python.org
https://mail.python.org/mailman/listinfo/pypy-dev

[pypy-dev] Running PyPy on top of CPython

[M A]

Hello, I read here (https://rpython.readthedocs.io/en/latest/rpython.html) that 
PyPy is runnable on top of CPython. How do I start PyPy so it runs in CPython?
___
pypy-dev mailing list
pypy-dev@python.org
https://mail.python.org/mailman/listinfo/pypy-dev