Re: [pypy-dev] what does the debug cycle look like?
Numpy's tests are a good starting point, but they tend to ignore corner cases such as: improper arguments, strange mixtures of options, passing in virtual ndarrays (lazy evaluation and slices), empty arrays or scalars, so we generally start with theirs and add more, trying to retain compatibility. Matti On 14/08/2012 8:15 AM, Brandon Rhodes wrote: ... I will have to ask him - or do you know? - whether we are writing our own tests, or just copying over NumPy's existing tests each time we add a new feature and want to be sure that it works? ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] what does the debug cycle look like?
Also, that 4GB Mac looks like it has at least 4 cores, so Make will fork 4 cc processes, each of which wants 700-1000MB RAM. Especially if it's only 2-core with hyperthreading, you'll want to limit the concurrent compiles to 2 (I forget the exact args to `translate` at the moment, on the subway). And keep in mind that translating with PyPy takes more ram than cpython! -- taa /*eof*/ On 2012-08-14, at 5:09, Armin Rigo wrote: > Hi, > > A quick note: > > On Tue, Aug 14, 2012 at 10:59 AM, Maciej Fijalkowski wrote: >> (not sure why your translation take so long) > > Because of swapping: translating on a 64-bit Mac requires 5-6 GB of RAM > ideally. > > > A bientôt, > > Armin. > ___ > pypy-dev mailing list > pypy-dev@python.org > http://mail.python.org/mailman/listinfo/pypy-dev ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] what does the debug cycle look like?
Hi, A quick note: On Tue, Aug 14, 2012 at 10:59 AM, Maciej Fijalkowski wrote: > (not sure why your translation take so long) Because of swapping: translating on a 64-bit Mac requires 5-6 GB of RAM ideally. A bientôt, Armin. ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] what does the debug cycle look like?
On Tue, Aug 14, 2012 at 7:11 AM, Brandon Rhodes wrote: > Alex Gaynor writes: > >> The trick is tests! Tests can be run without translating (actually >> the whole interpreter can, but we like tests :D). Just run `pytest.py >> pypy/module/numpypy/` and you're running the tests, they take like a >> minute or two, which is way too long IMO, but it's totally serviceable >> for iterative development. The only reason we ever translate is to >> asses the performance of something. > > Ah! That very nearly makes sense - tell me if this is what you meant: > the tests are not run against the translated, baked-in version of > numpypy sitting inside my pypy interpreter and that takes hours to > rebuild, but against the raw numpypy code - both rpython and normal > Python - that is sitting on the disk and that I can have just edited? > > -- > Brandon Rhodes bran...@rhodesmill.org http://rhodesmill.org/brandon > ___ > pypy-dev mailing list > pypy-dev@python.org > http://mail.python.org/mailman/listinfo/pypy-dev Yes. I appreciate you think we're super humans and can write code that can only be tested in 30min (not sure why your translation take so long), but we're not. We're very much into having a very short feedback loop. Cheers, fijal ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] what does the debug cycle look like?
Matti Picus writes: > Welcome, your help is needed and appreciated. One thing you might > want to try is adding the functionality as a pure python... Ah, yes, Maciej had mentioned something about that - that many NumPy functions wind up being pure Python even over in NumPy, and that for PyPy we can do the same thing. I will have to ask him - or do you know? - whether we are writing our own tests, or just copying over NumPy's existing tests each time we add a new feature and want to be sure that it works? -- Brandon Rhodes bran...@rhodesmill.org http://rhodesmill.org/brandon ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] what does the debug cycle look like?
Alex Gaynor writes: > The trick is tests! Tests can be run without translating (actually > the whole interpreter can, but we like tests :D). Just run `pytest.py > pypy/module/numpypy/` and you're running the tests, they take like a > minute or two, which is way too long IMO, but it's totally serviceable > for iterative development. The only reason we ever translate is to > asses the performance of something. Ah! That very nearly makes sense - tell me if this is what you meant: the tests are not run against the translated, baked-in version of numpypy sitting inside my pypy interpreter and that takes hours to rebuild, but against the raw numpypy code - both rpython and normal Python - that is sitting on the disk and that I can have just edited? -- Brandon Rhodes bran...@rhodesmill.org http://rhodesmill.org/brandon ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] what does the debug cycle look like?
Welcome, your help is needed and appreciated. One thing you might want to try is adding the functionality as a pure python. You should be able to add a load function somewhere in the the lib_pypy/numpypy/core/directory, and add tests to the lib_pypy/numpypy/tests directory. Then just run (from the top pypy directory) pypy/translator/goal/pypy pytest.py lib_pypy/numpypy/tests Only if speed become your real bottleneck, or if you simply cannot implement your function (load in this case) in pure python, would you want to dive into the murky depths of rpython. Matti On 14/08/2012 7:31 AM, Brandon Rhodes wrote: I need some help. I want to help add more numpypy goodness to PyPy, so that I can rewrite my PyEphem astronomy module as pure Python instead of having to always maintain a C extension. I will then tell the users who are doing large calculations, and need extra speed, that they can run their script on PyPy and have it run blazing fast. My problem is that I am not sure how to start adding code to PyPy in such a way that I can try out my code more than once within a single work day, because it is taking 5+ hours to build the translator on every platform to which I have access. This did not surprise me on my 2003 Dell PowerEdge 400SC, with its 2GB of kind-of-working RAM, which wound up taking 5h 28m to complete the build. But I just got finished trying the build on the fastest computer in the house - my wife's few-month-old iMac with 4GB of memory - and it actually took *longer*, finishing up a full 5h 53m (!) after it started. And, actually, that's not the big problem. The big problem is that whereas C Python uses Make, so it only re-compiles the files I have changed when I am in a tight development loop, it seems that PyPy starts the whole translation process over again when I run it. How does one actually iterate when working on part of PyPy and its libraries? Only now that it is past midnight, as I write this, do I realize that my subsequent builds of PyPy can be done with the PyPy interpreter itself, speeding things up considerably! But even if I can bring down the build time to under a half-hour, I can hardly imagine doing software development under the constraint of such delays - what am I missing? My first feat, should it success, will be writing load() for numpypy since that it the first missing feature I noticed when I tried running my prototype "jplephem" package - my first attempt at doing heavy-duty astronomy in pure Python. I had hoped to have the package run and walk away with a number to compare to the pure C performance, but instead I get the fun of contributing! :) ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] what does the debug cycle look like?
On Mon, Aug 13, 2012 at 9:31 PM, Brandon Rhodes wrote: > I need some help. > > I want to help add more numpypy goodness to PyPy, so that I can rewrite > my PyEphem astronomy module as pure Python instead of having to always > maintain a C extension. I will then tell the users who are doing large > calculations, and need extra speed, that they can run their script on > PyPy and have it run blazing fast. > > My problem is that I am not sure how to start adding code to PyPy in > such a way that I can try out my code more than once within a single > work day, because it is taking 5+ hours to build the translator on every > platform to which I have access. This did not surprise me on my 2003 > Dell PowerEdge 400SC, with its 2GB of kind-of-working RAM, which wound > up taking 5h 28m to complete the build. But I just got finished trying > the build on the fastest computer in the house - my wife's few-month-old > iMac with 4GB of memory - and it actually took *longer*, finishing up a > full 5h 53m (!) after it started. > > And, actually, that's not the big problem. The big problem is that > whereas C Python uses Make, so it only re-compiles the files I have > changed when I am in a tight development loop, it seems that PyPy starts > the whole translation process over again when I run it. How does one > actually iterate when working on part of PyPy and its libraries? > > Only now that it is past midnight, as I write this, do I realize that my > subsequent builds of PyPy can be done with the PyPy interpreter itself, > speeding things up considerably! But even if I can bring down the build > time to under a half-hour, I can hardly imagine doing software > development under the constraint of such delays - what am I missing? > > My first feat, should it success, will be writing load() for numpypy > since that it the first missing feature I noticed when I tried running > my prototype "jplephem" package - my first attempt at doing heavy-duty > astronomy in pure Python. I had hoped to have the package run and walk > away with a number to compare to the pure C performance, but instead I > get the fun of contributing! :) > > -- > Brandon Rhodes bran...@rhodesmill.org > http://rhodesmill.org/brandon > ___ > pypy-dev mailing list > pypy-dev@python.org > http://mail.python.org/mailman/listinfo/pypy-dev > The trick is tests! Tests can be run without translating (actually the whole interpreter can, but we like tests :D). Just run `pytest.py pypy/module/numpypy/` and you're running the tests, they take like a minute or two, which is way too long IMO, but it's totally serviceable for iterative development. The only reason we ever translate is to asses the performance of something. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
