Re: [pypy-dev] what does the debug cycle look like?

2012-08-14 Thread Matti Picus
Numpy's tests are a good starting point, but they tend to ignore corner 
cases such as:

improper arguments,
strange mixtures of options,
passing in virtual ndarrays (lazy evaluation and slices),
empty arrays or scalars,
so we generally start with theirs and add more, trying to retain 
compatibility.

Matti

On 14/08/2012 8:15 AM, Brandon Rhodes wrote:
... I will have to ask him - or do you know? - whether we are writing 
our own tests, or just copying over NumPy's existing tests each time 
we add a new feature and want to be sure that it works? 


___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] what does the debug cycle look like?

2012-08-14 Thread Taavi Burns
Also, that 4GB Mac looks like it has at least 4 cores, so Make will fork 4 cc 
processes, each of which wants 700-1000MB RAM.

Especially if it's only 2-core with hyperthreading, you'll want to limit the 
concurrent compiles to 2 (I forget the exact args to `translate` at the moment, 
on the subway).

And keep in mind that translating with PyPy takes more ram than cpython!

-- 
taa
/*eof*/

On 2012-08-14, at 5:09, Armin Rigo  wrote:

> Hi,
> 
> A quick note:
> 
> On Tue, Aug 14, 2012 at 10:59 AM, Maciej Fijalkowski  wrote:
>> (not sure why your translation take so long)
> 
> Because of swapping: translating on a 64-bit Mac requires 5-6 GB of RAM 
> ideally.
> 
> 
> A bientôt,
> 
> Armin.
> ___
> pypy-dev mailing list
> pypy-dev@python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] what does the debug cycle look like?

2012-08-14 Thread Armin Rigo
Hi,

A quick note:

On Tue, Aug 14, 2012 at 10:59 AM, Maciej Fijalkowski  wrote:
> (not sure why your translation take so long)

Because of swapping: translating on a 64-bit Mac requires 5-6 GB of RAM ideally.


A bientôt,

Armin.
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] what does the debug cycle look like?

2012-08-14 Thread Maciej Fijalkowski
On Tue, Aug 14, 2012 at 7:11 AM, Brandon Rhodes  wrote:
> Alex Gaynor  writes:
>
>> The trick is tests!  Tests can be run without translating (actually
>> the whole interpreter can, but we like tests :D).  Just run `pytest.py
>> pypy/module/numpypy/` and you're running the tests, they take like a
>> minute or two, which is way too long IMO, but it's totally serviceable
>> for iterative development.  The only reason we ever translate is to
>> asses the performance of something.
>
> Ah!  That very nearly makes sense - tell me if this is what you meant:
> the tests are not run against the translated, baked-in version of
> numpypy sitting inside my pypy interpreter and that takes hours to
> rebuild, but against the raw numpypy code - both rpython and normal
> Python - that is sitting on the disk and that I can have just edited?
>
> --
> Brandon Rhodes  bran...@rhodesmill.org  http://rhodesmill.org/brandon
> ___
> pypy-dev mailing list
> pypy-dev@python.org
> http://mail.python.org/mailman/listinfo/pypy-dev

Yes.

I appreciate you think we're super humans and can write code that can
only be tested in 30min (not sure why your translation take so long),
but we're not. We're very much into having a very short feedback loop.

Cheers,
fijal
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] what does the debug cycle look like?

2012-08-13 Thread Brandon Rhodes
Matti Picus  writes:

> Welcome, your help is needed and appreciated.  One thing you might
> want to try is adding the functionality as a pure python...

Ah, yes, Maciej had mentioned something about that - that many NumPy
functions wind up being pure Python even over in NumPy, and that for
PyPy we can do the same thing.

I will have to ask him - or do you know? - whether we are writing our
own tests, or just copying over NumPy's existing tests each time we add
a new feature and want to be sure that it works?

-- 
Brandon Rhodes  bran...@rhodesmill.org  http://rhodesmill.org/brandon
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] what does the debug cycle look like?

2012-08-13 Thread Brandon Rhodes
Alex Gaynor  writes:

> The trick is tests!  Tests can be run without translating (actually
> the whole interpreter can, but we like tests :D).  Just run `pytest.py
> pypy/module/numpypy/` and you're running the tests, they take like a
> minute or two, which is way too long IMO, but it's totally serviceable
> for iterative development.  The only reason we ever translate is to
> asses the performance of something.

Ah!  That very nearly makes sense - tell me if this is what you meant:
the tests are not run against the translated, baked-in version of
numpypy sitting inside my pypy interpreter and that takes hours to
rebuild, but against the raw numpypy code - both rpython and normal
Python - that is sitting on the disk and that I can have just edited?

-- 
Brandon Rhodes  bran...@rhodesmill.org  http://rhodesmill.org/brandon
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] what does the debug cycle look like?

2012-08-13 Thread Matti Picus

Welcome, your help is needed and appreciated.
One thing you might want to try is adding the functionality as a pure 
python. You should be
able to add a load function somewhere in the the 
lib_pypy/numpypy/core/directory, and

add tests to the lib_pypy/numpypy/tests directory.
Then just run (from the top pypy directory)
pypy/translator/goal/pypy pytest.py lib_pypy/numpypy/tests
Only if speed become your real bottleneck, or if you simply cannot 
implement your function
(load in this case) in pure python, would you want to dive into the 
murky depths of rpython.

Matti

On 14/08/2012 7:31 AM, Brandon Rhodes wrote:

I need some help.

I want to help add more numpypy goodness to PyPy, so that I can rewrite
my PyEphem astronomy module as pure Python instead of having to always
maintain a C extension.  I will then tell the users who are doing large
calculations, and need extra speed, that they can run their script on
PyPy and have it run blazing fast.

My problem is that I am not sure how to start adding code to PyPy in
such a way that I can try out my code more than once within a single
work day, because it is taking 5+ hours to build the translator on every
platform to which I have access.  This did not surprise me on my 2003
Dell PowerEdge 400SC, with its 2GB of kind-of-working RAM, which wound
up taking 5h 28m to complete the build.  But I just got finished trying
the build on the fastest computer in the house - my wife's few-month-old
iMac with 4GB of memory - and it actually took *longer*, finishing up a
full 5h 53m (!) after it started.

And, actually, that's not the big problem.  The big problem is that
whereas C Python uses Make, so it only re-compiles the files I have
changed when I am in a tight development loop, it seems that PyPy starts
the whole translation process over again when I run it.  How does one
actually iterate when working on part of PyPy and its libraries?

Only now that it is past midnight, as I write this, do I realize that my
subsequent builds of PyPy can be done with the PyPy interpreter itself,
speeding things up considerably!  But even if I can bring down the build
time to under a half-hour, I can hardly imagine doing software
development under the constraint of such delays - what am I missing?

My first feat, should it success, will be writing load() for numpypy
since that it the first missing feature I noticed when I tried running
my prototype "jplephem" package - my first attempt at doing heavy-duty
astronomy in pure Python.  I had hoped to have the package run and walk
away with a number to compare to the pure C performance, but instead I
get the fun of contributing! :)



___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] what does the debug cycle look like?

2012-08-13 Thread Alex Gaynor
On Mon, Aug 13, 2012 at 9:31 PM, Brandon Rhodes wrote:

> I need some help.
>
> I want to help add more numpypy goodness to PyPy, so that I can rewrite
> my PyEphem astronomy module as pure Python instead of having to always
> maintain a C extension.  I will then tell the users who are doing large
> calculations, and need extra speed, that they can run their script on
> PyPy and have it run blazing fast.
>
> My problem is that I am not sure how to start adding code to PyPy in
> such a way that I can try out my code more than once within a single
> work day, because it is taking 5+ hours to build the translator on every
> platform to which I have access.  This did not surprise me on my 2003
> Dell PowerEdge 400SC, with its 2GB of kind-of-working RAM, which wound
> up taking 5h 28m to complete the build.  But I just got finished trying
> the build on the fastest computer in the house - my wife's few-month-old
> iMac with 4GB of memory - and it actually took *longer*, finishing up a
> full 5h 53m (!) after it started.
>
> And, actually, that's not the big problem.  The big problem is that
> whereas C Python uses Make, so it only re-compiles the files I have
> changed when I am in a tight development loop, it seems that PyPy starts
> the whole translation process over again when I run it.  How does one
> actually iterate when working on part of PyPy and its libraries?
>
> Only now that it is past midnight, as I write this, do I realize that my
> subsequent builds of PyPy can be done with the PyPy interpreter itself,
> speeding things up considerably!  But even if I can bring down the build
> time to under a half-hour, I can hardly imagine doing software
> development under the constraint of such delays - what am I missing?
>
> My first feat, should it success, will be writing load() for numpypy
> since that it the first missing feature I noticed when I tried running
> my prototype "jplephem" package - my first attempt at doing heavy-duty
> astronomy in pure Python.  I had hoped to have the package run and walk
> away with a number to compare to the pure C performance, but instead I
> get the fun of contributing! :)
>
> --
> Brandon Rhodes  bran...@rhodesmill.org
> http://rhodesmill.org/brandon
> ___
> pypy-dev mailing list
> pypy-dev@python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>

The trick is tests!  Tests can be run without translating (actually the
whole interpreter can, but we like tests :D).  Just run `pytest.py
pypy/module/numpypy/` and you're running the tests, they take like a minute
or two, which is way too long IMO, but it's
totally serviceable for iterative development.  The only reason we ever
translate is to asses the performance of something.

Alex

-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev