Re: Running just core regression tests

[Karen Tracey]

On Mon, Oct 12, 2009 at 6:24 PM, Tim Chase
wrote:

>
> Last year, the wifey finally convinced me to let go of the even
> older machine (133MHz with 96 megs of memory and a 800x600 CRT :)
> running OpenBSD.
>
>
Yeah, you've got me beat.  I do still have the IBM AT I had in college, with
its roomy 10M (or was it 20M?) hard drive, mono screen, and I don't recall
how much memory, running DOS.  But it's been in a closet for more than a
decade (no, I don't have a good reason for keeping it).


> > Two other ideas -- what else are you running on this when you run the
> > tests?  The 15 minutes in my case was with basically nothing else running
> > (with just 384M memory it's easy to overload this poor box).  I tried a
> 2nd
> > run after starting a couple of browsers so that the memory was
> > overcommitted, and the testing wall clock time went up to 24 minutes.
>
> Thunderbird for mail and Midori for browsing are open in the
> background, but neither should be overly active.  I tried without
> Midori and it came out about 5 mins faster.  I'll try re-running
> with no other big contenders for the CPU, but they lie pretty
> dormant overnight.
>
>
I was more thinking memory than CPU.  Sure if they're idle and memory is
constrained they should get swapped out in favor of what's actively running,
but if they're waking up periodically to see what's happening in the world
that could be causing some swapping, which could be contributing to the
slowness.

Karen

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: Running just core regression tests

[Tim Chase]

>> On my oldest slowest machine (7 year old WinXP box with 384M of memory),
>> Then you're not far off -- my home work-horse is an 8-year-old
>> Celeron laptop clocked at 800MHz with 384M of memory running
>> Debian off a fairly slow HDD.
> 
> My 800MHz Celeron laptop got retired about a year ago when it quit wanting
> to stay powered on for more than 5 minutes. I thought I'd kept it long past
> its expire date but you've clearly got me beat.

Last year, the wifey finally convinced me to let go of the even 
older machine (133MHz with 96 megs of memory and a 800x600 CRT :) 
running OpenBSD.

> For me, tests for trunk using sqlite ran in 15 minutes.
> 
> My machine is faster than yours, 1.3GHz vs. 800MHz.  But that doesn't seem
> likely to account for your case taking four times longer to run.
> 
> Two other ideas -- what else are you running on this when you run the
> tests?  The 15 minutes in my case was with basically nothing else running
> (with just 384M memory it's easy to overload this poor box).  I tried a 2nd
> run after starting a couple of browsers so that the memory was
> overcommitted, and the testing wall clock time went up to 24 minutes.

Thunderbird for mail and Midori for browsing are open in the 
background, but neither should be overly active.  I tried without 
Midori and it came out about 5 mins faster.  I'll try re-running 
with no other big contenders for the CPU, but they lie pretty 
dormant overnight.

> Second, what version of Python?  The 15 minute result for me was with Python
> 2.6.  Backing off to Python 2.5 (plus keeping the memory overcommitted) made
> the wall clock time go up to 30 minutes.  

It's 2.5 which, if it takes double the time for you, may compound 
the clock-speed difference.  15 minutes * ~2x processor 
difference * ~2x penalty for 2.5 vs. 2.6 comes out to about an hour.

> So, making sure your machine is not swapping, if possible, and updating to a
> recent Python if you are currently running an older one, might help.

I'll see if I can get 2.6 from my Debian repositories which 
sounds like it can save me half the time.

Thanks for all your help!

-tim






--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: Running just core regression tests

[Karen Tracey]

On Sun, Oct 11, 2009 at 7:40 PM, Tim Chase
wrote:

>
> > On my oldest slowest machine (7 year old WinXP box with 384M of memory),
>
>  Then you're not far off -- my home work-horse is an 8-year-old
> Celeron laptop clocked at 800MHz with 384M of memory running
> Debian off a fairly slow HDD.


My 800MHz Celeron laptop got retired about a year ago when it quit wanting
to stay powered on for more than 5 minutes. I thought I'd kept it long past
its expire date but you've clearly got me beat.


> After "an hour and change" (fairly
> close to your 1.5hr benchmark) at lunch, I aborted on the
> assumption that there had to be a better way.  Especially for
> tight iterations on changes against trunk (working on a patch
> against core/trunk from a ticket I created).  I've svn-up'ed to
> the latest to ensure I've got everything as current as possible,
> but I'm fairly sure that was already the case.  I manually
> specified ":memory:" and sqlite, letting it run to completion
> this time and it took just over an hour:
>
>Ran 867 tests in 3775.195s
>
> I'll try Russel's suggestion to minimize the time spent running
> so I can do some quick(er) iterations before re-running the full
> suite.
>
>
Note my 1.5 hours was for the 1.0.X branch, not trunk.  That's why I was
thinking you must be using a DB that didn't support rollback, but if you are
using sqlite then that is not what's causing things to be so slow.  (You
also should not have to specify memory as the DB for testing sqlite, it
should do that automatically though I haven't verified that is working...so
far as I know it is working.)

For me, tests for trunk using sqlite ran in 15 minutes.

My machine is faster than yours, 1.3GHz vs. 800MHz.  But that doesn't seem
likely to account for your case taking four times longer to run.

Two other ideas -- what else are you running on this when you run the
tests?  The 15 minutes in my case was with basically nothing else running
(with just 384M memory it's easy to overload this poor box).  I tried a 2nd
run after starting a couple of browsers so that the memory was
overcommitted, and the testing wall clock time went up to 24 minutes.

Second, what version of Python?  The 15 minute result for me was with Python
2.6.  Backing off to Python 2.5 (plus keeping the memory overcommitted) made
the wall clock time go up to 30 minutes.  I also recall from running tests
against Python 2.3 to check for 2.3 compatibility that those ran what seemed
to be much slower than later levels (though I don't have Python 2.3 on this
particular machine to put a number on "much").  My general impression from
running tests against 2.3, 2.4, 2.5, 2.6 when doing compatibility testing
was that later Pythons were faster, though I never took specific
measurements to quantify that impression.

So, making sure your machine is not swapping, if possible, and updating to a
recent Python if you are currently running an older one, might help.

Finally as to the 50 missing tests noted in another thread -- some tests are
skipped if you don't have various optional bits (e.g. PIL, markdown, etc.)
installed.  I've got probably nothing optional installed on this machine
I've mentioned here and it is reporting 862 tests run.

Karen

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: Running just core regression tests

[Tim Chase]

>> t...@rubbish:~/django/trunk/tests$ python runtests.py
>> --settings=settings
>>
>> to execute the core set of tests.  However, it takes quite a
>> while (on the order of hours) on my old box.  Is there a way to
>> just run tests on the regressions/ subdirectory or even just a
>> particular file within?
>>
>> How do core devs typically make testing a subset faster?
>>
> Hours?  What database are you using?

For what I'm doing, it doesn't matter much currently, so whatever 
you want if it will make it faster. :)  However for the record, 
it's currently sqlite, though I suppose I could point it at 
:memory: instead.  I thought the test harness automatically 
detected sqlite and chooses :memory: as the database source for 
testing purposes,

> On my oldest slowest machine (7 year old WinXP box with 384M of memory),

Then you're not far off -- my home work-horse is an 8-year-old 
Celeron laptop clocked at 800MHz with 384M of memory running 
Debian off a fairly slow HDD.  After "an hour and change" (fairly 
close to your 1.5hr benchmark) at lunch, I aborted on the 
assumption that there had to be a better way.  Especially for 
tight iterations on changes against trunk (working on a patch 
against core/trunk from a ticket I created).  I've svn-up'ed to 
the latest to ensure I've got everything as current as possible, 
but I'm fairly sure that was already the case.  I manually 
specified ":memory:" and sqlite, letting it run to completion 
this time and it took just over an hour:

Ran 867 tests in 3775.195s

I'll try Russel's suggestion to minimize the time spent running 
so I can do some quick(er) iterations before re-running the full 
suite.

Thanks,

-tim




--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: Running just core regression tests

[Karen Tracey]

On Sat, Oct 10, 2009 at 11:41 AM, Tim Chase
wrote:

>
> Currently, I use
>
> t...@rubbish:~/django/trunk/tests$ python runtests.py
> --settings=settings
>
> to execute the core set of tests.  However, it takes quite a
> while (on the order of hours) on my old box.  Is there a way to
> just run tests on the regressions/ subdirectory or even just a
> particular file within?
>
> How do core devs typically make testing a subset faster?
>
> Hours?  What database are you using?  You show that you are running trunk
level code, so you should have the transaction-based test speedup that went
into 1.1.  However, this speedup only helps for database configurations that
support transactions, so if you are using MySQL with the MyISAM engine tests
will run very slowly.  If that is what is causing the slowness in your case,
switching to using sqlite for the database for testing would help
tremendously.

On my oldest slowest machine (7 year old WinXP box with 384M of memory),
current trunk tests take about 15 minutes to run using sqlite as the
database.  The 1.0.X branch, without the transaction speedups, takes 1.5
hours (even though it has 25% fewer tests).  So the hours time you cite
sounds to me like you must not be running a config that is using the test
speedup that went into 1.1?  Or do you really have a machine so much slower
than my oldest?

Karen

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: Running just core regression tests

[Russell Keith-Magee]

On Sat, Oct 10, 2009 at 11:41 PM, Tim Chase
 wrote:
>
> Currently, I use
>
> t...@rubbish:~/django/trunk/tests$ python runtests.py
> --settings=settings
>
> to execute the core set of tests.  However, it takes quite a
> while (on the order of hours) on my old box.  Is there a way to
> just run tests on the regressions/ subdirectory or even just a
> particular file within?
>
> How do core devs typically make testing a subset faster?

There's no way to say "just run the tests in the regression
directory", but you can use the same tricks that ./manage.py test
allows - that is, you can say:

./runtests.py --settings=xxx query aggregation_regress serializers_regress

and that will just run the query, aggregation_regress and
serializers_regress tests. For unit tests, you can go one step deeper
and narrow this down to a single TestCase, or a single test inside a
test case:

./runtests.py --settings=xxx test_client.ClientTest
./runtests.py --settings=xxx test_client.ClientTest.test_get_view

Following on from this, I personally have a couple of scripts where
I've gathered together subsets of tests that I know have a particular
focus - e.g., the tests that are testing core database functions, the
tests for the form library, tests for the admin, etc. These scripts
are effectively just aliases for calls to runtests.py with a list of
relevant test names. That way, I can easily run a subset of the full
test suite with a single script.

If I'm working on a particular feature, I may produce a custom script
for an even more reduced subset - for example, when I was working on
aggregation, I had a script to just run aggregation,
aggregation_regress, expressions, expressions_regress and query.

Of course, I still run the full test suite before I commit anything,
but these subsets are usually good enough during the development of a
particular feature that I know is "just database" or "just forms".

Yours,
Russ Magee %-)

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Running just core regression tests

[Tim Chase]

Currently, I use

t...@rubbish:~/django/trunk/tests$ python runtests.py 
--settings=settings

to execute the core set of tests.  However, it takes quite a 
while (on the order of hours) on my old box.  Is there a way to 
just run tests on the regressions/ subdirectory or even just a 
particular file within?

How do core devs typically make testing a subset faster?

-tkc





--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---