Benjamin,
Thanks for the followup!
I've been using MySQL. For normal (non-testing) use, it seems
fast enough on our development machines (Mac laptop) and in
our TEST and PROD environments (Linux on AWS). Page loads
are typically sub-second. It was only when I used the command:
% python manage.py test
that I saw the long delays.
I assume the delays are related to the default stuff happening in
setUp() and tearDown() for each test. Still happens with SQLite,
I assume. Just much faster. I may step through in the debugger
sometime to confirm this.
I read the slides at the URL you gave me. Good info! For now,
our site is fast enough and is not expected to scale dramatically,
but we do have a JIRA ticket for improving performance if/when
we need it. So far, I've tossed a bunch of idea in there, but not
needed to do any of them: profiling, caching, file minimization
and packaging, tuning mix of Ajax vs full page loads, etc.
I've added this URL to the JIRA ticket for further consideration
if/when we need it.
Is there a more current version of the slides? This copy was
presented to OSCON in 2011, and refers to Django 1.3 as new.
Here are some notes I added to our JIRA ticket along with this
URL for if/when we need it:
- Beware ManyToManyField
- Beware over-indexing
- Look into pg_stat_activity (PostgreSQL), or MySQL equivalent
- Prefer NULL to DEFAULT when migrating a large table
- Or do it 3 steps shown to avoid long lock on DB table
- Consider direct SQL sometimes instead of Django ORM
- Beware iterating over large querysets
- Beware filters with "__in" that match more than 10-15 items
- Beware OFFSET
- Use replication or a backup DB copy for large reporting operations
- Don't use DB as Celery job queue
- Use @commit_on_success or TransactionMiddleware
- Leave autocommit on (even thought it seems wrong)
- Modify tables in a fixed order. Same for rows of a table.
- Pass QuerySets directly to templates (to allow lazy evaluation)
- Cache templates and fragments
- Cache expensive query results
- Pool DB connections if over 100 concurrent users
- Use db_routers.py to write primary DB, read replica DB
Thanks again!
--Fred
------------------------------------------------------------------------
Fred Stluka -- mailto:f...@bristle.com -- http://bristle.com/~fred/
Bristle Software, Inc -- http://bristle.com -- Glad to be of service!
Open Source: Without walls and fences, we need no Windows or Gates.
------------------------------------------------------------------------
On 9/2/14 5:37 AM, Benjamin Scherrey wrote:
What DB were you testing on before? Unless you're going to launch this
thing into production with SQLite I would recommend you switch back to
whatever DB you're planning for production and test with it as you
probably have some serious performance considerations to work out
asap. If they're architectural rather than configuration-based issues
then the sooner you recognize and repair them the better. If you
happen to be using Postgres out of the box with no changes from your
distro's base install then it's going to be real slow because
Postgres' default config is to use as few resources as possible which
necessarily makes performance less than optimal. It's not hard to make
Postgres a very fast DB but you better start using it right to make
things go well when you launch.
See this article for some good advice:
http://thebuild.com/presentations/unbreaking-django.pdf It's a bit
dated and newer versions of Django improve on the old implementation
of the ORM but it's recommendations and things to look out for are
sound. You need to measure your actual performance and act accordingly.
Good luck,
-- Ben
On Sat, Aug 30, 2014 at 2:05 AM, Fred Stluka <f...@bristle.com
<mailto:f...@bristle.com>> wrote:
Benjamin,
I solved my problem. I switched to SQLite for testing and now
run 500+ tests in 30 secs instead of 75 minutes. Much better!
Thanks for your help!
--Fred
------------------------------------------------------------------------
Fred Stluka -- mailto:f...@bristle.com --
http://bristle.com/~fred/ <http://bristle.com/%7Efred/>
Bristle Software, Inc -- http://bristle.com -- Glad to be of service!
Open Source: Without walls and fences, we need no Windows or Gates.
------------------------------------------------------------------------
On 8/27/14 5:22 PM, Benjamin Scherrey wrote:
Postgres. best of luck.
On Thu, Aug 28, 2014 at 4:16 AM, Fred Stluka <f...@bristle.com
<mailto:f...@bristle.com>> wrote:
Ben,
Thanks! That's exactly the kind of ballpark figure I wanted.
Sounds like I should be expecting roughly 5-10 tests/sec, plus
maybe 10-20 secs DB setup/migration. I'm currently seeing 1
test per 7-9 secs, plus 50-60 secs DB setup time (syncdb, not
South, since I'm setting SOUTH_TESTS_MIGRATE = False).
Yeah, something must be wrong here. I'm running 100X slower
than expected.
BTW, are those numbers using SQLite, MySQL, PostgreSQL, other?
So far, I'm using MySQL, running locally on a pretty fast Mac
laptop.
I'm planning to switch to SQLite for testing, to make it run
faster,
but haven't gotten that to work yet.
My next steps will be try again with SQLite, and perhaps get the
test suite to run in the PyCharm debugger so I can step through
the code and see where all the time is going.
Thanks again!
--Fred
------------------------------------------------------------------------
Fred Stluka -- mailto:f...@bristle.com --
http://bristle.com/~fred/ <http://bristle.com/%7Efred/>
Bristle Software, Inc -- http://bristle.com -- Glad to be of
service!
Open Source: Without walls and fences, we need no Windows or
Gates.
------------------------------------------------------------------------
On 8/27/14 4:21 PM, Benjamin Scherrey wrote:
Clearly that depends on what your tests do. One of our
projects runs ~400 tests in about a minute. If your tests do
much database access or have complex setUp/tearDown then
that's not going to be as fast of course. The Django 1.7
project I'm working on at the moment takes a total of 12.7
seconds to complete but most of that time is the setup and
migrations execution for the test db. Once it starts the 36
tests I have now complete in 3.58 seconds in verbose mode
with full branch coverage turned on.
coverage run --branch
--source="partner,item,utils,importing,channel" manage.py
test --verbosity=2 && coverage report --show-missing
-- Ben
On Thu, Aug 28, 2014 at 3:13 AM, Fred Stluka
<f...@bristle.com <mailto:f...@bristle.com>> wrote:
Benjamin,
OK, Thanks! So roughly how fast would you expect? Hundreds
of trivial tests per minute? Thousands per minute?
Thanks!
--Fred
------------------------------------------------------------------------
Fred Stluka -- mailto:f...@bristle.com --
http://bristle.com/~fred/ <http://bristle.com/%7Efred/>
Bristle Software, Inc -- http://bristle.com -- Glad to
be of service!
Open Source: Without walls and fences, we need no
Windows or Gates.
------------------------------------------------------------------------
On 8/27/14 3:57 PM, Benjamin Scherrey wrote:
Something's definitely wrong. Except for the initial
setup for the test run (in 1.7 migrations run each time
for example), the individual tests should execute as
fast as any normal python unit test.
On Wed, Aug 27, 2014 at 9:06 PM, Fred Stluka
<f...@bristle.com <mailto:f...@bristle.com>> wrote:
How quickly do Django unit tests run?
Mine are taking 7-9 seconds each, even for trivial
tests like:
self.assertEqual(1 + 1, 2)
that are all in the same test class of the same app.
Is this typical? Or do I have something misconfigured.
Thanks!
--Fred
--
You received this message because you are
subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving
emails from it, send an email to
django-users+unsubscr...@googlegroups.com
<mailto:django-users%2bunsubscr...@googlegroups.com>.
To post to this group, send email to
django-users@googlegroups.com
<mailto:django-users@googlegroups.com>.
Visit this group at
http://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/53FDE5E7.6040506%40bristle.com.
For more options, visit
https://groups.google.com/d/optout.
--
Chief Systems Architect Proteus Technologies
<http://proteus-tech.com>
Chief Fan Biggest Fan Productions <http://biggestfan.net>
Personal blog where I am not your demographic
<http://notyourdemographic.com>.
This email intended solely for those who have received
it. If you have received this email by accident - well
lucky you!!
--
You received this message because you are subscribed to
the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving
emails from it, send an email to
django-users+unsubscr...@googlegroups.com
<mailto:django-users+unsubscr...@googlegroups.com>.
To post to this group, send email to
django-users@googlegroups.com
<mailto:django-users@googlegroups.com>.
Visit this group at
http://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/CAHN%3D9D4DNokh2VG47JE9m39FWuR%2Bcc%3D-inUL%3DeRJEr18YHnFRQ%40mail.gmail.com
<https://groups.google.com/d/msgid/django-users/CAHN%3D9D4DNokh2VG47JE9m39FWuR%2Bcc%3D-inUL%3DeRJEr18YHnFRQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to
the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails
from it, send an email to
django-users+unsubscr...@googlegroups.com
<mailto:django-users+unsubscr...@googlegroups.com>.
To post to this group, send email to
django-users@googlegroups.com
<mailto:django-users@googlegroups.com>.
Visit this group at
http://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/53FE3BDE.8020305%40bristle.com
<https://groups.google.com/d/msgid/django-users/53FE3BDE.8020305%40bristle.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
--
Chief Systems Architect Proteus Technologies
<http://proteus-tech.com>
Chief Fan Biggest Fan Productions <http://biggestfan.net>
Personal blog where I am not your demographic
<http://notyourdemographic.com>.
This email intended solely for those who have received it.
If you have received this email by accident - well lucky you!!
--
You received this message because you are subscribed to the
Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails
from it, send an email to
django-users+unsubscr...@googlegroups.com
<mailto:django-users+unsubscr...@googlegroups.com>.
To post to this group, send email to
django-users@googlegroups.com
<mailto:django-users@googlegroups.com>.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/CAHN%3D9D5vXm4N7Rn4hr_BR4hFqviuw5ve6OEY1ty5t0fxcyD1rg%40mail.gmail.com
<https://groups.google.com/d/msgid/django-users/CAHN%3D9D5vXm4N7Rn4hr_BR4hFqviuw5ve6OEY1ty5t0fxcyD1rg%40mail.gmail.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the
Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to
django-users+unsubscr...@googlegroups.com
<mailto:django-users+unsubscr...@googlegroups.com>.
To post to this group, send email to
django-users@googlegroups.com
<mailto:django-users@googlegroups.com>.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/53FE4ABD.4020500%40bristle.com
<https://groups.google.com/d/msgid/django-users/53FE4ABD.4020500%40bristle.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
--
Chief Systems Architect Proteus Technologies
<http://proteus-tech.com>
Chief Fan Biggest Fan Productions <http://biggestfan.net>
Personal blog where I am not your demographic
<http://notyourdemographic.com>.
This email intended solely for those who have received it. If you
have received this email by accident - well lucky you!!
--
You received this message because you are subscribed to the
Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to django-users+unsubscr...@googlegroups.com
<mailto:django-users+unsubscr...@googlegroups.com>.
To post to this group, send email to
django-users@googlegroups.com <mailto:django-users@googlegroups.com>.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/CAHN%3D9D496C_98XwuOfZfVaHCs9AQg7J9XgeDCDyOxmGz1y2Anw%40mail.gmail.com
<https://groups.google.com/d/msgid/django-users/CAHN%3D9D496C_98XwuOfZfVaHCs9AQg7J9XgeDCDyOxmGz1y2Anw%40mail.gmail.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to django-users+unsubscr...@googlegroups.com
<mailto:django-users+unsubscr...@googlegroups.com>.
To post to this group, send email to django-users@googlegroups.com
<mailto:django-users@googlegroups.com>.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/5400CEE7.4%40bristle.com
<https://groups.google.com/d/msgid/django-users/5400CEE7.4%40bristle.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
--
Chief Systems Architect Proteus Technologies <http://proteus-tech.com>
Chief Fan Biggest Fan Productions <http://biggestfan.net>
Personal blog where I am not your demographic
<http://notyourdemographic.com>.
This email intended solely for those who have received it. If you have
received this email by accident - well lucky you!!
--
You received this message because you are subscribed to the Google
Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to django-users+unsubscr...@googlegroups.com
<mailto:django-users+unsubscr...@googlegroups.com>.
To post to this group, send email to django-users@googlegroups.com
<mailto:django-users@googlegroups.com>.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/CAHN%3D9D6hKkKJOKYCWQ7onuE0kWC_8yrNFvi7Z7QovmSMb0tBSA%40mail.gmail.com
<https://groups.google.com/d/msgid/django-users/CAHN%3D9D6hKkKJOKYCWQ7onuE0kWC_8yrNFvi7Z7QovmSMb0tBSA%40mail.gmail.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Django
users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/5405C8C8.3070207%40bristle.com.
For more options, visit https://groups.google.com/d/optout.
