#16622: django.test.TestCase slow using PostgreSQL 8.4.4
-------------------------------------+-------------------------------------
Reporter: | Owner: nobody
valhallasw | Status: new
Type: | Component: Database layer
Cleanup/optimization | (models, ORM)
Milestone: | Severity: Normal
Version: 1.3 | Keywords:
Resolution: | Has patch: 0
Triage Stage: Accepted | Needs tests: 0
Needs documentation: 0 | Easy pickings: 0
Patch needs improvement: 0 |
UI/UX: 0 |
-------------------------------------+-------------------------------------
Changes (by valhallasw):
* type: Bug => Cleanup/optimization
* component: Testing framework => Database layer (models, ORM)
Comment:
tl;dr: it was our fault. sorry for wasting your time :-(
In the end, it was caused because we used a buggy way of adding a database
programmatically (i.e. not using the configuration screen) to be able to
test on different data sets (this is useful, for instance, for testing
statistical functions). This method did not call
[https://code.djangoproject.com/browser/django/tags/releases/1.3/django/db/backends/__init__.py#L335
features.confirm](). This, in turn means the testing framework got `None`
when it asked for `features.supports_transactions` .
which returned None (instead of True). This was intepreted as 'it's not
True', and thus transactions were not used.
I'm not quite sure how sensible returning None is: IMO is would make more
sense to raise an Exception if no data is available, or calling confrm()
to actually /get/ the data (although creating and deleting a table while
in a query might not be the most sensible thing to do).
To give you a full log of my debugging:
{{{
(Pdb) b /lib/python2.6/site-
packages/django/db/backends/postgresql_psycopg2/base.py:43, "TRUNCATE" in
query
}}}
the breakpoint is hit a few times. The first time, this is the stack
trace:
{{{
(Pdb) w
/usr/lib/python2.6/bdb.py(372)run()
-> exec cmd in globals, locals
<string>(1)<module>()
/home/project/src/manage.py(16)<module>()
-> execute_manager(settings)
/lib/python2.6/site-
packages/django/core/management/__init__.py(438)execute_manager()
-> utility.execute()
/lib/python2.6/site-
packages/django/core/management/__init__.py(379)execute()
-> self.fetch_command(subcommand).run_from_argv(self.argv)
/lib/python2.6/site-
packages/django/core/management/base.py(191)run_from_argv()
-> self.execute(*args, **options.__dict__)
/lib/python2.6/site-
packages/django/core/management/base.py(220)execute()
-> output = self.handle(*args, **options)
/lib/python2.6/site-
packages/django/core/management/commands/test.py(37)handle()
-> failures = test_runner.run_tests(test_labels)
/lib/python2.6/site-packages/django/test/simple.py(359)run_tests()
-> old_config = self.setup_databases()
/home/project/src/djatabase/testing/PassableTestRunner.py(16)setup_databases()
-> retval = DjangoTestSuiteRunner.setup_databases(self, **kwargs)
/lib/python2.6/site-packages/django/test/simple.py(296)setup_databases()
-> test_db_name = connection.creation.create_test_db(self.verbosity,
autoclobber=not self.interactive)
/lib/python2.6/site-
packages/django/db/backends/creation.py(376)create_test_db()
-> database=self.connection.alias)
/lib/python2.6/site-
packages/django/core/management/__init__.py(166)call_command()
-> return klass.execute(*args, **defaults)
/lib/python2.6/site-
packages/django/core/management/base.py(220)execute()
-> output = self.handle(*args, **options)
/lib/python2.6/site-packages/django/core/management/base.py(351)handle()
-> return self.handle_noargs(**options)
/lib/python2.6/site-
packages/django/core/management/commands/flush.py(54)handle_noargs()
-> cursor.execute(sql)
> /lib/python2.6/site-
packages/django/db/backends/postgresql_psycopg2/base.py(43)execute()
-> try:
}}}
which is the `TRUNCATE` called by the test database creation.
The second and all later times, this is the stack trace:
{{{
(Pdb) w
/usr/lib/python2.6/bdb.py(372)run()
-> exec cmd in globals, locals
<string>(1)<module>()
/home/project/src/manage.py(16)<module>()
-> execute_manager(settings)
/lib/python2.6/site-
packages/django/core/management/__init__.py(438)execute_manager()
-> utility.execute()
/lib/python2.6/site-
packages/django/core/management/__init__.py(379)execute()
-> self.fetch_command(subcommand).run_from_argv(self.argv)
/lib/python2.6/site-
packages/django/core/management/base.py(191)run_from_argv()
-> self.execute(*args, **options.__dict__)
/lib/python2.6/site-
packages/django/core/management/base.py(220)execute()
-> output = self.handle(*args, **options)
/lib/python2.6/site-
packages/django/core/management/commands/test.py(37)handle()
-> failures = test_runner.run_tests(test_labels)
/lib/python2.6/site-packages/django/test/simple.py(360)run_tests()
-> result = self.run_suite(suite)
/lib/python2.6/site-packages/django/test/simple.py(316)run_suite()
-> return unittest.TextTestRunner(verbosity=self.verbosity,
failfast=self.failfast).run(suite)
/lib/python2.6/site-packages/django/utils/unittest/runner.py(162)run()
-> test(result)
/lib/python2.6/site-
packages/django/utils/unittest/suite.py(64)__call__()
-> return self.run(*args, **kwds)
/lib/python2.6/site-packages/django/utils/unittest/suite.py(84)run()
-> self._wrapped_run(result)
/lib/python2.6/site-
packages/django/utils/unittest/suite.py(116)_wrapped_run()
-> test(result)
/lib/python2.6/site-packages/django/test/testcases.py(292)__call__()
-> self._pre_setup()
/lib/python2.6/site-packages/django/test/testcases.py(259)_pre_setup()
-> self._fixture_setup()
/lib/python2.6/site-
packages/django/test/testcases.py(569)_fixture_setup()
-> return super(TestCase, self)._fixture_setup()
/lib/python2.6/site-
packages/django/test/testcases.py(271)_fixture_setup()
-> call_command('flush', verbosity=0, interactive=False, database=db)
/lib/python2.6/site-
packages/django/core/management/__init__.py(166)call_command()
-> return klass.execute(*args, **defaults)
/lib/python2.6/site-
packages/django/core/management/base.py(220)execute()
-> output = self.handle(*args, **options)
/lib/python2.6/site-packages/django/core/management/base.py(351)handle()
-> return self.handle_noargs(**options)
/lib/python2.6/site-
packages/django/core/management/commands/flush.py(54)handle_noargs()
-> cursor.execute(sql)
> /lib/python2.6/site-
packages/django/db/backends/postgresql_psycopg2/base.py(43)execute()
-> try:
}}}
Traversing the chain up to
{{{
> /lib/python2.6/site-
packages/django/test/testcases.py(271)_fixture_setup()
-> call_command('flush', verbosity=0, interactive=False, database=db)
}}}
https://code.djangoproject.com/browser/django/tags/releases/1.3/django/test/testcases.py#L260
where it's clear flush is called if _fixture_setup() is called.
traversing one more step up to
{{{
-> self._fixture_setup()
/lib/python2.6/site-
packages/django/test/testcases.py(569)_fixture_setup()
-> return super(TestCase, self)._fixture_setup()
}}}
https://code.djangoproject.com/browser/django/tags/releases/1.3/django/test/testcases.py#L560
with the relevant code
{{{#!python
if not connections_support_transactions():
return super(TestCase, self)._fixture_setup()
}}}
and indeed:
{{{
(Pdb) connections_support_transactions()
False
}}}
[https://code.djangoproject.com/browser/django/tags/releases/1.3/django/test/testcases.py#L551
connections_support_transactions()]:
{{{#!python
return all(conn.features.supports_transactions
for conn in connections.all())
}}}
and if I request [conn.features.supports_transactions for conn in
connections.all()]:
{{{
(Pdb) [conn.features.supports_transactions for conn in connections.all()]
[True, None]
}}}
OK, so that is the reason I'm seeing flushes. But why is
`support_transactions` `None`? Apparently it's not being filled.
Calling
{{{
[conn.features.confirm() for conn in djconn.all()]
}}}
seems to fix that.
End result:
{{{
$ ./manage.py test
Creating test database for alias 'default'...
.....
----------------------------------------------------------------------
Ran 5 tests in 0.567s
OK
}}}
yay!
--
Ticket URL: <https://code.djangoproject.com/ticket/16622#comment:10>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-updates?hl=en.
