On Tue, Oct 6, 2009 at 4:43 PM, Simon Willison <si...@simonwillison.net> wrote: > > One of the things that has been established at DjangoCon is that, as a > community, we don't have a strong enough culture of testing. This is > despite Django shipping with some good testing tools (TestClient and > the like). Anything we can do to make testing more attractive is a > Good Thing.
Completely agreed on this point. > In my opinion, one of the biggest psychological barriers to testing is > this: .. > On a brand new project, the tests fail. This has been discussed > before: > > http://groups.google.com/group/django-developers/browse_thread/thread/ac888ea8567ba466 > http://code.djangoproject.com/ticket/7611 ... > The decision was to wontfix it, since the test failures are correct - > if you don't have those templates set up, the auth application won't > work. That certainly makes sense, but the larger issue remains: a > brand new project doesn't pass its tests out of the box. Again, completely agreed. > A number of potential solutions come to mind: ... > 6. Something else entirely I think we need something else. In particular, I think we need to address the problem at a slightly deeper level - we need to acknowledge that we don't differentiate between application tests and integration tests within Django's test framework. To clarify my terms - an app test is a test that validates that the application logic is correct. This means validating that the foo() view does what the foo() view should do when it has a correctly configured environment. An app test is entirely self contained - it should require nothing from the containing project in order to pass. On the other hand, an integration test validates that the app has been correctly deployed into the containing project. This includes ensuring that all required templates are available, required views are deployed and required settings are correctly defined. Integration tests don't validate the internal logic of the app - they only validate that the environment is correctly configured. The failing contrib.auth tests are strange beasts in this regard. On the one hand, they are app tests that validate that the change password view works as expected. However, they aren't self contained. They require the existence of project-level template definitions to work, so they also perform an integration testing role. My original wontfix from #7611 was driven by a desire for the contrib.auth tests to act as integration tests. However, in retrospect, that isn't what they do. A deployment of contrib.auth isn't _required_ to deploy the password change views, and if you don't deploy those views, you don't need the templates either. To make matters worse, the default empty project falls into this category of 'broken' configurations. So - here's my suggestion for option 6, in two parts: 1. Reverse the decision of #7611, and make the current contrib.auth test suite a genuine collection of app tests. 2. Add extra tests to validate the integration case. These tests would be conditional, depending on exactly what has been deployed. For example: one test would look to see if contrib.auth.views.password_change has been deployed in the project. If it has, then the test would try to GET the page. The test passes if the page renders without throwing a missing template warning. However, if the view isn't deployed, the test is either not run, or is passed as a no-op. Essentially, the integration tests should be trying to catch every way that you could misconfigure your deployment of an application in a project. Once we have made these changes for contrib.auth, we should probably revisit the rest of contrib to make sure the rest of the test suite behaves as expected. Making this happen will require two pieces of infrastructure: * Firstly, we need a way to make app tests completely independent of the project environment. We started down this path when we added TestCase.urls, and the patch on #7611 adds another little piece of the puzzle. What we really need is a way to address this consistently for _all_ application settings - including those provided by the application itself. However, it isn't as simple as creating a new settings object, because some settings - such as database settings - need to be inherited from the test environment. Doing this in a consistent fashion may mean deprecating TestCase.urls, but I'm OK with that. * Secondly, we need to make sure that we can easily establish if integration conditions need to be tested. reverse() already does much of the job here, but some helpers to make it easy wouldn't go astray. Consideration needs to be given to namespaces - should the contrib.auth tests validate that admin correctly deploys the password change view, or should the namespace boundary be considered the edge of responsibility for contrib.auth integration tests? Looking longer term, we could also look at marking individual test as being 'app' or 'integration' - then, we could modify the test runner so you can run the entire integration suite, or run the app suite for contrib.auth. The benefit here is that most users have no reason to run the contrib.auth app test suite - it should always pass as shipped. However, Django's test runner would run these tests in order to validate that the app works correctly. This could possibly be achieved using TestSuite instances, combined with the test execution capabilities of #11627. The contrib apps would all define a suite() method that that includes the integration tests, plus an app_suite() method that returns all the app tests. ./manage.py test auth would invoke the integration suite; ./manage.py test auth.app_suite would invoke the app suite. > I think this issue is well worth solving. If we DO solve it, we could > even think about adding some stuff about running "./manage.py test" to > the tutorial. This is orthogonal IMHO. There are many things that should be added to the tutorial, including testing. The failing test issue is annoying, but I don't think it should stop us from adding testing to the tutorial list. In fact, this bug is a good argument in favour of adding testing to the tutorial, as it gives us an obvious place to address the issue for new users. Yours, Russ Magee %-) --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~----------~----~----~----~------~----~------~--~---
