Re: Django 1.6 and migrating to the new test runner: note on user experience
Hi Ivan, On 09/27/2013 03:07 AM, Ivan Kharlamov wrote: > Maybe I'm a careless kind of guy, but when I test out new versions of > Django, reading all of the release notes is rarely the first thing that > I do... my bad. :) My first action is usually firing up the test suites > and looking at the number of failed tests and deprecation warnings. > > As we all know, Django-1.6 is special in this regard, since it doesn't > run all the tests that the previous version would've discovered. Since > it doesn't raise any deprecation warnings, it becomes really easy to > assume that everything is fine and that the project is Django-1.6 ready, > when in fact it is completely broken. > > Perhaps, Django-1.6 test runner could raise a warning and inform the > user that the number of tests that would've run on the previous > versions, was omitted. > > My wild guess is that the lack of a deprecation warning is just asking > for some wildly entertaining user stories. > > What do you think? Thanks for reporting on your experience. I see (and was already aware of) the problem, but all the solutions I see are ugly in one way or another. Some options: 1) The status quo. If people run the new "manage.py check" command, they'll get a warning directing them to the release notes. But given that "manage.py check" is new, its unlikely anyone would run it unless they've already read the release notes, so this isn't worth much. 2) The new test runner issues some kind of warning unconditionally on 1.6 (this seems to be what you're proposing). That would certainly catch the people we need to catch, but it seems like an awfully big hammer. Once you know about the new test runner and have adjusted your tests, wouldn't it be irritating to keep seeing that warning every single time? And people creating new 1.6 projects would be irritated too, since the warning is useless to them. 3) A first-run warning that can be turned off (and presumably would be off by default in new projects). The obvious mechanism for turning it off would be something in settings. That either means introducing a new setting specifically for this warning, or warning anybody who doesn't have an explicit TEST_RUNNER override (meaning all 1.6 projects would be required to needlessly specify the new test runner in their settings in order to silence the warning.) 4) A variant of (3) where we introduce a more general setting that we can use for similar situations in the future. Something like a SILENCE_UPGRADE_WARNINGS setting that's a list or set of well-known strings, or something: SILENCE_UPGRADE_WARNINGS = ['discovery-runner'] or some such. 5) The new test runner, every time its run, goes out and looks for tests in the locations where the new runner won't find them (models.py files, app tests/__init__.py files, tests.py in INSTALLED_APPS that are not within the CWD, doctests) and warns you if it finds any. This would be a fair chunk of code that would be easy to get wrong, and I think it's really too late to add it to 1.6. It would also slow down test-running unnecessarily on every run. So, like I said, a bunch of bad options. Which is the least bad? Personally I'd be open to considering (1), (3), or (4). Interested in others' thoughts. Carl -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
Re: Django 1.6 and migrating to the new test runner: note on user experience
On Friday 27 September 2013 20:01:00 Carl Meyer wrote: > > 1) The status quo. If people run the new "manage.py check" command, > they'll get a warning directing them to the release notes. But given > that "manage.py check" is new, its unlikely anyone would run it unless > they've already read the release notes, so this isn't worth much. > > 2) The new test runner issues some kind of warning unconditionally on > 1.6 (this seems to be what you're proposing). That would certainly catch > the people we need to catch, but it seems like an awfully big hammer. > Once you know about the new test runner and have adjusted your tests, > wouldn't it be irritating to keep seeing that warning every single time? > And people creating new 1.6 projects would be irritated too, since the > warning is useless to them. > > 3) A first-run warning that can be turned off (and presumably would be > off by default in new projects). The obvious mechanism for turning it > off would be something in settings. That either means introducing a new > setting specifically for this warning, or warning anybody who doesn't > have an explicit TEST_RUNNER override (meaning all 1.6 projects would be > required to needlessly specify the new test runner in their settings in > order to silence the warning.) > > 4) A variant of (3) where we introduce a more general setting that we > can use for similar situations in the future. Something like a > SILENCE_UPGRADE_WARNINGS setting that's a list or set of well-known > strings, or something: SILENCE_UPGRADE_WARNINGS = ['discovery-runner'] > or some such. > > 5) The new test runner, every time its run, goes out and looks for tests > in the locations where the new runner won't find them (models.py files, > app tests/__init__.py files, tests.py in INSTALLED_APPS that are not > within the CWD, doctests) and warns you if it finds any. This would be a > fair chunk of code that would be easy to get wrong, and I think it's > really too late to add it to 1.6. It would also slow down test-running > unnecessarily on every run. > > So, like I said, a bunch of bad options. Which is the least bad? > Personally I'd be open to considering (1), (3), or (4). Interested in > others' thoughts. > I think in view of the GSoC work and the check command, (4) as phrased above would be mostly duplication of effort. The 'check' command, IIRC, already includes mechanisms to silence specific warnings. I think my preferred solution is a mix of your (3) and (5) -- a search for tests that can be turned off. I think (though I'm not too familiar with that area of the code) that the coding here is not too hard -- essentially, run the old code as well as the new code for building the test-suite, and diff the results. Another idea is a variation on your (4) -- add a setting to specify the Django version expected by the project; so if a project says, for example, DJANGO_USED="1.5", we emit suitable upgrade warnings. This may have a problem -- it may create unwarranted expectations of backward- compatibility support; people may think it tells newer Django versions to use the behavior of older versions, rather than warn on breaking changes. But I do find it has a certain appeal -- it's a lot like including the protocol version in messages. My 2 cents, Shai. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
Re: Django 1.6 and migrating to the new test runner: note on user experience
Hi, Carl, On 09/27/2013 09:01 PM, Carl Meyer wrote: > Hi Ivan, > > On 09/27/2013 03:07 AM, Ivan Kharlamov wrote: >> Maybe I'm a careless kind of guy, but when I test out new versions of >> Django, reading all of the release notes is rarely the first thing that >> I do... my bad. :) My first action is usually firing up the test suites >> and looking at the number of failed tests and deprecation warnings. >> >> As we all know, Django-1.6 is special in this regard, since it doesn't >> run all the tests that the previous version would've discovered. Since >> it doesn't raise any deprecation warnings, it becomes really easy to >> assume that everything is fine and that the project is Django-1.6 ready, >> when in fact it is completely broken. >> >> Perhaps, Django-1.6 test runner could raise a warning and inform the >> user that the number of tests that would've run on the previous >> versions, was omitted. >> >> My wild guess is that the lack of a deprecation warning is just asking >> for some wildly entertaining user stories. >> >> What do you think? > > Thanks for reporting on your experience. I see (and was already aware > of) the problem, but all the solutions I see are ugly in one way or > another. Some options: > > 1) The status quo. If people run the new "manage.py check" command, > they'll get a warning directing them to the release notes. But given > that "manage.py check" is new, its unlikely anyone would run it unless > they've already read the release notes, so this isn't worth much. > > 2) The new test runner issues some kind of warning unconditionally on > 1.6 (this seems to be what you're proposing). That would certainly catch > the people we need to catch, but it seems like an awfully big hammer. > Once you know about the new test runner and have adjusted your tests, > wouldn't it be irritating to keep seeing that warning every single time? > And people creating new 1.6 projects would be irritated too, since the > warning is useless to them. > > 3) A first-run warning that can be turned off (and presumably would be > off by default in new projects). The obvious mechanism for turning it > off would be something in settings. That either means introducing a new > setting specifically for this warning, or warning anybody who doesn't > have an explicit TEST_RUNNER override (meaning all 1.6 projects would be > required to needlessly specify the new test runner in their settings in > order to silence the warning.) > > 4) A variant of (3) where we introduce a more general setting that we > can use for similar situations in the future. Something like a > SILENCE_UPGRADE_WARNINGS setting that's a list or set of well-known > strings, or something: SILENCE_UPGRADE_WARNINGS = ['discovery-runner'] > or some such. > > 5) The new test runner, every time its run, goes out and looks for tests > in the locations where the new runner won't find them (models.py files, > app tests/__init__.py files, tests.py in INSTALLED_APPS that are not > within the CWD, doctests) and warns you if it finds any. This would be a > fair chunk of code that would be easy to get wrong, and I think it's > really too late to add it to 1.6. It would also slow down test-running > unnecessarily on every run. > > So, like I said, a bunch of bad options. Which is the least bad? > Personally I'd be open to considering (1), (3), or (4). Interested in > others' thoughts. > > Carl > Thank you for considering a fix! Personally, I'd prefer a solution proposed by Shai, which is (5). As another alternative to creating a new variable in settings.py, there is an option, albeit also a little bit ugly, to add a command line parameter, so that the tests could be run with './manage.py test application --no-upgrade-warning'. With best wishes, Ivan P.S. I've recently found another minor bug in the new test runner: https://code.djangoproject.com/ticket/21206#ticket -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/524AC1D4.3040909%40gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Django 1.6 and migrating to the new test runner: note on user experience
Hi Shai, On 09/27/2013 05:42 PM, Shai Berger wrote: > On Friday 27 September 2013 20:01:00 Carl Meyer wrote: >> 3) A first-run warning that can be turned off (and presumably would be >> off by default in new projects). The obvious mechanism for turning it >> off would be something in settings. That either means introducing a new >> setting specifically for this warning, or warning anybody who doesn't >> have an explicit TEST_RUNNER override (meaning all 1.6 projects would be >> required to needlessly specify the new test runner in their settings in >> order to silence the warning.) >> >> 4) A variant of (3) where we introduce a more general setting that we >> can use for similar situations in the future. Something like a >> SILENCE_UPGRADE_WARNINGS setting that's a list or set of well-known >> strings, or something: SILENCE_UPGRADE_WARNINGS = ['discovery-runner'] >> or some such. >> >> 5) The new test runner, every time its run, goes out and looks for tests >> in the locations where the new runner won't find them (models.py files, >> app tests/__init__.py files, tests.py in INSTALLED_APPS that are not >> within the CWD, doctests) and warns you if it finds any. This would be a >> fair chunk of code that would be easy to get wrong, and I think it's >> really too late to add it to 1.6. It would also slow down test-running >> unnecessarily on every run. >> >> So, like I said, a bunch of bad options. Which is the least bad? >> Personally I'd be open to considering (1), (3), or (4). Interested in >> others' thoughts. >> > I think in view of the GSoC work and the check command, (4) as phrased above > would be mostly duplication of effort. The 'check' command, IIRC, already > includes mechanisms to silence specific warnings. > > I think my preferred solution is a mix of your (3) and (5) -- a search for > tests that can be turned off. I think (though I'm not too familiar with that > area of the code) that the coding here is not too hard -- essentially, run > the > old code as well as the new code for building the test-suite, and diff the > results. I think (5) -- searching for tests using the old and new runners and comparing results -- is more attractive in theory than it is useful in practice. Because the new test runner does not run tests in "external" apps by default, it will always run a different set of tests than the old runner (except in the rare case of a project that uses no contrib apps or pip-installed apps with tests). So if we compare the set of collected tests, we will basically always get different results, and always issue the warning. So in practice it reduces to almost the same thing as an always-issued warning, but with more code and moving parts to achieve the same result. I don't think it's worth it. Which brings us to the same problem we had before - the best way to turn off a warning you would otherwise always get. > Another idea is a variation on your (4) -- add a setting to specify the > Django > version expected by the project; so if a project says, for example, > DJANGO_USED="1.5", we emit suitable upgrade warnings. > > This may have a problem -- it may create unwarranted expectations of backward- > compatibility support; people may think it tells newer Django versions to use > the behavior of older versions, rather than warn on breaking changes. But I > do > find it has a certain appeal -- it's a lot like including the protocol > version > in messages. I agree that this idea would allow us to be a lot smarter about certain upgrade warnings. Carl -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/524B7BB8.4080803%40oddbird.net. For more options, visit https://groups.google.com/groups/opt_out.
Re: Django 1.6 and migrating to the new test runner: note on user experience
On 10/01/2013 06:36 AM, Ivan Kharlamov wrote: > On 09/27/2013 09:01 PM, Carl Meyer wrote: >> On 09/27/2013 03:07 AM, Ivan Kharlamov wrote: > Thank you for considering a fix! > > Personally, I'd prefer a solution proposed by Shai, which is (5). I commented in my reply to Shai why I don't think (5) is really any better than a warning you always get unless you turn it off. > As another alternative to creating a new variable in settings.py, there > is an option, albeit also a little bit ugly, to add a command line > parameter, so that the tests could be run with './manage.py test > application --no-upgrade-warning'. I think this is worse than a setting. A setting at least you can set once and not have to think about it again, rather than have to arrange to issue it every single time you run tests. Carl -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/524B7C29.5000409%40oddbird.net. For more options, visit https://groups.google.com/groups/opt_out.
Re: Django 1.6 and migrating to the new test runner: note on user experience
I was working at system check framework [1] during this holiday. I've done some changes that will solve all problems considered in this thread. Maybe I'm a careless kind of guy, but when I test out new versions of > Django, reading all of the release notes is rarely the first thing that > I do... my bad. :) My first action is usually firing up the test suites > and looking at the number of failed tests and deprecation warnings. > As we all know, Django-1.6 is special in this regard, since it doesn't > run all the tests that the previous version would've discovered. Since > it doesn't raise any deprecation warnings, it becomes really easy to > assume that everything is fine and that the project is Django-1.6 ready, > when in fact it is completely broken. The first change is that when you run tests, all checks are performed, so you don't have to know that there is new check command in order to be notified about the test runner issue. The second problem is disabling the warning. The warning is issued when TEST_RUNNER is not defined in settings.py file. If you set it, there is no warning even if the new value is DiscoverRunner (the default one). So the warning can be silenced by setting TEST_RUNNER. And there is new SILENCED_SYSTEM_CHECKS setting so there is another way to disable the warning. I guess that my branch will be merged with 1.7 version and we cannot do much with 1.6 release. [1] https://groups.google.com/forum/#!topic/django-developers/_sde4eoNRyQ -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/7b19b6af-a972-47e5-b6da-af18f5a319d3%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
