Re: turn on "string_if_invalid" as a ‘development default’

[James Schneider]

>
> We have no "pre-release final check/test". We run the whole test-suite
> twice daily. This is done automatically be jenkins.
> We try to automate as much as possible. Manual examination is possible,
> but who wants to do this? I would like to have it automated.
> All tests run in around 10 minutes.
>

Hence the reason I mentioned that it will be dependent on your workflow. If
this is something you want to automate, obviously you'll need to figure out
how to get it to work with your CI implementation.


>
> I guess we will see bugs like  four times per year.
>
With "like this" I mean:
>
>  * bugs caused by the template engine returning the empy string instead of
> an exception
>  * bugs which are not detected by CI and get into production
>

I did a bit of poking around and I found this:

https://docs.djangoproject.com/en/dev/topics/testing/tools/#overriding-settings


It actually looks like it wouldn't be too difficult to take specific
existing tests and re-run them in the same test run with
the string_if_invalid tweaked. That would save you from having to modify
your CI, and would provide coverage in both a 'production mirror' state and
the alternative 'template bug catcher' state. You can probably subclass
your existing test classes and just decorate them appropriately, although
you might have other things you're looking for in those cases.

Never tried it myself, but looks to be worth a shot in your case.

-James

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CA%2Be%2BciWQPAgUMK%3DrvBAHbzwCXvYGfVGZe3n25mcCQOpdxnbVSA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: turn on "string_if_invalid" as a ‘development default’

[guettli]

Am Montag, 18. Januar 2016 12:02:15 UTC+1 schrieb James Schneider:
>
> A second batch of tests?
>>
>> I am unsure if I understood you: Do you mean new tests, or running the 
>> same
>> tests again, with a different settings file?
>>
>
> I meant the latter. Sorry for not being clear.
>  
>
>>
>> If the first: write new tests: How should these tests be different from 
>> the existing tests?
>>
>> If the second: Why not run the test with the settings which have 
>> string_if_invalid enabled in the first run?
>> What do you gain by running the tests twice?
>>
>
> The two sets of tests could potentially provide two sets of differing 
> results. The development setting you want to change is known to negatively 
> interact with templates used by the Django admin and probably third-party 
> applications that provide templates, per the link that you provided. With 
> that setting changed, you are testing an application that intentionally 
> behaves differently than what will be seen in production. That makes me 
> uncomfortable.
>
>
Yes, I understand "...behaves differently than what will be seen in 
production. That makes me uncomfortable." I see it the same.

I would like  string_if_invalid to raise an exception immediately. Maybe 
even in prod, since: "...behaves differently than what will be seen in 
production. That makes me uncomfortable." 

Zen of Python:

> Errors should never pass silently. 
 

For me, that means running two sets of tests. One of them will closely 
> match the production settings (probably the settings file you're using 
> now), and the other can be tweaked to potentially catch edge-cases or other 
> uncommon bugs that you can/have run in to, or other failure points that may 
> not be easily exposed by the standard settings file. Perhaps the second 
> customized set of tests may not be something that you build into your CI, 
> but may be part of a manual examination performed at various milestones. It 
> would depend on your workflow and CI process, and how often you think these 
> particular bugs may present themselves. Maybe only as a pre-release final 
> check?
>
>
We have no "pre-release final check/test". We run the whole test-suite 
twice daily. This is done automatically be jenkins.
We try to automate as much as possible. Manual examination is possible, but 
who wants to do this? I would like to have it automated.
All tests run in around 10 minutes. 

I guess we will see bugs like  four times per year. 
With "like this" I mean:

 * bugs caused by the template engine returning the empy string instead of 
an exception 
 * bugs which are not detected by CI and get into production

 

> Or, if you don't use the admin or any 3rd party applications, you may be 
> safe in enabling it. I personally wouldn't unless I was also doing so in 
> production.
>
>

Yes, dev and prod should not be much different

 Thank you for your input.

Regards,
  Thomas

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/892db691-3437-461e-80bc-1396ef854834%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: turn on "string_if_invalid" as a ‘development default’

[James Schneider]

>
> A second batch of tests?
>
> I am unsure if I understood you: Do you mean new tests, or running the same
> tests again, with a different settings file?
>

I meant the latter. Sorry for not being clear.


>
> If the first: write new tests: How should these tests be different from
> the existing tests?
>
> If the second: Why not run the test with the settings which have
> string_if_invalid enabled in the first run?
> What do you gain by running the tests twice?
>

The two sets of tests could potentially provide two sets of differing
results. The development setting you want to change is known to negatively
interact with templates used by the Django admin and probably third-party
applications that provide templates, per the link that you provided. With
that setting changed, you are testing an application that intentionally
behaves differently than what will be seen in production. That makes me
uncomfortable.

For me, that means running two sets of tests. One of them will closely
match the production settings (probably the settings file you're using
now), and the other can be tweaked to potentially catch edge-cases or other
uncommon bugs that you can/have run in to, or other failure points that may
not be easily exposed by the standard settings file. Perhaps the second
customized set of tests may not be something that you build into your CI,
but may be part of a manual examination performed at various milestones. It
would depend on your workflow and CI process, and how often you think these
particular bugs may present themselves. Maybe only as a pre-release final
check?

Or, if you don't use the admin or any 3rd party applications, you may be
safe in enabling it. I personally wouldn't unless I was also doing so in
production.

-James

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CA%2Be%2BciXvN-sqPDdhjTkGxpRsqKqV0FQcySWeWAxG6dFSbRdRvQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: turn on "string_if_invalid" as a ‘development default’

[guettli]

A second batch of tests?

I am unsure if I understood you: Do you mean new tests, or running the same
tests again, with a different settings file?

If the first: write new tests: How should these tests be different from the 
existing tests?

If the second: Why not run the test with the settings which have 
string_if_invalid enabled in the first run?
What do you gain by running the tests twice?

Regards,
  Thomas Güttler


Am Donnerstag, 14. Januar 2016 18:33:13 UTC+1 schrieb James Schneider:
>
> On Jan 14, 2016 7:50 AM, "guettli"  > wrote:
> >
> > I would like to do this:
> >
> >   turn on "string_if_invalid" as a ‘development default’
> >
> > The django docs say:
> >
> >  While string_if_invalid can be a useful debugging tool, it is a bad 
> idea to turn it on as a ‘development default’.
> >
> > 
> https://docs.djangoproject.com/en/1.9/ref/templates/api/#how-invalid-variables-are-handled
> >
> > We were hit by some nasty bugs in production environment which were not 
> detected in our continous integration plattform.
> >
> > If "string_if_invalid" (or former TEMPLATE_STRING_IF_INVALID) would have 
> been activated,
> > we would have noticed the bug, and the broken code would have gone live.
> >
> > How do you handle this?
> >
>
> Why not have a second batch of tests that use a settings.py file with a 
> non default value for "string_if_invalid" that runs if the default settings 
> file passes? I've never dealt with a CI platform, but I'm assuming you can 
> do something like that.
>
> - James
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/c8fe8229-cf00-4c7e-b6b5-0bc9daf5fc69%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: turn on "string_if_invalid" as a ‘development default’

[James Schneider]

On Jan 14, 2016 7:50 AM, "guettli"  wrote:
>
> I would like to do this:
>
>   turn on "string_if_invalid" as a ‘development default’
>
> The django docs say:
>
>  While string_if_invalid can be a useful debugging tool, it is a bad idea
to turn it on as a ‘development default’.
>
>
https://docs.djangoproject.com/en/1.9/ref/templates/api/#how-invalid-variables-are-handled
>
> We were hit by some nasty bugs in production environment which were not
detected in our continous integration plattform.
>
> If "string_if_invalid" (or former TEMPLATE_STRING_IF_INVALID) would have
been activated,
> we would have noticed the bug, and the broken code would have gone live.
>
> How do you handle this?
>

Why not have a second batch of tests that use a settings.py file with a non
default value for "string_if_invalid" that runs if the default settings
file passes? I've never dealt with a CI platform, but I'm assuming you can
do something like that.

- James

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CA%2Be%2BciXRm4rdwB-icDSsNFwqAhk-qYes9DieOKQXK%3D8mPVCGmQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.