date:20170104

We typically shy away from endorsing third-party libraries in the Django 
docs. I don't think it makes much sense to stay something to the effect of 
"The built-in views are too complex so we recommend using other library 
instead."

On Wednesday, January 4, 2017 at 7:30:54 AM UTC-5, Asif Saifuddin wrote:
>
> Hi,
>
> I will update the doc pointing vanilla views as simpler alternative 
> implementation.
>
> Thanks
>
> On Tuesday, January 3, 2017 at 7:20:24 PM UTC+6, Adam Johnson wrote:
>>
>> I think this is probably too disruptive a change for Django core, 
>> especially after so long with the current GCBV implementations - it would 
>> require all users to rewrite their CBV's. Possibly the documentation could 
>> recommend django-vanilla-views?
>>
>> On 3 January 2017 at 13:02, Asif Saifuddin  wrote:
>>
>>> Hi,
>>>
>>> I have started work on https://github.com/django/django/pull/7783 for 
>>> converting django built in generic views to django-vanilla-views.
>>>
>>> I would like to hear what you think before I proceed for more.
>>>
>>> Thanks,
>>>
>>> Asif
>>>
>>> On Friday, September 16, 2016 at 1:37:36 AM UTC+6, Asif Saifuddin wrote:

 Hi Tom,

 I am basically +1 to see this change in the django core. The package is 
 3 years old and should be tested enough. If you/other core team members 
 thinks that now is a good time to include it to core and deprecation of 
 older API, then I will be willing to work and send PR for this.

 Looking for others opinions.

 Thanks,

 Asif

 On Thursday, October 3, 2013 at 3:09:58 PM UTC+6, Tom Christie wrote:
>
> Hi folks,
>
> I recently released an alternative implementation of Django's existing 
> class based views.  The intention was to mirror the *exact* same set of 
> functionality that we currently provide, but simplify the implementation 
> and API.  It's nothing cleverer or grander than a clean re-write, but the 
> end result is *significantly* less complex.  The class hierarchy is 
> trivial, the API is much smaller, and the flow control is much more 
> obvious.
>
> I won't go into any more here, as there's plenty of detail in the 
> documentation:
>
> http://django-vanilla-views.org
>
> There's also useful context in the related blog post, here:
>
> http://dabapps.com/blog/fixing-djangos-generic-class-based-views/
>
> The difficult thing here really is that there's no obvious approach to 
> introducing something like this in a backwards compatible way.
>
> It might be that we could take an incremental approach using the 
> standard deprecation process, but given the nature of the GCBVs it'd 
> likely 
> be pretty awkward.
> There might be some bits of deprecation process we simply wouldn't be 
> able to deal with in any sensible way, and we'd certainly end up with a 
> seriously gnarly implementation during the deprecation period.
>
> I'd be interested in getting some opinions from folks on the following:
>
> * If a simpler GCBV implementation along the lines of 
> django-vanilla-views is something we think we should working towards.
> * What approaches we might be able to take to dealing with backwards 
> compatibility if we did want to do so.
>
> Thanks for your time.
>
>   Tom
>
> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to django-develop...@googlegroups.com.
>>> To post to this group, send email to django-d...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/django-developers/5792872e-157c-4ba6-87c1-bf0f5a07d981%40googlegroups.com
>>>  
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>>
>> -- 
>> Adam
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/99823816-8352-40ba-99b7-bfe5006424a0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Methodology for increasing the number of PBKDF2 iterations

2017-01-04 Thread Tobias McNulty

Here's an interesting tidbit from Alex Gaynor in 2014:

https://github.com/django/django/commit/6732566967888f2c12efee1146940c85c0154e60#diff-dd9c116fcefaf3916ace2608656311e0

It's worth noting that, if I'm understanding this correctly, there are two
slow versions of pbkdf2 we have to worry about -- the one bundled in Django
(
https://github.com/django/django/blob/6732566967888f2c12efee1146940c85c0154e60/django/utils/crypto.py#L142,
which is used pre-2.7.8 and pre-3.4 and claims to be 5x slower) and the
Python fallback for pbkdf2_hmac (which I suppose is used if OpenSSL is
unavailable (?) and claims to be 3x slower).

Martin, is it possible your version of Python 3 is not linked against
OpenSSL and hence is missing the fast version of pbkdf2_hmac? I haven't had
a chance to try your benchmark yet, but in a quick test I don't see any
difference between Python 3.5.2 and Python 2.7.12 on a Mac.

Tobias

On Wed, Jan 4, 2017 at 3:22 AM, Aymeric Augustin <
aymeric.augus...@polytechnique.org> wrote:

> Still, this benchmark shows Python 3.5 being 3 times slower than Python
> 2.7.
>
> This is a surprisingly large regression for this time-sensitive function.
>
> --
> Aymeric.
>
> On 4 Jan 2017, at 02:06, Tim Graham  wrote:
>
> The PBKDF2 speed improvements are in Python 2.7.8 and 3.4+, so you'd need
> to use Python 2.7.7 or earlier to get the slower version.
>
> On Tuesday, January 3, 2017 at 7:56:35 PM UTC-5, Martin Koistinen wrote:
>>
>> H, I just tried this using a simple management command to do some
>> basic benchmarking of password hashing. I made this little package Py2/Py3
>> compatible. You can find it here: https://github.com/mkois
>> tinen/hash_benchmark
>>
>> (Just install it from the repo into an existing project, then add
>> 'hash_benchmark' to your INSTALLED_APPS and you now have the management
>> command `hash_benchmark`.)
>>
>> I was expecting to see Py3 out-perform Py2 here by roughly 3X based on
>> this thread. Instead, I see *the opposite*.
>>
>> Python: 2.7.10 (default, Jul 13 2015, 12:05:58) [GCC 4.2.1 Compatible
>> Apple LLVM 6.1.0 (clang-602.0.53)]
>>
>> Django: 1.9.7
>>
>> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification
>> takes, on average, 0.0955s
>>
>> vs.
>>
>> Python: 3.5.1 (v3.5.1:37a07cee5969, Dec  5 2015, 21:12:44) [GCC 4.2.1
>> (Apple Inc. build 5666) (dot 3)]
>>
>> Django: 1.10.3
>>
>> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification
>> takes, on average, 0.2751s
>>
>> What am I missing here?
>>
>> On Tuesday, January 3, 2017 at 12:45:42 PM UTC-5, Martin Koistinen wrote:
>>>
>>> I think the best practice is to set the iterations as high as you can
>>> tolerate without adversely affecting the user experience as they log-in.
>>> Iteration numbers as high as 200,000 for SHA-256 or even more are not
>>> unheard of these days. Without looking at an application's password
>>> expiration policies, there's really no "one size fits all" number here.
>>>
>>> But, to be consistent with Django 1.x going forward, let's define 36,000
>>> iterations as "acceptable performance" for a Python2 with Django 1.11
>>> install on a typical piece of server hardware today (beginning of 2017). A
>>> useful benchmark would be to determine how many iterations would yield the
>>> same delay on a Py3 + Django 1.11 install on the same server.
>>>
>>> This should probably server as a *baseline* default number of
>>> iterations and, IMHO, there should probably be reasonable amount of
>>> encouragement in the documentation to set the number of iterations to a
>>> value as high as the application can tolerate. Ideally, there could be some
>>> in-built benchmarking tools to make this easier for the admin.
>>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/8d383765-c41e-403c-9e85-
> 09f31582f58f%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.goo

Re: Methodology for increasing the number of PBKDF2 iterations

2017-01-04 Thread Joey Wilhelm

FWIW, here are my own results from that benchmark (I ran each 5 times just
to account for any other system activity):

Python: 2.7.12, Django: 1.10.4
Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification takes,
on average, 0.0884s
Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification takes,
on average, 0.0854s
Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification takes,
on average, 0.1034s
Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification takes,
on average, 0.1119s
Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification takes,
on average, 0.0949s

Python: 3.5.2, Django: 1.10.4
Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification takes,
on average, 0.0876s
Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification takes,
on average, 0.0857s
Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification takes,
on average, 0.0872s
Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification takes,
on average, 0.0847s
Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification takes,
on average, 0.0874s

Python: 3.6.0, Django: 1.10.4
Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification takes,
on average, 0.0861s
Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification takes,
on average, 0.0789s
Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification takes,
on average, 0.0803s
Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification takes,
on average, 0.0779s
Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification takes,
on average, 0.0815s

This appears to agree with Tobias' results; this is also on a Mac. I can
toss in an older Python 2.7 as well if necessary or desired to see the
slower implementation. But I think this shows that there's a near enough
negligible speed difference in recent Python versions. Aside from perhaps a
very slight speedup in 3.6.

-Joey Wilhelm

On Wed, Jan 4, 2017 at 9:32 AM, Tobias McNulty 
wrote:

> Here's an interesting tidbit from Alex Gaynor in 2014:
>
> https://github.com/django/django/commit/6732566967888f2c12efee1146940c
> 85c0154e60#diff-dd9c116fcefaf3916ace2608656311e0
>
> It's worth noting that, if I'm understanding this correctly, there are two
> slow versions of pbkdf2 we have to worry about -- the one bundled in Django
> (https://github.com/django/django/blob/6732566967888f2c12efee1146940c
> 85c0154e60/django/utils/crypto.py#L142, which is used pre-2.7.8 and
> pre-3.4 and claims to be 5x slower) and the Python fallback for pbkdf2_hmac
> (which I suppose is used if OpenSSL is unavailable (?) and claims to be 3x
> slower).
>
> Martin, is it possible your version of Python 3 is not linked against
> OpenSSL and hence is missing the fast version of pbkdf2_hmac? I haven't had
> a chance to try your benchmark yet, but in a quick test I don't see any
> difference between Python 3.5.2 and Python 2.7.12 on a Mac.
>
> Tobias
>
> On Wed, Jan 4, 2017 at 3:22 AM, Aymeric Augustin  polytechnique.org> wrote:
>
>> Still, this benchmark shows Python 3.5 being 3 times slower than Python
>> 2.7.
>>
>> This is a surprisingly large regression for this time-sensitive function.
>>
>> --
>> Aymeric.
>>
>> On 4 Jan 2017, at 02:06, Tim Graham  wrote:
>>
>> The PBKDF2 speed improvements are in Python 2.7.8 and 3.4+, so you'd need
>> to use Python 2.7.7 or earlier to get the slower version.
>>
>> On Tuesday, January 3, 2017 at 7:56:35 PM UTC-5, Martin Koistinen wrote:
>>>
>>> H, I just tried this using a simple management command to do some
>>> basic benchmarking of password hashing. I made this little package Py2/Py3
>>> compatible. You can find it here: https://github.com/mkois
>>> tinen/hash_benchmark
>>>
>>> (Just install it from the repo into an existing project, then add
>>> 'hash_benchmark' to your INSTALLED_APPS and you now have the management
>>> command `hash_benchmark`.)
>>>
>>> I was expecting to see Py3 out-perform Py2 here by roughly 3X based on
>>> this thread. Instead, I see *the opposite*.
>>>
>>> Python: 2.7.10 (default, Jul 13 2015, 12:05:58) [GCC 4.2.1 Compatible
>>> Apple LLVM 6.1.0 (clang-602.0.53)]
>>>
>>> Django: 1.9.7
>>>
>>> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification
>>> takes, on average, 0.0955s
>>>
>>> vs.
>>>
>>> Python: 3.5.1 (v3.5.1:37a07cee5969, Dec  5 2015, 21:12:44) [GCC 4.2.1
>>> (Apple Inc. build 5666) (dot 3)]
>>>
>>> Django: 1.10.3
>>>
>>> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification
>>> takes, on average, 0.2751s
>>>
>>> What am I missing here?
>>>
>>> On Tuesday, January 3, 2017 at 12:45:42 PM UTC-5, Martin Koistinen wrote:

 I think the best practice is to set the iterations as high as you can
 tolerate without adversely affecting the user experience as they log-in.
 Iteration numbers as high as 200,000 for SHA-256 or even more are not
 unheard of these days. Without looking at an application's password
 expi

Re: Methodology for increasing the number of PBKDF2 iterations

2017-01-04 Thread Alex Gaynor

Python 2.7.12 will look the same as 3.5.x, they both have the optimized
implementation. Only 2.7.X where X<8 will have the slow implementation.

If someone was motivated, they could look at the PyPI bigquery and see what
versions of 2.7 people are using to install django.

Alex

On Wed, Jan 4, 2017 at 12:39 PM, Joey Wilhelm  wrote:

> FWIW, here are my own results from that benchmark (I ran each 5 times just
> to account for any other system activity):
>
> Python: 2.7.12, Django: 1.10.4
> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification takes,
> on average, 0.0884s
> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification takes,
> on average, 0.0854s
> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification takes,
> on average, 0.1034s
> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification takes,
> on average, 0.1119s
> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification takes,
> on average, 0.0949s
>
> Python: 3.5.2, Django: 1.10.4
> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification takes,
> on average, 0.0876s
> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification takes,
> on average, 0.0857s
> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification takes,
> on average, 0.0872s
> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification takes,
> on average, 0.0847s
> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification takes,
> on average, 0.0874s
>
> Python: 3.6.0, Django: 1.10.4
> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification takes,
> on average, 0.0861s
> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification takes,
> on average, 0.0789s
> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification takes,
> on average, 0.0803s
> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification takes,
> on average, 0.0779s
> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification takes,
> on average, 0.0815s
>
> This appears to agree with Tobias' results; this is also on a Mac. I can
> toss in an older Python 2.7 as well if necessary or desired to see the
> slower implementation. But I think this shows that there's a near enough
> negligible speed difference in recent Python versions. Aside from perhaps a
> very slight speedup in 3.6.
>
> -Joey Wilhelm
>
> On Wed, Jan 4, 2017 at 9:32 AM, Tobias McNulty 
> wrote:
>
>> Here's an interesting tidbit from Alex Gaynor in 2014:
>>
>> https://github.com/django/django/commit/6732566967888f2c12ef
>> ee1146940c85c0154e60#diff-dd9c116fcefaf3916ace2608656311e0
>>
>> It's worth noting that, if I'm understanding this correctly, there are
>> two slow versions of pbkdf2 we have to worry about -- the one bundled in
>> Django (https://github.com/django/django/blob/6732566967888f2c12efe
>> e1146940c85c0154e60/django/utils/crypto.py#L142, which is used pre-2.7.8
>> and pre-3.4 and claims to be 5x slower) and the Python fallback for
>> pbkdf2_hmac (which I suppose is used if OpenSSL is unavailable (?) and
>> claims to be 3x slower).
>>
>> Martin, is it possible your version of Python 3 is not linked against
>> OpenSSL and hence is missing the fast version of pbkdf2_hmac? I haven't had
>> a chance to try your benchmark yet, but in a quick test I don't see any
>> difference between Python 3.5.2 and Python 2.7.12 on a Mac.
>>
>> Tobias
>>
>> On Wed, Jan 4, 2017 at 3:22 AM, Aymeric Augustin <
>> aymeric.augus...@polytechnique.org> wrote:
>>
>>> Still, this benchmark shows Python 3.5 being 3 times slower than Python
>>> 2.7.
>>>
>>> This is a surprisingly large regression for this time-sensitive function.
>>>
>>> --
>>> Aymeric.
>>>
>>> On 4 Jan 2017, at 02:06, Tim Graham  wrote:
>>>
>>> The PBKDF2 speed improvements are in Python 2.7.8 and 3.4+, so you'd
>>> need to use Python 2.7.7 or earlier to get the slower version.
>>>
>>> On Tuesday, January 3, 2017 at 7:56:35 PM UTC-5, Martin Koistinen wrote:

 H, I just tried this using a simple management command to do some
 basic benchmarking of password hashing. I made this little package Py2/Py3
 compatible. You can find it here: https://github.com/mkois
 tinen/hash_benchmark

 (Just install it from the repo into an existing project, then add
 'hash_benchmark' to your INSTALLED_APPS and you now have the management
 command `hash_benchmark`.)

 I was expecting to see Py3 out-perform Py2 here by roughly 3X based on
 this thread. Instead, I see *the opposite*.

 Python: 2.7.10 (default, Jul 13 2015, 12:05:58) [GCC 4.2.1 Compatible
 Apple LLVM 6.1.0 (clang-602.0.53)]

 Django: 1.9.7

 Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification
 takes, on average, 0.0955s

 vs.

 Python: 3.5.1 (v3.5.1:37a07cee5969, Dec  5 2015, 21:12:44) [GCC 4.2.1
 (Apple Inc. build 5666) (dot 3)]

 Django: 1.10.3

 Using cipher:

Re: Methodology for increasing the number of PBKDF2 iterations

Thanks Alex, TIL
https://mail.python.org/pipermail/distutils-sig/2016-May/028986.html

On 4 January 2017 at 17:42, Alex Gaynor  wrote:

> Python 2.7.12 will look the same as 3.5.x, they both have the optimized
> implementation. Only 2.7.X where X<8 will have the slow implementation.
>
> If someone was motivated, they could look at the PyPI bigquery and see
> what versions of 2.7 people are using to install django.
>
> Alex
>
> On Wed, Jan 4, 2017 at 12:39 PM, Joey Wilhelm 
> wrote:
>
>> FWIW, here are my own results from that benchmark (I ran each 5 times
>> just to account for any other system activity):
>>
>> Python: 2.7.12, Django: 1.10.4
>> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification
>> takes, on average, 0.0884s
>> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification
>> takes, on average, 0.0854s
>> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification
>> takes, on average, 0.1034s
>> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification
>> takes, on average, 0.1119s
>> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification
>> takes, on average, 0.0949s
>>
>> Python: 3.5.2, Django: 1.10.4
>> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification
>> takes, on average, 0.0876s
>> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification
>> takes, on average, 0.0857s
>> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification
>> takes, on average, 0.0872s
>> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification
>> takes, on average, 0.0847s
>> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification
>> takes, on average, 0.0874s
>>
>> Python: 3.6.0, Django: 1.10.4
>> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification
>> takes, on average, 0.0861s
>> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification
>> takes, on average, 0.0789s
>> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification
>> takes, on average, 0.0803s
>> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification
>> takes, on average, 0.0779s
>> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification
>> takes, on average, 0.0815s
>>
>> This appears to agree with Tobias' results; this is also on a Mac. I can
>> toss in an older Python 2.7 as well if necessary or desired to see the
>> slower implementation. But I think this shows that there's a near enough
>> negligible speed difference in recent Python versions. Aside from perhaps a
>> very slight speedup in 3.6.
>>
>> -Joey Wilhelm
>>
>> On Wed, Jan 4, 2017 at 9:32 AM, Tobias McNulty 
>> wrote:
>>
>>> Here's an interesting tidbit from Alex Gaynor in 2014:
>>>
>>> https://github.com/django/django/commit/6732566967888f2c12ef
>>> ee1146940c85c0154e60#diff-dd9c116fcefaf3916ace2608656311e0
>>>
>>> It's worth noting that, if I'm understanding this correctly, there are
>>> two slow versions of pbkdf2 we have to worry about -- the one bundled in
>>> Django (https://github.com/django/django/blob/6732566967888f2c12efe
>>> e1146940c85c0154e60/django/utils/crypto.py#L142, which is used
>>> pre-2.7.8 and pre-3.4 and claims to be 5x slower) and the Python fallback
>>> for pbkdf2_hmac (which I suppose is used if OpenSSL is unavailable (?) and
>>> claims to be 3x slower).
>>>
>>> Martin, is it possible your version of Python 3 is not linked against
>>> OpenSSL and hence is missing the fast version of pbkdf2_hmac? I haven't had
>>> a chance to try your benchmark yet, but in a quick test I don't see any
>>> difference between Python 3.5.2 and Python 2.7.12 on a Mac.
>>>
>>> Tobias
>>>
>>> On Wed, Jan 4, 2017 at 3:22 AM, Aymeric Augustin <
>>> aymeric.augus...@polytechnique.org> wrote:
>>>
 Still, this benchmark shows Python 3.5 being 3 times slower than Python
 2.7.

 This is a surprisingly large regression for this time-sensitive
 function.

 --
 Aymeric.

 On 4 Jan 2017, at 02:06, Tim Graham  wrote:

 The PBKDF2 speed improvements are in Python 2.7.8 and 3.4+, so you'd
 need to use Python 2.7.7 or earlier to get the slower version.

 On Tuesday, January 3, 2017 at 7:56:35 PM UTC-5, Martin Koistinen wrote:
>
> H, I just tried this using a simple management command to do some
> basic benchmarking of password hashing. I made this little package Py2/Py3
> compatible. You can find it here: https://github.com/mkois
> tinen/hash_benchmark
>
> (Just install it from the repo into an existing project, then add
> 'hash_benchmark' to your INSTALLED_APPS and you now have the management
> command `hash_benchmark`.)
>
> I was expecting to see Py3 out-perform Py2 here by roughly 3X based on
> this thread. Instead, I see *the opposite*.
>
> Python: 2.7.10 (default, Jul 13 2015, 12:05:58) [GCC 4.2.1 Compatible
> Apple LLVM 6.1.0 (clang-602.0.53)]
>
> Django: 1.9.7
>
> Using cipher: "p

Re: Methodology for increasing the number of PBKDF2 iterations

2017-01-04 Thread Joey Wilhelm

Okay, for good measure, here's with 2.7.7. And yeah, looks like almost 4x
slower.

Python: 2.7.7, Django: 1.10.4
Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification takes,
on average, 0.3050s
Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification takes,
on average, 0.3096s
Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification takes,
on average, 0.3064s
Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification takes,
on average, 0.3162s
Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification takes,
on average, 0.3054s

I'm not sure what conclusions this can help with, but at least they are
some solid-ish numbers to work with.

-Joey Wilhelm

On Wed, Jan 4, 2017 at 10:42 AM, Alex Gaynor  wrote:

> Python 2.7.12 will look the same as 3.5.x, they both have the optimized
> implementation. Only 2.7.X where X<8 will have the slow implementation.
>
> If someone was motivated, they could look at the PyPI bigquery and see
> what versions of 2.7 people are using to install django.
>
> Alex
>
> On Wed, Jan 4, 2017 at 12:39 PM, Joey Wilhelm 
> wrote:
>
>> FWIW, here are my own results from that benchmark (I ran each 5 times
>> just to account for any other system activity):
>>
>> Python: 2.7.12, Django: 1.10.4
>> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification
>> takes, on average, 0.0884s
>> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification
>> takes, on average, 0.0854s
>> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification
>> takes, on average, 0.1034s
>> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification
>> takes, on average, 0.1119s
>> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification
>> takes, on average, 0.0949s
>>
>> Python: 3.5.2, Django: 1.10.4
>> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification
>> takes, on average, 0.0876s
>> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification
>> takes, on average, 0.0857s
>> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification
>> takes, on average, 0.0872s
>> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification
>> takes, on average, 0.0847s
>> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification
>> takes, on average, 0.0874s
>>
>> Python: 3.6.0, Django: 1.10.4
>> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification
>> takes, on average, 0.0861s
>> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification
>> takes, on average, 0.0789s
>> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification
>> takes, on average, 0.0803s
>> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification
>> takes, on average, 0.0779s
>> Using cipher: "pbkdf2_sha256" with 100,000 iterations, verification
>> takes, on average, 0.0815s
>>
>> This appears to agree with Tobias' results; this is also on a Mac. I can
>> toss in an older Python 2.7 as well if necessary or desired to see the
>> slower implementation. But I think this shows that there's a near enough
>> negligible speed difference in recent Python versions. Aside from perhaps a
>> very slight speedup in 3.6.
>>
>> -Joey Wilhelm
>>
>> On Wed, Jan 4, 2017 at 9:32 AM, Tobias McNulty 
>> wrote:
>>
>>> Here's an interesting tidbit from Alex Gaynor in 2014:
>>>
>>> https://github.com/django/django/commit/6732566967888f2c12ef
>>> ee1146940c85c0154e60#diff-dd9c116fcefaf3916ace2608656311e0
>>>
>>> It's worth noting that, if I'm understanding this correctly, there are
>>> two slow versions of pbkdf2 we have to worry about -- the one bundled in
>>> Django (https://github.com/django/django/blob/6732566967888f2c12efe
>>> e1146940c85c0154e60/django/utils/crypto.py#L142, which is used
>>> pre-2.7.8 and pre-3.4 and claims to be 5x slower) and the Python fallback
>>> for pbkdf2_hmac (which I suppose is used if OpenSSL is unavailable (?) and
>>> claims to be 3x slower).
>>>
>>> Martin, is it possible your version of Python 3 is not linked against
>>> OpenSSL and hence is missing the fast version of pbkdf2_hmac? I haven't had
>>> a chance to try your benchmark yet, but in a quick test I don't see any
>>> difference between Python 3.5.2 and Python 2.7.12 on a Mac.
>>>
>>> Tobias
>>>
>>> On Wed, Jan 4, 2017 at 3:22 AM, Aymeric Augustin <
>>> aymeric.augus...@polytechnique.org> wrote:
>>>
 Still, this benchmark shows Python 3.5 being 3 times slower than Python
 2.7.

 This is a surprisingly large regression for this time-sensitive
 function.

 --
 Aymeric.

 On 4 Jan 2017, at 02:06, Tim Graham  wrote:

 The PBKDF2 speed improvements are in Python 2.7.8 and 3.4+, so you'd
 need to use Python 2.7.7 or earlier to get the slower version.

 On Tuesday, January 3, 2017 at 7:56:35 PM UTC-5, Martin Koistinen wrote:
>
> H, I just tried this using a simple management command to do some
> basic benchmarking of password has

Re: Methodology for increasing the number of PBKDF2 iterations

2017-01-04 Thread Martin Koistinen

I think this is a pretty solid guess. Bear in mind this was a direct 
install from Python.org.

The important thing here is, this demonstrates that we cannot just assume 
that all Python 3 installs have a "fast" PBKDF2 implementation =/

On Wednesday, January 4, 2017 at 11:33:17 AM UTC-5, Tobias McNulty wrote:

> ... 
>
Martin, is it possible your version of Python 3 is not linked against 
> OpenSSL and hence is missing the fast version of pbkdf2_hmac? I haven't had 
> a chance to try your benchmark yet, but in a quick test I don't see any 
> difference between Python 3.5.2 and Python 2.7.12 on a Mac.
>
> Tobias
>

 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/568d57ba-81c3-4b52-9fd4-99f3c036b6bc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Methodology for increasing the number of PBKDF2 iterations

2017-01-04 Thread Alex Gaynor

If anyone is curious about the breakdown of versions, I used the following
query:

SELECT
  REGEXP_EXTRACT(details.python, r"^([^\.]+\.[^\.]+\.[^\.]+)") as
python_version,
  COUNT(*) as download_count,
FROM
  TABLE_DATE_RANGE(
[the-psf:pypi.downloads],
DATE_ADD(CURRENT_TIMESTAMP(), -2, "week"),
CURRENT_TIMESTAMP()
  )
WHERE
  LOWER(file.project) = 'django'
GROUP BY
  python_version,
ORDER BY
  download_count DESC
LIMIT 100


And got these for the top 9:

Row python_version download_count
1 3.5.2 75888
2 2.7.12 65879
3 2.7.6 63925
4 null 56744
5 2.7.9 40378
6 2.7.10 25213
7 3.4.3 23223
8 2.7.13 20657
9 2.7.5 17256

(That's an HTML table, no clue what happens if you use a plaintext email
client)

Alex

On Wed, Jan 4, 2017 at 2:13 PM, Martin Koistinen 
wrote:

> I think this is a pretty solid guess. Bear in mind this was a direct
> install from Python.org.
>
> The important thing here is, this demonstrates that we cannot just assume
> that all Python 3 installs have a "fast" PBKDF2 implementation =/
>
> On Wednesday, January 4, 2017 at 11:33:17 AM UTC-5, Tobias McNulty wrote:
>
>> ...
>>
> Martin, is it possible your version of Python 3 is not linked against
>> OpenSSL and hence is missing the fast version of pbkdf2_hmac? I haven't had
>> a chance to try your benchmark yet, but in a quick test I don't see any
>> difference between Python 3.5.2 and Python 2.7.12 on a Mac.
>>
>> Tobias
>>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/568d57ba-81c3-4b52-9fd4-
> 99f3c036b6bc%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>



-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: D1B3 ADC0 E023 8CA6

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFRnB2XPoSwZT01W8sGA%3DpgzVBKq7bNCkb2Z8v_9ADxrzh2ENw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[ANNOUNCE] Django bugfix release: 1.10.5

Details are available on the Django project weblog:

https://www.djangoproject.com/weblog/2017/jan/04/bugfix-release/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/9d110110-7192-4ed5-8d02-be450e693b80%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Changing {% include %} to strip a trailing newline

Shortly after template widget rendering was merged, an issue about extra 
newlines appearing in the rendered output was reported [0]:

For example, from django-money:

UIC-Franc
\n\n
US Dollar
\n\n

The newlines aren't useful and they break assertions like this:

US Dollar in form.as_p

--- end report---

The reporter suggested removing the trailing newline in the attrs.html 
template but Adam Johnson reported: "POSIX states that all text files 
should end with a newline, and some tools break when operating on text 
files missing the final newline. That's why git has the warning \ No 
newline at end of file and Github has a warning symbol for the missing 
newline."

I suggested that perhaps {% include %} could do .rstrip('\n') on whatever 
it renders.

Preston pointed out that Jinja2 does something similar:

http://jinja.pocoo.org/docs/dev/templates/#whitespace-control

   - a single trailing newline is stripped if present
   - other whitespace (spaces, tabs, newlines etc.) is returned unchanged

I noted that the issue of {% include %} adding extra newlines was raised in 
#9198 [1] but Malcolm said, "For backwards compatibility reasons we cannot 
change the default behaviour now (there will be templates that rely on the 
newline being inserted)." I was skeptical this would be a burdensome change.

Carl replied:  "It seems quite likely to me that there are templates in the 
wild relying on preservation of the final newline. People render 
preformatted text (e.g. text emails) using DTL. I probably have some text 
email templates in older projects myself that would break with that change. 

We could add a new option to {% include %}, though." Adam also said, "I too 
have used DTL for text emails that would break under that behaviour. New 
option to include sounds good to me."


Me again: In the long run, having {% include %} remove the trailing newline 
seems like a more sensible default. For example, I wouldn't expect this 
code to have a newline inserted between it:


{% include "foo.txt" %}
{% include "bar.txt" %}
 

An option for {% include %} seems superfluous given that if you were writing


{% include "foo.txt" %}{% include "bar.txt" %}
 

now you can write the first thing which is much more intuitive.


How about a keep_trailing_newline TEMPLATES option for backwards 
compatibility for those who don't want to adapt their templates for the new 
behavior? Jinja2 has that option.

Carl replied: An engine option may be better than an option to {% include %}, 
though it doesn't allow us to ensure that we strip the newline in the 
specific case of attrs.

How we default the engine option I guess just depends on how seriously we 
take backwards compatibility. If we default it to strip and make people add 
the config to preserve the old behavior, that's not really backwards 
compatible. Historically (as seen in Malcolm's comment) we would choose to 
err on the side of actual backwards compatibility in a case like this, even 
if it didn't result in the ideal future behavior. But the adaptation isn't 
hard in this case, so I won't object if the choice is to break back-compat.


If it's not a per-include choice, of course, we have to break overall 
back-compat to preserve form-attr-rendering back-compat.


What do you think?

[0] https://github.com/django/django/pull/7769
[1] https://code.djangoproject.com/ticket/9198

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/8124d314-eb26-4465-9fc8-5b04fe2ba618%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Working towards a simpler GCBV implementation?

Fair enough, there are other community resources for this.

The PR adding a pointer in the docs to ccbv.co.uk in
https://github.com/django/django/pull/7785 is a good idea that came from
this thread though.

On 4 January 2017 at 13:08, Tim Graham  wrote:

> We typically shy away from endorsing third-party libraries in the Django
> docs. I don't think it makes much sense to stay something to the effect of
> "The built-in views are too complex so we recommend using other library
> instead."
>
>
> On Wednesday, January 4, 2017 at 7:30:54 AM UTC-5, Asif Saifuddin wrote:
>>
>> Hi,
>>
>> I will update the doc pointing vanilla views as simpler alternative
>> implementation.
>>
>> Thanks
>>
>> On Tuesday, January 3, 2017 at 7:20:24 PM UTC+6, Adam Johnson wrote:
>>>
>>> I think this is probably too disruptive a change for Django core,
>>> especially after so long with the current GCBV implementations - it would
>>> require all users to rewrite their CBV's. Possibly the documentation could
>>> recommend django-vanilla-views?
>>>
>>> On 3 January 2017 at 13:02, Asif Saifuddin  wrote:
>>>
 Hi,

 I have started work on https://github.com/django/django/pull/7783 for
 converting django built in generic views to django-vanilla-views.

 I would like to hear what you think before I proceed for more.

 Thanks,

 Asif

 On Friday, September 16, 2016 at 1:37:36 AM UTC+6, Asif Saifuddin wrote:
>
> Hi Tom,
>
> I am basically +1 to see this change in the django core. The package
> is 3 years old and should be tested enough. If you/other core team members
> thinks that now is a good time to include it to core and deprecation of
> older API, then I will be willing to work and send PR for this.
>
> Looking for others opinions.
>
> Thanks,
>
> Asif
>
> On Thursday, October 3, 2013 at 3:09:58 PM UTC+6, Tom Christie wrote:
>>
>> Hi folks,
>>
>> I recently released an alternative implementation of Django's
>> existing class based views.  The intention was to mirror the *exact* same
>> set of functionality that we currently provide, but simplify the
>> implementation and API.  It's nothing cleverer or grander than a clean
>> re-write, but the end result is *significantly* less complex.  The class
>> hierarchy is trivial, the API is much smaller, and the flow control is 
>> much
>> more obvious.
>>
>> I won't go into any more here, as there's plenty of detail in the
>> documentation:
>>
>> http://django-vanilla-views.org
>>
>> There's also useful context in the related blog post, here:
>>
>> http://dabapps.com/blog/fixing-djangos-generic-class-based-views/
>>
>> The difficult thing here really is that there's no obvious approach
>> to introducing something like this in a backwards compatible way.
>>
>> It might be that we could take an incremental approach using the
>> standard deprecation process, but given the nature of the GCBVs it'd 
>> likely
>> be pretty awkward.
>> There might be some bits of deprecation process we simply wouldn't be
>> able to deal with in any sensible way, and we'd certainly end up with a
>> seriously gnarly implementation during the deprecation period.
>>
>> I'd be interested in getting some opinions from folks on the
>> following:
>>
>> * If a simpler GCBV implementation along the lines of
>> django-vanilla-views is something we think we should working towards.
>> * What approaches we might be able to take to dealing with backwards
>> compatibility if we did want to do so.
>>
>> Thanks for your time.
>>
>>   Tom
>>
>> --
 You received this message because you are subscribed to the Google
 Groups "Django developers (Contributions to Django itself)" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to django-develop...@googlegroups.com.
 To post to this group, send email to django-d...@googlegroups.com.
 Visit this group at https://groups.google.com/group/django-developers.
 To view this discussion on the web visit https://groups.google.com/d/ms
 gid/django-developers/5792872e-157c-4ba6-87c1-bf0f5a07d981%
 40googlegroups.com
 
 .
 For more options, visit https://groups.google.com/d/optout.

>>>
>>>
>>>
>>> --
>>> Adam
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://gr

Re: Changing {% include %} to strip a trailing newline

2017-01-04 Thread Anthony King

I think adding the option would be the better approach, and add it in as
default for new projects.
Having used templates for text emails, I do find it unintuitive to write
them because of the newline behaviour.

On 4 Jan 2017 19:58, "Tim Graham"  wrote:

> Shortly after template widget rendering was merged, an issue about extra
> newlines appearing in the rendered output was reported [0]:
>
> For example, from django-money:
>
> UIC-Franc
> \n\n
> US Dollar
> \n\n
>
> The newlines aren't useful and they break assertions like this:
>
> US Dollar in form.as_p
>
> --- end report---
>
> The reporter suggested removing the trailing newline in the attrs.html
> template but Adam Johnson reported: "POSIX states that all text files
> should end with a newline, and some tools break when operating on text
> files missing the final newline. That's why git has the warning \ No
> newline at end of file and Github has a warning symbol for the missing
> newline."
>
> I suggested that perhaps {% include %} could do .rstrip('\n') on whatever
> it renders.
>
> Preston pointed out that Jinja2 does something similar:
>
> http://jinja.pocoo.org/docs/dev/templates/#whitespace-control
>
>- a single trailing newline is stripped if present
>- other whitespace (spaces, tabs, newlines etc.) is returned unchanged
>
> I noted that the issue of {% include %} adding extra newlines was raised
> in #9198 [1] but Malcolm said, "For backwards compatibility reasons we
> cannot change the default behaviour now (there will be templates that rely
> on the newline being inserted)." I was skeptical this would be a burdensome
> change.
>
> Carl replied:  "It seems quite likely to me that there are templates in
> the wild relying on preservation of the final newline. People render
> preformatted text (e.g. text emails) using DTL. I probably have some text
> email templates in older projects myself that would break with that change.
>
> We could add a new option to {% include %}, though." Adam also said, "I
> too have used DTL for text emails that would break under that behaviour.
> New option to include sounds good to me."
>
>
> Me again: In the long run, having {% include %} remove the trailing
> newline seems like a more sensible default. For example, I wouldn't expect
> this code to have a newline inserted between it:
>
>
> {% include "foo.txt" %}
> {% include "bar.txt" %}
>
>
> An option for {% include %} seems superfluous given that if you were
> writing
>
>
> {% include "foo.txt" %}{% include "bar.txt" %}
>
>
> now you can write the first thing which is much more intuitive.
>
>
> How about a keep_trailing_newline TEMPLATES option for backwards
> compatibility for those who don't want to adapt their templates for the new
> behavior? Jinja2 has that option.
>
> Carl replied: An engine option may be better than an option to {% include
> %}, though it doesn't allow us to ensure that we strip the newline in the
> specific case of attrs.
>
> How we default the engine option I guess just depends on how seriously we
> take backwards compatibility. If we default it to strip and make people add
> the config to preserve the old behavior, that's not really backwards
> compatible. Historically (as seen in Malcolm's comment) we would choose to
> err on the side of actual backwards compatibility in a case like this, even
> if it didn't result in the ideal future behavior. But the adaptation isn't
> hard in this case, so I won't object if the choice is to break back-compat.
>
>
> If it's not a per-include choice, of course, we have to break overall
> back-compat to preserve form-attr-rendering back-compat.
> 
>
> What do you think?
>
> [0] https://github.com/django/django/pull/7769
> [1] https://code.djangoproject.com/ticket/9198
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/8124d314-eb26-4465-9fc8-
> 5b04fe2ba618%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion o

Re: Changing {% include %} to strip a trailing newline

2017-01-04 Thread Sam Willis

Could this be set within the template rather than the include tag? So for 
example have a new tag such as {% strip_final_new_line %} that when 
included at the end of a template immediately before the final new line 
would instruct it to be striped.

This stops the user from having to remember to use a special option on the 
include tag for each use of a template that requires it - it also work on a 
normal (not included) template render as well, if that is wanted.

An alternative tag could be {% endtemplate %} that can be placed anywhere 
in the template, forcing the end of rendering?

On Wednesday, January 4, 2017 at 7:58:42 PM UTC, Tim Graham wrote:
>
> Shortly after template widget rendering was merged, an issue about extra 
> newlines appearing in the rendered output was reported [0]:
>
> For example, from django-money:
>
> UIC-Franc
> \n\n
> US Dollar
> \n\n
>
> The newlines aren't useful and they break assertions like this:
>
> US Dollar in form.as_p
>
> --- end report---
>
> The reporter suggested removing the trailing newline in the attrs.html 
> template but Adam Johnson reported: "POSIX states that all text files 
> should end with a newline, and some tools break when operating on text 
> files missing the final newline. That's why git has the warning \ No 
> newline at end of file and Github has a warning symbol for the missing 
> newline."
>
> I suggested that perhaps {% include %} could do .rstrip('\n') on whatever 
> it renders.
>
> Preston pointed out that Jinja2 does something similar:
>
> http://jinja.pocoo.org/docs/dev/templates/#whitespace-control
>
>- a single trailing newline is stripped if present
>- other whitespace (spaces, tabs, newlines etc.) is returned unchanged
>
> I noted that the issue of {% include %} adding extra newlines was raised 
> in #9198 [1] but Malcolm said, "For backwards compatibility reasons we 
> cannot change the default behaviour now (there will be templates that rely 
> on the newline being inserted)." I was skeptical this would be a burdensome 
> change.
>
> Carl replied:  "It seems quite likely to me that there are templates in 
> the wild relying on preservation of the final newline. People render 
> preformatted text (e.g. text emails) using DTL. I probably have some text 
> email templates in older projects myself that would break with that change. 
>
> We could add a new option to {% include %}, though." Adam also said, "I 
> too have used DTL for text emails that would break under that behaviour. 
> New option to include sounds good to me."
>
>
> Me again: In the long run, having {% include %} remove the trailing 
> newline seems like a more sensible default. For example, I wouldn't expect 
> this code to have a newline inserted between it:
>
>
> {% include "foo.txt" %}
> {% include "bar.txt" %}
>  
>
> An option for {% include %} seems superfluous given that if you were 
> writing
>
>
> {% include "foo.txt" %}{% include "bar.txt" %}
>  
>
> now you can write the first thing which is much more intuitive.
>
>
> How about a keep_trailing_newline TEMPLATES option for backwards 
> compatibility for those who don't want to adapt their templates for the new 
> behavior? Jinja2 has that option.
>
> Carl replied: An engine option may be better than an option to {% include 
> %}, though it doesn't allow us to ensure that we strip the newline in the 
> specific case of attrs.
>
> How we default the engine option I guess just depends on how seriously we 
> take backwards compatibility. If we default it to strip and make people add 
> the config to preserve the old behavior, that's not really backwards 
> compatible. Historically (as seen in Malcolm's comment) we would choose to 
> err on the side of actual backwards compatibility in a case like this, even 
> if it didn't result in the ideal future behavior. But the adaptation isn't 
> hard in this case, so I won't object if the choice is to break back-compat.
>
>
> If it's not a per-include choice, of course, we have to break overall 
> back-compat to preserve form-attr-rendering back-compat.
> 
>
> What do you think?
>
> [0] https://github.com/django/django/pull/7769
> [1] https://code.djangoproject.com/ticket/9198
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1819d705-7b6a-46f0-9ede-dee3cf5d7b69%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Changing {% include %} to strip a trailing newline

Anthony, I assume you're referring to the keep_trailing_newline  option? If 
we don't enable it for existing projects, then forms will render with extra 
newlines. I think it's better to have projects opt-out of the behavior if 
they're unable to audit their non-HTML templates due to time constraints. 
Only enabling it for new projects means that the tests for projects like 
django-money will break until the option is enabled. I consider the k
eep_trailing_newline option a temporary backwards compatibility measure and 
would like to eventually remove it.

Sam, re: "Could this be set within the template rather than the include 
tag?"
I'm not an expert in the inner workings of the Django template language, 
but based on how template tags usually work, that doesn't feel feasible. To 
be clear, I'm not proposing to add an option to the {% include %} tag. That 
seems like the wrong approach to me since if {% include %} already strips 
the trailing newline of its output, then you can control newlines by how 
you structure the {% include %} tag. This seems far more intuitive and 
readable.

I also considered a technique we've used to change template tag behaviors 
previously: {% load include from future %} however, I feel that would be 
really disruptive to force all projects to go through that deprecation.

On Wednesday, January 4, 2017 at 3:15:31 PM UTC-5, Sam Willis wrote:
>
> Could this be set within the template rather than the include tag? So for 
> example have a new tag such as {% strip_final_new_line %} that when 
> included at the end of a template immediately before the final new line 
> would instruct it to be striped.
>
> This stops the user from having to remember to use a special option on the 
> include tag for each use of a template that requires it - it also work on a 
> normal (not included) template render as well, if that is wanted.
>
> An alternative tag could be {% endtemplate %} that can be placed anywhere 
> in the template, forcing the end of rendering?
>
> On Wednesday, January 4, 2017 at 7:58:42 PM UTC, Tim Graham wrote:
>>
>> Shortly after template widget rendering was merged, an issue about extra 
>> newlines appearing in the rendered output was reported [0]:
>>
>> For example, from django-money:
>>
>> UIC-Franc
>> \n\n
>> US Dollar
>> \n\n
>>
>> The newlines aren't useful and they break assertions like this:
>>
>> US Dollar in form.as_p
>>
>> --- end report---
>>
>> The reporter suggested removing the trailing newline in the attrs.html 
>> template but Adam Johnson reported: "POSIX states that all text files 
>> should end with a newline, and some tools break when operating on text 
>> files missing the final newline. That's why git has the warning \ No 
>> newline at end of file and Github has a warning symbol for the missing 
>> newline."
>>
>> I suggested that perhaps {% include %} could do .rstrip('\n') on 
>> whatever it renders.
>>
>> Preston pointed out that Jinja2 does something similar:
>>
>> http://jinja.pocoo.org/docs/dev/templates/#whitespace-control
>>
>>- a single trailing newline is stripped if present
>>- other whitespace (spaces, tabs, newlines etc.) is returned unchanged
>>
>> I noted that the issue of {% include %} adding extra newlines was raised 
>> in #9198 [1] but Malcolm said, "For backwards compatibility reasons we 
>> cannot change the default behaviour now (there will be templates that rely 
>> on the newline being inserted)." I was skeptical this would be a burdensome 
>> change.
>>
>> Carl replied:  "It seems quite likely to me that there are templates in 
>> the wild relying on preservation of the final newline. People render 
>> preformatted text (e.g. text emails) using DTL. I probably have some text 
>> email templates in older projects myself that would break with that change. 
>>
>> We could add a new option to {% include %}, though." Adam also said, "I 
>> too have used DTL for text emails that would break under that behaviour. 
>> New option to include sounds good to me."
>>
>>
>> Me again: In the long run, having {% include %} remove the trailing 
>> newline seems like a more sensible default. For example, I wouldn't expect 
>> this code to have a newline inserted between it:
>>
>>
>> {% include "foo.txt" %}
>> {% include "bar.txt" %}
>>  
>>
>> An option for {% include %} seems superfluous given that if you were 
>> writing
>>
>>
>> {% include "foo.txt" %}{% include "bar.txt" %}
>>  
>>
>> now you can write the first thing which is much more intuitive.
>>
>>
>> How about a keep_trailing_newline TEMPLATES option for backwards 
>> compatibility for those who don't want to adapt their templates for the new 
>> behavior? Jinja2 has that option.
>>
>> Carl replied: An engine option may be better than an option to {% 
>> include %}, though it doesn't allow us to ensure that we strip the 
>> newline in the specific case of attrs.
>>
>> How we default the engine option I guess just depends on how seriously we 
>> take backwards compatibility. I

Add custom autoreload file tracking options setting

Hi,

Right now, Django only tracks Python module files for autoreload during
development. As a project starts to include more custom include files, such as
Javascript, SQL, Makefiles, etc.., the autoreload function doesn't apply to
these.

For my use case, I have custom view functions that call in separate SQL &
Javascript files. (I don’t use the template system.)

If I edit these Javascript & SQL files, the Django server doesn’t autoreload.

So, I made a pull-request where we can add a manual list of files to track for
autoreload: https://github.com/django/django/pull/7791

In your project's settings.py file, assign a variable TRACK_FILES containing a
list of full file paths to track. This will track files to autoreload the
development run server as these files are updated.

Is this OK? This pull request is a basic option and I’m sure it can get more
complicated than that (directory tracking, Makefiles, Javscript builds, etc..)

-bobby

--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/77BE6024-04E1-4DC1-89BD-24091F151FFC%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Add custom autoreload file tracking options setting

Could you give us a code snippet (sample view, perhaps) demonstrating how 
this caching happens?

On Wednesday, January 4, 2017 at 3:57:31 PM UTC-5, Bobby Mozumder wrote:
>
> Hi, 
>
> Right now, Django only tracks Python module files for autoreload during 
> development. As a project starts to include more custom include files, such 
> as Javascript, SQL, Makefiles, etc.., the autoreload function doesn't apply 
> to these. 
>
> For my use case, I have custom view functions that call in separate SQL & 
> Javascript files.  (I don’t use the template system.) 
>
> If I edit these Javascript & SQL files, the Django server doesn’t 
> autoreload. 
>
> So, I made a pull-request where we can add a manual list of files to track 
> for autoreload: https://github.com/django/django/pull/7791 
>
> In your project's settings.py file, assign a variable TRACK_FILES 
> containing a list of full file paths to track. This will track files to 
> autoreload the development run server as these files are updated. 
>
> Is this OK?  This pull request is a basic option and I’m sure it can get 
> more complicated than that (directory tracking, Makefiles, Javscript 
> builds, etc..) 
>
> -bobby

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/c857c334-6388-4e10-8367-ffbee08acc10%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Add custom autoreload file tracking options setting

OK here is some example code snippet where I load prepared SQL statements:


class FastDetailView(DetailView,FastView):

c = connection.cursor()

SQL_VIEW_DIRS = {
'fashion': (
'include/sql/materializedviews/headlines',
'include/sql/materializedviews/latestCollections',
'include/sql/materializedviews/allSeasons',
'include/sql/materializedviews/fullSeason',
'include/sql/materializedviews/gallery',
'include/sql/materializedviews/indexView',
'include/sql/materializedviews/cover',
'include/sql/materializedviews/latestSeasonView',
'include/sql/materializedviews/seasonView',
'include/sql/materializedviews/collectionView',
'include/sql/materializedviews/latestCollectionsJSON',
'include/sql/materializedviews/collectionCardJSON',
'include/sql/materializedviews/indexJSON',
'include/sql/materializedviews/categoryJSON',
'include/sql/materializedviews/articleJSON',
'include/sql/triggers/globals',
'include/sql/triggers/brand',
'include/sql/triggers/collection',
'include/sql/triggers/collectionlookassignment',
'include/sql/triggers/cover',
'include/sql/triggers/look',
'include/sql/triggers/photo',
'include/sql/triggers/season',
'include/sql/triggers/fashion_headlinesviewmat',
'include/sql/triggers/fashion_latestcollectionsviewmat',
'include/sql/triggers/fashion_allseasonsviewmat',
'include/sql/triggers/fashion_fullseasonviewmat',
'include/sql/triggers/fashion_galleryviewmat',
'include/sql/triggers/fashion_coverviewmat',
'include/sql/triggers/fashion_indexviewmat',
'include/sql/triggers/fashion_latestseasonviewmat',
'include/sql/triggers/fashion_seasonviewmat',
'include/sql/triggers/fashion_collectionviewmat',
'include/sql/triggers/fashion_collectioncardjsonviewmat',
),
'analytics': (
'include/sql/analytics',
),
}

MATERIALIZED_VIEWS = True

@classmethod
def prepare_db_queries(self):
logger.info('Reading fashion prepared SQL statements')
cursor = connection.cursor()
for sql_view_dir in SQL_VIEW_DIRS['fashion']:
file_name = sql_view_dir + '/prepare.sql'
try:
with open(file_name, 'r') as file:
sql_prepare=file.read().strip()
if sql_prepare:
cursor.execute(sql_prepare)
except (OSError, IOError) as e:
pass
except e:
logger.info('Error reading SQL file: %s' % file_name)
raise e
if MATERIALIZED_VIEWS:
file_name = sql_view_dir + '/prepare_materialized.sql'
try:
with open(file_name, 'r') as file:
sql_prepare=file.read().strip()
if sql_prepare:
cursor.execute(sql_prepare)
except (OSError, IOError) as e:
pass
except e:
logger.info('Error reading SQL file: %s' % file_name)
raise e


It’s a custom view class that basically reads SQL from a separate list of files 
on initialization, and executes those SQL files.  

If I edit these SQL files, it won't restart the development server.

-bobby

> On Jan 4, 2017, at 4:03 PM, Tim Graham  wrote:
> 
> Could you give us a code snippet (sample view, perhaps) demonstrating how 
> this caching happens?
> 
> On Wednesday, January 4, 2017 at 3:57:31 PM UTC-5, Bobby Mozumder wrote:
> Hi, 
> 
> Right now, Django only tracks Python module files for autoreload during 
> development. As a project starts to include more custom include files, such 
> as Javascript, SQL, Makefiles, etc.., the autoreload function doesn't apply 
> to these. 
> 
> For my use case, I have custom view functions that call in separate SQL & 
> Javascript files.  (I don’t use the template system.) 
> 
> If I edit these Javascript & SQL files, the Django server doesn’t autoreload. 
> 
> So, I made a pull-request where we can add a manual list of files to track 
> for autoreload: https://github.com/django/django/pull/7791 
>  
> 
> In your project's settings.py file, assign a variable TRACK_FILES containing 
> a list of full file paths to track. This will track files to autoreload the 
> development run server as these files are updated. 
> 
> Is this OK?  This pull request is a basic option and I’m sure it can get more 
> complicated than that (directory tracking, Makefiles, Javscript builds, 
> etc..) 
> 
> -bobby
> 
> -- 
> You received this message because you are subscribed to the Google Groups

Re: Add custom autoreload file tracking options setting

When is prepare_db_queries() called? During a request/response cycle? I 
doesn't look like any caching is happening so I still doesn't see why the 
server needs to restart to pickup changes to the SQL files.

On Wednesday, January 4, 2017 at 4:12:27 PM UTC-5, Bobby Mozumder wrote:
>
> OK here is some example code snippet where I load prepared SQL statements:
>
> class FastDetailView(DetailView,FastView):
>
> c = connection.cursor()
>
> SQL_VIEW_DIRS = {
> 'fashion': (
> 'include/sql/materializedviews/headlines',
> 'include/sql/materializedviews/latestCollections',
> 'include/sql/materializedviews/allSeasons',
> 'include/sql/materializedviews/fullSeason',
> 'include/sql/materializedviews/gallery',
> 'include/sql/materializedviews/indexView',
> 'include/sql/materializedviews/cover',
> 'include/sql/materializedviews/latestSeasonView',
> 'include/sql/materializedviews/seasonView',
> 'include/sql/materializedviews/collectionView',
> 'include/sql/materializedviews/latestCollectionsJSON',
> 'include/sql/materializedviews/collectionCardJSON',
> 'include/sql/materializedviews/indexJSON',
> 'include/sql/materializedviews/categoryJSON',
> 'include/sql/materializedviews/articleJSON',
> 'include/sql/triggers/globals',
> 'include/sql/triggers/brand',
> 'include/sql/triggers/collection',
> 'include/sql/triggers/collectionlookassignment',
> 'include/sql/triggers/cover',
> 'include/sql/triggers/look',
> 'include/sql/triggers/photo',
> 'include/sql/triggers/season',
> 'include/sql/triggers/fashion_headlinesviewmat',
> 'include/sql/triggers/fashion_latestcollectionsviewmat',
> 'include/sql/triggers/fashion_allseasonsviewmat',
> 'include/sql/triggers/fashion_fullseasonviewmat',
> 'include/sql/triggers/fashion_galleryviewmat',
> 'include/sql/triggers/fashion_coverviewmat',
> 'include/sql/triggers/fashion_indexviewmat',
> 'include/sql/triggers/fashion_latestseasonviewmat',
> 'include/sql/triggers/fashion_seasonviewmat',
> 'include/sql/triggers/fashion_collectionviewmat',
> 'include/sql/triggers/fashion_collectioncardjsonviewmat',
> ),
> 'analytics': (
> 'include/sql/analytics',
> ),
> }
>
> MATERIALIZED_VIEWS = True
>
> @classmethod
> def prepare_db_queries(self):
> logger.info('Reading fashion prepared SQL statements')
> cursor = connection.cursor()
> for sql_view_dir in SQL_VIEW_DIRS['fashion']:
> file_name = sql_view_dir + '/prepare.sql'
> try:
> with open(file_name, 'r') as file:
> sql_prepare=file.read().strip()
> if sql_prepare:
> cursor.execute(sql_prepare)
> except (OSError, IOError) as e:
> pass
> except e:
> logger.info('Error reading SQL file: %s' % file_name)
> raise e
> if MATERIALIZED_VIEWS:
> file_name = sql_view_dir + '/prepare_materialized.sql'
> try:
> with open(file_name, 'r') as file:
> sql_prepare=file.read().strip()
> if sql_prepare:
> cursor.execute(sql_prepare)
> except (OSError, IOError) as e:
> pass
> except e:
> logger.info('Error reading SQL file: %s' % file_name)
> raise e
>
>
> It’s a custom view class that basically reads SQL from a separate list of 
> files on initialization, and executes those SQL files.  
>
> If I edit these SQL files, it won't restart the development server.
>
> -bobby
>
> On Jan 4, 2017, at 4:03 PM, Tim Graham > 
> wrote:
>
> Could you give us a code snippet (sample view, perhaps) demonstrating how 
> this caching happens?
>
> On Wednesday, January 4, 2017 at 3:57:31 PM UTC-5, Bobby Mozumder wrote:
>>
>> Hi, 
>>
>> Right now, Django only tracks Python module files for autoreload during 
>> development. As a project starts to include more custom include files, such 
>> as Javascript, SQL, Makefiles, etc.., the autoreload function doesn't apply 
>> to these. 
>>
>> For my use case, I have custom view functions that call in separate SQL & 
>> Javascript files.  (I don’t use the template system.) 
>>
>> If I edit these Javascript & SQL files, the Django server doesn’t 
>> autoreload. 
>>
>> So, I made a pull-request where we can add a manual list of files to 
>> track for autoreload: https://github.com/django/django/pull/7791 
>>
>> In your project's settings.py file, assign a variable TRA

Re: Add custom autoreload file tracking options setting

For that use case I'd suggest just re-executing prepare_db_queries on every
page view when DEBUG=True. This is similar to how Django's template loaders
work without the cached loader wrapping them.

On 4 January 2017 at 21:12, Bobby Mozumder  wrote:

> OK here is some example code snippet where I load prepared SQL statements:
>
> class FastDetailView(DetailView,FastView):
>
> c = connection.cursor()
>
> SQL_VIEW_DIRS = {
> 'fashion': (
> 'include/sql/materializedviews/headlines',
> 'include/sql/materializedviews/latestCollections',
> 'include/sql/materializedviews/allSeasons',
> 'include/sql/materializedviews/fullSeason',
> 'include/sql/materializedviews/gallery',
> 'include/sql/materializedviews/indexView',
> 'include/sql/materializedviews/cover',
> 'include/sql/materializedviews/latestSeasonView',
> 'include/sql/materializedviews/seasonView',
> 'include/sql/materializedviews/collectionView',
> 'include/sql/materializedviews/latestCollectionsJSON',
> 'include/sql/materializedviews/collectionCardJSON',
> 'include/sql/materializedviews/indexJSON',
> 'include/sql/materializedviews/categoryJSON',
> 'include/sql/materializedviews/articleJSON',
> 'include/sql/triggers/globals',
> 'include/sql/triggers/brand',
> 'include/sql/triggers/collection',
> 'include/sql/triggers/collectionlookassignment',
> 'include/sql/triggers/cover',
> 'include/sql/triggers/look',
> 'include/sql/triggers/photo',
> 'include/sql/triggers/season',
> 'include/sql/triggers/fashion_headlinesviewmat',
> 'include/sql/triggers/fashion_latestcollectionsviewmat',
> 'include/sql/triggers/fashion_allseasonsviewmat',
> 'include/sql/triggers/fashion_fullseasonviewmat',
> 'include/sql/triggers/fashion_galleryviewmat',
> 'include/sql/triggers/fashion_coverviewmat',
> 'include/sql/triggers/fashion_indexviewmat',
> 'include/sql/triggers/fashion_latestseasonviewmat',
> 'include/sql/triggers/fashion_seasonviewmat',
> 'include/sql/triggers/fashion_collectionviewmat',
> 'include/sql/triggers/fashion_collectioncardjsonviewmat',
> ),
> 'analytics': (
> 'include/sql/analytics',
> ),
> }
>
> MATERIALIZED_VIEWS = True
>
> @classmethod
> def prepare_db_queries(self):
> logger.info('Reading fashion prepared SQL statements')
> cursor = connection.cursor()
> for sql_view_dir in SQL_VIEW_DIRS['fashion']:
> file_name = sql_view_dir + '/prepare.sql'
> try:
> with open(file_name, 'r') as file:
> sql_prepare=file.read().strip()
> if sql_prepare:
> cursor.execute(sql_prepare)
> except (OSError, IOError) as e:
> pass
> except e:
> logger.info('Error reading SQL file: %s' % file_name)
> raise e
> if MATERIALIZED_VIEWS:
> file_name = sql_view_dir + '/prepare_materialized.sql'
> try:
> with open(file_name, 'r') as file:
> sql_prepare=file.read().strip()
> if sql_prepare:
> cursor.execute(sql_prepare)
> except (OSError, IOError) as e:
> pass
> except e:
> logger.info('Error reading SQL file: %s' % file_name)
> raise e
>
>
> It’s a custom view class that basically reads SQL from a separate list of
> files on initialization, and executes those SQL files.
>
> If I edit these SQL files, it won't restart the development server.
>
> -bobby
>
> On Jan 4, 2017, at 4:03 PM, Tim Graham  wrote:
>
> Could you give us a code snippet (sample view, perhaps) demonstrating how
> this caching happens?
>
> On Wednesday, January 4, 2017 at 3:57:31 PM UTC-5, Bobby Mozumder wrote:
>>
>> Hi,
>>
>> Right now, Django only tracks Python module files for autoreload during
>> development. As a project starts to include more custom include files, such
>> as Javascript, SQL, Makefiles, etc.., the autoreload function doesn't apply
>> to these.
>>
>> For my use case, I have custom view functions that call in separate SQL &
>> Javascript files.  (I don’t use the template system.)
>>
>> If I edit these Javascript & SQL files, the Django server doesn’t
>> autoreload.
>>
>> So, I made a pull-request where we can add a manual list of files to
>> track for autoreload: https://github.com/django/django/pull/7791
>>
>> In your project's settings.py file, assign a variable TRACK_FILES
>> containing a list of full file paths to

Re: Changing {% include %} to strip a trailing newline

2017-01-04 Thread Anthony King

Tim, yes I meant the keep_trailing_newline option.

Digging through the code (as I mentioned on IRC), it's possible to have the
form rendering to use different options.
Here is a proof of concept [0], which forces removal of trailing whitespace
in forms, and adds it as an option for settings.

I'm unsure of the behaviour in Jinja. The naming seems wrong for it to be
for {% include %} only, however this is just a proof of concept.

If you're happy with it, I'll contribute it to the currently open PR.

[0] https://gist.github.com/anonymous/098c076f626466dc3154e3a6da02d996

On 4 January 2017 at 20:32, Tim Graham  wrote:

> Anthony, I assume you're referring to the keep_trailing_newline  option?
> If we don't enable it for existing projects, then forms will render with
> extra newlines. I think it's better to have projects opt-out of the
> behavior if they're unable to audit their non-HTML templates due to time
> constraints. Only enabling it for new projects means that the tests for
> projects like django-money will break until the option is enabled. I
> consider the keep_trailing_newline option a temporary backwards
> compatibility measure and would like to eventually remove it.
>
> Sam, re: "Could this be set within the template rather than the include
> tag?"
> I'm not an expert in the inner workings of the Django template language,
> but based on how template tags usually work, that doesn't feel feasible. To
> be clear, I'm not proposing to add an option to the {% include %} tag. That
> seems like the wrong approach to me since if {% include %} already strips
> the trailing newline of its output, then you can control newlines by how
> you structure the {% include %} tag. This seems far more intuitive and
> readable.
>
> I also considered a technique we've used to change template tag behaviors
> previously: {% load include from future %} however, I feel that would be
> really disruptive to force all projects to go through that deprecation.
>
>
> On Wednesday, January 4, 2017 at 3:15:31 PM UTC-5, Sam Willis wrote:
>>
>> Could this be set within the template rather than the include tag? So for
>> example have a new tag such as {% strip_final_new_line %} that when
>> included at the end of a template immediately before the final new line
>> would instruct it to be striped.
>>
>> This stops the user from having to remember to use a special option on
>> the include tag for each use of a template that requires it - it also work
>> on a normal (not included) template render as well, if that is wanted.
>>
>> An alternative tag could be {% endtemplate %} that can be placed anywhere
>> in the template, forcing the end of rendering?
>>
>> On Wednesday, January 4, 2017 at 7:58:42 PM UTC, Tim Graham wrote:
>>>
>>> Shortly after template widget rendering was merged, an issue about extra
>>> newlines appearing in the rendered output was reported [0]:
>>>
>>> For example, from django-money:
>>>
>>> UIC-Franc
>>> \n\n
>>> US Dollar
>>> \n\n
>>>
>>> The newlines aren't useful and they break assertions like this:
>>>
>>> US Dollar in form.as_p
>>>
>>> --- end report---
>>>
>>> The reporter suggested removing the trailing newline in the attrs.html
>>> template but Adam Johnson reported: "POSIX states that all text files
>>> should end with a newline, and some tools break when operating on text
>>> files missing the final newline. That's why git has the warning \ No
>>> newline at end of file and Github has a warning symbol for the missing
>>> newline."
>>>
>>> I suggested that perhaps {% include %} could do .rstrip('\n') on
>>> whatever it renders.
>>>
>>> Preston pointed out that Jinja2 does something similar:
>>>
>>> http://jinja.pocoo.org/docs/dev/templates/#whitespace-control
>>>
>>>- a single trailing newline is stripped if present
>>>- other whitespace (spaces, tabs, newlines etc.) is returned
>>>unchanged
>>>
>>> I noted that the issue of {% include %} adding extra newlines was raised
>>> in #9198 [1] but Malcolm said, "For backwards compatibility reasons we
>>> cannot change the default behaviour now (there will be templates that rely
>>> on the newline being inserted)." I was skeptical this would be a burdensome
>>> change.
>>>
>>> Carl replied:  "It seems quite likely to me that there are templates in
>>> the wild relying on preservation of the final newline. People render
>>> preformatted text (e.g. text emails) using DTL. I probably have some text
>>> email templates in older projects myself that would break with that change.
>>>
>>> We could add a new option to {% include %}, though." Adam also said, "I
>>> too have used DTL for text emails that would break under that behaviour.
>>> New option to include sounds good to me."
>>>
>>>
>>> Me again: In the long run, having {% include %} remove the trailing
>>> newline seems like a more sensible default. For example, I wouldn't expect
>>> this code to have a newline inserted between it:
>>>
>>>
>>> {% include "foo.txt" %}
>>> {% include "bar.txt" %}
>>>
>>

Re: Add custom autoreload file tracking options setting

It’s actually called once on app startup during DB connection via a Signal.

Here is my app.py:

from django.apps import AppConfig
from .signals import *
from django.utils import autoreload

class FashionAppConfig(AppConfig):
name = 'fashion'
verbose_name = "Fashion"

And here is my signals.py:

from django.dispatch import receiver
from django.db.backends.signals import connection_created
from .page import factory as pageFactory
from .api import APIFactory


@receiver(connection_created, dispatch_uid="dbConnectionInitiated")
def prepareSQL(sender, **kwargs):
pageFactory.prepare_db_queries()
APIFactory.prepare_db_queries()
pass


My FastDetailView class is a subclass of factory , and this is where 
prepare_db_queries happens.

I also have separate code that loads in Javascript into this FastDetailView 
class.

Basically I don’t read html from the template at all during the 
Request/Response cycle.  I instead pre-load all my DB Queries and Javascript.

(This whole process helps speed up the view response generation times.  I can 
generate a full page in about 4ms, and that includes GZip.)

-bobby

> On Jan 4, 2017, at 4:17 PM, Tim Graham  wrote:
> 
> When is prepare_db_queries() called? During a request/response cycle? I 
> doesn't look like any caching is happening so I still doesn't see why the 
> server needs to restart to pickup changes to the SQL files.
> 
> On Wednesday, January 4, 2017 at 4:12:27 PM UTC-5, Bobby Mozumder wrote:
> OK here is some example code snippet where I load prepared SQL statements:
> 
> 
> class FastDetailView(DetailView,FastView):
> 
> c = connection.cursor()
> 
> SQL_VIEW_DIRS = {
> 'fashion': (
> 'include/sql/materializedviews/headlines',
> 'include/sql/materializedviews/latestCollections',
> 'include/sql/materializedviews/allSeasons',
> 'include/sql/materializedviews/fullSeason',
> 'include/sql/materializedviews/gallery',
> 'include/sql/materializedviews/indexView',
> 'include/sql/materializedviews/cover',
> 'include/sql/materializedviews/latestSeasonView',
> 'include/sql/materializedviews/seasonView',
> 'include/sql/materializedviews/collectionView',
> 'include/sql/materializedviews/latestCollectionsJSON',
> 'include/sql/materializedviews/collectionCardJSON',
> 'include/sql/materializedviews/indexJSON',
> 'include/sql/materializedviews/categoryJSON',
> 'include/sql/materializedviews/articleJSON',
> 'include/sql/triggers/globals',
> 'include/sql/triggers/brand',
> 'include/sql/triggers/collection',
> 'include/sql/triggers/collectionlookassignment',
> 'include/sql/triggers/cover',
> 'include/sql/triggers/look',
> 'include/sql/triggers/photo',
> 'include/sql/triggers/season',
> 'include/sql/triggers/fashion_headlinesviewmat',
> 'include/sql/triggers/fashion_latestcollectionsviewmat',
> 'include/sql/triggers/fashion_allseasonsviewmat',
> 'include/sql/triggers/fashion_fullseasonviewmat',
> 'include/sql/triggers/fashion_galleryviewmat',
> 'include/sql/triggers/fashion_coverviewmat',
> 'include/sql/triggers/fashion_indexviewmat',
> 'include/sql/triggers/fashion_latestseasonviewmat',
> 'include/sql/triggers/fashion_seasonviewmat',
> 'include/sql/triggers/fashion_collectionviewmat',
> 'include/sql/triggers/fashion_collectioncardjsonviewmat',
> ),
> 'analytics': (
> 'include/sql/analytics',
> ),
> }
> 
> MATERIALIZED_VIEWS = True
> 
> @classmethod
> def prepare_db_queries(self):
> logger.info ('Reading fashion prepared SQL 
> statements')
> cursor = connection.cursor()
> for sql_view_dir in SQL_VIEW_DIRS['fashion']:
> file_name = sql_view_dir + '/prepare.sql'
> try:
> with open(file_name, 'r') as file:
> sql_prepare=file.read().strip()
> if sql_prepare:
> cursor.execute(sql_prepare)
> except (OSError, IOError) as e:
> pass
> except e:
> logger.info ('Error reading SQL file: 
> %s' % file_name)
> raise e
> if MATERIALIZED_VIEWS:
> file_name = sql_view_dir + '/prepare_materialized.sql'
> try:
> with open(file_name, 'r') as file:
> sql_prepare=file.read().strip()
> if sql_prepare:
> cursor.execute(sql_prepare)
> except (OSError, IOError) as e:
> pass
> except e:
>

Re: Changing {% include %} to strip a trailing newline

2017-01-04 Thread Aymeric Augustin

If I understand correctly, we have to choose between:

1. breaking backwards compatibility for {% include %}
2. breaking backwards compatibility for widgets HTML
3. having a handful of single-line, non-newline-terminated files

I don’t think option 1 is reasonable. Whitespace control in templates is an 
exercise in frustration. In my experience more flexible tools such as {%-, {%+, 
-%}, and +%} in Jinja2 increase the frustration. I don’t think Django should 
get itself into this mess just for fixing the problem discussed here.

Option 2 seems acceptable to me. This is the "do nothing” option. I don’t think 
the exact HTML string rendered by widgets is considered a public API. We made 
changes to that output in the past, for example when we switched to HTML5.

Option 3 also seems reasonable to me. Files could end with {# no newline, see 
issue #xxx #} and no newline. A test could validate that the rendered output 
doesn’t contain a newline. This would be enough to catch accidental regressions 
caused by editors automatically adding the newline.

The quote from the POSIX standard says “should”, not “must”. If we interpret it 
as if it was written in capitals in a RFC, this means we can make an exception 
if we have a good reason.

(At this point surely someone will suggest that we can write an editorconfig to 
specify that these files mustn’t end with a newline. Whatever works.)

I don’t have a strong preference for 2 or 3, however, I have a strong distaste 
for anything more complicated.

Best regards,

-- 
Aymeric.

> On 4 Jan 2017, at 20:58, Tim Graham  wrote:
> 
> Shortly after template widget rendering was merged, an issue about extra 
> newlines appearing in the rendered output was reported [0]:
> 
> For example, from django-money:
> 
> UIC-Franc
> \n\n
> US Dollar
> \n\n
> 
> The newlines aren't useful and they break assertions like this:
> 
> US Dollar in form.as_p
> 
> --- end report---
> 
> 
> The reporter suggested removing the trailing newline in the attrs.html 
> template but Adam Johnson reported: "POSIX states that all text files should 
> end with a newline, and some tools break when operating on text files missing 
> the final newline. That's why git has the warning \ No newline at end of file 
> and Github has a warning symbol for the missing newline."
> 
> I suggested that perhaps {% include %} could do .rstrip('\n') on whatever it 
> renders.
> 
> Preston pointed out that Jinja2 does something similar:
> 
> http://jinja.pocoo.org/docs/dev/templates/#whitespace-control 
> 
> a single trailing newline is stripped if present
> other whitespace (spaces, tabs, newlines etc.) is returned unchanged
> I noted that the issue of {% include %} adding extra newlines was raised in 
> #9198 [1] but Malcolm said, "For backwards compatibility reasons we cannot 
> change the default behaviour now (there will be templates that rely on the 
> newline being inserted)." I was skeptical this would be a burdensome change.
> 
> Carl replied:  "It seems quite likely to me that there are templates in the 
> wild relying on preservation of the final newline. People render preformatted 
> text (e.g. text emails) using DTL. I probably have some text email templates 
> in older projects myself that would break with that change.
> We could add a new option to {% include %}, though." Adam also said, "I too 
> have used DTL for text emails that would break under that behaviour. New 
> option to include sounds good to me."
> 
> 
> 
> Me again: In the long run, having {% include %} remove the trailing newline 
> seems like a more sensible default. For example, I wouldn't expect this code 
> to have a newline inserted between it:
> 
> 
> {% include "foo.txt" %}
> {% include "bar.txt" %}
>  
> An option for {% include %} seems superfluous given that if you were writing
> 
> 
> 
> {% include "foo.txt" %}{% include "bar.txt" %}
>  
> now you can write the first thing which is much more intuitive.
> 
> 
> 
> How about a keep_trailing_newline TEMPLATES option for backwards 
> compatibility for those who don't want to adapt their templates for the new 
> behavior? Jinja2 has that option.
> 
> 
> Carl replied: An engine option may be better than an option to {% include %}, 
> though it doesn't allow us to ensure that we strip the newline in the 
> specific case of attrs.
> 
> How we default the engine option I guess just depends on how seriously we 
> take backwards compatibility. If we default it to strip and make people add 
> the config to preserve the old behavior, that's not really backwards 
> compatible. Historically (as seen in Malcolm's comment) we would choose to 
> err on the side of actual backwards compatibility in a case like this, even 
> if it didn't result in the ideal future behavior. But the adaptation isn't 
> hard in this case, so I won't object if the choice is to break back-compat.
> 
> 
> 
> If it's not a per-include choice, of course, we have

Re: Add custom autoreload file tracking options setting

That’ll make development server access times really slow.  There’s a pretty 
long Makefile also that builds Javascript.  We really shouldn’t have to rebuild 
Javascript on every page view, especially since I call interactive API requests 
with these views.

-bobby

> On Jan 4, 2017, at 4:18 PM, Adam Johnson  wrote:
> 
> For that use case I'd suggest just re-executing prepare_db_queries on every 
> page view when DEBUG=True. This is similar to how Django's template loaders 
> work without the cached loader wrapping them.
> 
> On 4 January 2017 at 21:12, Bobby Mozumder  > wrote:
> OK here is some example code snippet where I load prepared SQL statements:
> 
> 
> class FastDetailView(DetailView,FastView):
> 
> c = connection.cursor()
> 
> SQL_VIEW_DIRS = {
> 'fashion': (
> 'include/sql/materializedviews/headlines',
> 'include/sql/materializedviews/latestCollections',
> 'include/sql/materializedviews/allSeasons',
> 'include/sql/materializedviews/fullSeason',
> 'include/sql/materializedviews/gallery',
> 'include/sql/materializedviews/indexView',
> 'include/sql/materializedviews/cover',
> 'include/sql/materializedviews/latestSeasonView',
> 'include/sql/materializedviews/seasonView',
> 'include/sql/materializedviews/collectionView',
> 'include/sql/materializedviews/latestCollectionsJSON',
> 'include/sql/materializedviews/collectionCardJSON',
> 'include/sql/materializedviews/indexJSON',
> 'include/sql/materializedviews/categoryJSON',
> 'include/sql/materializedviews/articleJSON',
> 'include/sql/triggers/globals',
> 'include/sql/triggers/brand',
> 'include/sql/triggers/collection',
> 'include/sql/triggers/collectionlookassignment',
> 'include/sql/triggers/cover',
> 'include/sql/triggers/look',
> 'include/sql/triggers/photo',
> 'include/sql/triggers/season',
> 'include/sql/triggers/fashion_headlinesviewmat',
> 'include/sql/triggers/fashion_latestcollectionsviewmat',
> 'include/sql/triggers/fashion_allseasonsviewmat',
> 'include/sql/triggers/fashion_fullseasonviewmat',
> 'include/sql/triggers/fashion_galleryviewmat',
> 'include/sql/triggers/fashion_coverviewmat',
> 'include/sql/triggers/fashion_indexviewmat',
> 'include/sql/triggers/fashion_latestseasonviewmat',
> 'include/sql/triggers/fashion_seasonviewmat',
> 'include/sql/triggers/fashion_collectionviewmat',
> 'include/sql/triggers/fashion_collectioncardjsonviewmat',
> ),
> 'analytics': (
> 'include/sql/analytics',
> ),
> }
> 
> MATERIALIZED_VIEWS = True
> 
> @classmethod
> def prepare_db_queries(self):
> logger.info ('Reading fashion prepared SQL 
> statements')
> cursor = connection.cursor()
> for sql_view_dir in SQL_VIEW_DIRS['fashion']:
> file_name = sql_view_dir + '/prepare.sql'
> try:
> with open(file_name, 'r') as file:
> sql_prepare=file.read().strip()
> if sql_prepare:
> cursor.execute(sql_prepare)
> except (OSError, IOError) as e:
> pass
> except e:
> logger.info ('Error reading SQL file: 
> %s' % file_name)
> raise e
> if MATERIALIZED_VIEWS:
> file_name = sql_view_dir + '/prepare_materialized.sql'
> try:
> with open(file_name, 'r') as file:
> sql_prepare=file.read().strip()
> if sql_prepare:
> cursor.execute(sql_prepare)
> except (OSError, IOError) as e:
> pass
> except e:
> logger.info ('Error reading SQL 
> file: %s' % file_name)
> raise e
> 
> 
> It’s a custom view class that basically reads SQL from a separate list of 
> files on initialization, and executes those SQL files.  
> 
> If I edit these SQL files, it won't restart the development server.
> 
> -bobby
> 
>> On Jan 4, 2017, at 4:03 PM, Tim Graham > > wrote:
>> 
>> Could you give us a code snippet (sample view, perhaps) demonstrating how 
>> this caching happens?
>> 
>> On Wednesday, January 4, 2017 at 3:57:31 PM UTC-5, Bobby Mozumder wrote:
>> Hi, 
>> 
>> Right now, Django only tracks Python module files for autoreload during 
>> development. As a project starts to include more custom include files, such 
>> as Javascript, SQL, Makefiles, etc.., the autoreload function doesn't apply

Re: Add custom autoreload file tracking options setting

2017-01-04 Thread Aymeric Augustin

Hello Bobby,

> On 4 Jan 2017, at 22:25, Bobby Mozumder  wrote:
> 
> It’s actually called once on app startup during DB connection via a Signal.

Unless I missed something, since the development server creates a new 
connection to the database for each request — Python’s threaded socket server 
creates a new thread for each connection and database connections are thread 
local in Django — prepared statements will be refreshed for each request. So I 
don’t think you need autoreloading here.

> (This whole process helps speed up the view response generation times.  I can 
> generate a full page in about 4ms, and that includes GZip.)

Off topic, but I’m jealous…

I still believe we should stop maintaining an autoreloader as soon as possible. 
Django’s autoreloader is annoyingly slow, highly inefficient, moderately well 
designed, and a gigantic pain to maintain. I’m more scared of 
django.utils.autoreload than of django.db.models.related before it was cleaned 
up.

I wish one day someone will take the time to write a good autoreloading dev 
server based on watchman. This would solve the problem discussed here because 
watchman watches all files in the current directory. The correct way to do this 
is to throw away the current design and start from scratch.

Watchman is smart enough to wait until you’ve finished a git operation to 
trigger a reload. Once such polished tech has become available, trying to 
compete with a thread that checks the mtime of all known files every second 
isn’t funny anymore. In fact it’s just sad.

While I don’t care much about django.utils.autoreloader because I want to kill 
it, I’m reluctant to add public APIs such as the proposed setting, which may 
not make sense with an hypothetical better autoreloader.

I’m -0 on the change. I could move to +0 if I understood why the use case 
described here requires watching additional files.

Best regards,

-- 
Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/BAAB82B3-974C-4240-B9C6-E81F04D9604F%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.

Re: Add custom autoreload file tracking options setting

> On Jan 4, 2017, at 4:47 PM, Aymeric Augustin 
>  wrote:
> 
> Hello Bobby,
> 
>> On 4 Jan 2017, at 22:25, Bobby Mozumder  wrote:
>> 
>> It’s actually called once on app startup during DB connection via a Signal.
> 
> Unless I missed something, since the development server creates a new 
> connection to the database for each request — Python’s threaded socket server 
> creates a new thread for each connection and database connections are thread 
> local in Django — prepared statements will be refreshed for each request. So 
> I don’t think you need autoreloading here.

Hmm I thought the test server used permanent connections if CONN_MAX_AGE was 
set to None?  Could have sworn it was that way for a while?

In any case, I just tried the Javascript & CSS Makefile with a simple one line 
change to one file, and it took 4 seconds.  That’s going to be way too long to 
do on each request.

How do people serve development Javascript & CSS files?  These days Javascript 
& CSS involves a large build process.  Are we forced to manually restart the 
development server every time Javascript changes?  

I think it should be just like changing Python files and it autoreloads 
Javascript & CSS as it changes.

Also, I haven’t used the Django Template system or Jinja in over a year.  Does 
the Django development server restart if you edit Templates?  Because it should 
also do that.

This is what I’m getting at with this autoreload.. it should include the full 
system, not just the Python files.

> 
>> (This whole process helps speed up the view response generation times.  I 
>> can generate a full page in about 4ms, and that includes GZip.)
> 
> Off topic, but I’m jealous…
> 

Also, my average page generates in 1-2ms, and fully cached html serves in 200 
microseconds, and that include Gzip, since I cache Gzip (which makes my cache 
10x bigger).   There’s lots that I can contribute here, including non-blocking 
analytics stored in teh database.  If I can make it “generic” enough I’ll try 
to publish it, but the code is really manual.  Maybe instead of a formal 
toolset, I could post a methodology manual?  It’s really tedious though, 
involving materialized database views & all sorts of other techniques.  

BTW My site is at http://www.futureclaw.com 

> 
> 
> I still believe we should stop maintaining an autoreloader as soon as 
> possible. Django’s autoreloader is annoyingly slow, highly inefficient, 
> moderately well designed, and a gigantic pain to maintain. I’m more scared of 
> django.utils.autoreload than of django.db.models.related before it was 
> cleaned up.
> 
> I wish one day someone will take the time to write a good autoreloading dev 
> server based on watchman. This would solve the problem discussed here because 
> watchman watches all files in the current directory. The correct way to do 
> this is to throw away the current design and start from scratch.
> 
> Watchman is smart enough to wait until you’ve finished a git operation to 
> trigger a reload. Once such polished tech has become available, trying to 
> compete with a thread that checks the mtime of all known files every second 
> isn’t funny anymore. In fact it’s just sad.
> 
> 
> 
> While I don’t care much about django.utils.autoreloader because I want to 
> kill it, I’m reluctant to add public APIs such as the proposed setting, which 
> may not make sense with an hypothetical better autoreloader.
> 
> I’m -0 on the change. I could move to +0 if I understood why the use case 
> described here requires watching additional files.
> 

I guess I could just use Watchman to restart the Django development server as 
needed?

I’m OK with that.. my only goal is to make sure I don’t have to restart Django 
manually every time I edit CSS/Javascript/SQL/HTML.

Is that the recommended workflow for this?  I never looked into outside tools 
to manage this, but it seems it would be a lot easier to add a list of files or 
directories to our settings and have Django manage autoreload, than to bring in 
an outside tool.  (I can add directories to watch to the pull request if needed)

> Best regards,
> 
> -- 
> Aymeric.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/BAAB82B3-974C-4240-B9C6-E81F04D9604F%40polytechnique.org.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, se

Re: Add custom autoreload file tracking options setting

>
> How do people serve development Javascript & CSS files?  These days
> Javascript & CSS involves a large build process.  Are we forced to manually
> restart the development server every time Javascript changes?


Django just serves them from the filesystem, and doesn't cache them itself.
You reload the page in your browser, and Django serves the files from disk

Does the Django development server restart if you edit Templates?  Because
> it should also do that.


Similarly, by default templates aren't cached, so on each view the dev
server loads the templates from disk, parses them, and renders them.


On 4 January 2017 at 22:31, Bobby Mozumder  wrote:

>
> > On Jan 4, 2017, at 4:47 PM, Aymeric Augustin  polytechnique.org> wrote:
> >
> > Hello Bobby,
> >
> >> On 4 Jan 2017, at 22:25, Bobby Mozumder  wrote:
> >>
> >> It’s actually called once on app startup during DB connection via a
> Signal.
> >
> > Unless I missed something, since the development server creates a new
> connection to the database for each request — Python’s threaded socket
> server creates a new thread for each connection and database connections
> are thread local in Django — prepared statements will be refreshed for each
> request. So I don’t think you need autoreloading here.
>
> Hmm I thought the test server used permanent connections if CONN_MAX_AGE
> was set to None?  Could have sworn it was that way for a while?
>
> In any case, I just tried the Javascript & CSS Makefile with a simple one
> line change to one file, and it took 4 seconds.  That’s going to be way too
> long to do on each request.
>
> How do people serve development Javascript & CSS files?  These days
> Javascript & CSS involves a large build process.  Are we forced to manually
> restart the development server every time Javascript changes?
>
> I think it should be just like changing Python files and it autoreloads
> Javascript & CSS as it changes.
>
> Also, I haven’t used the Django Template system or Jinja in over a year.
> Does the Django development server restart if you edit Templates?  Because
> it should also do that.
>
> This is what I’m getting at with this autoreload.. it should include the
> full system, not just the Python files.
>
> >
> >> (This whole process helps speed up the view response generation times.
> I can generate a full page in about 4ms, and that includes GZip.)
> >
> > Off topic, but I’m jealous…
> >
>
> Also, my average page generates in 1-2ms, and fully cached html serves in
> 200 microseconds, and that include Gzip, since I cache Gzip (which makes my
> cache 10x bigger).   There’s lots that I can contribute here, including
> non-blocking analytics stored in teh database.  If I can make it “generic”
> enough I’ll try to publish it, but the code is really manual.  Maybe
> instead of a formal toolset, I could post a methodology manual?  It’s
> really tedious though, involving materialized database views & all sorts of
> other techniques.
>
> BTW My site is at http://www.futureclaw.com
>
> > 
> >
> > I still believe we should stop maintaining an autoreloader as soon as
> possible. Django’s autoreloader is annoyingly slow, highly inefficient,
> moderately well designed, and a gigantic pain to maintain. I’m more scared
> of django.utils.autoreload than of django.db.models.related before it was
> cleaned up.
> >
> > I wish one day someone will take the time to write a good autoreloading
> dev server based on watchman. This would solve the problem discussed here
> because watchman watches all files in the current directory. The correct
> way to do this is to throw away the current design and start from scratch.
> >
> > Watchman is smart enough to wait until you’ve finished a git operation
> to trigger a reload. Once such polished tech has become available, trying
> to compete with a thread that checks the mtime of all known files every
> second isn’t funny anymore. In fact it’s just sad.
> >
> > 
> >
> > While I don’t care much about django.utils.autoreloader because I want
> to kill it, I’m reluctant to add public APIs such as the proposed setting,
> which may not make sense with an hypothetical better autoreloader.
> >
> > I’m -0 on the change. I could move to +0 if I understood why the use
> case described here requires watching additional files.
> >
>
> I guess I could just use Watchman to restart the Django development server
> as needed?
>
> I’m OK with that.. my only goal is to make sure I don’t have to restart
> Django manually every time I edit CSS/Javascript/SQL/HTML.
>
> Is that the recommended workflow for this?  I never looked into outside
> tools to manage this, but it seems it would be a lot easier to add a list
> of files or directories to our settings and have Django manage autoreload,
> than to bring in an outside tool.  (I can add directories to watch to the
> pull request if needed)
>
> > Best regards,
> >
> > --
> > Aymeric.
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "Dj

Re: django-formtools is neglected/unmaintained

2017-01-04 Thread eric . verner

Hi Tim,

You make some good points. Basically, my situation is that I want to use 
some features of Django v1.10 for a project at my company, but I am unable 
to because I use django-formtools for a FormPreview, and it is not 
compatible with v.1.10. I am also worried about using Django for future 
projects given that there seem to be problems finding people to contribute 
to what seems to be an important package. Whether I contribute depends on 
the answers to several questions: (1) How much work is necessary to get 
formtools compatible with v1.10? (2) Would my contribution make a 
difference, given that I would be a new contributor, and I don't have a ton 
of time to give? (3) Is there another package that people are using for 
FormPreviews that I'm not aware of? I couldn't find anything else that was 
more up-to-date than django-formtools. (4) Is this a sign that people 
simply aren't using Django for this task, and are maybe instead using some 
kind of Javascript library instead? If you have any information these 
questions, please let me know.

Thanks,
Eric

On Tuesday, January 3, 2017 at 6:04:59 PM UTC-7, Tim Graham wrote:
>
> Is the situation bad enough that you would volunteer to maintain the 
> project?
>
> The Django team is a collection of volunteers (excepting me) who work on 
> Django. It seems that no one on the team or reading this mailing list has 
> time or interest in maintaining the project. We need to clarify the 
> maintenance status, but that doesn't necessarily mean continuing the 
> project in its current form if there's little interest.
>
> Absent volunteers, another idea could be for an interested freelancer to 
> start a kickstarter-style fundraiser to raise money to fund some time to 
> work on some issue in the backlog. At least it would help determine if any 
> users think the project is worth paying for.
>
> On Tuesday, January 3, 2017 at 7:28:59 PM UTC-5, 
> eric@datalyticsolutions.com wrote:
>>
>> This is really bad. django-formtools used to be part of the core of 
>> Django. Is this getting the attention it deserves from the Django 
>> Foundation?
>>
>> On Monday, November 28, 2016 at 9:55:48 PM UTC-7, Asif Saifuddin wrote:
>>>
>>> Hi Tim,
>>>
>>> In case there is lack of active maintainers for the project then you 
>>> could add me to the list of maintainers. I do contribute to django eco 
>>> system regularly.
>>>
>>> my github activities:
>>>
>>> github.com/auvipy
>>>
>>>
>>> And about moving to jazzband, I do take part in django-admin2 
>>> maintenance there with the two other old maintainers, The fact is moving it 
>>> to jazzband haven't increased the active co maintainers.
>>>
>>> Thanks,
>>>
>>> Asif
>>>
>>> On Tuesday, November 29, 2016 at 5:45:59 AM UTC+6, Tim Graham wrote:

 Hi,

 django-formtools seems neglected. It's been several months since the 
 release of Django 1.10 but a compatible version of formtools hasn't been 
 released to PyPI.

 Personally, I don't know if it's important to maintain it as an 
 "official project." An alternative could be to "donate" it to JazzBand (
 https://jazzband.co/) and see if the community maintains it.

 If you have an interest in this package (especially if you want to 
 maintain it), let us know.

>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5e728ee6-f722-4286-ade5-facfb81d3899%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: django-formtools is neglected/unmaintained

1) Tim added testing on Django 1.10 in July 2016, it seems to work?
https://github.com/django/django-formtools/commits/master
2) New contributors are always welcome
3) I don't know of other packages, you can check https://djangopackages.org/
4) It's true that many websites are built with pure JS frontends these
days, there are probably more recently developed tools there

On 4 January 2017 at 22:42,  wrote:

> Hi Tim,
>
> You make some good points. Basically, my situation is that I want to use
> some features of Django v1.10 for a project at my company, but I am unable
> to because I use django-formtools for a FormPreview, and it is not
> compatible with v.1.10. I am also worried about using Django for future
> projects given that there seem to be problems finding people to contribute
> to what seems to be an important package. Whether I contribute depends on
> the answers to several questions: (1) How much work is necessary to get
> formtools compatible with v1.10? (2) Would my contribution make a
> difference, given that I would be a new contributor, and I don't have a ton
> of time to give? (3) Is there another package that people are using for
> FormPreviews that I'm not aware of? I couldn't find anything else that was
> more up-to-date than django-formtools. (4) Is this a sign that people
> simply aren't using Django for this task, and are maybe instead using some
> kind of Javascript library instead? If you have any information these
> questions, please let me know.
>
> Thanks,
> Eric
>
> On Tuesday, January 3, 2017 at 6:04:59 PM UTC-7, Tim Graham wrote:
>>
>> Is the situation bad enough that you would volunteer to maintain the
>> project?
>>
>> The Django team is a collection of volunteers (excepting me) who work on
>> Django. It seems that no one on the team or reading this mailing list has
>> time or interest in maintaining the project. We need to clarify the
>> maintenance status, but that doesn't necessarily mean continuing the
>> project in its current form if there's little interest.
>>
>> Absent volunteers, another idea could be for an interested freelancer to
>> start a kickstarter-style fundraiser to raise money to fund some time to
>> work on some issue in the backlog. At least it would help determine if any
>> users think the project is worth paying for.
>>
>> On Tuesday, January 3, 2017 at 7:28:59 PM UTC-5,
>> eric@datalyticsolutions.com wrote:
>>>
>>> This is really bad. django-formtools used to be part of the core of
>>> Django. Is this getting the attention it deserves from the Django
>>> Foundation?
>>>
>>> On Monday, November 28, 2016 at 9:55:48 PM UTC-7, Asif Saifuddin wrote:

 Hi Tim,

 In case there is lack of active maintainers for the project then you
 could add me to the list of maintainers. I do contribute to django eco
 system regularly.

 my github activities:

 github.com/auvipy


 And about moving to jazzband, I do take part in django-admin2
 maintenance there with the two other old maintainers, The fact is moving it
 to jazzband haven't increased the active co maintainers.

 Thanks,

 Asif

 On Tuesday, November 29, 2016 at 5:45:59 AM UTC+6, Tim Graham wrote:
>
> Hi,
>
> django-formtools seems neglected. It's been several months since the
> release of Django 1.10 but a compatible version of formtools hasn't been
> released to PyPI.
>
> Personally, I don't know if it's important to maintain it as an
> "official project." An alternative could be to "donate" it to JazzBand (
> https://jazzband.co/) and see if the community maintains it.
>
> If you have an interest in this package (especially if you want to
> maintain it), let us know.
>
 --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/5e728ee6-f722-4286-ade5-
> facfb81d3899%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Adam

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django

Re: Changing {% include %} to strip a trailing newline

Aymeric, we have a difference of opinion. I feel that if {% include %} 
removed the trailing newline, it would result in far more intuitive 
whitespace control (which may be needed in plain text email templates, for 
example). I think the change has merits outside the context of this issue. 
For example, I checked view-source on a djangoproject.com page that uses a 
{% for ... %}{% include %}{% endfor %} construct and noticed that it's much 
shorter without all those extra blank lines. I'm not advocating for 
additional controls to craft well formatted HTML but I think there's some 
advantages to not having blank lines everywhere. (I don't like the 
keep_trailing_newline option either, I'd rather call it a bugfix and make a 
backwards-incompatible change but my projects are unaffected as far as I 
checked.)

There are many questions on Stackoverflow about how to suppress newlines 
from {% include %} and even a django-include-strip-tag third-party app that 
offers the feature.

In the context of multiple template engines, I think it would be useful if 
DTL templates could be written in a similar style to Jinja2.

I'd like to know from Carl, Adam, and others, how much effort would be 
required to adapt the templates in your project for the new behavior. I 
imagine a script could be written to add newlines after all {% include %} 
in all plain text templates, for example.

It won't be the end of the world if we can't get consensus to change the 
behavior -- I'm okay with removing the trailing newlines in the source 
files as needed, although it will add some inconvenience.

On Wednesday, January 4, 2017 at 4:26:43 PM UTC-5, Aymeric Augustin wrote:
>
> If I understand correctly, we have to choose between:
>
> 1. breaking backwards compatibility for {% include %}
> 2. breaking backwards compatibility for widgets HTML
> 3. having a handful of single-line, non-newline-terminated files
>
> I don’t think option 1 is reasonable. Whitespace control in templates is 
> an exercise in frustration. In my experience more flexible tools such as 
> {%-, {%+, -%}, and +%} in Jinja2 increase the frustration. I don’t think 
> Django should get itself into this mess just for fixing the problem 
> discussed here.
>
> Option 2 seems acceptable to me. This is the "do nothing” option. I don’t 
> think the exact HTML string rendered by widgets is considered a public API. 
> We made changes to that output in the past, for example when we switched to 
> HTML5.
>
> Option 3 also seems reasonable to me. Files could end with {# no newline, 
> see issue #xxx #} and no newline. A test could validate that the rendered 
> output doesn’t contain a newline. This would be enough to catch accidental 
> regressions caused by editors automatically adding the newline.
>
> The quote from the POSIX standard says “should”, not “must”. If we 
> interpret it as if it was written in capitals in a RFC, this means we can 
> make an exception if we have a good reason.
>
> (At this point surely someone will suggest that we can write an 
> editorconfig to specify that these files mustn’t end with a newline. 
> Whatever works.)
>
> I don’t have a strong preference for 2 or 3, however, I have a strong 
> distaste for anything more complicated.
>
> Best regards,
>
> -- 
> Aymeric.
>
> On 4 Jan 2017, at 20:58, Tim Graham > 
> wrote:
>
> Shortly after template widget rendering was merged, an issue about extra 
> newlines appearing in the rendered output was reported [0]:
>
> For example, from django-money:
>
> UIC-Franc
> \n\n
> US Dollar
> \n\n
>
> The newlines aren't useful and they break assertions like this:
>
> US Dollar in form.as_p
>
> --- end report---
>
> The reporter suggested removing the trailing newline in the attrs.html 
> template but Adam Johnson reported: "POSIX states that all text files 
> should end with a newline, and some tools break when operating on text 
> files missing the final newline. That's why git has the warning \ No 
> newline at end of file and Github has a warning symbol for the missing 
> newline."
>
> I suggested that perhaps {% include %} could do .rstrip('\n') on whatever 
> it renders.
>
> Preston pointed out that Jinja2 does something similar:
>
> http://jinja.pocoo.org/docs/dev/templates/#whitespace-control
>
>- a single trailing newline is stripped if present
>- other whitespace (spaces, tabs, newlines etc.) is returned unchanged
>
> I noted that the issue of {% include %} adding extra newlines was raised 
> in #9198 [1] but Malcolm said, "For backwards compatibility reasons we 
> cannot change the default behaviour now (there will be templates that rely 
> on the newline being inserted)." I was skeptical this would be a burdensome 
> change.
>
> Carl replied:  "It seems quite likely to me that there are templates in 
> the wild relying on preservation of the final newline. People render 
> preformatted text (e.g. text emails) using DTL. I probably have some text 
> email templates in older projects myself that wo

Re: django-formtools is neglected/unmaintained

According to https://github.com/django/django-formtools/issues/75, there's 
a change in master that's needed for 1.10 compatibility.

I'll try to do a release if I can get access to the PyPI record. I pinged 
the owner (jezdez) in #django-dev about it.

On Wednesday, January 4, 2017 at 5:51:56 PM UTC-5, Adam Johnson wrote:
>
> 1) Tim added testing on Django 1.10 in July 2016, it seems to work? 
> https://github.com/django/django-formtools/commits/master
> 2) New contributors are always welcome
> 3) I don't know of other packages, you can check 
> https://djangopackages.org/
> 4) It's true that many websites are built with pure JS frontends these 
> days, there are probably more recently developed tools there
>
> On 4 January 2017 at 22:42, 
> > wrote:
>
>> Hi Tim,
>>
>> You make some good points. Basically, my situation is that I want to use 
>> some features of Django v1.10 for a project at my company, but I am unable 
>> to because I use django-formtools for a FormPreview, and it is not 
>> compatible with v.1.10. I am also worried about using Django for future 
>> projects given that there seem to be problems finding people to contribute 
>> to what seems to be an important package. Whether I contribute depends on 
>> the answers to several questions: (1) How much work is necessary to get 
>> formtools compatible with v1.10? (2) Would my contribution make a 
>> difference, given that I would be a new contributor, and I don't have a ton 
>> of time to give? (3) Is there another package that people are using for 
>> FormPreviews that I'm not aware of? I couldn't find anything else that was 
>> more up-to-date than django-formtools. (4) Is this a sign that people 
>> simply aren't using Django for this task, and are maybe instead using some 
>> kind of Javascript library instead? If you have any information these 
>> questions, please let me know.
>>
>> Thanks,
>> Eric
>>
>> On Tuesday, January 3, 2017 at 6:04:59 PM UTC-7, Tim Graham wrote:
>>>
>>> Is the situation bad enough that you would volunteer to maintain the 
>>> project?
>>>
>>> The Django team is a collection of volunteers (excepting me) who work on 
>>> Django. It seems that no one on the team or reading this mailing list has 
>>> time or interest in maintaining the project. We need to clarify the 
>>> maintenance status, but that doesn't necessarily mean continuing the 
>>> project in its current form if there's little interest.
>>>
>>> Absent volunteers, another idea could be for an interested freelancer to 
>>> start a kickstarter-style fundraiser to raise money to fund some time to 
>>> work on some issue in the backlog. At least it would help determine if any 
>>> users think the project is worth paying for.
>>>
>>> On Tuesday, January 3, 2017 at 7:28:59 PM UTC-5, 
>>> eric@datalyticsolutions.com wrote:

 This is really bad. django-formtools used to be part of the core of 
 Django. Is this getting the attention it deserves from the Django 
 Foundation?

 On Monday, November 28, 2016 at 9:55:48 PM UTC-7, Asif Saifuddin wrote:
>
> Hi Tim,
>
> In case there is lack of active maintainers for the project then you 
> could add me to the list of maintainers. I do contribute to django eco 
> system regularly.
>
> my github activities:
>
> github.com/auvipy
>
>
> And about moving to jazzband, I do take part in django-admin2 
> maintenance there with the two other old maintainers, The fact is moving 
> it 
> to jazzband haven't increased the active co maintainers.
>
> Thanks,
>
> Asif
>
> On Tuesday, November 29, 2016 at 5:45:59 AM UTC+6, Tim Graham wrote:
>>
>> Hi,
>>
>> django-formtools seems neglected. It's been several months since the 
>> release of Django 1.10 but a compatible version of formtools hasn't been 
>> released to PyPI.
>>
>> Personally, I don't know if it's important to maintain it as an 
>> "official project." An alternative could be to "donate" it to JazzBand (
>> https://jazzband.co/) and see if the community maintains it.
>>
>> If you have an interest in this package (especially if you want to 
>> maintain it), let us know.
>>
> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-develop...@googlegroups.com .
>> To post to this group, send email to django-d...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/5e728ee6-f722-4286-ade5-facfb81d3899%40googlegroups.com
>>  
>> 
>> .
>>
>>

Re: Changing {% include %} to strip a trailing newline

>
> Whitespace control in templates is an exercise in frustration. In my
> experience more flexible tools such as {%-, {%+, -%}, and +%} in Jinja2
> increase the frustration.


I really enjoy the {%- etc. operators in Jinja 2 in the context of Ansible,
there are often cases when templating obscure configuration files that they
come in useful.


> I'd like to know from Carl, Adam, and others, how much effort would be
> required to adapt the templates in your project for the new behavior. I
> imagine a script could be written to add newlines after all {% include %}
> in all plain text templates, for example.


You're right, it's not too hard to fix for the text email template usecase.
But the backwards compatible option is probably necessary for some more
advanced uses of templates out there, e.g. templating whitespace sensitive
file formats.

On 4 January 2017 at 22:53, Tim Graham  wrote:

> Aymeric, we have a difference of opinion. I feel that if {% include %}
> removed the trailing newline, it would result in far more intuitive
> whitespace control (which may be needed in plain text email templates, for
> example). I think the change has merits outside the context of this issue.
> For example, I checked view-source on a djangoproject.com page that uses
> a {% for ... %}{% include %}{% endfor %} construct and noticed that it's
> much shorter without all those extra blank lines. I'm not advocating for
> additional controls to craft well formatted HTML but I think there's some
> advantages to not having blank lines everywhere. (I don't like the
> keep_trailing_newline option either, I'd rather call it a bugfix and make a
> backwards-incompatible change but my projects are unaffected as far as I
> checked.)
>
> There are many questions on Stackoverflow about how to suppress newlines
> from {% include %} and even a django-include-strip-tag third-party app that
> offers the feature.
>
> In the context of multiple template engines, I think it would be useful if
> DTL templates could be written in a similar style to Jinja2.
>
> I'd like to know from Carl, Adam, and others, how much effort would be
> required to adapt the templates in your project for the new behavior. I
> imagine a script could be written to add newlines after all {% include %}
> in all plain text templates, for example.
>
> It won't be the end of the world if we can't get consensus to change the
> behavior -- I'm okay with removing the trailing newlines in the source
> files as needed, although it will add some inconvenience.
>
> On Wednesday, January 4, 2017 at 4:26:43 PM UTC-5, Aymeric Augustin wrote:
>>
>> If I understand correctly, we have to choose between:
>>
>> 1. breaking backwards compatibility for {% include %}
>> 2. breaking backwards compatibility for widgets HTML
>> 3. having a handful of single-line, non-newline-terminated files
>>
>> I don’t think option 1 is reasonable. Whitespace control in templates is
>> an exercise in frustration. In my experience more flexible tools such as
>> {%-, {%+, -%}, and +%} in Jinja2 increase the frustration. I don’t think
>> Django should get itself into this mess just for fixing the problem
>> discussed here.
>>
>> Option 2 seems acceptable to me. This is the "do nothing” option. I don’t
>> think the exact HTML string rendered by widgets is considered a public API.
>> We made changes to that output in the past, for example when we switched to
>> HTML5.
>>
>> Option 3 also seems reasonable to me. Files could end with {# no newline,
>> see issue #xxx #} and no newline. A test could validate that the rendered
>> output doesn’t contain a newline. This would be enough to catch accidental
>> regressions caused by editors automatically adding the newline.
>>
>> The quote from the POSIX standard says “should”, not “must”. If we
>> interpret it as if it was written in capitals in a RFC, this means we can
>> make an exception if we have a good reason.
>>
>> (At this point surely someone will suggest that we can write an
>> editorconfig to specify that these files mustn’t end with a newline.
>> Whatever works.)
>>
>> I don’t have a strong preference for 2 or 3, however, I have a strong
>> distaste for anything more complicated.
>>
>> Best regards,
>>
>> --
>> Aymeric.
>>
>> On 4 Jan 2017, at 20:58, Tim Graham  wrote:
>>
>> Shortly after template widget rendering was merged, an issue about extra
>> newlines appearing in the rendered output was reported [0]:
>>
>> For example, from django-money:
>>
>> UIC-Franc
>> \n\n
>> US Dollar
>> \n\n
>>
>> The newlines aren't useful and they break assertions like this:
>>
>> US Dollar in form.as_p
>>
>> --- end report---
>>
>> The reporter suggested removing the trailing newline in the attrs.html
>> template but Adam Johnson reported: "POSIX states that all text files
>> should end with a newline, and some tools break when operating on text
>> files missing the final newline. That's why git has the warning \ No
>> newline at end of file and Github has a warning symbol fo

Re: Changing {% include %} to strip a trailing newline

2017-01-04 Thread Curtis Maloney




On 05/01/17 10:09, Adam Johnson wrote:

Whitespace control in templates is an exercise in frustration. In my
experience more flexible tools such as {%-, {%+, -%}, and +%} in
Jinja2 increase the frustration.


I really enjoy the {%- etc. operators in Jinja 2 in the context of
Ansible, there are often cases when templating obscure configuration
files that they come in useful.


I believe SmileyChris had a patch to add this to DTL...

--
Curtis

--
You received this message because you are subscribed to the Google Groups "Django 
developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/3c0f36e6-74ea-2bf5-6fa5-fe616440e560%40tinbrain.net.
For more options, visit https://groups.google.com/d/optout.

Re: Changing {% include %} to strip a trailing newline

2017-01-04 Thread Carl Meyer

On 01/04/2017 02:53 PM, Tim Graham wrote:
> I'd like to know from Carl, Adam, and others, how much effort would be
> required to adapt the templates in your project for the new behavior. I
> imagine a script could be written to add newlines after all {% include
> %} in all plain text templates, for example.

Not much, I'd expect. Plain text templates aren't the common case, and
many of them don't use {% include %} anyway. And as you say, it is a
scriptable change if necessary.

That said, IMO the perceived cost of backwards-incompatible changes at
upgrade time scales with the number of changes required, even when
individually they don't require much work to fix. In theory, we promise
backwards-compatibility, and I don't think "call it a bugfix" can be a
universal escape hatch; it's clear that the current behavior has been
known and accepted for years. If we change this, it's a small but
incompatible change in the known and intended behavior, it's not fixing
a bug. But we've bitten the bullet and made far larger
backwards-incompatible changes in the past, so...

There is also the possibility that someone is relying on this behavior
in a more intentional way than I've ever done myself; e.g. has an
include file with multiple trailing newlines or other trailing
whitespace, specifically because it's desired in multiple different
places (in a preformatted text context). This (admittedly hypothetical)
case isn't quite as trivial to work around, if you propose to strip all
trailing whitespace. Or do you propose to strip only exactly one
trailing newline character?

In the end, I'm happy to abstain from this decision; I wouldn't really
be upset by any of the three options Aymeric listed.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d0dd9fb3-a009-bbe6-64fa-906d5cc44ff3%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.

signature.asc
Description: OpenPGP digital signature

Re: Changing {% include %} to strip a trailing newline

About "the backwards compatible option is probably necessary for some more 
advanced uses of templates out there, e.g. templating whitespace sensitive 
file formats." -- I'm not following why a similar find/replace approach 
wouldn't be sufficient to adapt those templates?

On Wednesday, January 4, 2017 at 6:10:12 PM UTC-5, Adam Johnson wrote:
>
> Whitespace control in templates is an exercise in frustration. In my 
>> experience more flexible tools such as {%-, {%+, -%}, and +%} in Jinja2 
>> increase the frustration.
>
>
> I really enjoy the {%- etc. operators in Jinja 2 in the context of 
> Ansible, there are often cases when templating obscure configuration files 
> that they come in useful.
>  
>
>> I'd like to know from Carl, Adam, and others, how much effort would be 
>> required to adapt the templates in your project for the new behavior. I 
>> imagine a script could be written to add newlines after all {% include %} 
>> in all plain text templates, for example.
>
>
> You're right, it's not too hard to fix for the text email template 
> usecase. But the backwards compatible option is probably necessary for some 
> more advanced uses of templates out there, e.g. templating whitespace 
> sensitive file formats.
>
> On 4 January 2017 at 22:53, Tim Graham > 
> wrote:
>
>> Aymeric, we have a difference of opinion. I feel that if {% include %} 
>> removed the trailing newline, it would result in far more intuitive 
>> whitespace control (which may be needed in plain text email templates, for 
>> example). I think the change has merits outside the context of this issue. 
>> For example, I checked view-source on a djangoproject.com page that uses 
>> a {% for ... %}{% include %}{% endfor %} construct and noticed that it's 
>> much shorter without all those extra blank lines. I'm not advocating for 
>> additional controls to craft well formatted HTML but I think there's some 
>> advantages to not having blank lines everywhere. (I don't like the 
>> keep_trailing_newline option either, I'd rather call it a bugfix and make a 
>> backwards-incompatible change but my projects are unaffected as far as I 
>> checked.)
>>
>> There are many questions on Stackoverflow about how to suppress newlines 
>> from {% include %} and even a django-include-strip-tag third-party app that 
>> offers the feature.
>>
>> In the context of multiple template engines, I think it would be useful 
>> if DTL templates could be written in a similar style to Jinja2.
>>
>> I'd like to know from Carl, Adam, and others, how much effort would be 
>> required to adapt the templates in your project for the new behavior. I 
>> imagine a script could be written to add newlines after all {% include %} 
>> in all plain text templates, for example.
>>
>> It won't be the end of the world if we can't get consensus to change the 
>> behavior -- I'm okay with removing the trailing newlines in the source 
>> files as needed, although it will add some inconvenience.
>>
>> On Wednesday, January 4, 2017 at 4:26:43 PM UTC-5, Aymeric Augustin wrote:
>>>
>>> If I understand correctly, we have to choose between:
>>>
>>> 1. breaking backwards compatibility for {% include %}
>>> 2. breaking backwards compatibility for widgets HTML
>>> 3. having a handful of single-line, non-newline-terminated files
>>>
>>> I don’t think option 1 is reasonable. Whitespace control in templates is 
>>> an exercise in frustration. In my experience more flexible tools such as 
>>> {%-, {%+, -%}, and +%} in Jinja2 increase the frustration. I don’t think 
>>> Django should get itself into this mess just for fixing the problem 
>>> discussed here.
>>>
>>> Option 2 seems acceptable to me. This is the "do nothing” option. I 
>>> don’t think the exact HTML string rendered by widgets is considered a 
>>> public API. We made changes to that output in the past, for example when we 
>>> switched to HTML5.
>>>
>>> Option 3 also seems reasonable to me. Files could end with {# no 
>>> newline, see issue #xxx #} and no newline. A test could validate that the 
>>> rendered output doesn’t contain a newline. This would be enough to catch 
>>> accidental regressions caused by editors automatically adding the newline.
>>>
>>> The quote from the POSIX standard says “should”, not “must”. If we 
>>> interpret it as if it was written in capitals in a RFC, this means we can 
>>> make an exception if we have a good reason.
>>>
>>> (At this point surely someone will suggest that we can write an 
>>> editorconfig to specify that these files mustn’t end with a newline. 
>>> Whatever works.)
>>>
>>> I don’t have a strong preference for 2 or 3, however, I have a strong 
>>> distaste for anything more complicated.
>>>
>>> Best regards,
>>>
>>> -- 
>>> Aymeric.
>>>
>>> On 4 Jan 2017, at 20:58, Tim Graham  wrote:
>>>
>>> Shortly after template widget rendering was merged, an issue about extra 
>>> newlines appearing in the rendered output was reported [0]:
>>>
>>> For example, from django-money:
>>>
>>> UIC-Franc
>>>

Re: django-formtools is neglected/unmaintained

2017-01-04 Thread eric . verner

Thanks a lot, Tim! If you think I could help, then let me know. Otherwise, 
I'll just stay out of your way :)

On Wednesday, January 4, 2017 at 4:02:29 PM UTC-7, Tim Graham wrote:
>
> According to https://github.com/django/django-formtools/issues/75, 
> there's a change in master that's needed for 1.10 compatibility.
>
> I'll try to do a release if I can get access to the PyPI record. I pinged 
> the owner (jezdez) in #django-dev about it.
>
> On Wednesday, January 4, 2017 at 5:51:56 PM UTC-5, Adam Johnson wrote:
>>
>> 1) Tim added testing on Django 1.10 in July 2016, it seems to work? 
>> https://github.com/django/django-formtools/commits/master
>> 2) New contributors are always welcome
>> 3) I don't know of other packages, you can check 
>> https://djangopackages.org/
>> 4) It's true that many websites are built with pure JS frontends these 
>> days, there are probably more recently developed tools there
>>
>> On 4 January 2017 at 22:42,  wrote:
>>
>>> Hi Tim,
>>>
>>> You make some good points. Basically, my situation is that I want to use 
>>> some features of Django v1.10 for a project at my company, but I am unable 
>>> to because I use django-formtools for a FormPreview, and it is not 
>>> compatible with v.1.10. I am also worried about using Django for future 
>>> projects given that there seem to be problems finding people to contribute 
>>> to what seems to be an important package. Whether I contribute depends on 
>>> the answers to several questions: (1) How much work is necessary to get 
>>> formtools compatible with v1.10? (2) Would my contribution make a 
>>> difference, given that I would be a new contributor, and I don't have a ton 
>>> of time to give? (3) Is there another package that people are using for 
>>> FormPreviews that I'm not aware of? I couldn't find anything else that was 
>>> more up-to-date than django-formtools. (4) Is this a sign that people 
>>> simply aren't using Django for this task, and are maybe instead using some 
>>> kind of Javascript library instead? If you have any information these 
>>> questions, please let me know.
>>>
>>> Thanks,
>>> Eric
>>>
>>> On Tuesday, January 3, 2017 at 6:04:59 PM UTC-7, Tim Graham wrote:

 Is the situation bad enough that you would volunteer to maintain the 
 project?

 The Django team is a collection of volunteers (excepting me) who work 
 on Django. It seems that no one on the team or reading this mailing list 
 has time or interest in maintaining the project. We need to clarify the 
 maintenance status, but that doesn't necessarily mean continuing the 
 project in its current form if there's little interest.

 Absent volunteers, another idea could be for an interested freelancer 
 to start a kickstarter-style fundraiser to raise money to fund some time 
 to 
 work on some issue in the backlog. At least it would help determine if any 
 users think the project is worth paying for.

 On Tuesday, January 3, 2017 at 7:28:59 PM UTC-5, 
 eric@datalyticsolutions.com wrote:
>
> This is really bad. django-formtools used to be part of the core of 
> Django. Is this getting the attention it deserves from the Django 
> Foundation?
>
> On Monday, November 28, 2016 at 9:55:48 PM UTC-7, Asif Saifuddin wrote:
>>
>> Hi Tim,
>>
>> In case there is lack of active maintainers for the project then you 
>> could add me to the list of maintainers. I do contribute to django eco 
>> system regularly.
>>
>> my github activities:
>>
>> github.com/auvipy
>>
>>
>> And about moving to jazzband, I do take part in django-admin2 
>> maintenance there with the two other old maintainers, The fact is moving 
>> it 
>> to jazzband haven't increased the active co maintainers.
>>
>> Thanks,
>>
>> Asif
>>
>> On Tuesday, November 29, 2016 at 5:45:59 AM UTC+6, Tim Graham wrote:
>>>
>>> Hi,
>>>
>>> django-formtools seems neglected. It's been several months since the 
>>> release of Django 1.10 but a compatible version of formtools hasn't 
>>> been 
>>> released to PyPI.
>>>
>>> Personally, I don't know if it's important to maintain it as an 
>>> "official project." An alternative could be to "donate" it to JazzBand (
>>> https://jazzband.co/) and see if the community maintains it.
>>>
>>> If you have an interest in this package (especially if you want to 
>>> maintain it), let us know.
>>>
>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to django-develop...@googlegroups.com.
>>> To post to this group, send email to django-d...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/django-developers.
>>> To view this discu

Re: Future of the development server's auto-reloading

So this idea doesn't get lost, I created a ticket for "Allow autoreloader 
to use watchman"
https://code.djangoproject.com/ticket/27685#ticket

On Monday, June 27, 2016 at 8:23:05 PM UTC-4, Tim Graham wrote:
>
> A pull request is proposed to add a new setting to allow specifying a 
> custom reloader:
>
> https://github.com/django/django/pull/6719
>
> Is this something anyone else would find useful and does it seems like we 
> could continue support that option even if autoreloading is refactored 
> based on the ideas here?
>
> On Saturday, June 4, 2016 at 8:37:25 PM UTC-4, Elf Sternberg wrote:
>>
>>
>>
>> On Saturday, August 8, 2015 at 2:53:32 PM UTC-7, Aymeric Augustin wrote:
>>>
>>> Hello,
>>>
>>> While writing some horrific code [1] related to the development server’s 
>>> auto-reloading, I realized that its design is fundamentally flawed and I 
>>> think that we should reconsider it.
>>>
>>> The current implementation walks through the list of all imported Python 
>>> modules, attempts to figure out which file they come from and builds a list 
>>> of these files. Then it watches all these files for changes by checking if 
>>> their mtime changed every second or, since 1.7, with inotify on Linux. I 
>>> tried to do it with kqueue on BSD (read: OS X) but I failed. This stuff is 
>>> hard.
>>>
>>> In addition to the source of imported Python modules, the autoreloader 
>>> watches .mo files in order to catch translation changes and source files 
>>> for modules that failed to import — it extracts them from exception 
>>> tracebacks. This shows the limits of basing the list on imported Python 
>>> modules. Even with such hacks, the auto-reloader will never handle all 
>>> cases. Two examples:
>>>
>>> - It doesn’t survive a syntax error in the settings module. I have 
>>> reasons to believe that this would be extremely messy to fix.
>>> - If a module reads a configuration file on disk at startup and caches 
>>> it, the auto-reloader doesn’t detect changes to that file.
>>>
>>> So we’re stuck with cases where the development server doesn’t reload. I 
>>> can’t say they’re edge cases; who never made a typo in a settings file? 
>>> People who run tutorials report that it’s a very common cause of 
>>> frustration for beginners.
>>>
>>> It's also wasteful. There’s little need to watch every file of every 
>>> dependency in the virtualenv for changes.
>>>
>>> I think it would be much better to remove the auto-reloading logic from 
>>> Django and have an external process watch the current working directory for 
>>> changes. (When virtualenv is considered appropriate for people with exactly 
>>> 0 days of Django experience under their belt, I think we can require a 
>>> dependency for the development server’s auto-reloading.)
>>>
>>> The behavior would be much easier to describe: if you change something 
>>> in your source tree, it reloads automatically; if you change something 
>>> somewhere else, you must reload manually. It would also be quite easy to 
>>> customize with a standard include / exclude mechanism to select which files 
>>> should be watched.
>>>
>>> We aren’t in the business of writing production web servers, neither are 
>>> we in the business of writing a filesystem watcher :-) Watchman [2] appears 
>>> to be a robust solution. However I would prefer a pure-Python 
>>> implementation that could easily be installed in a virtualenv. Does someone 
>>> have experience in this area?
>>>
>>> I’m not making a concrete proposal as I haven’t studied the details. It 
>>> this point I’d just like to hear what you think of the general concept.
>>>
>>> Thanks!
>>>
>>> -- 
>>> Aymeric.
>>>
>>> [1] https://github.com/django/django/pull/5106
>>> [2] https://github.com/facebook/watchman
>>> [3] 
>>> http://tutorial.djangogirls.org/en/django_installation/index.html#virtual-environment
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/8ccda5a4-25dc-47d9-951c-f3175d927f7a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Changing {% include %} to strip a trailing newline

Actually you're right it's probably the same

> Or do you propose to strip only exactly one
trailing newline character?

I thought the proposal was only for exactly one


On Wed, 4 Jan 2017 at 23:17, Tim Graham  wrote:

> About "the backwards compatible option is probably necessary for some more
>
> advanced uses of templates out there, e.g. templating whitespace
>
> sensitive file formats." -- I'm not following why a similar find/replace
> approach wouldn't be sufficient to adapt those templates?
>
>
> On Wednesday, January 4, 2017 at 6:10:12 PM UTC-5, Adam Johnson wrote:
>
> Whitespace control in templates is an exercise in frustration. In my
> experience more flexible tools such as {%-, {%+, -%}, and +%} in Jinja2
> increase the frustration.
>
>
> I really enjoy the {%- etc. operators in Jinja 2 in the context of
> Ansible, there are often cases when templating obscure configuration files
> that they come in useful.
>
>
> I'd like to know from Carl, Adam, and others, how much effort would be
> required to adapt the templates in your project for the new behavior. I
> imagine a script could be written to add newlines after all {% include %}
> in all plain text templates, for example.
>
>
> You're right, it's not too hard to fix for the text email template
> usecase. But the backwards compatible option is probably necessary for some
> more advanced uses of templates out there, e.g. templating whitespace
> sensitive file formats.
>
> On 4 January 2017 at 22:53, Tim Graham  wrote:
>
> Aymeric, we have a difference of opinion. I feel that if {% include %}
> removed the trailing newline, it would result in far more intuitive
> whitespace control (which may be needed in plain text email templates, for
> example). I think the change has merits outside the context of this issue.
> For example, I checked view-source on a djangoproject.com page that uses
> a {% for ... %}{% include %}{% endfor %} construct and noticed that it's
> much shorter without all those extra blank lines. I'm not advocating for
> additional controls to craft well formatted HTML but I think there's some
> advantages to not having blank lines everywhere. (I don't like the
> keep_trailing_newline option either, I'd rather call it a bugfix and make a
> backwards-incompatible change but my projects are unaffected as far as I
> checked.)
>
> There are many questions on Stackoverflow about how to suppress newlines
> from {% include %} and even a django-include-strip-tag third-party app that
> offers the feature.
>
> In the context of multiple template engines, I think it would be useful if
> DTL templates could be written in a similar style to Jinja2.
>
> I'd like to know from Carl, Adam, and others, how much effort would be
> required to adapt the templates in your project for the new behavior. I
> imagine a script could be written to add newlines after all {% include %}
> in all plain text templates, for example.
>
> It won't be the end of the world if we can't get consensus to change the
> behavior -- I'm okay with removing the trailing newlines in the source
> files as needed, although it will add some inconvenience.
>
> On Wednesday, January 4, 2017 at 4:26:43 PM UTC-5, Aymeric Augustin wrote:
>
> If I understand correctly, we have to choose between:
>
> 1. breaking backwards compatibility for {% include %}
> 2. breaking backwards compatibility for widgets HTML
> 3. having a handful of single-line, non-newline-terminated files
>
> I don’t think option 1 is reasonable. Whitespace control in templates is
> an exercise in frustration. In my experience more flexible tools such as
> {%-, {%+, -%}, and +%} in Jinja2 increase the frustration. I don’t think
> Django should get itself into this mess just for fixing the problem
> discussed here.
>
> Option 2 seems acceptable to me. This is the "do nothing” option. I don’t
> think the exact HTML string rendered by widgets is considered a public API.
> We made changes to that output in the past, for example when we switched to
> HTML5.
>
> Option 3 also seems reasonable to me. Files could end with {# no newline,
> see issue #xxx #} and no newline. A test could validate that the rendered
> output doesn’t contain a newline. This would be enough to catch accidental
> regressions caused by editors automatically adding the newline.
>
> The quote from the POSIX standard says “should”, not “must”. If we
> interpret it as if it was written in capitals in a RFC, this means we can
> make an exception if we have a good reason.
>
> (At this point surely someone will suggest that we can write an
> editorconfig to specify that these files mustn’t end with a newline.
> Whatever works.)
>
> I don’t have a strong preference for 2 or 3, however, I have a strong
> distaste for anything more complicated.
>
> Best regards,
>
>
>
> --
> Aymeric.
>
>
>
>
>
>
>
> On 4 Jan 2017, at 20:58, Tim Graham  wrote:
>
> Shortly after template widget rendering was merged, an issue about extra
> newlines appearing in the rendered output was

Re: Add custom autoreload file tracking options setting


> On Jan 4, 2017, at 5:40 PM, Adam Johnson  wrote:
> 
> How do people serve development Javascript & CSS files?  These days 
> Javascript & CSS involves a large build process.  Are we forced to manually 
> restart the development server every time Javascript changes?
> 
> Django just serves them from the filesystem, and doesn't cache them itself. 
> You reload the page in your browser, and Django serves the files from disk
> 
> Does the Django development server restart if you edit Templates?  Because it 
> should also do that.
> 
> Similarly, by default templates aren't cached, so on each view the dev server 
> loads the templates from disk, parses them, and renders them.
>  

Does it do the same under production environments - read and interpret each 
template file on every request?  

Is there interest in making a much faster version of this flow?

> 
> On 4 January 2017 at 22:31, Bobby Mozumder  > wrote:
> 
> > On Jan 4, 2017, at 4:47 PM, Aymeric Augustin 
> >  > > wrote:
> >
> > Hello Bobby,
> >
> >> On 4 Jan 2017, at 22:25, Bobby Mozumder  >> > wrote:
> >>
> >> It’s actually called once on app startup during DB connection via a Signal.
> >
> > Unless I missed something, since the development server creates a new 
> > connection to the database for each request — Python’s threaded socket 
> > server creates a new thread for each connection and database connections 
> > are thread local in Django — prepared statements will be refreshed for each 
> > request. So I don’t think you need autoreloading here.
> 
> Hmm I thought the test server used permanent connections if CONN_MAX_AGE was 
> set to None?  Could have sworn it was that way for a while?
> 
> In any case, I just tried the Javascript & CSS Makefile with a simple one 
> line change to one file, and it took 4 seconds.  That’s going to be way too 
> long to do on each request.
> 
> How do people serve development Javascript & CSS files?  These days 
> Javascript & CSS involves a large build process.  Are we forced to manually 
> restart the development server every time Javascript changes?
> 
> I think it should be just like changing Python files and it autoreloads 
> Javascript & CSS as it changes.
> 
> Also, I haven’t used the Django Template system or Jinja in over a year.  
> Does the Django development server restart if you edit Templates?  Because it 
> should also do that.
> 
> This is what I’m getting at with this autoreload.. it should include the full 
> system, not just the Python files.
> 
> >
> >> (This whole process helps speed up the view response generation times.  I 
> >> can generate a full page in about 4ms, and that includes GZip.)
> >
> > Off topic, but I’m jealous…
> >
> 
> Also, my average page generates in 1-2ms, and fully cached html serves in 200 
> microseconds, and that include Gzip, since I cache Gzip (which makes my cache 
> 10x bigger).   There’s lots that I can contribute here, including 
> non-blocking analytics stored in teh database.  If I can make it “generic” 
> enough I’ll try to publish it, but the code is really manual.  Maybe instead 
> of a formal toolset, I could post a methodology manual?  It’s really tedious 
> though, involving materialized database views & all sorts of other techniques.
> 
> BTW My site is at http://www.futureclaw.com 
> 
> > 
> >
> > I still believe we should stop maintaining an autoreloader as soon as 
> > possible. Django’s autoreloader is annoyingly slow, highly inefficient, 
> > moderately well designed, and a gigantic pain to maintain. I’m more scared 
> > of django.utils.autoreload than of django.db.models.related before it was 
> > cleaned up.
> >
> > I wish one day someone will take the time to write a good autoreloading dev 
> > server based on watchman. This would solve the problem discussed here 
> > because watchman watches all files in the current directory. The correct 
> > way to do this is to throw away the current design and start from scratch.
> >
> > Watchman is smart enough to wait until you’ve finished a git operation to 
> > trigger a reload. Once such polished tech has become available, trying to 
> > compete with a thread that checks the mtime of all known files every second 
> > isn’t funny anymore. In fact it’s just sad.
> >
> > 
> >
> > While I don’t care much about django.utils.autoreloader because I want to 
> > kill it, I’m reluctant to add public APIs such as the proposed setting, 
> > which may not make sense with an hypothetical better autoreloader.
> >
> > I’m -0 on the change. I could move to +0 if I understood why the use case 
> > described here requires watching additional files.
> >
> 
> I guess I could just use Watchman to restart the Django development server as 
> needed?
> 
> I’m OK with that.. my only goal is to make sure I don’t have to restart 
> Django manually every time I edit CSS/Javascript/SQL/HTML.
> 
>

Re: Add custom autoreload file tracking options setting