Re: delegating our static file serving

[David Evans]

Hi Aleksej,

Thanks for your work on this, and sorry I haven't replied sooner -- just 
been busy with lots of other things ;)

My plan for integrating WhiteNoise was actually to merge it in as part of 
contrib.staticfiles and basically stop development on it as an independent 
project. When I first started WhiteNoise I went to quite a lot of effort to 
make the core framework-agnostic and to have all the Django specific code 
as an extension to the core. However this makes the code more complex than 
if it was written for Django alone. I'm also not sure it gets a lot of use 
outside of the Django world as other frameworks have their own way of doing 
things and the most useful features of WhiteNoise are Django only.

So the end-goal in my mind is to have a stripped down and simplified 
version of the WhiteNoise code bundled as part of Django itself, which I 
could then help maintain as part of the wider Django project. However, I 
don't think this process would be an ideal project for someone who is a new 
contributor to Django as it would require quite a detailed understanding of 
the design decisions and architectural tradeoffs behind WhiteNoise. Really 
this is the sort of thing I ought to be doing but it's been quite a busy 
year (new baby, work, etc etc) and I just haven't had the time.

If you're able to bundle WhiteNoise as it currently is with Django in a way 
that the core devs are happy with then that's great, and obviously I 
wouldn't stand in your way. But I can imagine that there might be some 
resistance to creating a hard dependency on WhiteNoise, while it's still a 
standalone project.

Again, thanks for the work you've put into this and sorry I don't have a 
more helpful response.

Dave

On Wednesday, 22 March 2017 16:14:58 UTC, Aleksej Manaev wrote:
>
> Am I on the right track with the integration of WhiteNoise into Django? 
> Can Someone lead me to the next steps that need to be done?
>
>
> https://github.com/django/django/compare/master...AleksejManaev:ticket_27325?expand=1
>
> I think "django/views/static.py" can't be deprecated because it is used 
> for providing MEDIA_FILES during development and WhiteNoise is only 
> responsible for STATIC_FILES.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/cfbf04a4-9346-4fd7-95bf-e3802f7a8e5d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: delegating our static file serving

[David Evans]

I've been intending to work on this for a while (as the original author of 
WhiteNoise it would make the most sense for me to do the integration work) 
but as ever the problem has been finding the time. I've been hoping to make 
some progress on this over the Christmas period, but we'll see what happens!

On Tuesday, 20 December 2016 21:42:55 UTC, Guilherme Leal wrote:
>
> I'm willing to cooperate on this one. Anyone is working on this and need 
> help?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/9a6e143d-9bf9-4e26-89ca-cfc9944ba794%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Revisiting lazy middleware initialization

[David Evans]

Great!

Well the two line patch I suggested to `get_wsgi_application` solves the
problem. But it still leaves the lazy loading mechanism in the code base.
Getting rid of it would obviously be preferable, but is more complex. I'll
work on producing a patch for that though.

Dave
On 25 Mar 2016 10:08 a.m., "Florian Apolloner" <f.apollo...@gmail.com>
wrote:

> +1 -- Patches welcome :D
>
> On Friday, March 25, 2016 at 11:02:26 AM UTC+1, Aymeric Augustin wrote:
>>
>> On 24 Mar 2016, at 17:39, David Evans <drh@gmail.com> wrote:
>>
>>
>> Currently, middleware is initialized lazily on serving the first request,
>> rather than on application start. There may well have been good reasons for
>> this historically, but I don't think they apply any longer.
>>
>>
>> Indeed, since 1.7 and the app-loading refactor, Django does its best to
>> crash when it starts rather than when it serves the first request. +1 to
>> making this change.
>>
>> --
>> Aymeric.
>>
>> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/django-developers/iIm7M6aUJmU/unsubscribe
> .
> To unsubscribe from this group and all its topics, send an email to
> django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/3d63f9d8-8781-4b9e-8498-9cc7c091831b%40googlegroups.com
> <https://groups.google.com/d/msgid/django-developers/3d63f9d8-8781-4b9e-8498-9cc7c091831b%40googlegroups.com?utm_medium=email_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAHbVmPyedP2p6YmMyJuChks8JPZyG2eyVx3cr8Vaavyfm64nFg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Revisiting lazy middleware initialization

[David Evans]

Hi all,

Currently, middleware is initialized lazily on serving the first request, 
rather than on application start. There may well have been good reasons for 
this historically, but I don't think they apply any longer. Lazy 
initialization is unhelpful if a middleware class throws an error (e.g to 
report a misconfiguration) because the application will appear to start 
successfully and only later report the error when a request is made.

I'd like to propose initializing middleware when `get_wsgi_application` is 
called. This solves the problem described above and, as far as I can tell, 
raises no backwards compatibility issues.

More details on all this below.


### 1. Specific example of the problem

I recently wrote an adapter for the WhiteNoise static file server so it 
could function as Django middleware as well as WSGI middleware 
(https://github.com/evansd/whitenoise). WhiteNoise may be unusual in doing 
a non-trivial amount of work on initialization, but it doesn't seem 
unreasonable. When used as WSGI middleware any errors are triggered 
immediately on start up, but not so when used as Django middleware. This 
makes for a worse developer experience and an increased chance of 
deployment errors.


### 2. Reasons previously given for lazy initialization

There was some brief discussion in this ticket 4 years ago:
https://code.djangoproject.com/ticket/18577

The reason given there is that "resolving on first request makes most 
sense, especially for the case where you might not be serving requests at 
all". Presumably this refers to running non-http-related management 
commands. But in those cases we never instantiate a WSGI application anyway 
(wsgi.py is just never imported) so this is no reason not to initialize 
eagerly when constructing the WSGI application. (Of course, things may have 
been different 4 years ago.)

Another reason is given in the comments in django.core.handles.wsgi:
https://github.com/django/django/blob/3c1b572f1815c295878795b183b1957d0df2ca39/django/core/handlers/wsgi.py#L154

This says "Set up middleware if needed. We couldn't do this earlier, 
because settings weren't available". However `get_wsgi_application` (the 
only public WSGI API) now calls `django.setup()` before constructing the 
handler so settings are in fact available.


### 3. Proposed solution

My proposal is simply to amend `get_wsgi_application` as follows:

def get_wsgi_application():
django.setup(set_prefix=False)
handler = WSGIHandler()
handler.load_middleware()
return handler

It's possible that this logic could be moved into the handler's __init__ 
method. This caused no problems with existing application when I tried it, 
however it did cause problems with the test suite which seems to rely on 
the old behaviour in places. The above proposal passes all existing tests 
as is.


### 4. Backwards compatibility issues

Middleware constructors have no means of accessing the request object or 
anything that depends on it. They are called right at the start of the 
handler's `__call__` method before the `request_started` signal is sent and 
before the `script_prefix` thread-local is set. Therefore it cannot matter, 
from the middleware class's perspective, whether it is instantiated before 
or after the first request comes in.


I'm aware this issue probably isn't high on anyone else's priority list, 
but I think it would count as a genuine -- if small -- improvement to 
Django.

Thanks,

Dave

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/a8b2fc46-5bc3-4797-9e94-052eb3e35e8a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Feedback on Django Channels

[David Evans]

On the static files question, I'm about to release v3 of WhiteNoise 
(http://whitenoise.evans.io/en/latest/changelog.html) which provides the 
option to integrate via Django middlware, rather than wsgi middleware. (It 
uses the FileResponse class, which didn't exist when I first wrote 
WhiteNoise.) I'm hoping (though I haven't tested it yet) that this will 
work out of the box with Channels, as it's only using standard Django APIs.

Dave

On Thursday, 17 March 2016 16:42:05 UTC, Jacob Kaplan-Moss wrote:
>
> Hi folks (and especially Andrew):
>
> I've just completed writing an example Channels app [1] for an article 
> about Channels [2]. Overall it was a super-pleasant experience: Channels 
> seems pretty solid, the APIs make sense to me, and I couldn't be more 
> excited about the new things this'll let me do! 
>
> In the interests of making this thing as solid as possible before we merge 
> it into Django, I do have some feedback on some of the hiccups I 
> encountered. Roughly in order of severity (as I perceive it), they are:
>
> 1. Debugging errors:
>
> I found debugging errors that happen in a consumer to be *really* 
> difficult -- errors mostly presented as things just silently not working. 
> It took a ton of messing around with logging setups before I could get 
> anything of use dumped to the console. This isn't strictly a Channels issue 
> (I've noted similar problems with AJAX views and errors in Celery tasks), 
> but I think Channels sorta brings the issue to a head. 
>
> I think we need some better defaults, and simple, clear documentation, to 
> make sure that exceptions go somewhere useful.
>
> 2. Static files:
>
> I had trouble getting static files served. I'm used to using Whitenoise (
> http://whitenoise.evans.io/en/stable/) for small-to-medium-ish sites that 
> don't need a proper static server, but of course it doesn't work with 
> Channels since Channels doesn't use WSGI! I found the (undocumented) 
> StaticFilesConsumer (
> https://github.com/jacobian/channels-example/blob/master/chat/routing.py#L5-L8),
>  
> but that feels less than ideal.
>
> I think this might be an opportunity, however. If Daphne learns how to 
> serve static files (perhaps via optional integration with Whitenoise?), 
> this would actually make static media in Django a bit easier by default.
>
> [I would be happy to work on this if I get a thumbsup.]
>
> 3. WebSocket routing:
>
> Channels routes all WebSocket connections to a single set of consumers 
> (the `websocket.*` consumers). This means that if you want multiple 
> WebSocket URLs in a single app you need to manually parse the path. And, to 
> make things more complicated, you only get the WebSocket path passed in the 
> connection message, so you have to also use a channel session to keep track.
>
> This feels like a distinct step back from the URL routing we already have 
> in Django, and it was surprising to me to have to do this by hand. It 
> definitely felt like Channels is missing some sort of WebSocket URL router. 
>
> I had a brief chat with Andrew, who indicates that he'd planned for this 
> to be a post-1.0 feature. I'm not sure I agree - it feels pretty 
> fundamental - but I'd like to hear other thoughts. 
>
> [This is another thing I'd be interested in working on, assuming a thumbs.]
>
> ---
>
> Has anyone else here played with Channels? Are there other things I'm 
> missing that might need to be included before we merge this?
>
> Jacob
>
> [1] https://github.com/jacobian/channels-example
>
> [2] 
> https://blog.heroku.com/archives/2016/3/17/in_deep_with_django_channels_the_future_of_real_time_apps_in_django
> ?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ba29cc7a-90fc-4cf2-a65d-da782e83b3ca%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: delegating our static file serving

[David Evans]

I've just pushed a simple middleware wrapper that lets you install 
WhiteNoise by adding:

MIDDLEWARE_CLASSES = (
   ...
'whitenoise.middleware.StaticFilesMiddleware',
   ...
)


See: https://github.com/evansd/whitenoise/blob/master/whitenoise/middleware.py

No messing with wsgi.py!

This approach will be marginally less efficient than handling things at the 
WSGI layer, but the integration is much cleaner.

On Tuesday, 29 December 2015 17:31:02 UTC, David Evans wrote:
>
> I'd be very happy to help with this. Two things to mention:
>
> 1. When I first wrote WhiteNoise the FileResponse class didn't exist and 
> so the only way I could access `wsgi.file_wrapper` was by wrapping the wsgi 
> application directly and bypassing Django. Now we have the FileResponse 
> class it would be possible to implement WhiteNoise as standard Django 
> middleware without having to edit wsgi.py. I think this would be a much 
> cleaner approach, and I don't think it will be too much work to implement. 
> I should have time to look at this next week.
>
> 2. Serving media files is a slightly different case. WhiteNoise was 
> designed around the assumption that it's serving a set of 
> developer-supplied, public files which remain unchanged during the lifetime 
> of the process. This simplifies a lot of performance and security concerns 
> that come with serving mutable, user-supplied files. At present, if you use 
> `add_files(settings.MEDIA_ROOT, prefix=settings.MEDIA_URL)` as you 
> suggested then WhiteNoise won't pick up any files that were uploaded after 
> the application was started -- at least, not unless you enable the 
> "autorefresh" setting which I explicitly don't recommend for production.
>
> Obviously it's possible to support this if we decide this is an important 
> goal but it will need more thought and will definitely be more work than 
> just supporting static files.
>
> On Tuesday, 29 December 2015 00:36:06 UTC, Tim Graham wrote:
>>
>> I'd like to work together with Dave to develop a proof of concept that 
>> integrates whitenoise into Django. I spent about an hour looking through 
>> whitenoise and our own static file serving code, and I think integrating 
>> whitenoise will yield a simpler user experience with about the same amount 
>> of code as have now.
>>
>> Essentially, we'd recommend adding something like this to existing 
>> wsgi.py files (it would be in the default startproject template)
>>
>> from whitenoise.django import DjangoWhiteNoise
>> application = DjangoWhiteNoise(application)
>> application.add_files(settings.MEDIA_ROOT, prefix=settings.MEDIA_URL)
>>
>> which would have the benefit of working out of the box in production too, 
>> I think. Of course, you could disable that based on settings.DEBUG or some 
>> other toggle.
>>
>> We could then deprecate:
>>
>> * django/contrib/staticfiles/views.py
>> * django/contrib/staticfiles/management/commands/runserver.py
>> * django/contrib/staticfiles/handlers.py
>> * django/views/static.py
>>
>> Any objections to doing further investigation in this area?
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/e24d4590-e8e7-437d-a441-b1ba51e1f535%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: delegating our static file serving

[David Evans]

I'd be very happy to help with this. Two things to mention:

1. When I first wrote WhiteNoise the FileResponse class didn't exist and so 
the only way I could access `wsgi.file_wrapper` was by wrapping the wsgi 
application directly and bypassing Django. Now we have the FileResponse 
class it would be possible to implement WhiteNoise as standard Django 
middleware without having to edit wsgi.py. I think this would be a much 
cleaner approach, and I don't think it will be too much work to implement. 
I should have time to look at this next week.

2. Serving media files is a slightly different case. WhiteNoise was 
designed around the assumption that it's serving a set of 
developer-supplied, public files which remain unchanged during the lifetime 
of the process. This simplifies a lot of performance and security concerns 
that come with serving mutable, user-supplied files. At present, if you use 
`add_files(settings.MEDIA_ROOT, prefix=settings.MEDIA_URL)` as you 
suggested then WhiteNoise won't pick up any files that were uploaded after 
the application was started -- at least, not unless you enable the 
"autorefresh" setting which I explicitly don't recommend for production.

Obviously it's possible to support this if we decide this is an important 
goal but it will need more thought and will definitely be more work than 
just supporting static files.

On Tuesday, 29 December 2015 00:36:06 UTC, Tim Graham wrote:
>
> I'd like to work together with Dave to develop a proof of concept that 
> integrates whitenoise into Django. I spent about an hour looking through 
> whitenoise and our own static file serving code, and I think integrating 
> whitenoise will yield a simpler user experience with about the same amount 
> of code as have now.
>
> Essentially, we'd recommend adding something like this to existing wsgi.py 
> files (it would be in the default startproject template)
>
> from whitenoise.django import DjangoWhiteNoise
> application = DjangoWhiteNoise(application)
> application.add_files(settings.MEDIA_ROOT, prefix=settings.MEDIA_URL)
>
> which would have the benefit of working out of the box in production too, 
> I think. Of course, you could disable that based on settings.DEBUG or some 
> other toggle.
>
> We could then deprecate:
>
> * django/contrib/staticfiles/views.py
> * django/contrib/staticfiles/management/commands/runserver.py
> * django/contrib/staticfiles/handlers.py
> * django/views/static.py
>
> Any objections to doing further investigation in this area?
>
> On Saturday, June 20, 2015 at 8:09:11 AM UTC-4, David Evans wrote:
>>
>> On Friday, 5 December 2014 19:14:29 UTC, Carl Meyer wrote:
>>>
>>> On 12/04/2014 10:33 PM, Collin Anderson wrote: 
>>> > Hi All, 
>>> > 
>>> > I'm pretty interested in getting secure and _somewhat_ efficient 
>>> static 
>>> > file serving in Django. 
>>> > 
>>> > Quick history: 
>>> > 2005 - Jacob commits #428: a "static pages" view.  Note that this view 
>>> > should only be used for testing!" 
>>> > 2010 - Jannis adds staticfiles. Serving via django is considered 
>>> "grossly 
>>> > inefficient and probably insecure". 
>>> > 2011 - Graham Dumpleton adds wsgi.file_wrapper to Gunicorn. 
>>> > 2012 - Aymeric adds StreamingHttpResponse and now files are read in 
>>> chunks 
>>> > rather than reading the entire file into memory. (No longer grossly 
>>> > inefficient IMHO.) 
>>> > 
>>> > I propose: 
>>> > - Deprecate the "show_indexes" parameter of static.serve() (unless 
>>> people 
>>> > actually use it). 
>>> > - Have people report security issues to secu...@djangoproject.com 
>>> (like 
>>> > always) 
>>> > - Audit the code and possibly add more security checks and tests. 
>>> > - add wsgi.file_wrapper support to responses (5-line proof of concept: 
>>> > https://github.com/django/django/pull/3650 ) 
>>> > - support serving static files in production, but still recommend 
>>> > nginx/apache or a cdn for performance. 
>>> > - make serving static files in production an opt-in, but put the view 
>>> in 
>>> > project_template/project_name/urls.py 
>>> > 
>>> > I think it's a huge win for low-traffic sites or sites in the "just 
>>> trying 
>>> > to deploy and get something live" phase. You can always optimize later 
>>> by 
>>> > serving via nginx or cdn. 
>>> > We already have the views, api, and logic around for finding and 
>>> serving 
&g

Re: delegating our static file serving

[David Evans]

On Friday, 5 December 2014 19:14:29 UTC, Carl Meyer wrote:
>
> On 12/04/2014 10:33 PM, Collin Anderson wrote: 
> > Hi All, 
> > 
> > I'm pretty interested in getting secure and _somewhat_ efficient static 
> > file serving in Django. 
> > 
> > Quick history: 
> > 2005 - Jacob commits #428: a "static pages" view.  Note that this view 
> > should only be used for testing!" 
> > 2010 - Jannis adds staticfiles. Serving via django is considered 
> "grossly 
> > inefficient and probably insecure". 
> > 2011 - Graham Dumpleton adds wsgi.file_wrapper to Gunicorn. 
> > 2012 - Aymeric adds StreamingHttpResponse and now files are read in 
> chunks 
> > rather than reading the entire file into memory. (No longer grossly 
> > inefficient IMHO.) 
> > 
> > I propose: 
> > - Deprecate the "show_indexes" parameter of static.serve() (unless 
> people 
> > actually use it). 
> > - Have people report security issues to secu...@djangoproject.com 
>  (like 
> > always) 
> > - Audit the code and possibly add more security checks and tests. 
> > - add wsgi.file_wrapper support to responses (5-line proof of concept: 
> > https://github.com/django/django/pull/3650 ) 
> > - support serving static files in production, but still recommend 
> > nginx/apache or a cdn for performance. 
> > - make serving static files in production an opt-in, but put the view in 
> > project_template/project_name/urls.py 
> > 
> > I think it's a huge win for low-traffic sites or sites in the "just 
> trying 
> > to deploy and get something live" phase. You can always optimize later 
> by 
> > serving via nginx or cdn. 
> > We already have the views, api, and logic around for finding and serving 
> > the correct files. 
> > We can be just as efficient and secure as static/dj-static without 
> needing 
> > to make people install and configure wsgi middleware to the application. 
> > We could have staticfiles classes implement more complicated features 
> like 
> > giving cache recommendations, and serving pre-gzipped files. 
> > 
> > Is this a good idea? I realize it's not totally thought through. I'm 
> fine 
> > with waiting until 1.9 if needed. 
>
> I also think this is a good plan. It certainly makes sense to look at 
> "static" and "whitenoise" for ideas and compare their code to ours to 
> see where we could be more efficient or secure, but it's much less churn 
> for Django users if we simply improve our existing code rather than pull 
> in something wholly new. 
>
> Carl 
>
>
>  
Sorry to revive an old thread here, but I just wanted to add that v2.0 of 
WhiteNoise now supports serving development files, providing the same 
behaviour as runserver currently does in DEBUG mode. (There were enough 
people wanting to do their development using gunicorn rather than runserver 
to make this worthwhile.)

This means that WhiteNoise is now a one-stop-shop for static file handling 
in Django. If there's still an appetite for integrating it, or something 
equivalent, into core I'd be happy to help out.

Dave

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/c4b46672-276e-4177-802e-df0f09389887%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: delegating our static file serving

[David Evans]

+1 to this: I'm all in favour of supporting production-adequate static file
handling in core. A couple of small points:


We already have the views, api, and logic around for finding and serving
> the correct files.


One question that needs to be thought through is the role of
`collectstatic` and `STATIC_ROOT`.

The existing static.serve view just ignores `STATIC_ROOT` completely and
uses the configured StaticFileFinders to locate files. This means if you
use the ManifestStaticFilesStorage backend -- or anything that uses the
`post_process` hook -- it won't find your processed files at all.

Probably the simplest thing is to switch on DEBUG, and serve files from
`STATIC_ROOT` (falling back to using the finders) when DEBUG is False, and
use the existing behaviour otherwise.


We could have staticfiles classes implement more complicated features like
> giving cache recommendations, and serving pre-gzipped files.


Definitely +1 to using classes (CBVs, presumably) to handle this. The
existing views don't easily support customising behaviour.

Regards,

Dave





On 5 December 2014 at 05:33, Collin Anderson  wrote:

> Hi All,
>
> I'm pretty interested in getting secure and _somewhat_ efficient static
> file serving in Django.
>
> Quick history:
> 2005 - Jacob commits #428: a "static pages" view.  Note that this view
> should only be used for testing!"
> 2010 - Jannis adds staticfiles. Serving via django is considered "grossly
> inefficient and probably insecure".
> 2011 - Graham Dumpleton adds wsgi.file_wrapper to Gunicorn.
> 2012 - Aymeric adds StreamingHttpResponse and now files are read in chunks
> rather than reading the entire file into memory. (No longer grossly
> inefficient IMHO.)
>
> I propose:
> - Deprecate the "show_indexes" parameter of static.serve() (unless people
> actually use it).
> - Have people report security issues to secur...@djangoproject.com (like
> always)
> - Audit the code and possibly add more security checks and tests.
> - add wsgi.file_wrapper support to responses (5-line proof of concept:
> https://github.com/django/django/pull/3650 )
> - support serving static files in production, but still recommend
> nginx/apache or a cdn for performance.
> - make serving static files in production an opt-in, but put the view in
> project_template/project_name/urls.py
>
> I think it's a huge win for low-traffic sites or sites in the "just trying
> to deploy and get something live" phase. You can always optimize later by
> serving via nginx or cdn.
> We already have the views, api, and logic around for finding and serving
> the correct files.
> We can be just as efficient and secure as static/dj-static without needing
> to make people install and configure wsgi middleware to the application.
> We could have staticfiles classes implement more complicated features like
> giving cache recommendations, and serving pre-gzipped files.
>
> Is this a good idea? I realize it's not totally thought through. I'm fine
> with waiting until 1.9 if needed.
>
> Collin
>
> On Saturday, November 29, 2014 6:07:05 PM UTC-5, Collin Anderson wrote:
>>
>> Hi All,
>>
>> I think doing something here is really good idea. I'm happy with any of
>> the solutions mentioned so far.
>>
>> My question is: what does static/dj-static do that our built-in code
>> doesn't do? What makes it more secure? It seems to me we're only missing is
>> wsgi.file_wrapper and maybe a few more security checks. Why don't we just
>> make our own code secure and start supporting it?
>> Here's basic wsgi.file_wrapper support: https://github.com/django/
>> django/pull/3650
>>
>> We could then, over time, start supporting more extensions ourselves:
>> ranges, pre-gziped files, urls with never-changing content, etc. That way
>> we get very, very deep django integration. It seems to me this is a piece
>> that a web framework should be able to support itself.
>>
>> Collin
>>
>>
>> On Friday, November 28, 2014 9:15:03 AM UTC-5, Tim Graham wrote:
>>>
>>> Berker has worked on integrating gunicorn with runserver
>>>  so that we might be able
>>> to deprecate our own homegrown webserver. Windows support for gunicorn is
>>> supposedly coming soon which
>>> may actually make the idea feasible. This way we provide a more secure
>>> solution out of the box (anecdotes indicate that runserver is widely used
>>> in production despite our documentation warnings against doing so).
>>>
>>> On the pull request, Anssi had an idea to use dj-static
>>>  to serve static/media
>>> files. My understanding is that we would basically integrate the code for
>>> dj-static into Django and then add a dependency on static
>>> . It could be an optional
>>> dependency since you might want to serve static files differently in
>>> production, but it would likely be more or less universally used 

Re: delegating our static file serving

[David Evans]

It's worth flagging up that part of what makes WhiteNoise good for serving 
static files in production also makes it not quite as good for using in 
development.

Most file servers work by taking the requested URL, constructing a local 
path from it, and then checking whether a file exists at that particular 
path. WhiteNoise on the other hand, searches for all available static files 
on startup and then builds a dictionary of files which it assumes won't 
change until the code is reloaded. This has a couple of big advantages: one 
is security -- there's a whole class of path-traversal vulnerabilities that 
I don't have to worry about because I never have to generate a path from a 
user-supplied URL. The other is that WhiteNoise can serve files from 
arbitrary URLs, rather than just within a fixed prefix like `/static/`, 
without adding an extra hit on the filesystem for every single request that 
comes in.

While this works well in production, it breaks down in development where 
the static files change without causing a reload of the server. I've 
started getting bug reports from users who are trying to use WhiteNoise in 
development and are getting confused by this.

So, while I'd obviously be delighted to see some of WhiteNoise's features 
merged into core I think there will need to be a slightly different 
approach for serving files in development.

Dave

On Saturday, 29 November 2014 23:07:05 UTC, Collin Anderson wrote:
>
> Hi All,
>
> I think doing something here is really good idea. I'm happy with any of 
> the solutions mentioned so far.
>
> My question is: what does static/dj-static do that our built-in code 
> doesn't do? What makes it more secure? It seems to me we're only missing is 
> wsgi.file_wrapper and maybe a few more security checks. Why don't we just 
> make our own code secure and start supporting it?
> Here's basic wsgi.file_wrapper support: 
> https://github.com/django/django/pull/3650
>
> We could then, over time, start supporting more extensions ourselves: 
> ranges, pre-gziped files, urls with never-changing content, etc. That way 
> we get very, very deep django integration. It seems to me this is a piece 
> that a web framework should be able to support itself.
>
> Collin
>
>
> On Friday, November 28, 2014 9:15:03 AM UTC-5, Tim Graham wrote:
>>
>> Berker has worked on integrating gunicorn with runserver 
>>  so that we might be able to 
>> deprecate our own homegrown webserver. Windows support for gunicorn is 
>> supposedly coming soon which 
>> may actually make the idea feasible. This way we provide a more secure 
>> solution out of the box (anecdotes indicate that runserver is widely used 
>> in production despite our documentation warnings against doing so).
>>
>> On the pull request, Anssi had an idea to use dj-static 
>>  to serve static/media files. 
>> My understanding is that we would basically integrate the code for 
>> dj-static into Django and then add a dependency on static 
>> . It could be an optional 
>> dependency since you might want to serve static files differently in 
>> production, but it would likely be more or less universally used in 
>> development. We could then say that django.views.static.serve (and its 
>> counterpart in staticfiles) is okay to use in production (and I guess 
>> people are already using them in production despite our warnings that they 
>> are not hardened for production use).
>>
>> What do you think of this plan?
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/a4efc639-ee61-4c6b-88b4-2866f86bdc24%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Proposal to add a secure JSON encoding template tag

[David Evans]

Thanks Chris. Yes, I do share your feeling that the declare-a-global-var
way of doing things is bit icky. My reason for wanting to support it was
that in order to use the "proper" method you need to be able to parse JSON
and that means that if you want to support older browsers you've now got a
dependency on some external JSON parsing library.

Also, although I called them different "modes" of operation, they're really
just minor syntactic variants from the point of view of the template tag so
it seemed a bit of shame to force the user down one route when the other
could be so easily supported. I didn't want a situation where people ended
up using another, insecure method because the official method forced them
to do things in a way that seemed too complicated.

That said, I'd be very open to just supporting the CSP-compatible method if
that was the general view.


On 24 May 2014 06:37, Chris Beaven <smileych...@gmail.com> wrote:

> I like the idea, I've been using a custom script that does the first mode
> of this tag nearly exactly the same way (with the same security escaping).
>
> Not the biggest fan of the second mode of operation since like you say,
> it's not compatible with strict CSP. Why not just encourage people to do it
> the correct way? Having a separate mode of operation for the same tag
> necessary, perhaps it'd be less controversial just getting the first mode
> in?
>
> A related side note, it's good practice for JS templates to live within
> script blocks too. I have {% scriptblock %}{% endscriptblock %} that is
> sits next to {% jsonblock %} in my custom library since it uses the same
> escaping methods.
>
>
>
> On Tuesday, May 13, 2014 10:03:43 PM UTC+12, David Evans wrote:
>
>> There was some discussion previously (see https://code.djangoproject.
>> com/ticket/17419) of adding a JSON encoding filter to Django. This was
>> rejected as being impossible (or very difficult) to do securely. However
>> the requirement to embed JSON in an HTML page is quite a common one, and
>> it's easy to get wrong and create XSS vulnerabilities. We should make it
>> easy for people to do the right thing.
>>
>> I propose a ``json`` tag (implementation here
>> <https://gist.github.com/evansd/41ea9dfc90d87f6afde1>) which outputs the
>> entire script element as well as the JSON data. By enforcing the context in
>> which in the JSON is output, it's possible to escape it securely.
>>
>> It would have two basic modes of operation. The first, and recommended,
>> one would look like this:
>>
>> {% json data id="initial-data" %}
>>
>>
>> and would produce HTML like this:
>>
>> 
>>   {"foo": "bar"}
>> 
>>
>>
>> The resulting data would be accessed in JavaScript like this:
>>
>> var el = document.getElementById('initial-data');
>> var initialData = JSON.parse(el.textContent || el.innerText);
>>
>>
>> This is compatible with a strict Content Security Policy which prohibits
>> all in-page script execution and maintains a clean separation between
>> passive data and executable code.
>>
>> The second mode of operation would look like this:
>>
>> {% json data var="initialData" %}
>>
>>
>> and would produce HTML like this:
>>
>>
>> 
>>   var initialData = {"foo": "bar"};
>> 
>>
>>
>> This isn't compatible with strict CSP but it is perhaps simpler and more
>> familiar to many developers, and not fundamentally insecure, so it should
>> still be supported.
>>
>> Of course, the key issue is whether this can be done securely. In the
>> gist below is a proposed implementation with links to the sources I've used
>> to ensure I'm escaping things correctly:
>> https://gist.github.com/evansd/41ea9dfc90d87f6afde1
>>
>> If people are happy with it then I can create a proper pull request with
>> docs etc.
>>
>> Thanks,
>>
>> Dave
>>
>  --
> You received this message because you are subscribed to a topic in the
> Google Groups "Django developers" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/django-developers/RNMs5YbKeRY/unsubscribe
> .
> To unsubscribe from this group and all its topics, send an email to
> django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/1d3f399a-dbc3-4f2f-b9e3-01c37

Re: Proposal to add a secure JSON encoding template tag

[David Evans]

Thanks. This is a good example of the problem actually as that library is 
vulnerable to XSS. If someone can get the string "]]>" into the 
JSON they break out of the CDATA block and the script element and can then 
execute arbitrary code.

On Wednesday, 14 May 2014 14:57:06 UTC+1, Michael Mior wrote:
>
> I've used django-jsonify (https://pypi.python.org/pypi/django-jsonify/) 
> in the the past for this successfully. I'm not certain of the security of 
> the code since unfortunately I didn't have the time to do a proper audit, 
> but it seemed to handle some common cases.
>
> --
> Michael Mior
>
> On Tuesday, May 13, 2014 6:03:43 AM UTC-4, David Evans wrote:
>>
>> There was some discussion previously (see 
>> https://code.djangoproject.com/ticket/17419) of adding a JSON encoding 
>> filter to Django. This was rejected as being impossible (or very difficult) 
>> to do securely. However the requirement to embed JSON in an HTML page is 
>> quite a common one, and it's easy to get wrong and create XSS 
>> vulnerabilities. We should make it easy for people to do the right thing.
>>
>> I propose a ``json`` tag (implementation 
>> here<https://gist.github.com/evansd/41ea9dfc90d87f6afde1>) 
>> which outputs the entire script element as well as the JSON data. By 
>> enforcing the context in which in the JSON is output, it's possible to 
>> escape it securely.
>>
>> It would have two basic modes of operation. The first, and recommended, 
>> one would look like this:
>>
>> {% json data id="initial-data" %}
>>
>>
>> and would produce HTML like this:
>>
>> 
>>   {"foo": "bar"}
>> 
>>
>>
>> The resulting data would be accessed in JavaScript like this:
>>
>> var el = document.getElementById('initial-data');
>> var initialData = JSON.parse(el.textContent || el.innerText);
>>
>>
>> This is compatible with a strict Content Security Policy which prohibits 
>> all in-page script execution and maintains a clean separation between 
>> passive data and executable code.
>>
>> The second mode of operation would look like this:
>>
>> {% json data var="initialData" %}
>>
>>
>> and would produce HTML like this:
>>
>>
>> 
>>   var initialData = {"foo": "bar"};
>> 
>>
>>
>> This isn't compatible with strict CSP but it is perhaps simpler and more 
>> familiar to many developers, and not fundamentally insecure, so it should 
>> still be supported.
>>
>> Of course, the key issue is whether this can be done securely. In the 
>> gist below is a proposed implementation with links to the sources I've used 
>> to ensure I'm escaping things correctly: 
>> https://gist.github.com/evansd/41ea9dfc90d87f6afde1
>>
>> If people are happy with it then I can create a proper pull request with 
>> docs etc.
>>
>> Thanks,
>>
>> Dave
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/8dfe3cf8-f55b-4b51-bb04-a362fc4876a1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Proposal to add a secure JSON encoding template tag

[David Evans]

There was some discussion previously (see 
https://code.djangoproject.com/ticket/17419) of adding a JSON encoding 
filter to Django. This was rejected as being impossible (or very difficult) 
to do securely. However the requirement to embed JSON in an HTML page is 
quite a common one, and it's easy to get wrong and create XSS 
vulnerabilities. We should make it easy for people to do the right thing.

I propose a ``json`` tag (implementation 
here) 
which outputs the entire script element as well as the JSON data. By 
enforcing the context in which in the JSON is output, it's possible to 
escape it securely.

It would have two basic modes of operation. The first, and recommended, one 
would look like this:

{% json data id="initial-data" %}


and would produce HTML like this:


  {"foo": "bar"}



The resulting data would be accessed in JavaScript like this:

var el = document.getElementById('initial-data');
var initialData = JSON.parse(el.textContent || el.innerText);


This is compatible with a strict Content Security Policy which prohibits 
all in-page script execution and maintains a clean separation between 
passive data and executable code.

The second mode of operation would look like this:

{% json data var="initialData" %}


and would produce HTML like this:



  var initialData = {"foo": "bar"};



This isn't compatible with strict CSP but it is perhaps simpler and more 
familiar to many developers, and not fundamentally insecure, so it should 
still be supported.

Of course, the key issue is whether this can be done securely. In the gist 
below is a proposed implementation with links to the sources I've used to 
ensure I'm escaping things correctly: 
https://gist.github.com/evansd/41ea9dfc90d87f6afde1

If people are happy with it then I can create a proper pull request with 
docs etc.

Thanks,

Dave

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/187d31c1-6b47-4fc5-adc0-1d93bf4d53fd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.