Re: delegating our static file serving

[David Evans]

Hi Aleksej,

Thanks for your work on this, and sorry I haven't replied sooner -- just 
been busy with lots of other things ;)

My plan for integrating WhiteNoise was actually to merge it in as part of 
contrib.staticfiles and basically stop development on it as an independent 
project. When I first started WhiteNoise I went to quite a lot of effort to 
make the core framework-agnostic and to have all the Django specific code 
as an extension to the core. However this makes the code more complex than 
if it was written for Django alone. I'm also not sure it gets a lot of use 
outside of the Django world as other frameworks have their own way of doing 
things and the most useful features of WhiteNoise are Django only.

So the end-goal in my mind is to have a stripped down and simplified 
version of the WhiteNoise code bundled as part of Django itself, which I 
could then help maintain as part of the wider Django project. However, I 
don't think this process would be an ideal project for someone who is a new 
contributor to Django as it would require quite a detailed understanding of 
the design decisions and architectural tradeoffs behind WhiteNoise. Really 
this is the sort of thing I ought to be doing but it's been quite a busy 
year (new baby, work, etc etc) and I just haven't had the time.

If you're able to bundle WhiteNoise as it currently is with Django in a way 
that the core devs are happy with then that's great, and obviously I 
wouldn't stand in your way. But I can imagine that there might be some 
resistance to creating a hard dependency on WhiteNoise, while it's still a 
standalone project.

Again, thanks for the work you've put into this and sorry I don't have a 
more helpful response.

Dave

On Wednesday, 22 March 2017 16:14:58 UTC, Aleksej Manaev wrote:
>
> Am I on the right track with the integration of WhiteNoise into Django? 
> Can Someone lead me to the next steps that need to be done?
>
>
> https://github.com/django/django/compare/master...AleksejManaev:ticket_27325?expand=1
>
> I think "django/views/static.py" can't be deprecated because it is used 
> for providing MEDIA_FILES during development and WhiteNoise is only 
> responsible for STATIC_FILES.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/cfbf04a4-9346-4fd7-95bf-e3802f7a8e5d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: delegating our static file serving

[Florian Apolloner]

I think there is no need to vendor nowadays, we can just depend on 
whitenoise. (like we do for pytz)

On Wednesday, March 22, 2017 at 5:14:58 PM UTC+1, Aleksej Manaev wrote:
>
> Am I on the right track with the integration of WhiteNoise into Django? 
> Can Someone lead me to the next steps that need to be done?
>
>
> https://github.com/django/django/compare/master...AleksejManaev:ticket_27325?expand=1
>
> I think "django/views/static.py" can't be deprecated because it is used 
> for providing MEDIA_FILES during development and WhiteNoise is only 
> responsible for STATIC_FILES.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d37a3380-e706-40c5-b32d-adc3291f67e0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: delegating our static file serving

[Aleksej Manaev]

Am I on the right track with the integration of WhiteNoise into Django? Can 
Someone lead me to the next steps that need to be done?

https://github.com/django/django/compare/master...AleksejManaev:ticket_27325?expand=1

I think "django/views/static.py" can't be deprecated because it is used for 
providing MEDIA_FILES during development and WhiteNoise is only responsible 
for STATIC_FILES.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/61e59680-a1b9-418c-add2-4a7a79d70bd6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: delegating our static file serving

[Aleksej Manaev]

I started looking into this but have progressed slowely. At first I put the 
whitenoise project into the contrib folder of django. 
Tim Graham suggested not to change much of the integration with the old 
WSGI interface, so I think the new middleware way would be alright too. 

On Tuesday, 29 December 2015 00:36:06 UTC, Tim Graham wrote:

> I'd like to work together with Dave to develop a proof of concept that 
> integrates whitenoise into Django. I spent about an hour looking through 
> whitenoise and our own static file serving code, and I think integrating 
> whitenoise will yield a simpler user experience with about the same amount 
> of code as have now.
>
> Essentially, we'd recommend adding something like this to existing wsgi.py 
> files (it would be in the default startproject template)
>
> from whitenoise.django import DjangoWhiteNoise
> application = DjangoWhiteNoise(application)
> application.add_files(
> settings.MEDIA_ROOT, prefix=settings.MEDIA_URL)
>
> which would have the benefit of working out of the box in production too, 
> I think. Of course, you could disable that based on settings.DEBUG or some 
> other toggle.
>
> We could then deprecate:
>
> * django/contrib/staticfiles/views.py
> * django/contrib/staticfiles/management/commands/runserver.py
> * django/contrib/staticfiles/handlers.py
> * django/views/static.py
>
> Any objections to doing further investigation in this area?
>


As I tried to remove the suggested deprecated files I encoutered the 
dependency from StaticLiveServerTestCase to StaticFilesHandler.
StaticLiveServerTestCase allows the execution of tests without using 
collectstatic first which I think is not possible in Whitenoise. Do we add 
this functionality in Whitenoise or force the user to use collectstatic? 
Whitenoise depends on the staticfileapp and maybe a merge of the two 
components would be cleaner.
I wanted to do this as a project in my university and would like to offer 
my help.




-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/87756f6f-2629-444c-9cdf-1b3ae76241ad%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: delegating our static file serving

[Guilherme Leal]

If you need some help, I'm as available as I can be.

Em ter, 20 de dez de 2016 19:50, David Evans  escreveu:

> I've been intending to work on this for a while (as the original author of
> WhiteNoise it would make the most sense for me to do the integration work)
> but as ever the problem has been finding the time. I've been hoping to make
> some progress on this over the Christmas period, but we'll see what happens!
>
>
> On Tuesday, 20 December 2016 21:42:55 UTC, Guilherme Leal wrote:
>
> I'm willing to cooperate on this one. Anyone is working on this and need
> help?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/9a6e143d-9bf9-4e26-89ca-cfc9944ba794%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAOs3Lp7qGjXiw7Ew7BPb%3D1h97gmOUMzorPJiEyjhXh2-uRTPCA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: delegating our static file serving

[David Evans]

I've been intending to work on this for a while (as the original author of 
WhiteNoise it would make the most sense for me to do the integration work) 
but as ever the problem has been finding the time. I've been hoping to make 
some progress on this over the Christmas period, but we'll see what happens!

On Tuesday, 20 December 2016 21:42:55 UTC, Guilherme Leal wrote:
>
> I'm willing to cooperate on this one. Anyone is working on this and need 
> help?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/9a6e143d-9bf9-4e26-89ca-cfc9944ba794%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: delegating our static file serving

[Guilherme Leal]

I'm willing to cooperate on this one. Anyone is working on this and need help?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/fc0f3cd5-dbd6-4e04-844e-8f850be060a9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: delegating our static file serving

[Tim Graham]

Hi, I don't know of any updates on this issue outside of this thread. I 
think you could read through the thread and summarize the problems and 
possible solutions just as well as I could. The solution might involve 
integrating whitenoise in Django, as well an adding some enhancements. The 
whitenoise author, David Evans, might have some more thoughts about 
specifics.

On Monday, October 3, 2016 at 1:08:02 PM UTC-4, Aleksej Manaev wrote:
>
> Hello, I would like to work on this in the scope of my study project. Is 
> someone already working on this?
> I am fairly new to contributing to the framework, can you provide me more 
> information on the current state of the issue, which problems need to be 
> solved and which possible solutions there are available? Are there some 
> resources I can read to educate myself in this topic?
> I have already read the comments in this discussion and Django 
> documentation 
>
> https://docs.djangoproject.com/en/1.10/howto/static-files/ 
> andhttps://docs.djangoproject.com/en/1.10/howto/static-files/deployment/.
>
>
> Am Freitag, 28. November 2014 15:15:03 UTC+1 schrieb Tim Graham:
>>
>> Berker has worked on integrating gunicorn with runserver 
>>  so that we might be able to 
>> deprecate our own homegrown webserver. Windows support for gunicorn is 
>> supposedly coming soon which 
>> may actually make the idea feasible. This way we provide a more secure 
>> solution out of the box (anecdotes indicate that runserver is widely used 
>> in production despite our documentation warnings against doing so).
>>
>> On the pull request, Anssi had an idea to use dj-static 
>>  to serve static/media files. 
>> My understanding is that we would basically integrate the code for 
>> dj-static into Django and then add a dependency on static 
>> . It could be an optional 
>> dependency since you might want to serve static files differently in 
>> production, but it would likely be more or less universally used in 
>> development. We could then say that django.views.static.serve (and its 
>> counterpart in staticfiles) is okay to use in production (and I guess 
>> people are already using them in production despite our warnings that they 
>> are not hardened for production use).
>>
>> What do you think of this plan?
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/8cd02d9d-960a-42fc-a711-b2baec279cad%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: delegating our static file serving

[Aleksej Manaev]

Hello, I would like to work on this in the scope of my study project. Is 
someone already working on this?
I am fairly new to contributing to the framework, can you provide me more 
information on the current state of the issue, which problems need to be 
solved and which possible solutions there are available? Are there some 
resources I can read to educate myself in this topic?
I have already read the comments in this discussion and Django 
documentation 

https://docs.djangoproject.com/en/1.10/howto/static-files/ 
andhttps://docs.djangoproject.com/en/1.10/howto/static-files/deployment/.


Am Freitag, 28. November 2014 15:15:03 UTC+1 schrieb Tim Graham:
>
> Berker has worked on integrating gunicorn with runserver 
>  so that we might be able to 
> deprecate our own homegrown webserver. Windows support for gunicorn is 
> supposedly coming soon which 
> may actually make the idea feasible. This way we provide a more secure 
> solution out of the box (anecdotes indicate that runserver is widely used 
> in production despite our documentation warnings against doing so).
>
> On the pull request, Anssi had an idea to use dj-static 
>  to serve static/media files. 
> My understanding is that we would basically integrate the code for 
> dj-static into Django and then add a dependency on static 
> . It could be an optional dependency 
> since you might want to serve static files differently in production, but 
> it would likely be more or less universally used in development. We could 
> then say that django.views.static.serve (and its counterpart in 
> staticfiles) is okay to use in production (and I guess people are already 
> using them in production despite our warnings that they are not hardened 
> for production use).
>
> What do you think of this plan?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/342138dd-953d-4aa4-8c7d-f4627591005a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: delegating our static file serving

[Thomas C]

Hello Curtis and Fabio, 

You can have a look at django- private-media, which implement a private 
storage engine :
https://github.com/RacingTadpole/django-private-media

Basic usage :

> from private_media.storages import PrivateMediaStorage 

class Car(models.Model):
> photo = models.ImageField(storage=PrivateMediaStorage())


Regards,


Thomas.


On Friday, January 1, 2016 at 1:30:05 PM UTC+4, Curtis Maloney wrote:
>
> On 01/01/16 15:49, Fabio Caritas Barrionuevo da Luz wrote: 
> > A question: are there any plans to also improve MEDIA files (user 
> > uploaded files) in any foreseeable future? 
> > 
> > Perhaps this is outside the scope of Django, but I believe Django could 
> > provide by default, any option to get a little more control over who can 
> > and can not access the MEDIA files. 
>
> I certainly agree that at least some basic form of access control ought 
> be provided in core/contrib. 
>
> > Most Django deployment tutorials with Nginx and Apache that saw out 
> > there do not say anything on the issue of data security. 
> > 
> > I believe it is a common use case, that not every file sent by a User 
> > should be available and accessible to anyone on the web. 
>
> I was talking with Tom Eastman about a related topic at PyConAU this 
> year, and I believe the solution is multiple storage engines. 
>
> Basically, for each realm of access, have a separate file storage 
> instance with its own access policy. 
>
> We could then register each storage engine we want published with the 
> static/media server machinery, with its associated policy... 
>
> Hmm... I think I've just created myself a new project... 
>
> -- 
> Curtis 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ec589ee2-0a0f-4566-9161-7ca1474104fc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: delegating our static file serving

[Curtis Maloney]

I should probably finish off my toy working on the same lines... but 
introducing my earlier comments about per-source paths and permissions 
mixins...


Very much a WIP, but I'm sure you'll see the idea:

https://gist.github.com/funkybob/a89c10e24716a4ce55cf

--
C

On 18/01/16 09:58, Gordon wrote:

I put together a simple django middleware based off the functionality of
whitenoise except that it uses the staticfiles storage for
finding/serving files.  I am sure there are lots of improvements to be
made on it but it demonstrates using the filestorage api to accommodate
remote static files.  It is roughly 100 lines.
https://gist.github.com/thenewguy/a92e05a3568fb6f1cc7c

On Saturday, January 16, 2016 at 6:11:46 PM UTC-5, Gordon wrote:

Apologies if this has already been mentioned but I read this thread
in full awhile ago and now everything is collapsed.

If staticfiles serving is brought into django proper it would be
nice if the storage api was used so that staticfiles can be stored
remotely.  I just ran into this looking at whitenoise so I figured I
would mention it here to be taken into consideration when putting an
implementation together.

On Friday, January 8, 2016 at 12:11:12 PM UTC-5, David Evans wrote:

I've just pushed a simple middleware wrapper that lets you
install WhiteNoise by adding:

MIDDLEWARE_CLASSES = (
...
 'whitenoise.middleware.StaticFilesMiddleware',
...
)


See:

https://github.com/evansd/whitenoise/blob/master/whitenoise/middleware.py



No messing with wsgi.py!

This approach will be marginally less efficient than handling
things at the WSGI layer, but the integration is much cleaner.

On Tuesday, 29 December 2015 17:31:02 UTC, David Evans wrote:

I'd be very happy to help with this. Two things to mention:

1. When I first wrote WhiteNoise the FileResponse class
didn't exist and so the only way I could access
`wsgi.file_wrapper` was by wrapping the wsgi application
directly and bypassing Django. Now we have the FileResponse
class it would be possible to implement WhiteNoise as
standard Django middleware without having to edit wsgi.py. I
think this would be a much cleaner approach, and I don't
think it will be too much work to implement. I should have
time to look at this next week.

2. Serving media files is a slightly different case.
WhiteNoise was designed around the assumption that it's
serving a set of developer-supplied, public files which
remain unchanged during the lifetime of the process. This
simplifies a lot of performance and security concerns that
come with serving mutable, user-supplied files. At present,
if you use `add_files(settings.MEDIA_ROOT,
prefix=settings.MEDIA_URL)` as you suggested then WhiteNoise
won't pick up any files that were uploaded after the
application was started -- at least, not unless you enable
the "autorefresh" setting which I explicitly don't recommend
for production.

Obviously it's possible to support this if we decide this is
an important goal but it will need more thought and will
definitely be more work than just supporting static files.

On Tuesday, 29 December 2015 00:36:06 UTC, Tim Graham wrote:

I'd like to work together with Dave to develop a proof
of concept that integrates whitenoise into Django. I
spent about an hour looking through whitenoise and our
own static file serving code, and I think integrating
whitenoise will yield a simpler user experience with
about the same amount of code as have now.

Essentially, we'd recommend adding something like this
to existing wsgi.py files (it would be in the default
startproject template)

from whitenoise.django import DjangoWhiteNoise
application = DjangoWhiteNoise(application)
application.add_files(settings.MEDIA_ROOT,
prefix=settings.MEDIA_URL)

which would have the benefit of working out of the box
in production too, I think. Of course, you could disable
that based on settings.DEBUG or some other toggle.

We could then deprecate:

* django/contrib/staticfiles/views.py
*
django/contrib/staticfiles/management/commands/runserver.py
* django/contrib/staticfiles/handlers.py
* django/views/stati

Re: delegating our static file serving

[Gordon]

I put together a simple django middleware based off the functionality of 
whitenoise except that it uses the staticfiles storage for finding/serving 
files.  I am sure there are lots of improvements to be made on it but it 
demonstrates using the filestorage api to accommodate remote static files.  
It is roughly 100 lines.  
https://gist.github.com/thenewguy/a92e05a3568fb6f1cc7c

On Saturday, January 16, 2016 at 6:11:46 PM UTC-5, Gordon wrote:
>
> Apologies if this has already been mentioned but I read this thread in 
> full awhile ago and now everything is collapsed.
>
> If staticfiles serving is brought into django proper it would be nice if 
> the storage api was used so that staticfiles can be stored remotely.  I 
> just ran into this looking at whitenoise so I figured I would mention it 
> here to be taken into consideration when putting an implementation together.
>
> On Friday, January 8, 2016 at 12:11:12 PM UTC-5, David Evans wrote:
>>
>> I've just pushed a simple middleware wrapper that lets you install 
>> WhiteNoise by adding:
>>
>> MIDDLEWARE_CLASSES = (
>>...
>> 'whitenoise.middleware.StaticFilesMiddleware',
>>...
>> )
>>
>>
>> See: 
>> https://github.com/evansd/whitenoise/blob/master/whitenoise/middleware.py
>>
>> No messing with wsgi.py!
>>
>> This approach will be marginally less efficient than handling things at 
>> the WSGI layer, but the integration is much cleaner.
>>
>> On Tuesday, 29 December 2015 17:31:02 UTC, David Evans wrote:
>>>
>>> I'd be very happy to help with this. Two things to mention:
>>>
>>> 1. When I first wrote WhiteNoise the FileResponse class didn't exist and 
>>> so the only way I could access `wsgi.file_wrapper` was by wrapping the wsgi 
>>> application directly and bypassing Django. Now we have the FileResponse 
>>> class it would be possible to implement WhiteNoise as standard Django 
>>> middleware without having to edit wsgi.py. I think this would be a much 
>>> cleaner approach, and I don't think it will be too much work to implement. 
>>> I should have time to look at this next week.
>>>
>>> 2. Serving media files is a slightly different case. WhiteNoise was 
>>> designed around the assumption that it's serving a set of 
>>> developer-supplied, public files which remain unchanged during the lifetime 
>>> of the process. This simplifies a lot of performance and security concerns 
>>> that come with serving mutable, user-supplied files. At present, if you use 
>>> `add_files(settings.MEDIA_ROOT, prefix=settings.MEDIA_URL)` as you 
>>> suggested then WhiteNoise won't pick up any files that were uploaded after 
>>> the application was started -- at least, not unless you enable the 
>>> "autorefresh" setting which I explicitly don't recommend for production.
>>>
>>> Obviously it's possible to support this if we decide this is an 
>>> important goal but it will need more thought and will definitely be more 
>>> work than just supporting static files.
>>>
>>> On Tuesday, 29 December 2015 00:36:06 UTC, Tim Graham wrote:

 I'd like to work together with Dave to develop a proof of concept that 
 integrates whitenoise into Django. I spent about an hour looking through 
 whitenoise and our own static file serving code, and I think integrating 
 whitenoise will yield a simpler user experience with about the same amount 
 of code as have now.

 Essentially, we'd recommend adding something like this to existing 
 wsgi.py files (it would be in the default startproject template)

 from whitenoise.django import DjangoWhiteNoise
 application = DjangoWhiteNoise(application)
 application.add_files(settings.MEDIA_ROOT, prefix=settings.MEDIA_URL)

 which would have the benefit of working out of the box in production 
 too, I think. Of course, you could disable that based on settings.DEBUG or 
 some other toggle.

 We could then deprecate:

 * django/contrib/staticfiles/views.py
 * django/contrib/staticfiles/management/commands/runserver.py
 * django/contrib/staticfiles/handlers.py
 * django/views/static.py

 Any objections to doing further investigation in this area?

>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/c2b85443-4d50-425d-8b33-a05a048d3957%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: delegating our static file serving

[Gordon]

Apologies if this has already been mentioned but I read this thread in full 
awhile ago and now everything is collapsed.

If staticfiles serving is brought into django proper it would be nice if 
the storage api was used so that staticfiles can be stored remotely.  I 
just ran into this looking at whitenoise so I figured I would mention it 
here to be taken into consideration when putting an implementation together.

On Friday, January 8, 2016 at 12:11:12 PM UTC-5, David Evans wrote:
>
> I've just pushed a simple middleware wrapper that lets you install 
> WhiteNoise by adding:
>
> MIDDLEWARE_CLASSES = (
>...
> 'whitenoise.middleware.StaticFilesMiddleware',
>...
> )
>
>
> See: 
> https://github.com/evansd/whitenoise/blob/master/whitenoise/middleware.py
>
> No messing with wsgi.py!
>
> This approach will be marginally less efficient than handling things at 
> the WSGI layer, but the integration is much cleaner.
>
> On Tuesday, 29 December 2015 17:31:02 UTC, David Evans wrote:
>>
>> I'd be very happy to help with this. Two things to mention:
>>
>> 1. When I first wrote WhiteNoise the FileResponse class didn't exist and 
>> so the only way I could access `wsgi.file_wrapper` was by wrapping the wsgi 
>> application directly and bypassing Django. Now we have the FileResponse 
>> class it would be possible to implement WhiteNoise as standard Django 
>> middleware without having to edit wsgi.py. I think this would be a much 
>> cleaner approach, and I don't think it will be too much work to implement. 
>> I should have time to look at this next week.
>>
>> 2. Serving media files is a slightly different case. WhiteNoise was 
>> designed around the assumption that it's serving a set of 
>> developer-supplied, public files which remain unchanged during the lifetime 
>> of the process. This simplifies a lot of performance and security concerns 
>> that come with serving mutable, user-supplied files. At present, if you use 
>> `add_files(settings.MEDIA_ROOT, prefix=settings.MEDIA_URL)` as you 
>> suggested then WhiteNoise won't pick up any files that were uploaded after 
>> the application was started -- at least, not unless you enable the 
>> "autorefresh" setting which I explicitly don't recommend for production.
>>
>> Obviously it's possible to support this if we decide this is an important 
>> goal but it will need more thought and will definitely be more work than 
>> just supporting static files.
>>
>> On Tuesday, 29 December 2015 00:36:06 UTC, Tim Graham wrote:
>>>
>>> I'd like to work together with Dave to develop a proof of concept that 
>>> integrates whitenoise into Django. I spent about an hour looking through 
>>> whitenoise and our own static file serving code, and I think integrating 
>>> whitenoise will yield a simpler user experience with about the same amount 
>>> of code as have now.
>>>
>>> Essentially, we'd recommend adding something like this to existing 
>>> wsgi.py files (it would be in the default startproject template)
>>>
>>> from whitenoise.django import DjangoWhiteNoise
>>> application = DjangoWhiteNoise(application)
>>> application.add_files(settings.MEDIA_ROOT, prefix=settings.MEDIA_URL)
>>>
>>> which would have the benefit of working out of the box in production 
>>> too, I think. Of course, you could disable that based on settings.DEBUG or 
>>> some other toggle.
>>>
>>> We could then deprecate:
>>>
>>> * django/contrib/staticfiles/views.py
>>> * django/contrib/staticfiles/management/commands/runserver.py
>>> * django/contrib/staticfiles/handlers.py
>>> * django/views/static.py
>>>
>>> Any objections to doing further investigation in this area?
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d1348727-da81-4d34-ac18-8ae6ca339fbb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: delegating our static file serving

[David Evans]

I've just pushed a simple middleware wrapper that lets you install 
WhiteNoise by adding:

MIDDLEWARE_CLASSES = (
   ...
'whitenoise.middleware.StaticFilesMiddleware',
   ...
)


See: https://github.com/evansd/whitenoise/blob/master/whitenoise/middleware.py

No messing with wsgi.py!

This approach will be marginally less efficient than handling things at the 
WSGI layer, but the integration is much cleaner.

On Tuesday, 29 December 2015 17:31:02 UTC, David Evans wrote:
>
> I'd be very happy to help with this. Two things to mention:
>
> 1. When I first wrote WhiteNoise the FileResponse class didn't exist and 
> so the only way I could access `wsgi.file_wrapper` was by wrapping the wsgi 
> application directly and bypassing Django. Now we have the FileResponse 
> class it would be possible to implement WhiteNoise as standard Django 
> middleware without having to edit wsgi.py. I think this would be a much 
> cleaner approach, and I don't think it will be too much work to implement. 
> I should have time to look at this next week.
>
> 2. Serving media files is a slightly different case. WhiteNoise was 
> designed around the assumption that it's serving a set of 
> developer-supplied, public files which remain unchanged during the lifetime 
> of the process. This simplifies a lot of performance and security concerns 
> that come with serving mutable, user-supplied files. At present, if you use 
> `add_files(settings.MEDIA_ROOT, prefix=settings.MEDIA_URL)` as you 
> suggested then WhiteNoise won't pick up any files that were uploaded after 
> the application was started -- at least, not unless you enable the 
> "autorefresh" setting which I explicitly don't recommend for production.
>
> Obviously it's possible to support this if we decide this is an important 
> goal but it will need more thought and will definitely be more work than 
> just supporting static files.
>
> On Tuesday, 29 December 2015 00:36:06 UTC, Tim Graham wrote:
>>
>> I'd like to work together with Dave to develop a proof of concept that 
>> integrates whitenoise into Django. I spent about an hour looking through 
>> whitenoise and our own static file serving code, and I think integrating 
>> whitenoise will yield a simpler user experience with about the same amount 
>> of code as have now.
>>
>> Essentially, we'd recommend adding something like this to existing 
>> wsgi.py files (it would be in the default startproject template)
>>
>> from whitenoise.django import DjangoWhiteNoise
>> application = DjangoWhiteNoise(application)
>> application.add_files(settings.MEDIA_ROOT, prefix=settings.MEDIA_URL)
>>
>> which would have the benefit of working out of the box in production too, 
>> I think. Of course, you could disable that based on settings.DEBUG or some 
>> other toggle.
>>
>> We could then deprecate:
>>
>> * django/contrib/staticfiles/views.py
>> * django/contrib/staticfiles/management/commands/runserver.py
>> * django/contrib/staticfiles/handlers.py
>> * django/views/static.py
>>
>> Any objections to doing further investigation in this area?
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/e24d4590-e8e7-437d-a441-b1ba51e1f535%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: delegating our static file serving

[Aymeric Augustin]

Hello Collin,

> On 31 déc. 2015, at 23:44, Collin Anderson  wrote:
> 
> I feel like the insecure flag could get renamed to inefficient or something 
> like that.

This view is insecure for serving user-uploaded files, for reasons Florian 
hinted at.

Browsers “helpfully” get a lot things wrong for legacy reasons. As a 
consequence,
downloading a file is one of the most insecure things you can do in general.

I can’t say for sure if it’s insecure for serving static files. I believe 
Django has some
countermeasures against directory traversal attacks for example. However, since
the code wasn’t written with security in mind from the beginning, rewriting it 
from a
security perspective may be the most efficient way to make sure it’s secure.

Best regards,

-- 
Aymeric.




-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/745A1E1A-AE02-410C-B2B1-705BC08378CC%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.

Re: delegating our static file serving

[Curtis Maloney]

On 01/01/16 15:49, Fabio Caritas Barrionuevo da Luz wrote:

A question: are there any plans to also improve MEDIA files (user
uploaded files) in any foreseeable future?

Perhaps this is outside the scope of Django, but I believe Django could
provide by default, any option to get a little more control over who can
and can not access the MEDIA files.


I certainly agree that at least some basic form of access control ought 
be provided in core/contrib.



Most Django deployment tutorials with Nginx and Apache that saw out
there do not say anything on the issue of data security.

I believe it is a common use case, that not every file sent by a User
should be available and accessible to anyone on the web.


I was talking with Tom Eastman about a related topic at PyConAU this 
year, and I believe the solution is multiple storage engines.


Basically, for each realm of access, have a separate file storage 
instance with its own access policy.


We could then register each storage engine we want published with the 
static/media server machinery, with its associated policy...


Hmm... I think I've just created myself a new project...

--
Curtis

--
You received this message because you are subscribed to the Google Groups "Django 
developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/56864707.5030007%40tinbrain.net.
For more options, visit https://groups.google.com/d/optout.

Re: delegating our static file serving

[Fabio Caritas Barrionuevo da Luz]

A question: are there any plans to also improve MEDIA files (user uploaded 
files) in any foreseeable future?

Perhaps this is outside the scope of Django, but I believe Django could 
provide by default, any option to get a little more control over who can 
and can not access the MEDIA files.

Most Django deployment tutorials with Nginx and Apache that saw out there 
do not say anything on the issue of data security.

I believe it is a common use case, that not every file sent by a User 
should be available and accessible to anyone on the web.

Out there, I found these two implementations:

https://github.com/benoitbryon/django-downloadview

https://github.com/johnsensible/django-sendfile


-- 
Fábio C. Barrionuevo da Luz
Palmas - Tocantins - Brasil - América do Sul

http://pythonclub.com.br/



Em quinta-feira, 31 de dezembro de 2015 19:44:01 UTC-3, Collin Anderson 
escreveu:
>
> I'm still +1 to django providing a good file solution, whether it be 
> whitenoise, dj-static, or whatever.
>
> Here's all I came up with the last time I looked into this. It basically 
> involves passing insecure=True into our current code and sending cache 
> headers. (I feel like the insecure flag could get renamed to inefficient or 
> something like that.) I feel like the code we have really isn't all that 
> bad. Using something like whitenoise or dj-static allows you to bypass 
> middleware, which can really improve performance (if you have inefficient 
> middleware installed), but on the other hand, some people may want to use 
> the currently logged in user to determine whether or not to serve the file.
>
> Collin
>
> import re
>
> from django.conf import settings
> from django.conf.urls import url
> from django.contrib.staticfiles.views import serve as static_serve
> from django.core.exceptions import ImproperlyConfigured
> from django.views.decorators.cache import cache_control
> from django.views.static import serve
>
>
> def static(prefix, view=serve, **kwargs):
> if not prefix:
> raise ImproperlyConfigured("Empty static prefix not permitted")
> if '://' in prefix:
> return []
> if not settings.DEBUG:
> view = cache_control(max_age=86400 * 30)(view)
> return [
> url(r'^%s(?P.*)$' % re.escape(prefix.lstrip('/')), view, 
> kwargs=kwargs),
> ]
>
> urlpatterns = static(settings.STATIC_URL, view=static_serve, insecure=True
> )
>
> if settings.MEDIA_URL and settings.MEDIA_ROOT:
> urlpatterns += static(settings.MEDIA_URL, document_root=settings.
> MEDIA_ROOT)
>
>
> On Thursday, December 31, 2015 at 7:42:54 AM UTC-5, Aymeric Augustin wrote:
>>
>> 2015-12-31 12:52 GMT+01:00 Anssi Kääriäinen :
>>
>>> In my opinion one of the most important goals for Django is to make
>>> deploying tiny projects securely and easily a trivial matter. We
>>> aren't there right now.
>>>
>>
>> +1
>>
>> I'm maintaining three such sites (my wife's, my consultancy's and mine). 
>> I'm
>> clearly not going to bother configuring a CDN. I deployed them with 
>> whitenoise
>> — which isn't nearly as well known as it should be. If I hadn't known 
>> about
>> it, I suspect I would have rolled a poor man's solution to serve static 
>> files
>> in Python, because that's the easiest when deploying to a PaaS.
>>
>> Regarding the question of media files raised earlier in this thread, I 
>> think
>> we shouldn't attempt to cache all existing files on start-up because even
>> small sites could easily have tens of thousands e.g. a forum that stores
>> users' avatars. We should look up the file at each query, which will be
>> slightly less efficient, but functional and not subject to scalability 
>> issues.
>>
>> For both static and media files, we care about simplicity and security. It
>> appears that whitenoise gives us great performance for static files as 
>> well,
>> which is good. If we can't have it for media files, it's still fine.
>>
>> -- 
>> Aymeric.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/6b3b995c-ca28-4a61-a8b6-023314ca8b70%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: delegating our static file serving

[Collin Anderson]

I'm still +1 to django providing a good file solution, whether it be 
whitenoise, dj-static, or whatever.

Here's all I came up with the last time I looked into this. It basically 
involves passing insecure=True into our current code and sending cache 
headers. (I feel like the insecure flag could get renamed to inefficient or 
something like that.) I feel like the code we have really isn't all that 
bad. Using something like whitenoise or dj-static allows you to bypass 
middleware, which can really improve performance (if you have inefficient 
middleware installed), but on the other hand, some people may want to use 
the currently logged in user to determine whether or not to serve the file.

Collin

import re

from django.conf import settings
from django.conf.urls import url
from django.contrib.staticfiles.views import serve as static_serve
from django.core.exceptions import ImproperlyConfigured
from django.views.decorators.cache import cache_control
from django.views.static import serve


def static(prefix, view=serve, **kwargs):
if not prefix:
raise ImproperlyConfigured("Empty static prefix not permitted")
if '://' in prefix:
return []
if not settings.DEBUG:
view = cache_control(max_age=86400 * 30)(view)
return [
url(r'^%s(?P.*)$' % re.escape(prefix.lstrip('/')), view, 
kwargs=kwargs),
]

urlpatterns = static(settings.STATIC_URL, view=static_serve, insecure=True)

if settings.MEDIA_URL and settings.MEDIA_ROOT:
urlpatterns += static(settings.MEDIA_URL, document_root=settings.
MEDIA_ROOT)


On Thursday, December 31, 2015 at 7:42:54 AM UTC-5, Aymeric Augustin wrote:
>
> 2015-12-31 12:52 GMT+01:00 Anssi Kääriäinen  >:
>
>> In my opinion one of the most important goals for Django is to make
>> deploying tiny projects securely and easily a trivial matter. We
>> aren't there right now.
>>
>
> +1
>
> I'm maintaining three such sites (my wife's, my consultancy's and mine). 
> I'm
> clearly not going to bother configuring a CDN. I deployed them with 
> whitenoise
> — which isn't nearly as well known as it should be. If I hadn't known about
> it, I suspect I would have rolled a poor man's solution to serve static 
> files
> in Python, because that's the easiest when deploying to a PaaS.
>
> Regarding the question of media files raised earlier in this thread, I 
> think
> we shouldn't attempt to cache all existing files on start-up because even
> small sites could easily have tens of thousands e.g. a forum that stores
> users' avatars. We should look up the file at each query, which will be
> slightly less efficient, but functional and not subject to scalability 
> issues.
>
> For both static and media files, we care about simplicity and security. It
> appears that whitenoise gives us great performance for static files as 
> well,
> which is good. If we can't have it for media files, it's still fine.
>
> -- 
> Aymeric.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/9fd6dee0-3293-43bb-9eee-f281a7f14921%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: delegating our static file serving

[Aymeric Augustin]

2015-12-31 12:52 GMT+01:00 Anssi Kääriäinen :

> In my opinion one of the most important goals for Django is to make
> deploying tiny projects securely and easily a trivial matter. We
> aren't there right now.
>

+1

I'm maintaining three such sites (my wife's, my consultancy's and mine). I'm
clearly not going to bother configuring a CDN. I deployed them with
whitenoise
— which isn't nearly as well known as it should be. If I hadn't known about
it, I suspect I would have rolled a poor man's solution to serve static
files
in Python, because that's the easiest when deploying to a PaaS.

Regarding the question of media files raised earlier in this thread, I think
we shouldn't attempt to cache all existing files on start-up because even
small sites could easily have tens of thousands e.g. a forum that stores
users' avatars. We should look up the file at each query, which will be
slightly less efficient, but functional and not subject to scalability
issues.

For both static and media files, we care about simplicity and security. It
appears that whitenoise gives us great performance for static files as well,
which is good. If we can't have it for media files, it's still fine.

-- 
Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CANE-7mV_KwtmtgV9Oow4fgf9CaSPO4PsE_NL-NEqbOGC1kfk_A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: delegating our static file serving

[Guilherme Leal]

>>Anything that improves the development experience for tiny sites gets
>>a big +1 from me.

As a solo developer that usualy deploy a diferent application every 3 weeks
or so, cant agree more.

Big +1 for me too.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAOs3Lp5RXYZA6aSA3gUkrwsGOM9a3ebMgAhG5DZZLy1GXyo2DA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: delegating our static file serving

[Anssi Kääriäinen]

In my opinion one of the most important goals for Django is to make
deploying tiny projects securely and easily a trivial matter. We
aren't there right now.

If you have a site where traffic is less than one hit per minute, you
shouldn't care about performance that much.

With Django, it is pretty easy to write a decent "one-off" application
in an hour. Unfortunately, proper (that is, secure and automated)
deployment takes a lot longer than that.

Maybe part of the problem is that core developers tend to work on
larger sites, where you likely want a customized deployment anyways,
and working on deployment scripts doesn't take such a large portion of
the total time.

Anything that improves the development experience for tiny sites gets
a big +1 from me.

 - Anssi

On Thu, Dec 31, 2015 at 1:17 PM, Jannis Leidel  wrote:
>
>> On 29 Dec 2015, at 01:36, Tim Graham  wrote:
>>
>> I'd like to work together with Dave to develop a proof of concept that 
>> integrates whitenoise into Django. I spent about an hour looking through 
>> whitenoise and our own static file serving code, and I think integrating 
>> whitenoise will yield a simpler user experience with about the same amount 
>> of code as have now.
>>
>> Essentially, we'd recommend adding something like this to existing wsgi.py 
>> files (it would be in the default startproject template)
>>
>> from whitenoise.django import DjangoWhiteNoise
>> application = DjangoWhiteNoise(application)
>> application.add_files(settings.MEDIA_ROOT, prefix=settings.MEDIA_URL)
>>
>> which would have the benefit of working out of the box in production too, I 
>> think. Of course, you could disable that based on settings.DEBUG or some 
>> other toggle.
>>
>> We could then deprecate:
>>
>> * django/contrib/staticfiles/views.py
>> * django/contrib/staticfiles/management/commands/runserver.py
>> * django/contrib/staticfiles/handlers.py
>> * django/views/static.py
>>
>> Any objections to doing further investigation in this area?
>
> None, this sounds like a great combination and a logical continuation of what 
> I did back then with staticfiles.
>
> The fact that staticfiles had all this custom logic to handle file serving in 
> staticfiles was a necessary evil to stay backward compatible and lower the 
> risk of staticfiles becoming a failure -- it wasn’t clear at all if it’d 
> match the workflow for most of Django’s user base (and it still hasn’t 
> completely I think). Whitenoise is agnostic enough about where the served 
> files come from so this fits nicely.
>
>> On Saturday, June 20, 2015 at 8:09:11 AM UTC-4, David Evans wrote:
>> On Friday, 5 December 2014 19:14:29 UTC, Carl Meyer wrote:
>> On 12/04/2014 10:33 PM, Collin Anderson wrote:
>> > Hi All,
>> >
>> > I'm pretty interested in getting secure and _somewhat_ efficient static
>> > file serving in Django.
>> >
>> > Quick history:
>> > 2005 - Jacob commits #428: a "static pages" view.  Note that this view
>> > should only be used for testing!"
>> > 2010 - Jannis adds staticfiles. Serving via django is considered "grossly
>> > inefficient and probably insecure".
>> > 2011 - Graham Dumpleton adds wsgi.file_wrapper to Gunicorn.
>> > 2012 - Aymeric adds StreamingHttpResponse and now files are read in chunks
>> > rather than reading the entire file into memory. (No longer grossly
>> > inefficient IMHO.)
>> >
>> > I propose:
>> > - Deprecate the "show_indexes" parameter of static.serve() (unless people
>> > actually use it).
>> > - Have people report security issues to secu...@djangoproject.com (like
>> > always)
>> > - Audit the code and possibly add more security checks and tests.
>> > - add wsgi.file_wrapper support to responses (5-line proof of concept:
>> > https://github.com/django/django/pull/3650 )
>> > - support serving static files in production, but still recommend
>> > nginx/apache or a cdn for performance.
>> > - make serving static files in production an opt-in, but put the view in
>> > project_template/project_name/urls.py
>> >
>> > I think it's a huge win for low-traffic sites or sites in the "just trying
>> > to deploy and get something live" phase. You can always optimize later by
>> > serving via nginx or cdn.
>> > We already have the views, api, and logic around for finding and serving
>> > the correct files.
>> > We can be just as efficient and secure as static/dj-static without needing
>> > to make people install and configure wsgi middleware to the application.
>> > We could have staticfiles classes implement more complicated features like
>> > giving cache recommendations, and serving pre-gzipped files.
>> >
>> > Is this a good idea? I realize it's not totally thought through. I'm fine
>> > with waiting until 1.9 if needed.
>>
>> I also think this is a good plan. It certainly makes sense to look at
>> "static" and "whitenoise" for ideas and compare their code to ours to
>> see where we could be more efficient or secure, but it's much less churn
>> for Django users if we simply improve our existing code rathe

Re: delegating our static file serving

[Jannis Leidel]

> On 29 Dec 2015, at 01:36, Tim Graham  wrote:
> 
> I'd like to work together with Dave to develop a proof of concept that 
> integrates whitenoise into Django. I spent about an hour looking through 
> whitenoise and our own static file serving code, and I think integrating 
> whitenoise will yield a simpler user experience with about the same amount of 
> code as have now.
> 
> Essentially, we'd recommend adding something like this to existing wsgi.py 
> files (it would be in the default startproject template)
> 
> from whitenoise.django import DjangoWhiteNoise
> application = DjangoWhiteNoise(application)
> application.add_files(settings.MEDIA_ROOT, prefix=settings.MEDIA_URL)
> 
> which would have the benefit of working out of the box in production too, I 
> think. Of course, you could disable that based on settings.DEBUG or some 
> other toggle.
> 
> We could then deprecate:
> 
> * django/contrib/staticfiles/views.py
> * django/contrib/staticfiles/management/commands/runserver.py
> * django/contrib/staticfiles/handlers.py
> * django/views/static.py
> 
> Any objections to doing further investigation in this area?

None, this sounds like a great combination and a logical continuation of what I 
did back then with staticfiles.

The fact that staticfiles had all this custom logic to handle file serving in 
staticfiles was a necessary evil to stay backward compatible and lower the risk 
of staticfiles becoming a failure -- it wasn’t clear at all if it’d match the 
workflow for most of Django’s user base (and it still hasn’t completely I 
think). Whitenoise is agnostic enough about where the served files come from so 
this fits nicely.

> On Saturday, June 20, 2015 at 8:09:11 AM UTC-4, David Evans wrote:
> On Friday, 5 December 2014 19:14:29 UTC, Carl Meyer wrote:
> On 12/04/2014 10:33 PM, Collin Anderson wrote: 
> > Hi All, 
> > 
> > I'm pretty interested in getting secure and _somewhat_ efficient static 
> > file serving in Django. 
> > 
> > Quick history: 
> > 2005 - Jacob commits #428: a "static pages" view.  Note that this view 
> > should only be used for testing!" 
> > 2010 - Jannis adds staticfiles. Serving via django is considered "grossly 
> > inefficient and probably insecure". 
> > 2011 - Graham Dumpleton adds wsgi.file_wrapper to Gunicorn. 
> > 2012 - Aymeric adds StreamingHttpResponse and now files are read in chunks 
> > rather than reading the entire file into memory. (No longer grossly 
> > inefficient IMHO.) 
> > 
> > I propose: 
> > - Deprecate the "show_indexes" parameter of static.serve() (unless people 
> > actually use it). 
> > - Have people report security issues to secu...@djangoproject.com (like 
> > always) 
> > - Audit the code and possibly add more security checks and tests. 
> > - add wsgi.file_wrapper support to responses (5-line proof of concept: 
> > https://github.com/django/django/pull/3650 ) 
> > - support serving static files in production, but still recommend 
> > nginx/apache or a cdn for performance. 
> > - make serving static files in production an opt-in, but put the view in 
> > project_template/project_name/urls.py 
> > 
> > I think it's a huge win for low-traffic sites or sites in the "just trying 
> > to deploy and get something live" phase. You can always optimize later by 
> > serving via nginx or cdn. 
> > We already have the views, api, and logic around for finding and serving 
> > the correct files. 
> > We can be just as efficient and secure as static/dj-static without needing 
> > to make people install and configure wsgi middleware to the application. 
> > We could have staticfiles classes implement more complicated features like 
> > giving cache recommendations, and serving pre-gzipped files. 
> > 
> > Is this a good idea? I realize it's not totally thought through. I'm fine 
> > with waiting until 1.9 if needed. 
> 
> I also think this is a good plan. It certainly makes sense to look at 
> "static" and "whitenoise" for ideas and compare their code to ours to 
> see where we could be more efficient or secure, but it's much less churn 
> for Django users if we simply improve our existing code rather than pull 
> in something wholly new. 
> 
> Carl 
> 
> 
>  
> Sorry to revive an old thread here, but I just wanted to add that v2.0 of 
> WhiteNoise now supports serving development files, providing the same 
> behaviour as runserver currently does in DEBUG mode. (There were enough 
> people wanting to do their development using gunicorn rather than runserver 
> to make this worthwhile.)
> 
> This means that WhiteNoise is now a one-stop-shop for static file handling in 
> Django. If there's still an appetite for integrating it, or something 
> equivalent, into core I'd be happy to help out.
> 
> Dave
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubsc

Re: delegating our static file serving

[Florian Apolloner]

You are free to use what your PaaS provider supplies, but if you are using 
heroku (and many others) you do not have a choice. If AWS allows you to use 
apache 2.4 in event mode, then by all means use it! But there are plenty of 
platforms out there where you cannot easily serve static files and/or 
performance does not matter too much. So in the end it boils down if we can 
remove code from Django, if yes it is a win, even if it is just used for 
the development server.

On Thursday, December 31, 2015 at 1:00:13 AM UTC+1, Cristiano Coelho wrote:
>
> Just curious, about PaaS, I know AWS (Amazon) deploys python/django on 
> apache (which is quite decent), and apache also serves static files 
> decently. What would be wrong with this?
> The idea of having Python serving static files (and potentially gziping 
> it), when python is one of the slowest languages out there, I don't think 
> it is really a good idea. You should be really serving static files with 
> something implemented in C/C++ or similar high performant language (read 
> nginx or apache). How wrong am I?
>
> El viernes, 5 de diciembre de 2014, 5:19:45 (UTC-3), Aymeric Augustin 
> escribió:
>>
>> Yes, I support this plan.
>>
>> "Serve your files with nginx!" doesn't fly in the age of PaaS.
>>
>> Serving static files with Django and having a CDN cache them is a 
>> reasonable setup as far as I know.
>>
>> I don't know if the "probably insecure" argument still holds. Are there 
>> some specific security risks in serving files that don't exist in serving 
>> dynamic content? I'd say dynamic content is more difficult. This main 
>> problem I can imagine when serving files is directory traversal. We already 
>> have protections against this class of attacks in several features. (Usual 
>> disclaimer: my background is security is mostly theoretical.)
>>
>> -- 
>> Aymeric.
>>
>>
>> 2014-12-05 6:33 GMT+01:00 Collin Anderson :
>>
>>> Hi All,
>>>
>>> I'm pretty interested in getting secure and _somewhat_ efficient static 
>>> file serving in Django.
>>>
>>> Quick history:
>>> 2005 - Jacob commits #428: a "static pages" view.  Note that this view 
>>> should only be used for testing!"
>>> 2010 - Jannis adds staticfiles. Serving via django is considered "
>>> grossly inefficient and probably insecure".
>>> 2011 - Graham Dumpleton adds wsgi.file_wrapper to Gunicorn.
>>> 2012 - Aymeric adds StreamingHttpResponse and now files are read in 
>>> chunks rather than reading the entire file into memory. (No longer grossly 
>>> inefficient IMHO.)
>>>
>>> I propose:
>>> - Deprecate the "show_indexes" parameter of static.serve() (unless 
>>> people actually use it).
>>> - Have people report security issues to secu...@djangoproject.com (like 
>>> always)
>>> - Audit the code and possibly add more security checks and tests.
>>> - add wsgi.file_wrapper support to responses (5-line proof of concept: 
>>> https://github.com/django/django/pull/3650 )
>>> - support serving static files in production, but still recommend 
>>> nginx/apache or a cdn for performance.
>>> - make serving static files in production an opt-in, but put the view in 
>>> project_template/project_name/urls.py
>>>
>>> I think it's a huge win for low-traffic sites or sites in the "just 
>>> trying to deploy and get something live" phase. You can always optimize 
>>> later by serving via nginx or cdn.
>>> We already have the views, api, and logic around for finding and serving 
>>> the correct files.
>>> We can be just as efficient and secure as static/dj-static without 
>>> needing to make people install and configure wsgi middleware to the 
>>> application.
>>> We could have staticfiles classes implement more complicated features 
>>> like giving cache recommendations, and serving pre-gzipped files.
>>>
>>> Is this a good idea? I realize it's not totally thought through. I'm 
>>> fine with waiting until 1.9 if needed.
>>>
>>> Collin
>>>
>>> On Saturday, November 29, 2014 6:07:05 PM UTC-5, Collin Anderson wrote:

 Hi All,

 I think doing something here is really good idea. I'm happy with any of 
 the solutions mentioned so far.

 My question is: what does static/dj-static do that our built-in code 
 doesn't do? What makes it more secure? It seems to me we're only missing 
 is 
 wsgi.file_wrapper and maybe a few more security checks. Why don't we just 
 make our own code secure and start supporting it?
 Here's basic wsgi.file_wrapper support: https://github.com/django/
 django/pull/3650

 We could then, over time, start supporting more extensions ourselves: 
 ranges, pre-gziped files, urls with never-changing content, etc. That way 
 we get very, very deep django integration. It seems to me this is a piece 
 that a web framework should be able to support itself.

 Collin


 On Friday, November 28, 2014 9:15:03 AM UTC-5, Tim Graham wrote:
>
> Berker has worked on integrating gunicorn with runserver 
> 

Re: delegating our static file serving

[Cristiano Coelho]

Just curious, about PaaS, I know AWS (Amazon) deploys python/django on 
apache (which is quite decent), and apache also serves static files 
decently. What would be wrong with this?
The idea of having Python serving static files (and potentially gziping 
it), when python is one of the slowest languages out there, I don't think 
it is really a good idea. You should be really serving static files with 
something implemented in C/C++ or similar high performant language (read 
nginx or apache). How wrong am I?

El viernes, 5 de diciembre de 2014, 5:19:45 (UTC-3), Aymeric Augustin 
escribió:
>
> Yes, I support this plan.
>
> "Serve your files with nginx!" doesn't fly in the age of PaaS.
>
> Serving static files with Django and having a CDN cache them is a 
> reasonable setup as far as I know.
>
> I don't know if the "probably insecure" argument still holds. Are there 
> some specific security risks in serving files that don't exist in serving 
> dynamic content? I'd say dynamic content is more difficult. This main 
> problem I can imagine when serving files is directory traversal. We already 
> have protections against this class of attacks in several features. (Usual 
> disclaimer: my background is security is mostly theoretical.)
>
> -- 
> Aymeric.
>
>
> 2014-12-05 6:33 GMT+01:00 Collin Anderson  >:
>
>> Hi All,
>>
>> I'm pretty interested in getting secure and _somewhat_ efficient static 
>> file serving in Django.
>>
>> Quick history:
>> 2005 - Jacob commits #428: a "static pages" view.  Note that this view 
>> should only be used for testing!"
>> 2010 - Jannis adds staticfiles. Serving via django is considered "grossly 
>> inefficient and probably insecure".
>> 2011 - Graham Dumpleton adds wsgi.file_wrapper to Gunicorn.
>> 2012 - Aymeric adds StreamingHttpResponse and now files are read in 
>> chunks rather than reading the entire file into memory. (No longer grossly 
>> inefficient IMHO.)
>>
>> I propose:
>> - Deprecate the "show_indexes" parameter of static.serve() (unless people 
>> actually use it).
>> - Have people report security issues to secu...@djangoproject.com 
>>  (like always)
>> - Audit the code and possibly add more security checks and tests.
>> - add wsgi.file_wrapper support to responses (5-line proof of concept: 
>> https://github.com/django/django/pull/3650 )
>> - support serving static files in production, but still recommend 
>> nginx/apache or a cdn for performance.
>> - make serving static files in production an opt-in, but put the view in 
>> project_template/project_name/urls.py
>>
>> I think it's a huge win for low-traffic sites or sites in the "just 
>> trying to deploy and get something live" phase. You can always optimize 
>> later by serving via nginx or cdn.
>> We already have the views, api, and logic around for finding and serving 
>> the correct files.
>> We can be just as efficient and secure as static/dj-static without 
>> needing to make people install and configure wsgi middleware to the 
>> application.
>> We could have staticfiles classes implement more complicated features 
>> like giving cache recommendations, and serving pre-gzipped files.
>>
>> Is this a good idea? I realize it's not totally thought through. I'm fine 
>> with waiting until 1.9 if needed.
>>
>> Collin
>>
>> On Saturday, November 29, 2014 6:07:05 PM UTC-5, Collin Anderson wrote:
>>>
>>> Hi All,
>>>
>>> I think doing something here is really good idea. I'm happy with any of 
>>> the solutions mentioned so far.
>>>
>>> My question is: what does static/dj-static do that our built-in code 
>>> doesn't do? What makes it more secure? It seems to me we're only missing is 
>>> wsgi.file_wrapper and maybe a few more security checks. Why don't we just 
>>> make our own code secure and start supporting it?
>>> Here's basic wsgi.file_wrapper support: https://github.com/django/
>>> django/pull/3650
>>>
>>> We could then, over time, start supporting more extensions ourselves: 
>>> ranges, pre-gziped files, urls with never-changing content, etc. That way 
>>> we get very, very deep django integration. It seems to me this is a piece 
>>> that a web framework should be able to support itself.
>>>
>>> Collin
>>>
>>>
>>> On Friday, November 28, 2014 9:15:03 AM UTC-5, Tim Graham wrote:

 Berker has worked on integrating gunicorn with runserver 
  so that we might be able 
 to deprecate our own homegrown webserver. Windows support for gunicorn is 
 supposedly coming soon 
 which 
 may actually make the idea feasible. This way we provide a more secure 
 solution out of the box (anecdotes indicate that runserver is widely used 
 in production despite our documentation warnings against doing so).

 On the pull request, Anssi had an idea to use dj-static 
  to serve static/media 
 files. My understanding is that we would basically integrat

Re: delegating our static file serving

[Chris Cogdon]

Also consider the media files could number into the millions (or 
bajizillions?). Particularly for, say, a image hosting application. Clearly 
it would not be feasible to enumerate all the files, and there would 
clearly be regular additions. It might be that this use case is simply 
"beyond the scope" of a django-supplied static files server, and the 
developer should instead use nginx, lighttpd or something similar.

On Tuesday, December 29, 2015 at 9:31:02 AM UTC-8, David Evans wrote:
>
> 2. Serving media files is a slightly different case. WhiteNoise was 
> designed around the assumption that it's serving a set of 
> developer-supplied, public files which remain unchanged during the lifetime 
> of the process. This simplifies a lot of performance and security concerns 
> that come with serving mutable, user-supplied files. At present, if you use 
> `add_files(settings.MEDIA_ROOT, prefix=settings.MEDIA_URL)` as you 
> suggested then WhiteNoise won't pick up any files that were uploaded after 
> the application was started -- at least, not unless you enable the 
> "autorefresh" setting which I explicitly don't recommend for production.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d489de11-018a-40e0-8419-40b4d18de9a0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: delegating our static file serving

[Chris Cogdon]

Also consider the media files could number into the millions (or 
bajizillions?). Particularly for, say, a image hosting application. Clearly 
it would not be feasible to enumerate all the files, and there would 
clearly be regular additions. It might be that this use case is simply 
"beyond the scope" of a django-supplied static files server, and the 
developer should instead use nginx, lighttpd or something similar.


On Tuesday, December 29, 2015 at 9:31:02 AM UTC-8, David Evans wrote:
>
> 2. Serving media files is a slightly different case. WhiteNoise was 
> designed around the assumption that it's serving a set of 
> developer-supplied, public files which remain unchanged during the lifetime 
> of the process. This simplifies a lot of performance and security concerns 
> that come with serving mutable, user-supplied files. At present, if you use 
> `add_files(settings.MEDIA_ROOT, prefix=settings.MEDIA_URL)` as you 
> suggested then WhiteNoise won't pick up any files that were uploaded after 
> the application was started -- at least, not unless you enable the 
> "autorefresh" setting which I explicitly don't recommend for production.
>
> Obviously it's possible to support this if we decide this is an important 
> goal but it will need more thought and will definitely be more work than 
> just supporting static files.
>
> On Tuesday, 29 December 2015 00:36:06 UTC, Tim Graham wrote:
>>
>> I'd like to work together with Dave to develop a proof of concept that 
>> integrates whitenoise into Django. I spent about an hour looking through 
>> whitenoise and our own static file serving code, and I think integrating 
>> whitenoise will yield a simpler user experience with about the same amount 
>> of code as have now.
>>
>> Essentially, we'd recommend adding something like this to existing 
>> wsgi.py files (it would be in the default startproject template)
>>
>> from whitenoise.django import DjangoWhiteNoise
>> application = DjangoWhiteNoise(application)
>> application.add_files(settings.MEDIA_ROOT, prefix=settings.MEDIA_URL)
>>
>> which would have the benefit of working out of the box in production too, 
>> I think. Of course, you could disable that based on settings.DEBUG or some 
>> other toggle.
>>
>> We could then deprecate:
>>
>> * django/contrib/staticfiles/views.py
>> * django/contrib/staticfiles/management/commands/runserver.py
>> * django/contrib/staticfiles/handlers.py
>> * django/views/static.py
>>
>> Any objections to doing further investigation in this area?
>>
>> On Saturday, June 20, 2015 at 8:09:11 AM UTC-4, David Evans wrote:
>>>
>>> On Friday, 5 December 2014 19:14:29 UTC, Carl Meyer wrote:

 On 12/04/2014 10:33 PM, Collin Anderson wrote: 
 > Hi All, 
 > 
 > I'm pretty interested in getting secure and _somewhat_ efficient 
 static 
 > file serving in Django. 
 > 
 > Quick history: 
 > 2005 - Jacob commits #428: a "static pages" view.  Note that this 
 view 
 > should only be used for testing!" 
 > 2010 - Jannis adds staticfiles. Serving via django is considered 
 "grossly 
 > inefficient and probably insecure". 
 > 2011 - Graham Dumpleton adds wsgi.file_wrapper to Gunicorn. 
 > 2012 - Aymeric adds StreamingHttpResponse and now files are read in 
 chunks 
 > rather than reading the entire file into memory. (No longer grossly 
 > inefficient IMHO.) 
 > 
 > I propose: 
 > - Deprecate the "show_indexes" parameter of static.serve() (unless 
 people 
 > actually use it). 
 > - Have people report security issues to secu...@djangoproject.com 
 (like 
 > always) 
 > - Audit the code and possibly add more security checks and tests. 
 > - add wsgi.file_wrapper support to responses (5-line proof of 
 concept: 
 > https://github.com/django/django/pull/3650 ) 
 > - support serving static files in production, but still recommend 
 > nginx/apache or a cdn for performance. 
 > - make serving static files in production an opt-in, but put the view 
 in 
 > project_template/project_name/urls.py 
 > 
 > I think it's a huge win for low-traffic sites or sites in the "just 
 trying 
 > to deploy and get something live" phase. You can always optimize 
 later by 
 > serving via nginx or cdn. 
 > We already have the views, api, and logic around for finding and 
 serving 
 > the correct files. 
 > We can be just as efficient and secure as static/dj-static without 
 needing 
 > to make people install and configure wsgi middleware to the 
 application. 
 > We could have staticfiles classes implement more complicated features 
 like 
 > giving cache recommendations, and serving pre-gzipped files. 
 > 
 > Is this a good idea? I realize it's not totally thought through. I'm 
 fine 
 > with waiting until 1.9 if needed. 

 I also think this is a good plan. It certainly makes sense to look at 

Re: delegating our static file serving

[Florian Apolloner]

On Tuesday, December 29, 2015 at 6:31:02 PM UTC+1, David Evans wrote:
>
> 2. Serving media files is a slightly different case. WhiteNoise was 
> designed around the assumption that it's serving a set of 
> developer-supplied, public files which remain unchanged during the lifetime 
> of the process. This simplifies a lot of performance and security concerns 
> that come with serving mutable, user-supplied files. 
>

Speaking from a  security perspective: It is also important that WhiteNoise 
would serve Media files in a safe way, ie serving most files as attachments 
to prevent uploaded HTML files from starting an attack, preventing browser 
content type sniffing etc…

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/db7e7fe6-c735-43bb-a3f7-2bc809707032%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: delegating our static file serving

[David Evans]

I'd be very happy to help with this. Two things to mention:

1. When I first wrote WhiteNoise the FileResponse class didn't exist and so 
the only way I could access `wsgi.file_wrapper` was by wrapping the wsgi 
application directly and bypassing Django. Now we have the FileResponse 
class it would be possible to implement WhiteNoise as standard Django 
middleware without having to edit wsgi.py. I think this would be a much 
cleaner approach, and I don't think it will be too much work to implement. 
I should have time to look at this next week.

2. Serving media files is a slightly different case. WhiteNoise was 
designed around the assumption that it's serving a set of 
developer-supplied, public files which remain unchanged during the lifetime 
of the process. This simplifies a lot of performance and security concerns 
that come with serving mutable, user-supplied files. At present, if you use 
`add_files(settings.MEDIA_ROOT, prefix=settings.MEDIA_URL)` as you 
suggested then WhiteNoise won't pick up any files that were uploaded after 
the application was started -- at least, not unless you enable the 
"autorefresh" setting which I explicitly don't recommend for production.

Obviously it's possible to support this if we decide this is an important 
goal but it will need more thought and will definitely be more work than 
just supporting static files.

On Tuesday, 29 December 2015 00:36:06 UTC, Tim Graham wrote:
>
> I'd like to work together with Dave to develop a proof of concept that 
> integrates whitenoise into Django. I spent about an hour looking through 
> whitenoise and our own static file serving code, and I think integrating 
> whitenoise will yield a simpler user experience with about the same amount 
> of code as have now.
>
> Essentially, we'd recommend adding something like this to existing wsgi.py 
> files (it would be in the default startproject template)
>
> from whitenoise.django import DjangoWhiteNoise
> application = DjangoWhiteNoise(application)
> application.add_files(settings.MEDIA_ROOT, prefix=settings.MEDIA_URL)
>
> which would have the benefit of working out of the box in production too, 
> I think. Of course, you could disable that based on settings.DEBUG or some 
> other toggle.
>
> We could then deprecate:
>
> * django/contrib/staticfiles/views.py
> * django/contrib/staticfiles/management/commands/runserver.py
> * django/contrib/staticfiles/handlers.py
> * django/views/static.py
>
> Any objections to doing further investigation in this area?
>
> On Saturday, June 20, 2015 at 8:09:11 AM UTC-4, David Evans wrote:
>>
>> On Friday, 5 December 2014 19:14:29 UTC, Carl Meyer wrote:
>>>
>>> On 12/04/2014 10:33 PM, Collin Anderson wrote: 
>>> > Hi All, 
>>> > 
>>> > I'm pretty interested in getting secure and _somewhat_ efficient 
>>> static 
>>> > file serving in Django. 
>>> > 
>>> > Quick history: 
>>> > 2005 - Jacob commits #428: a "static pages" view.  Note that this view 
>>> > should only be used for testing!" 
>>> > 2010 - Jannis adds staticfiles. Serving via django is considered 
>>> "grossly 
>>> > inefficient and probably insecure". 
>>> > 2011 - Graham Dumpleton adds wsgi.file_wrapper to Gunicorn. 
>>> > 2012 - Aymeric adds StreamingHttpResponse and now files are read in 
>>> chunks 
>>> > rather than reading the entire file into memory. (No longer grossly 
>>> > inefficient IMHO.) 
>>> > 
>>> > I propose: 
>>> > - Deprecate the "show_indexes" parameter of static.serve() (unless 
>>> people 
>>> > actually use it). 
>>> > - Have people report security issues to secu...@djangoproject.com 
>>> (like 
>>> > always) 
>>> > - Audit the code and possibly add more security checks and tests. 
>>> > - add wsgi.file_wrapper support to responses (5-line proof of concept: 
>>> > https://github.com/django/django/pull/3650 ) 
>>> > - support serving static files in production, but still recommend 
>>> > nginx/apache or a cdn for performance. 
>>> > - make serving static files in production an opt-in, but put the view 
>>> in 
>>> > project_template/project_name/urls.py 
>>> > 
>>> > I think it's a huge win for low-traffic sites or sites in the "just 
>>> trying 
>>> > to deploy and get something live" phase. You can always optimize later 
>>> by 
>>> > serving via nginx or cdn. 
>>> > We already have the views, api, and logic around for finding and 
>>> serving 
>>> > the correct files. 
>>> > We can be just as efficient and secure as static/dj-static without 
>>> needing 
>>> > to make people install and configure wsgi middleware to the 
>>> application. 
>>> > We could have staticfiles classes implement more complicated features 
>>> like 
>>> > giving cache recommendations, and serving pre-gzipped files. 
>>> > 
>>> > Is this a good idea? I realize it's not totally thought through. I'm 
>>> fine 
>>> > with waiting until 1.9 if needed. 
>>>
>>> I also think this is a good plan. It certainly makes sense to look at 
>>> "static" and "whitenoise" for ideas and compare their code to ours 

Re: delegating our static file serving

[Tim Graham]

I'd like to work together with Dave to develop a proof of concept that 
integrates whitenoise into Django. I spent about an hour looking through 
whitenoise and our own static file serving code, and I think integrating 
whitenoise will yield a simpler user experience with about the same amount 
of code as have now.

Essentially, we'd recommend adding something like this to existing wsgi.py 
files (it would be in the default startproject template)

from whitenoise.django import DjangoWhiteNoise
application = DjangoWhiteNoise(application)
application.add_files(settings.MEDIA_ROOT, prefix=settings.MEDIA_URL)

which would have the benefit of working out of the box in production too, I 
think. Of course, you could disable that based on settings.DEBUG or some 
other toggle.

We could then deprecate:

* django/contrib/staticfiles/views.py
* django/contrib/staticfiles/management/commands/runserver.py
* django/contrib/staticfiles/handlers.py
* django/views/static.py

Any objections to doing further investigation in this area?

On Saturday, June 20, 2015 at 8:09:11 AM UTC-4, David Evans wrote:
>
> On Friday, 5 December 2014 19:14:29 UTC, Carl Meyer wrote:
>>
>> On 12/04/2014 10:33 PM, Collin Anderson wrote: 
>> > Hi All, 
>> > 
>> > I'm pretty interested in getting secure and _somewhat_ efficient static 
>> > file serving in Django. 
>> > 
>> > Quick history: 
>> > 2005 - Jacob commits #428: a "static pages" view.  Note that this view 
>> > should only be used for testing!" 
>> > 2010 - Jannis adds staticfiles. Serving via django is considered 
>> "grossly 
>> > inefficient and probably insecure". 
>> > 2011 - Graham Dumpleton adds wsgi.file_wrapper to Gunicorn. 
>> > 2012 - Aymeric adds StreamingHttpResponse and now files are read in 
>> chunks 
>> > rather than reading the entire file into memory. (No longer grossly 
>> > inefficient IMHO.) 
>> > 
>> > I propose: 
>> > - Deprecate the "show_indexes" parameter of static.serve() (unless 
>> people 
>> > actually use it). 
>> > - Have people report security issues to secu...@djangoproject.com 
>> (like 
>> > always) 
>> > - Audit the code and possibly add more security checks and tests. 
>> > - add wsgi.file_wrapper support to responses (5-line proof of concept: 
>> > https://github.com/django/django/pull/3650 ) 
>> > - support serving static files in production, but still recommend 
>> > nginx/apache or a cdn for performance. 
>> > - make serving static files in production an opt-in, but put the view 
>> in 
>> > project_template/project_name/urls.py 
>> > 
>> > I think it's a huge win for low-traffic sites or sites in the "just 
>> trying 
>> > to deploy and get something live" phase. You can always optimize later 
>> by 
>> > serving via nginx or cdn. 
>> > We already have the views, api, and logic around for finding and 
>> serving 
>> > the correct files. 
>> > We can be just as efficient and secure as static/dj-static without 
>> needing 
>> > to make people install and configure wsgi middleware to the 
>> application. 
>> > We could have staticfiles classes implement more complicated features 
>> like 
>> > giving cache recommendations, and serving pre-gzipped files. 
>> > 
>> > Is this a good idea? I realize it's not totally thought through. I'm 
>> fine 
>> > with waiting until 1.9 if needed. 
>>
>> I also think this is a good plan. It certainly makes sense to look at 
>> "static" and "whitenoise" for ideas and compare their code to ours to 
>> see where we could be more efficient or secure, but it's much less churn 
>> for Django users if we simply improve our existing code rather than pull 
>> in something wholly new. 
>>
>> Carl 
>>
>>
>>  
> Sorry to revive an old thread here, but I just wanted to add that v2.0 of 
> WhiteNoise now supports serving development files, providing the same 
> behaviour as runserver currently does in DEBUG mode. (There were enough 
> people wanting to do their development using gunicorn rather than runserver 
> to make this worthwhile.)
>
> This means that WhiteNoise is now a one-stop-shop for static file handling 
> in Django. If there's still an appetite for integrating it, or something 
> equivalent, into core I'd be happy to help out.
>
> Dave
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/a2136cbb-2dfb-4269-8bb9-64a55f8c7b6a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: delegating our static file serving

[David Evans]

On Friday, 5 December 2014 19:14:29 UTC, Carl Meyer wrote:
>
> On 12/04/2014 10:33 PM, Collin Anderson wrote: 
> > Hi All, 
> > 
> > I'm pretty interested in getting secure and _somewhat_ efficient static 
> > file serving in Django. 
> > 
> > Quick history: 
> > 2005 - Jacob commits #428: a "static pages" view.  Note that this view 
> > should only be used for testing!" 
> > 2010 - Jannis adds staticfiles. Serving via django is considered 
> "grossly 
> > inefficient and probably insecure". 
> > 2011 - Graham Dumpleton adds wsgi.file_wrapper to Gunicorn. 
> > 2012 - Aymeric adds StreamingHttpResponse and now files are read in 
> chunks 
> > rather than reading the entire file into memory. (No longer grossly 
> > inefficient IMHO.) 
> > 
> > I propose: 
> > - Deprecate the "show_indexes" parameter of static.serve() (unless 
> people 
> > actually use it). 
> > - Have people report security issues to secu...@djangoproject.com 
>  (like 
> > always) 
> > - Audit the code and possibly add more security checks and tests. 
> > - add wsgi.file_wrapper support to responses (5-line proof of concept: 
> > https://github.com/django/django/pull/3650 ) 
> > - support serving static files in production, but still recommend 
> > nginx/apache or a cdn for performance. 
> > - make serving static files in production an opt-in, but put the view in 
> > project_template/project_name/urls.py 
> > 
> > I think it's a huge win for low-traffic sites or sites in the "just 
> trying 
> > to deploy and get something live" phase. You can always optimize later 
> by 
> > serving via nginx or cdn. 
> > We already have the views, api, and logic around for finding and serving 
> > the correct files. 
> > We can be just as efficient and secure as static/dj-static without 
> needing 
> > to make people install and configure wsgi middleware to the application. 
> > We could have staticfiles classes implement more complicated features 
> like 
> > giving cache recommendations, and serving pre-gzipped files. 
> > 
> > Is this a good idea? I realize it's not totally thought through. I'm 
> fine 
> > with waiting until 1.9 if needed. 
>
> I also think this is a good plan. It certainly makes sense to look at 
> "static" and "whitenoise" for ideas and compare their code to ours to 
> see where we could be more efficient or secure, but it's much less churn 
> for Django users if we simply improve our existing code rather than pull 
> in something wholly new. 
>
> Carl 
>
>
>  
Sorry to revive an old thread here, but I just wanted to add that v2.0 of 
WhiteNoise now supports serving development files, providing the same 
behaviour as runserver currently does in DEBUG mode. (There were enough 
people wanting to do their development using gunicorn rather than runserver 
to make this worthwhile.)

This means that WhiteNoise is now a one-stop-shop for static file handling 
in Django. If there's still an appetite for integrating it, or something 
equivalent, into core I'd be happy to help out.

Dave

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/c4b46672-276e-4177-802e-df0f09389887%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: delegating our static file serving

[Carl Meyer]

On 12/04/2014 10:33 PM, Collin Anderson wrote:
> Hi All,
> 
> I'm pretty interested in getting secure and _somewhat_ efficient static 
> file serving in Django.
> 
> Quick history:
> 2005 - Jacob commits #428: a "static pages" view.  Note that this view 
> should only be used for testing!"
> 2010 - Jannis adds staticfiles. Serving via django is considered "grossly 
> inefficient and probably insecure".
> 2011 - Graham Dumpleton adds wsgi.file_wrapper to Gunicorn.
> 2012 - Aymeric adds StreamingHttpResponse and now files are read in chunks 
> rather than reading the entire file into memory. (No longer grossly 
> inefficient IMHO.)
> 
> I propose:
> - Deprecate the "show_indexes" parameter of static.serve() (unless people 
> actually use it).
> - Have people report security issues to secur...@djangoproject.com (like 
> always)
> - Audit the code and possibly add more security checks and tests.
> - add wsgi.file_wrapper support to responses (5-line proof of concept: 
> https://github.com/django/django/pull/3650 )
> - support serving static files in production, but still recommend 
> nginx/apache or a cdn for performance.
> - make serving static files in production an opt-in, but put the view in 
> project_template/project_name/urls.py
> 
> I think it's a huge win for low-traffic sites or sites in the "just trying 
> to deploy and get something live" phase. You can always optimize later by 
> serving via nginx or cdn.
> We already have the views, api, and logic around for finding and serving 
> the correct files.
> We can be just as efficient and secure as static/dj-static without needing 
> to make people install and configure wsgi middleware to the application.
> We could have staticfiles classes implement more complicated features like 
> giving cache recommendations, and serving pre-gzipped files.
> 
> Is this a good idea? I realize it's not totally thought through. I'm fine 
> with waiting until 1.9 if needed.

I also think this is a good plan. It certainly makes sense to look at
"static" and "whitenoise" for ideas and compare their code to ours to
see where we could be more efficient or secure, but it's much less churn
for Django users if we simply improve our existing code rather than pull
in something wholly new.

Carl


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/548203D3.5070809%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature

Re: delegating our static file serving

[David Evans]

+1 to this: I'm all in favour of supporting production-adequate static file
handling in core. A couple of small points:


We already have the views, api, and logic around for finding and serving
> the correct files.


One question that needs to be thought through is the role of
`collectstatic` and `STATIC_ROOT`.

The existing static.serve view just ignores `STATIC_ROOT` completely and
uses the configured StaticFileFinders to locate files. This means if you
use the ManifestStaticFilesStorage backend -- or anything that uses the
`post_process` hook -- it won't find your processed files at all.

Probably the simplest thing is to switch on DEBUG, and serve files from
`STATIC_ROOT` (falling back to using the finders) when DEBUG is False, and
use the existing behaviour otherwise.


We could have staticfiles classes implement more complicated features like
> giving cache recommendations, and serving pre-gzipped files.


Definitely +1 to using classes (CBVs, presumably) to handle this. The
existing views don't easily support customising behaviour.

Regards,

Dave





On 5 December 2014 at 05:33, Collin Anderson  wrote:

> Hi All,
>
> I'm pretty interested in getting secure and _somewhat_ efficient static
> file serving in Django.
>
> Quick history:
> 2005 - Jacob commits #428: a "static pages" view.  Note that this view
> should only be used for testing!"
> 2010 - Jannis adds staticfiles. Serving via django is considered "grossly
> inefficient and probably insecure".
> 2011 - Graham Dumpleton adds wsgi.file_wrapper to Gunicorn.
> 2012 - Aymeric adds StreamingHttpResponse and now files are read in chunks
> rather than reading the entire file into memory. (No longer grossly
> inefficient IMHO.)
>
> I propose:
> - Deprecate the "show_indexes" parameter of static.serve() (unless people
> actually use it).
> - Have people report security issues to secur...@djangoproject.com (like
> always)
> - Audit the code and possibly add more security checks and tests.
> - add wsgi.file_wrapper support to responses (5-line proof of concept:
> https://github.com/django/django/pull/3650 )
> - support serving static files in production, but still recommend
> nginx/apache or a cdn for performance.
> - make serving static files in production an opt-in, but put the view in
> project_template/project_name/urls.py
>
> I think it's a huge win for low-traffic sites or sites in the "just trying
> to deploy and get something live" phase. You can always optimize later by
> serving via nginx or cdn.
> We already have the views, api, and logic around for finding and serving
> the correct files.
> We can be just as efficient and secure as static/dj-static without needing
> to make people install and configure wsgi middleware to the application.
> We could have staticfiles classes implement more complicated features like
> giving cache recommendations, and serving pre-gzipped files.
>
> Is this a good idea? I realize it's not totally thought through. I'm fine
> with waiting until 1.9 if needed.
>
> Collin
>
> On Saturday, November 29, 2014 6:07:05 PM UTC-5, Collin Anderson wrote:
>>
>> Hi All,
>>
>> I think doing something here is really good idea. I'm happy with any of
>> the solutions mentioned so far.
>>
>> My question is: what does static/dj-static do that our built-in code
>> doesn't do? What makes it more secure? It seems to me we're only missing is
>> wsgi.file_wrapper and maybe a few more security checks. Why don't we just
>> make our own code secure and start supporting it?
>> Here's basic wsgi.file_wrapper support: https://github.com/django/
>> django/pull/3650
>>
>> We could then, over time, start supporting more extensions ourselves:
>> ranges, pre-gziped files, urls with never-changing content, etc. That way
>> we get very, very deep django integration. It seems to me this is a piece
>> that a web framework should be able to support itself.
>>
>> Collin
>>
>>
>> On Friday, November 28, 2014 9:15:03 AM UTC-5, Tim Graham wrote:
>>>
>>> Berker has worked on integrating gunicorn with runserver
>>>  so that we might be able
>>> to deprecate our own homegrown webserver. Windows support for gunicorn is
>>> supposedly coming soon which
>>> may actually make the idea feasible. This way we provide a more secure
>>> solution out of the box (anecdotes indicate that runserver is widely used
>>> in production despite our documentation warnings against doing so).
>>>
>>> On the pull request, Anssi had an idea to use dj-static
>>>  to serve static/media
>>> files. My understanding is that we would basically integrate the code for
>>> dj-static into Django and then add a dependency on static
>>> . It could be an optional
>>> dependency since you might want to serve static files differently in
>>> production, but it would likely be more or less universally used in
>>> development. We

Re: delegating our static file serving

[Michael Manfre]

Requiring fewer moving parts for small sites is definitely an improvement.
+1 to Collin's plan.

Regards,
Michael Manfre

On Fri, Dec 5, 2014 at 2:02 AM, Loïc Bistuer  wrote:

> I'm +1 on this plan, "native" support for wsgi.file_wrapper makes a lot of
> sense.
>
> --
> Loïc
>
> > On Dec 5, 2014, at 12:33 PM, Collin Anderson 
> wrote:
> >
> > Hi All,
> >
> > I'm pretty interested in getting secure and _somewhat_ efficient static
> file serving in Django.
> >
> > Quick history:
> > 2005 - Jacob commits #428: a "static pages" view.  Note that this view
> should only be used for testing!"
> > 2010 - Jannis adds staticfiles. Serving via django is considered
> "grossly inefficient and probably insecure".
> > 2011 - Graham Dumpleton adds wsgi.file_wrapper to Gunicorn.
> > 2012 - Aymeric adds StreamingHttpResponse and now files are read in
> chunks rather than reading the entire file into memory. (No longer grossly
> inefficient IMHO.)
> >
> > I propose:
> > - Deprecate the "show_indexes" parameter of static.serve() (unless
> people actually use it).
> > - Have people report security issues to secur...@djangoproject.com
> (like always)
> > - Audit the code and possibly add more security checks and tests.
> > - add wsgi.file_wrapper support to responses (5-line proof of concept:
> https://github.com/django/django/pull/3650 )
> > - support serving static files in production, but still recommend
> nginx/apache or a cdn for performance.
> > - make serving static files in production an opt-in, but put the view in
> project_template/project_name/urls.py
> >
> > I think it's a huge win for low-traffic sites or sites in the "just
> trying to deploy and get something live" phase. You can always optimize
> later by serving via nginx or cdn.
> > We already have the views, api, and logic around for finding and serving
> the correct files.
> > We can be just as efficient and secure as static/dj-static without
> needing to make people install and configure wsgi middleware to the
> application.
> > We could have staticfiles classes implement more complicated features
> like giving cache recommendations, and serving pre-gzipped files.
> >
> > Is this a good idea? I realize it's not totally thought through. I'm
> fine with waiting until 1.9 if needed.
> >
> > Collin
> >
> > On Saturday, November 29, 2014 6:07:05 PM UTC-5, Collin Anderson wrote:
> > Hi All,
> >
> > I think doing something here is really good idea. I'm happy with any of
> the solutions mentioned so far.
> >
> > My question is: what does static/dj-static do that our built-in code
> doesn't do? What makes it more secure? It seems to me we're only missing is
> wsgi.file_wrapper and maybe a few more security checks. Why don't we just
> make our own code secure and start supporting it?
> > Here's basic wsgi.file_wrapper support:
> https://github.com/django/django/pull/3650
> >
> > We could then, over time, start supporting more extensions ourselves:
> ranges, pre-gziped files, urls with never-changing content, etc. That way
> we get very, very deep django integration. It seems to me this is a piece
> that a web framework should be able to support itself.
> >
> > Collin
> >
> >
> > On Friday, November 28, 2014 9:15:03 AM UTC-5, Tim Graham wrote:
> > Berker has worked on integrating gunicorn with runserver so that we
> might be able to deprecate our own homegrown webserver. Windows support for
> gunicorn is supposedly coming soon which may actually make the idea
> feasible. This way we provide a more secure solution out of the box
> (anecdotes indicate that runserver is widely used in production despite our
> documentation warnings against doing so).
> >
> > On the pull request, Anssi had an idea to use dj-static to serve
> static/media files. My understanding is that we would basically integrate
> the code for dj-static into Django and then add a dependency on static. It
> could be an optional dependency since you might want to serve static files
> differently in production, but it would likely be more or less universally
> used in development. We could then say that django.views.static.serve (and
> its counterpart in staticfiles) is okay to use in production (and I guess
> people are already using them in production despite our warnings that they
> are not hardened for production use).
> >
> > What do you think of this plan?
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "Django developers (Contributions to Django itself)" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to django-developers+unsubscr...@googlegroups.com.
> > To post to this group, send email to django-developers@googlegroups.com.
> > Visit this group at http://groups.google.com/group/django-developers.
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/bfc1eb0d-8b69-4450-bfe0-7147ef729317%40googlegroups.com
> .
> > For more options, visit https://groups.google.com/d/opt

Re: delegating our static file serving

[Loïc Bistuer]

I'm +1 on this plan, "native" support for wsgi.file_wrapper makes a lot of 
sense.

-- 
Loïc

> On Dec 5, 2014, at 12:33 PM, Collin Anderson  wrote:
> 
> Hi All,
> 
> I'm pretty interested in getting secure and _somewhat_ efficient static file 
> serving in Django.
> 
> Quick history:
> 2005 - Jacob commits #428: a "static pages" view.  Note that this view should 
> only be used for testing!"
> 2010 - Jannis adds staticfiles. Serving via django is considered "grossly 
> inefficient and probably insecure".
> 2011 - Graham Dumpleton adds wsgi.file_wrapper to Gunicorn.
> 2012 - Aymeric adds StreamingHttpResponse and now files are read in chunks 
> rather than reading the entire file into memory. (No longer grossly 
> inefficient IMHO.)
> 
> I propose:
> - Deprecate the "show_indexes" parameter of static.serve() (unless people 
> actually use it).
> - Have people report security issues to secur...@djangoproject.com (like 
> always)
> - Audit the code and possibly add more security checks and tests.
> - add wsgi.file_wrapper support to responses (5-line proof of concept: 
> https://github.com/django/django/pull/3650 )
> - support serving static files in production, but still recommend 
> nginx/apache or a cdn for performance.
> - make serving static files in production an opt-in, but put the view in 
> project_template/project_name/urls.py
> 
> I think it's a huge win for low-traffic sites or sites in the "just trying to 
> deploy and get something live" phase. You can always optimize later by 
> serving via nginx or cdn.
> We already have the views, api, and logic around for finding and serving the 
> correct files.
> We can be just as efficient and secure as static/dj-static without needing to 
> make people install and configure wsgi middleware to the application.
> We could have staticfiles classes implement more complicated features like 
> giving cache recommendations, and serving pre-gzipped files.
> 
> Is this a good idea? I realize it's not totally thought through. I'm fine 
> with waiting until 1.9 if needed.
> 
> Collin
> 
> On Saturday, November 29, 2014 6:07:05 PM UTC-5, Collin Anderson wrote:
> Hi All,
> 
> I think doing something here is really good idea. I'm happy with any of the 
> solutions mentioned so far.
> 
> My question is: what does static/dj-static do that our built-in code doesn't 
> do? What makes it more secure? It seems to me we're only missing is 
> wsgi.file_wrapper and maybe a few more security checks. Why don't we just 
> make our own code secure and start supporting it?
> Here's basic wsgi.file_wrapper support: 
> https://github.com/django/django/pull/3650
> 
> We could then, over time, start supporting more extensions ourselves: ranges, 
> pre-gziped files, urls with never-changing content, etc. That way we get 
> very, very deep django integration. It seems to me this is a piece that a web 
> framework should be able to support itself.
> 
> Collin
> 
> 
> On Friday, November 28, 2014 9:15:03 AM UTC-5, Tim Graham wrote:
> Berker has worked on integrating gunicorn with runserver so that we might be 
> able to deprecate our own homegrown webserver. Windows support for gunicorn 
> is supposedly coming soon which may actually make the idea feasible. This way 
> we provide a more secure solution out of the box (anecdotes indicate that 
> runserver is widely used in production despite our documentation warnings 
> against doing so).
> 
> On the pull request, Anssi had an idea to use dj-static to serve static/media 
> files. My understanding is that we would basically integrate the code for 
> dj-static into Django and then add a dependency on static. It could be an 
> optional dependency since you might want to serve static files differently in 
> production, but it would likely be more or less universally used in 
> development. We could then say that django.views.static.serve (and its 
> counterpart in staticfiles) is okay to use in production (and I guess people 
> are already using them in production despite our warnings that they are not 
> hardened for production use).
> 
> What do you think of this plan?
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/bfc1eb0d-8b69-4450-bfe0-7147ef729317%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-develope

Re: delegating our static file serving

[Aymeric Augustin]

Yes, I support this plan.

"Serve your files with nginx!" doesn't fly in the age of PaaS.

Serving static files with Django and having a CDN cache them is a
reasonable setup as far as I know.

I don't know if the "probably insecure" argument still holds. Are there
some specific security risks in serving files that don't exist in serving
dynamic content? I'd say dynamic content is more difficult. This main
problem I can imagine when serving files is directory traversal. We already
have protections against this class of attacks in several features. (Usual
disclaimer: my background is security is mostly theoretical.)

-- 
Aymeric.


2014-12-05 6:33 GMT+01:00 Collin Anderson :

> Hi All,
>
> I'm pretty interested in getting secure and _somewhat_ efficient static
> file serving in Django.
>
> Quick history:
> 2005 - Jacob commits #428: a "static pages" view.  Note that this view
> should only be used for testing!"
> 2010 - Jannis adds staticfiles. Serving via django is considered "grossly
> inefficient and probably insecure".
> 2011 - Graham Dumpleton adds wsgi.file_wrapper to Gunicorn.
> 2012 - Aymeric adds StreamingHttpResponse and now files are read in chunks
> rather than reading the entire file into memory. (No longer grossly
> inefficient IMHO.)
>
> I propose:
> - Deprecate the "show_indexes" parameter of static.serve() (unless people
> actually use it).
> - Have people report security issues to secur...@djangoproject.com (like
> always)
> - Audit the code and possibly add more security checks and tests.
> - add wsgi.file_wrapper support to responses (5-line proof of concept:
> https://github.com/django/django/pull/3650 )
> - support serving static files in production, but still recommend
> nginx/apache or a cdn for performance.
> - make serving static files in production an opt-in, but put the view in
> project_template/project_name/urls.py
>
> I think it's a huge win for low-traffic sites or sites in the "just trying
> to deploy and get something live" phase. You can always optimize later by
> serving via nginx or cdn.
> We already have the views, api, and logic around for finding and serving
> the correct files.
> We can be just as efficient and secure as static/dj-static without needing
> to make people install and configure wsgi middleware to the application.
> We could have staticfiles classes implement more complicated features like
> giving cache recommendations, and serving pre-gzipped files.
>
> Is this a good idea? I realize it's not totally thought through. I'm fine
> with waiting until 1.9 if needed.
>
> Collin
>
> On Saturday, November 29, 2014 6:07:05 PM UTC-5, Collin Anderson wrote:
>>
>> Hi All,
>>
>> I think doing something here is really good idea. I'm happy with any of
>> the solutions mentioned so far.
>>
>> My question is: what does static/dj-static do that our built-in code
>> doesn't do? What makes it more secure? It seems to me we're only missing is
>> wsgi.file_wrapper and maybe a few more security checks. Why don't we just
>> make our own code secure and start supporting it?
>> Here's basic wsgi.file_wrapper support: https://github.com/django/
>> django/pull/3650
>>
>> We could then, over time, start supporting more extensions ourselves:
>> ranges, pre-gziped files, urls with never-changing content, etc. That way
>> we get very, very deep django integration. It seems to me this is a piece
>> that a web framework should be able to support itself.
>>
>> Collin
>>
>>
>> On Friday, November 28, 2014 9:15:03 AM UTC-5, Tim Graham wrote:
>>>
>>> Berker has worked on integrating gunicorn with runserver
>>>  so that we might be able
>>> to deprecate our own homegrown webserver. Windows support for gunicorn is
>>> supposedly coming soon which
>>> may actually make the idea feasible. This way we provide a more secure
>>> solution out of the box (anecdotes indicate that runserver is widely used
>>> in production despite our documentation warnings against doing so).
>>>
>>> On the pull request, Anssi had an idea to use dj-static
>>>  to serve static/media
>>> files. My understanding is that we would basically integrate the code for
>>> dj-static into Django and then add a dependency on static
>>> . It could be an optional
>>> dependency since you might want to serve static files differently in
>>> production, but it would likely be more or less universally used in
>>> development. We could then say that django.views.static.serve (and its
>>> counterpart in staticfiles) is okay to use in production (and I guess
>>> people are already using them in production despite our warnings that they
>>> are not hardened for production use).
>>>
>>> What do you think of this plan?
>>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To uns

Re: delegating our static file serving

[Collin Anderson]

Hi All,

I'm pretty interested in getting secure and _somewhat_ efficient static 
file serving in Django.

Quick history:
2005 - Jacob commits #428: a "static pages" view.  Note that this view 
should only be used for testing!"
2010 - Jannis adds staticfiles. Serving via django is considered "grossly 
inefficient and probably insecure".
2011 - Graham Dumpleton adds wsgi.file_wrapper to Gunicorn.
2012 - Aymeric adds StreamingHttpResponse and now files are read in chunks 
rather than reading the entire file into memory. (No longer grossly 
inefficient IMHO.)

I propose:
- Deprecate the "show_indexes" parameter of static.serve() (unless people 
actually use it).
- Have people report security issues to secur...@djangoproject.com (like 
always)
- Audit the code and possibly add more security checks and tests.
- add wsgi.file_wrapper support to responses (5-line proof of concept: 
https://github.com/django/django/pull/3650 )
- support serving static files in production, but still recommend 
nginx/apache or a cdn for performance.
- make serving static files in production an opt-in, but put the view in 
project_template/project_name/urls.py

I think it's a huge win for low-traffic sites or sites in the "just trying 
to deploy and get something live" phase. You can always optimize later by 
serving via nginx or cdn.
We already have the views, api, and logic around for finding and serving 
the correct files.
We can be just as efficient and secure as static/dj-static without needing 
to make people install and configure wsgi middleware to the application.
We could have staticfiles classes implement more complicated features like 
giving cache recommendations, and serving pre-gzipped files.

Is this a good idea? I realize it's not totally thought through. I'm fine 
with waiting until 1.9 if needed.

Collin

On Saturday, November 29, 2014 6:07:05 PM UTC-5, Collin Anderson wrote:
>
> Hi All,
>
> I think doing something here is really good idea. I'm happy with any of 
> the solutions mentioned so far.
>
> My question is: what does static/dj-static do that our built-in code 
> doesn't do? What makes it more secure? It seems to me we're only missing is 
> wsgi.file_wrapper and maybe a few more security checks. Why don't we just 
> make our own code secure and start supporting it?
> Here's basic wsgi.file_wrapper support: 
> https://github.com/django/django/pull/3650
>
> We could then, over time, start supporting more extensions ourselves: 
> ranges, pre-gziped files, urls with never-changing content, etc. That way 
> we get very, very deep django integration. It seems to me this is a piece 
> that a web framework should be able to support itself.
>
> Collin
>
>
> On Friday, November 28, 2014 9:15:03 AM UTC-5, Tim Graham wrote:
>>
>> Berker has worked on integrating gunicorn with runserver 
>>  so that we might be able to 
>> deprecate our own homegrown webserver. Windows support for gunicorn is 
>> supposedly coming soon which 
>> may actually make the idea feasible. This way we provide a more secure 
>> solution out of the box (anecdotes indicate that runserver is widely used 
>> in production despite our documentation warnings against doing so).
>>
>> On the pull request, Anssi had an idea to use dj-static 
>>  to serve static/media files. 
>> My understanding is that we would basically integrate the code for 
>> dj-static into Django and then add a dependency on static 
>> . It could be an optional 
>> dependency since you might want to serve static files differently in 
>> production, but it would likely be more or less universally used in 
>> development. We could then say that django.views.static.serve (and its 
>> counterpart in staticfiles) is okay to use in production (and I guess 
>> people are already using them in production despite our warnings that they 
>> are not hardened for production use).
>>
>> What do you think of this plan?
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/bfc1eb0d-8b69-4450-bfe0-7147ef729317%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: delegating our static file serving

[David Evans]

It's worth flagging up that part of what makes WhiteNoise good for serving 
static files in production also makes it not quite as good for using in 
development.

Most file servers work by taking the requested URL, constructing a local 
path from it, and then checking whether a file exists at that particular 
path. WhiteNoise on the other hand, searches for all available static files 
on startup and then builds a dictionary of files which it assumes won't 
change until the code is reloaded. This has a couple of big advantages: one 
is security -- there's a whole class of path-traversal vulnerabilities that 
I don't have to worry about because I never have to generate a path from a 
user-supplied URL. The other is that WhiteNoise can serve files from 
arbitrary URLs, rather than just within a fixed prefix like `/static/`, 
without adding an extra hit on the filesystem for every single request that 
comes in.

While this works well in production, it breaks down in development where 
the static files change without causing a reload of the server. I've 
started getting bug reports from users who are trying to use WhiteNoise in 
development and are getting confused by this.

So, while I'd obviously be delighted to see some of WhiteNoise's features 
merged into core I think there will need to be a slightly different 
approach for serving files in development.

Dave

On Saturday, 29 November 2014 23:07:05 UTC, Collin Anderson wrote:
>
> Hi All,
>
> I think doing something here is really good idea. I'm happy with any of 
> the solutions mentioned so far.
>
> My question is: what does static/dj-static do that our built-in code 
> doesn't do? What makes it more secure? It seems to me we're only missing is 
> wsgi.file_wrapper and maybe a few more security checks. Why don't we just 
> make our own code secure and start supporting it?
> Here's basic wsgi.file_wrapper support: 
> https://github.com/django/django/pull/3650
>
> We could then, over time, start supporting more extensions ourselves: 
> ranges, pre-gziped files, urls with never-changing content, etc. That way 
> we get very, very deep django integration. It seems to me this is a piece 
> that a web framework should be able to support itself.
>
> Collin
>
>
> On Friday, November 28, 2014 9:15:03 AM UTC-5, Tim Graham wrote:
>>
>> Berker has worked on integrating gunicorn with runserver 
>>  so that we might be able to 
>> deprecate our own homegrown webserver. Windows support for gunicorn is 
>> supposedly coming soon which 
>> may actually make the idea feasible. This way we provide a more secure 
>> solution out of the box (anecdotes indicate that runserver is widely used 
>> in production despite our documentation warnings against doing so).
>>
>> On the pull request, Anssi had an idea to use dj-static 
>>  to serve static/media files. 
>> My understanding is that we would basically integrate the code for 
>> dj-static into Django and then add a dependency on static 
>> . It could be an optional 
>> dependency since you might want to serve static files differently in 
>> production, but it would likely be more or less universally used in 
>> development. We could then say that django.views.static.serve (and its 
>> counterpart in staticfiles) is okay to use in production (and I guess 
>> people are already using them in production despite our warnings that they 
>> are not hardened for production use).
>>
>> What do you think of this plan?
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/a4efc639-ee61-4c6b-88b4-2866f86bdc24%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: delegating our static file serving

[Collin Anderson]

Hi All,

I think doing something here is really good idea. I'm happy with any of the 
solutions mentioned so far.

My question is: what does static/dj-static do that our built-in code 
doesn't do? What makes it more secure? It seems to me we're only missing is 
wsgi.file_wrapper and maybe a few more security checks. Why don't we just 
make our own code secure and start supporting it?
Here's basic wsgi.file_wrapper support: 
https://github.com/django/django/pull/3650

We could then, over time, start supporting more extensions ourselves: 
ranges, pre-gziped files, urls with never-changing content, etc. That way 
we get very, very deep django integration. It seems to me this is a piece 
that a web framework should be able to support itself.

Collin


On Friday, November 28, 2014 9:15:03 AM UTC-5, Tim Graham wrote:
>
> Berker has worked on integrating gunicorn with runserver 
>  so that we might be able to 
> deprecate our own homegrown webserver. Windows support for gunicorn is 
> supposedly coming soon which 
> may actually make the idea feasible. This way we provide a more secure 
> solution out of the box (anecdotes indicate that runserver is widely used 
> in production despite our documentation warnings against doing so).
>
> On the pull request, Anssi had an idea to use dj-static 
>  to serve static/media files. 
> My understanding is that we would basically integrate the code for 
> dj-static into Django and then add a dependency on static 
> . It could be an optional dependency 
> since you might want to serve static files differently in production, but 
> it would likely be more or less universally used in development. We could 
> then say that django.views.static.serve (and its counterpart in 
> staticfiles) is okay to use in production (and I guess people are already 
> using them in production despite our warnings that they are not hardened 
> for production use).
>
> What do you think of this plan?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/e3be33a3-3675-4397-b017-e0565bda84fb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: delegating our static file serving

[Jannis Leidel]

> On 28 Nov 2014, at 15:15, Tim Graham  wrote:
> 
> Berker has worked on integrating gunicorn with runserver so that we might be 
> able to deprecate our own homegrown webserver. Windows support for gunicorn 
> is supposedly coming soon which may actually make the idea feasible. This way 
> we provide a more secure solution out of the box (anecdotes indicate that 
> runserver is widely used in production despite our documentation warnings 
> against doing so).
> 
> On the pull request, Anssi had an idea to use dj-static to serve static/media 
> files. My understanding is that we would basically integrate the code for 
> dj-static into Django and then add a dependency on static. It could be an 
> optional dependency since you might want to serve static files differently in 
> production, but it would likely be more or less universally used in 
> development. We could then say that django.views.static.serve (and its 
> counterpart in staticfiles) is okay to use in production (and I guess people 
> are already using them in production despite our warnings that they are not 
> hardened for production use).
> 
> What do you think of this plan?

I don't think using dj-static is a good idea (after having fixed a few things 
myself in it). If we'd want to do some form of "simpler" file serving we should 
be going with whitenoise (http://whitenoise.evans.io/) because:

- it has a deeper integration with Django (e.g. knows about the manifest 
storage backend in staticfiles)
- has a more flexible API than the Cling based dj-static for custom paths like 
file uploads and more
- works great with CDN/load balancer backed deployments via far-future cache 
headers for content that doesn't change
- supports serving gzipped content when wanted

To me this is currently the one-stop-solution when using staticfiles and I 
would be stoked to see it or its techniques be adopted.

Jannis

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/9B11AB1B-2850-401D-97BA-FB7C73268672%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: delegating our static file serving

[Carl Meyer]

On 11/28/2014 07:15 AM, Tim Graham wrote:
> Berker has worked on integrating gunicorn with runserver
>  so that we might be able to
> deprecate our own homegrown webserver. Windows support for gunicorn is
> supposedly coming soon
> which may actually make
> the idea feasible. This way we provide a more secure solution out of the
> box (anecdotes indicate that runserver is widely used in production
> despite our documentation warnings against doing so).
> 
> On the pull request, Anssi had an idea to use dj-static
>  to serve static/media files.
> My understanding is that we would basically integrate the code for
> dj-static into Django and then add a dependency on static
> . It could be an optional
> dependency since you might want to serve static files differently in
> production, but it would likely be more or less universally used in
> development. We could then say that django.views.static.serve (and its
> counterpart in staticfiles) is okay to use in production (and I guess
> people are already using them in production despite our warnings that
> they are not hardened for production use).
> 
> What do you think of this plan?

I think it's a good plan.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5478A567.2060703%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature

delegating our static file serving

[Tim Graham]

Berker has worked on integrating gunicorn with runserver 
 so that we might be able to 
deprecate our own homegrown webserver. Windows support for gunicorn is 
supposedly coming soon which 
may actually make the idea feasible. This way we provide a more secure 
solution out of the box (anecdotes indicate that runserver is widely used 
in production despite our documentation warnings against doing so).

On the pull request, Anssi had an idea to use dj-static 
 to serve static/media files. My 
understanding is that we would basically integrate the code for dj-static 
into Django and then add a dependency on static 
. It could be an optional dependency 
since you might want to serve static files differently in production, but 
it would likely be more or less universally used in development. We could 
then say that django.views.static.serve (and its counterpart in 
staticfiles) is okay to use in production (and I guess people are already 
using them in production despite our warnings that they are not hardened 
for production use).

What do you think of this plan?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5f1efe14-5858-4fb0-ac39-7f4fd582cd31%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.