Re: State of X-Sendfile support?

[Gabriel Hurley]

FWIW, I vaguely recall how the last thread on X-sendfile and the files API 
got conflated (derailed?) and as far as I understood it then and as far as I 
understand it now, they're related only because some backends (which we 
currently don't directly support) are difficult to work with without such 
conflation. That doesn't mean we can't solve one problem at a time, however. 
Especially when one problem *is* simple, and the other is not.

They're both good goals, and problems worth solving, but IMHO there's no 
reason that they couldn't be two separate features.

+1 on a simple solution for X-sendfile, +0 on making the file API more 
compatible with a broader range of backends, -1 for solving both issues 
together.

- Gabriel

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: State of X-Sendfile support?

[Jacob Kaplan-Moss]

On Mon, Mar 28, 2011 at 1:13 PM, Waldemar Kornewald
 wrote:
> That's a good goal and as long as you only focus on file downloads
> it's possible to reuse the middlewares setting. However, if you ever
> want to provide an abstract file uploads API we're back to the same
> problem.

I'm trying to talk about downloads and you keep bringing up uploads. I
still don't get why they're at all related. Am I just an idiot? What
am I missing?

> Doesn't your proposal require a change to the admin?

Does it? I shouldn't think so. The admin doesn't need to use
X-Sendfile and friends.

Either way, I think we're going 'round in circles here. I hear that
you think this problem is more complex that I think it is, and I hope
you hear that I don't. I'm going to remain -1 on your patch until
someone demonstrates that the simplistic approach won't work.

Jacob

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: State of X-Sendfile support?

[Waldemar Kornewald]

On Mon, Mar 28, 2011 at 6:40 PM, Jacob Kaplan-Moss  wrote:
> On Mon, Mar 28, 2011 at 11:22 AM, Waldemar Kornewald
>  wrote:
>> I do agree that it's too complicated (esp., the forms) and I plan to
>> improve django-filetransfers in this regard. The biggest complexity
>> comes largely from file upload handling (which I understand isn't a
>> problem you're trying to solve right now). If your main goal is to
>> only handle private file downloads then django-filetransfers already
>> provides a serve_file() view which does exactly that (the only
>> difference is that it doesn't use a middleware-based solution).
>
> Right, it introduces Yet Another Setting, which I'm really trying to
> discourage. It's not your fault at all, but I'm going to look very
> skeptically upon any patch that introduces new settings without a
> great reason. I'm really trying to avoid setting creep: Django 1.1 had
> 87 settings, Django 1.2 had 123, and Django 1.3 has 137. This has to
> stop.

That's a good goal and as long as you only focus on file downloads
it's possible to reuse the middlewares setting. However, if you ever
want to provide an abstract file uploads API we're back to the same
problem.

> More importantly, your approach, again, *requires* a patch to Django;
> my proposed one doesn't. This is a good thing since it'll allow anyone
> using older versions of Django access to the feature. If I'm thinking
> about it right, a HttpResponse + middleware approach could work all
> the way back to 1.0. (Maybe?)

Doesn't your proposal require a change to the admin? My patch is no
different. If it weren't for the admin my patch wouldn't be necessary
(the remaining changes are just cosmetic). Maybe the admin should just
provide an easy way to write custom code for file handling, so
3rd-party file handling apps can provide a FileModelAdmin that deals
with file uploads/downloads?

>> However, there are other use-cases and hosting setups. For instance,
> [...]
>
> Look, you're probably right. But I'd prefer to try the simple way
> first and only do something complicated if the simple approach doesn't
> work. Your approach, to my eyes, mixes a lot of unrelated stuff
> together. If I've got that wrong, you need to explain to me (and
> anyone else) why uploads and downloads belong together in the same
> patch and why a simple "just support X-Sendfile and friends" patch
> can't possibly work.

It's just a matter of providing a complete solution. With some
services (e.g., GAE Blobstore, Amazon S3 POST) file uploads and
downloads are tightly coupled. Only providing a download mechanism
wouldn't be sufficient because you still can't upload those files in
the first place. So, when you use those services the simple solution
already doesn't work and I hope that this won't just be ignored for
the sake of simplicity.

Bye,
Waldemar

-- 
Django on App Engine, MongoDB, ...? Browser-side Python? It's open-source:
http://www.allbuttonspressed.com/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: State of X-Sendfile support?

[Christophe Pettus]

On Mar 28, 2011, at 9:40 AM, Jacob Kaplan-Moss wrote:

> If I've got that wrong, you need to explain to me (and
> anyone else) why uploads and downloads belong together in the same
> patch and why a simple "just support X-Sendfile and friends" patch
> can't possibly work.

+1.  It's entirely possible my brain is three sizes too small, but I don't see 
the obvious correlation between X-Sendfile (speaking generically) and the other 
features in the patch.  This is a very, very useful feature, but not one that 
has an obvious home in the Django core, especially given the varying 
implementations for specific environments.

--
-- Christophe Pettus
   x...@thebuild.com

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: State of X-Sendfile support?

[Jacob Kaplan-Moss]

On Mon, Mar 28, 2011 at 11:22 AM, Waldemar Kornewald
 wrote:
> I do agree that it's too complicated (esp., the forms) and I plan to
> improve django-filetransfers in this regard. The biggest complexity
> comes largely from file upload handling (which I understand isn't a
> problem you're trying to solve right now). If your main goal is to
> only handle private file downloads then django-filetransfers already
> provides a serve_file() view which does exactly that (the only
> difference is that it doesn't use a middleware-based solution).

Right, it introduces Yet Another Setting, which I'm really trying to
discourage. It's not your fault at all, but I'm going to look very
skeptically upon any patch that introduces new settings without a
great reason. I'm really trying to avoid setting creep: Django 1.1 had
87 settings, Django 1.2 had 123, and Django 1.3 has 137. This has to
stop.

More importantly, your approach, again, *requires* a patch to Django;
my proposed one doesn't. This is a good thing since it'll allow anyone
using older versions of Django access to the feature. If I'm thinking
about it right, a HttpResponse + middleware approach could work all
the way back to 1.0. (Maybe?)

> However, there are other use-cases and hosting setups. For instance,
[...]

Look, you're probably right. But I'd prefer to try the simple way
first and only do something complicated if the simple approach doesn't
work. Your approach, to my eyes, mixes a lot of unrelated stuff
together. If I've got that wrong, you need to explain to me (and
anyone else) why uploads and downloads belong together in the same
patch and why a simple "just support X-Sendfile and friends" patch
can't possibly work.

Jacob

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: State of X-Sendfile support?

[Waldemar Kornewald]

On Mon, Mar 28, 2011 at 5:05 PM, Jacob Kaplan-Moss  wrote:
> On Sat, Mar 26, 2011 at 8:45 AM, Waldemar Kornewald
>  wrote:
>> That's pretty much exactly what django-filetransfers tries to do on
>> the download side:
>> http://www.allbuttonspressed.com/projects/django-filetransfers
>> Hotever it's not only for X-Sendfile, but also for any other file
>> serving mechanisms and it's not only for downloads, but also for
>> uploads. There's also a corresponding patch (which still needs lots of
>> work) for including that functionality in Django
>> (http://code.djangoproject.com/ticket/13960).
>>
>> Is this what you were thinking of?
>
> Maybe... but there's a *lot* more going on in that patch that I don't
> understand. There seems to be no documentation so I'm having to puzzle
> through the patch -- all this prepare_upload stuff, backends,
> serve_file indirection just confuses me. Not to mention this new "form
> traits" mechanism I don't quite get, either.
>
> I think the big difference between my proposal and this patch is that
> my way should be doable without a patch to Django. That is, no new
> setting, no internals changes, etc. -- it should all be possible with
> a HttpResponse subclass and a set of middleware. I'm not quite sure
> what all the additionally complication your patch gains.

I do agree that it's too complicated (esp., the forms) and I plan to
improve django-filetransfers in this regard. The biggest complexity
comes largely from file upload handling (which I understand isn't a
problem you're trying to solve right now). If your main goal is to
only handle private file downloads then django-filetransfers already
provides a serve_file() view which does exactly that (the only
difference is that it doesn't use a middleware-based solution).

However, there are other use-cases and hosting setups. For instance,
you might want publicly accessible uploads (e.g., an image for an
article) to be downloaded directly from an external file server (e.g.
S3) instead of going through the Django instance (which would only
redirected to S3, anyway). You might even combine two different file
hosting solutions (e.g., one for public videos served via a CDN and
the other for private gallery uploads). It would be nice to have an
abstract API that can handle all this, so (even for internal apps) one
doesn't have to maintain several forks with custom file
upload/download handling code. The patch attempts to solve this and
django-filetransfers does the same as a separate app (with the
disadvantage that the admin doesn't use it). Both solutions have lots
of room for improvement, though.

Bye,
Waldemar

-- 
Django on App Engine, MongoDB, ...? Browser-side Python? It's open-source:
http://www.allbuttonspressed.com/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: State of X-Sendfile support?

[Jacob Kaplan-Moss]

On Sat, Mar 26, 2011 at 8:45 AM, Waldemar Kornewald
 wrote:
> That's pretty much exactly what django-filetransfers tries to do on
> the download side:
> http://www.allbuttonspressed.com/projects/django-filetransfers
> Hotever it's not only for X-Sendfile, but also for any other file
> serving mechanisms and it's not only for downloads, but also for
> uploads. There's also a corresponding patch (which still needs lots of
> work) for including that functionality in Django
> (http://code.djangoproject.com/ticket/13960).
>
> Is this what you were thinking of?

Maybe... but there's a *lot* more going on in that patch that I don't
understand. There seems to be no documentation so I'm having to puzzle
through the patch -- all this prepare_upload stuff, backends,
serve_file indirection just confuses me. Not to mention this new "form
traits" mechanism I don't quite get, either.

I think the big difference between my proposal and this patch is that
my way should be doable without a patch to Django. That is, no new
setting, no internals changes, etc. -- it should all be possible with
a HttpResponse subclass and a set of middleware. I'm not quite sure
what all the additionally complication your patch gains.

Jacob

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: State of X-Sendfile support?

[Graham Dumpleton]

On Monday, March 28, 2011 9:15:45 PM UTC+11, Gustavo Narea wrote:
>
> On 26/03/11 11:31, Graham Dumpleton wrote:
> >
> > Yes and no as nginx also has a X-Sendfile module, again possibly
> > optional (can't remember).
>
> I didn't know there was an X-Sendfile module for Nginx -- Search results
> for "nginx xsendfile" point me to X-Accel-Redirect
>

You could be right and there isn't a X-Sendfile for module for nginx. As I 
said I couldn't remember. :-) 

> >
> > Add to this wsgi.file_wrapper and you have three different schemes one
> > can use. For the case of URL mechanism, there also various headers and
> > consumers of the headers.
>
> I think wsgi.file_wrapper is a different beast. I would not try to
> support it along with X-Sendfile/X-Accel-Redirect/Location.
>

But wsgi.file_wrapper, and its graceful fallback as documented in WSGI PEP, 
is the only one actually guaranteed to work without a silent failure. Ie., 
you are always guaranteed to get a response still where as with the others, 
if no front end web server actually supports it or is enabled properly, then 
you just end up with an empty response and the header passed back to the 
client.

Thus it actually potentially makes much more sense to have the default be to 
take a filesystem path and then use wsgi.file_wrapper to return content. 
This could then optionally be overridden to use one of the other mechanisms, 
with a user supplying a mapping function as necessary to translate the file 
system path into a URL in the case of Location, X-Accel-Redirect or the 
headers used by the other front end load balancing products. Conceptually, 
the configuration mechanism could even support the concept of certain file 
system path prefixes being served up in different ways by different parts of 
the front end infrastructure. This configuration could be defined as part of 
the Django settings file for a project.

Graham 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: State of X-Sendfile support?

[Gustavo Narea]

On 26/03/11 11:31, Graham Dumpleton wrote:
>
> Yes and no as nginx also has a X-Sendfile module, again possibly
> optional (can't remember).

I didn't know there was an X-Sendfile module for Nginx -- Search results
for "nginx xsendfile" point me to X-Accel-Redirect.


> In Apache/mod_wsgi when using daemon mode you have CGI style
> 200/Location to use as well.
>
> So, you have a mix of file system path and URL based schemes. In the
> case of file system based schemes, because the code processing
> X-Sendfile is potentially a different process, then you also have to
> deal with issues of that other process not having the required
> permissions to access the file even though the original application did.
>
> For URL based schemes, nginx has an easy way to designate the URLs
> which are the target of X-Accel-Redirect are private and only
> accessible from a sub request triggered by the header. For Apache when
> using 200/Location, you have to use a mod_rewrite rule to effect
> private URLs.
>
> Other proxy/load balancing front ends such a Pound and Squid (???)
> also support their own headers for this sort of URL sub requests.
>
> Add to this wsgi.file_wrapper and you have three different schemes one
> can use. For the case of URL mechanism, there also various headers and
> consumers of the headers.

I think wsgi.file_wrapper is a different beast. I would not try to
support it along with X-Sendfile/X-Accel-Redirect/Location.


> In short, it is all a mess and trying to provide support for it in one
> bit of code is possibly asking a bit much.

I agree with you. However, I think trying to offer some kind of common
abstraction may still be beneficial.

Cheers.

-- 
Gustavo Narea.
Software Developer.
2degrees, Ltd. .

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: State of X-Sendfile support?

[Waldemar Kornewald]

On Sat, Mar 26, 2011 at 2:08 PM, Jacob Kaplan-Moss  wrote:
> On Sat, Mar 26, 2011 at 6:31 AM, Graham Dumpleton
>  wrote:
>> In short, it is all a mess and trying to provide support for it in one bit
>> of code is possibly asking a bit much.
>
> One possible solution would be to split the problem up a bit. Django
> could provide an HttpFileResponse object. This would use the new lazy
> response mechanism in 1.3 to render the file locally (by doing a naive
> open/read) on demand. But then we could also ship a collection of
> middleware classes that know how to translate a HttpFileResponse for
> each various backend (FileResponseToXSendFile, FileResponseToXAccell,
> etc). Users could then select the correct piece of middleware for
> their upstream server or write their own for ones not supported out of
> the box.
>
> The nice thing about this approach is that it could be explored as a
> third-party app and made available to anyone using 1.3 or above.
>
> Can anyone see a reason why this wouldn't work? Want to throw some
> code together and give it a try?

That's pretty much exactly what django-filetransfers tries to do on
the download side:
http://www.allbuttonspressed.com/projects/django-filetransfers
Hotever it's not only for X-Sendfile, but also for any other file
serving mechanisms and it's not only for downloads, but also for
uploads. There's also a corresponding patch (which still needs lots of
work) for including that functionality in Django
(http://code.djangoproject.com/ticket/13960).

Is this what you were thinking of?

Bye,
Waldemar

-- 
Django on App Engine, MongoDB, ...? Browser-side Python? It's open-source:
http://www.allbuttonspressed.com/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: State of X-Sendfile support?

[Jacob Kaplan-Moss]

On Sat, Mar 26, 2011 at 6:31 AM, Graham Dumpleton
 wrote:
> In short, it is all a mess and trying to provide support for it in one bit
> of code is possibly asking a bit much.

One possible solution would be to split the problem up a bit. Django
could provide an HttpFileResponse object. This would use the new lazy
response mechanism in 1.3 to render the file locally (by doing a naive
open/read) on demand. But then we could also ship a collection of
middleware classes that know how to translate a HttpFileResponse for
each various backend (FileResponseToXSendFile, FileResponseToXAccell,
etc). Users could then select the correct piece of middleware for
their upstream server or write their own for ones not supported out of
the box.

The nice thing about this approach is that it could be explored as a
third-party app and made available to anyone using 1.3 or above.

Can anyone see a reason why this wouldn't work? Want to throw some
code together and give it a try?

Jacob

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: State of X-Sendfile support?

[Graham Dumpleton]

On Saturday, March 26, 2011 9:15:48 PM UTC+11, Gustavo Narea wrote:
>
>  On 26/03/11 00:17, Graham Dumpleton wrote: 
>
>  Why guess in the first place? Apache and Nginx both support 'X-Sendfile:
>> ' don't they? (older nginx seemed to only support their
>> own syntax, though).
>>
> Apache requires a separate module to be installed which isn't part of the 
> core, so no you can't rely on it being present.
>
>
> Indeed. Plus, the meaning of the path you give to the server has a totally 
> different meaning in Nginx, compared to that Apache module.
>

Yes and no as nginx also has a X-Sendfile module, again possibly optional 
(can't remember).

In Apache/mod_wsgi when using daemon mode you have CGI style 200/Location to 
use as well.

So, you have a mix of file system path and URL based schemes. In the case of 
file system based schemes, because the code processing X-Sendfile is 
potentially a different process, then you also have to deal with issues of 
that other process not having the required permissions to access the file 
even though the original application did.

For URL based schemes, nginx has an easy way to designate the URLs which are 
the target of X-Accel-Redirect are private and only accessible from a sub 
request triggered by the header. For Apache when using 200/Location, you 
have to use a mod_rewrite rule to effect private URLs.

Other proxy/load balancing front ends such a Pound and Squid (???) also 
support their own headers for this sort of URL sub requests.

Add to this wsgi.file_wrapper and you have three different schemes one can 
use. For the case of URL mechanism, there also various headers and consumers 
of the headers.

One of the Django tickets about this has an explanation about all the 
various options.

In short, it is all a mess and trying to provide support for it in one bit 
of code is possibly asking a bit much.

Graham

 

>
> In that Apache module and Lighttpd, the path set in the "X-Sendfile" header 
> is a path on the filesystem. In Nginx (header "X-Accel-Redirect") and CGI 
> (heder "Location"), the path is a URL path within the same host.
>
> -- 
> Gustavo Narea.
> Software Developer.
> 2degrees, Ltd.  
> .
>
>  

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: State of X-Sendfile support?

[Gustavo Narea]

On 26/03/11 00:17, Graham Dumpleton wrote:
>
> Why guess in the first place? Apache and Nginx both support
> 'X-Sendfile:
> ' don't they? (older nginx seemed to only support their
> own syntax, though).
>
> Apache requires a separate module to be installed which isn't part of
> the core, so no you can't rely on it being present.

Indeed. Plus, the meaning of the path you give to the server has a
totally different meaning in Nginx, compared to that Apache module.

In that Apache module and Lighttpd, the path set in the "X-Sendfile"
header is a path on the filesystem. In Nginx (header "X-Accel-Redirect")
and CGI (heder "Location"), the path is a URL path within the same host.

-- 
Gustavo Narea.
Software Developer.
2degrees, Ltd. .

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: State of X-Sendfile support?

[Gustavo Narea]

On 25/03/11 14:50, Thomas Guettler wrote:
> Maybe you are right. Guessing is bad. But I think to write "nginx" into
> the application code is bad, too. Something like this could be in settings.py.

I think you're diverging: My point is simply to illustrate how to use
wsgi-xsendfile. You wouldn't hard code the path to the file either, and
yet that's how I've done it in my example.

-- 
Gustavo Narea.
Software Developer.
2degrees, Ltd. .

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: State of X-Sendfile support?

[Graham Dumpleton]

On Friday, March 25, 2011 11:43:39 PM UTC+11, Pascal Germroth wrote:
>
> On Fri, 2011-03-25 at 12:55 +0100, Łukasz Rekucki wrote:
> > On 25 March 2011 12:34, Thomas Guettler  wrote:
> > > I look at your example code. You need to name the webserver in the
> > > code. That's not very nice. I guess it should be possible to
> > > guess the webserver (apache vs nginx) by looking at request.META.
> > >
> > 
> > -1 on guessing anything, especially something that doesn't change.
> > Also, a fairly common configuration is Apache behind an nginx proxy.
> > In this setup, I want nginx will be the one serving static files, not
> > Apache.
>
> Why guess in the first place? Apache and Nginx both support 'X-Sendfile:
> ' don't they? (older nginx seemed to only support their
> own syntax, though).
>
Apache requires a separate module to be installed which isn't part of the 
core, so no you can't rely on it being present.

Graham

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: State of X-Sendfile support?

[Thomas Guettler]

On 25.03.2011 12:55, Łukasz Rekucki wrote:
> On 25 March 2011 12:34, Thomas Guettler  wrote:
>> Hi,
>>
>> I look at your example code. You need to name the webserver in the
>> code. That's not very nice. I guess it should be possible to
>> guess the webserver (apache vs nginx) by looking at request.META.
>>
> 
> -1 on guessing anything, especially something that doesn't change.
> Also, a fairly common configuration is Apache behind an nginx proxy.
> In this setup, I want nginx will be the one serving static files, not
> Apache.

Maybe you are right. Guessing is bad. But I think to write "nginx" into
the application code is bad, too. Something like this could be in settings.py.

  Thomas

-- 
Thomas Guettler, http://www.thomas-guettler.de/
E-Mail: guettli (*) thomas-guettler + de

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: State of X-Sendfile support?

[Pascal Germroth]

On Fri, 2011-03-25 at 12:55 +0100, Łukasz Rekucki wrote:
> On 25 March 2011 12:34, Thomas Guettler  wrote:
> > I look at your example code. You need to name the webserver in the
> > code. That's not very nice. I guess it should be possible to
> > guess the webserver (apache vs nginx) by looking at request.META.
> >
> 
> -1 on guessing anything, especially something that doesn't change.
> Also, a fairly common configuration is Apache behind an nginx proxy.
> In this setup, I want nginx will be the one serving static files, not
> Apache.

Why guess in the first place? Apache and Nginx both support 'X-Sendfile:
' don't they? (older nginx seemed to only support their
own syntax, though).

-- 
pascal

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: State of X-Sendfile support?

[Łukasz Rekucki]

On 25 March 2011 12:34, Thomas Guettler  wrote:
> Hi,
>
> I look at your example code. You need to name the webserver in the
> code. That's not very nice. I guess it should be possible to
> guess the webserver (apache vs nginx) by looking at request.META.
>

-1 on guessing anything, especially something that doesn't change.
Also, a fairly common configuration is Apache behind an nginx proxy.
In this setup, I want nginx will be the one serving static files, not
Apache.


-- 
Łukasz Rekucki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: State of X-Sendfile support?

[Thomas Guettler]

Hi,

I look at your example code. You need to name the webserver in the
code. That's not very nice. I guess it should be possible to
guess the webserver (apache vs nginx) by looking at request.META.

Documentation would be nice

  Thomas

On 25.03.2011 10:23, Gustavo Narea wrote:
> Hi all,
> 
> Just to let you know that there's an X-Sendfile implementation for
> WSGI apps (inc. Django), which also works with Nginx:
> https://launchpad.net/wsgi-xsendfile
> 
> You can use it in Django views via twod.wsgi. For example:
> 
> """
> from twod.wsgi import call_wsgi_app
> from xsendfile import NginxSendfile
> 
> file_sender = NginxSendfile('/var/www/docs')
> file_sender_wsgi_app = XSendfileApplication('/media/documents',
> file_sender)
> 
> def django_view(request, article_id):
> article = Article.objects.get(article_id)
> if it's foggy:
> doc_response = call_wsgi_app(
> file_sender_wsgi_app,
> request,
> article.path,
> )
> else:
> doc_response = HttpResponseForbidden()
> 
> return doc_response
> """
> 
> That package is pretty stable, the only thing it's missing is
> documentation which should be sorted soon.
> 
> Cheers.
> 
>  - Gustavo.


-- 
Thomas Guettler, http://www.thomas-guettler.de/
E-Mail: guettli (*) thomas-guettler + de

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: State of X-Sendfile support?

[Gustavo Narea]

Hi all,

Just to let you know that there's an X-Sendfile implementation for
WSGI apps (inc. Django), which also works with Nginx:
https://launchpad.net/wsgi-xsendfile

You can use it in Django views via twod.wsgi. For example:

"""
from twod.wsgi import call_wsgi_app
from xsendfile import NginxSendfile

file_sender = NginxSendfile('/var/www/docs')
file_sender_wsgi_app = XSendfileApplication('/media/documents',
file_sender)

def django_view(request, article_id):
article = Article.objects.get(article_id)
if it's foggy:
doc_response = call_wsgi_app(
file_sender_wsgi_app,
request,
article.path,
)
else:
doc_response = HttpResponseForbidden()

return doc_response
"""

That package is pretty stable, the only thing it's missing is
documentation which should be sorted soon.

Cheers.

 - Gustavo.

On Mar 25, 6:50 am, Waldemar Kornewald  wrote:
> On Thursday, March 24, 2011 2:40:39 PM UTC+1, Kristaps Kūlis wrote:
>
> > I wish to note that Nginx implements this feature differently than
> > LigHTTPd and Apache2
> >http://wiki.nginx.org/XSendfile,
>
> > Should django implementation consider that ?
>
> > My proposal to implement would be:
> >  1. HttpFileResponse which takes file location (relative to MEDIA_URL ?)
> >  2. HttpFileResponse checks for settings.X_SENDFILE or
> > settings.X_ACCEL_REDIRECT and modifies sets revelant headers
> > (Content-Type, X-Sendfile, X-Accel-Redirect) etc. HttpFileResponse
> > should fallback to outputting file if no accelerated redirect is
> > available.
> >  3. Update docs, showing example server config .
> >  4. Tests
>
> > I could provide patch, if design idea is ok .
>
> You might also be interested in this ticket which goes a little bit further
> in abstracting file uploads and downloads not only for X-Sendfile, but also
> S3, App Engine, and other file hosting/serving 
> solutions:http://code.djangoproject.com/ticket/13960
>
> The patch is not great. It's too complicated, both implementation-wise (esp.
> in django.forms) and usability-wise (esp. the transfer_id stuff which is
> used for assigning backends to uploads/downloads).
>
> For the sake of completeness, here is a separate app with the same 
> purpose:http://www.allbuttonspressed.com/projects/django-filetransfers
> The API can certainly be simplified some more and I plan to do that probably
> in May.
>
> I hope some of this is useful to you.
>
> Bye,
> Waldemar
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: State of X-Sendfile support?

[Waldemar Kornewald]

On Thursday, March 24, 2011 2:40:39 PM UTC+1, Kristaps Kūlis wrote:
>
> I wish to note that Nginx implements this feature differently than
> LigHTTPd and Apache2
> http://wiki.nginx.org/XSendfile ,
>
> Should django implementation consider that ?
>
> My proposal to implement would be:
>  1. HttpFileResponse which takes file location (relative to MEDIA_URL ?)
>  2. HttpFileResponse checks for settings.X_SENDFILE or
> settings.X_ACCEL_REDIRECT and modifies sets revelant headers
> (Content-Type, X-Sendfile, X-Accel-Redirect) etc. HttpFileResponse
> should fallback to outputting file if no accelerated redirect is
> available.
>  3. Update docs, showing example server config .
>  4. Tests
>
> I could provide patch, if design idea is ok .
>
You might also be interested in this ticket which goes a little bit further 
in abstracting file uploads and downloads not only for X-Sendfile, but also 
S3, App Engine, and other file hosting/serving solutions:
http://code.djangoproject.com/ticket/13960

The patch is not great. It's too complicated, both implementation-wise (esp. 
in django.forms) and usability-wise (esp. the transfer_id stuff which is 
used for assigning backends to uploads/downloads).

For the sake of completeness, here is a separate app with the same purpose:
http://www.allbuttonspressed.com/projects/django-filetransfers
The API can certainly be simplified some more and I plan to do that probably 
in May.

I hope some of this is useful to you.

Bye,
Waldemar

>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: State of X-Sendfile support?

[Russell Keith-Magee]

On Thu, Mar 24, 2011 at 9:40 PM, Kristaps Kūlis
 wrote:
> I wish to note that Nginx implements this feature differently than
> LigHTTPd and Apache2
> http://wiki.nginx.org/XSendfile ,
>
> Should django implementation consider that ?
>
> My proposal to implement would be:
>  1. HttpFileResponse which takes file location (relative to MEDIA_URL ?)
>  2. HttpFileResponse checks for settings.X_SENDFILE or
> settings.X_ACCEL_REDIRECT and modifies sets revelant headers
> (Content-Type, X-Sendfile, X-Accel-Redirect) etc. HttpFileResponse
> should fallback to outputting file if no accelerated redirect is
> available.
>  3. Update docs, showing example server config .
>  4. Tests
>
> I could provide patch, if design idea is ok .

The fact that there are multiple interpretations and headers for this
general feature is well known to the core team, and if you search the
django-dev archives and the relevant tickets (esp #2131), there have
been extensive discussions about how this should be handled.

Yours,
Russ Magee %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: State of X-Sendfile support?

[David De La Harpe Golden]

On 24/03/11 13:40, Kristaps Kūlis  wrote:
> I wish to note that Nginx implements this feature differently than
> LigHTTPd and Apache2
> http://wiki.nginx.org/XSendfile ,
> 
> Should django implementation consider that ?
>

I'd also note that wsgi itself apparently has an api abstraction
wsgi.file_wrapper [1]

I really don't know much about it seeing as directly using apache's
x-sendfile works anyway (and we started out on fcgi then moved later to
wsgi) (at least in cases where we hadn't been storing our files in
postgresql bytea fields anyway...)

But for clean layering, django should maybe go via that in the wsgi case
if you're adding any explicit django-level support for such things.

[1]
http://www.python.org/dev/peps/pep-0333/#optional-platform-specific-file-handling

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: State of X-Sendfile support?

[Kristaps Kūlis]

I wish to note that Nginx implements this feature differently than
LigHTTPd and Apache2
http://wiki.nginx.org/XSendfile ,

Should django implementation consider that ?

My proposal to implement would be:
 1. HttpFileResponse which takes file location (relative to MEDIA_URL ?)
 2. HttpFileResponse checks for settings.X_SENDFILE or
settings.X_ACCEL_REDIRECT and modifies sets revelant headers
(Content-Type, X-Sendfile, X-Accel-Redirect) etc. HttpFileResponse
should fallback to outputting file if no accelerated redirect is
available.
 3. Update docs, showing example server config .
 4. Tests

I could provide patch, if design idea is ok .

On Thu, Mar 24, 2011 at 1:49 PM, Russell Keith-Magee
 wrote:
> On Wed, Mar 23, 2011 at 5:20 AM, Paul McMillan  wrote:
>> It's worth pointing out that manage.py runserver isn't likely to ever
>> support serving those files even if the header arrives in core. It's
>> explicitly not appropriate for any kind of production use, fallback or
>> otherwise. If you really need support for your development work (and
>> checking the headers on the response isn't good enough), you should be
>> developing in an environment closer to your production server.
>>
>> I would be very much -1 to any proposal to include code that actually
>> served the file in django core if the webserver doesn't support
>> X-Sendfile.
>>
>> That said, I'd very much like to see support for the header in core.
>> It's commonly used, and reimplementing it every time is silly.
>
> Agreed. We just need someone to do the work. Did I hear the sound of
> someone volunteering? :-)
>
> Yours,
> Russ Magee %-)
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Django developers" group.
> To post to this group, send email to django-developers@googlegroups.com.
> To unsubscribe from this group, send email to 
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/django-developers?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: State of X-Sendfile support?

[Russell Keith-Magee]

On Wed, Mar 23, 2011 at 5:20 AM, Paul McMillan  wrote:
> It's worth pointing out that manage.py runserver isn't likely to ever
> support serving those files even if the header arrives in core. It's
> explicitly not appropriate for any kind of production use, fallback or
> otherwise. If you really need support for your development work (and
> checking the headers on the response isn't good enough), you should be
> developing in an environment closer to your production server.
>
> I would be very much -1 to any proposal to include code that actually
> served the file in django core if the webserver doesn't support
> X-Sendfile.
>
> That said, I'd very much like to see support for the header in core.
> It's commonly used, and reimplementing it every time is silly.

Agreed. We just need someone to do the work. Did I hear the sound of
someone volunteering? :-)

Yours,
Russ Magee %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: State of X-Sendfile support?

[Paul McMillan]

It's worth pointing out that manage.py runserver isn't likely to ever
support serving those files even if the header arrives in core. It's
explicitly not appropriate for any kind of production use, fallback or
otherwise. If you really need support for your development work (and
checking the headers on the response isn't good enough), you should be
developing in an environment closer to your production server.

I would be very much -1 to any proposal to include code that actually
served the file in django core if the webserver doesn't support
X-Sendfile.

That said, I'd very much like to see support for the header in core.
It's commonly used, and reimplementing it every time is silly.

-Paul

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: State of X-Sendfile support?

[Russell Keith-Magee]

On Sunday, March 20, 2011, Pascal Germroth  wrote:
> Hi,
>
> I want to use X-Sendfile to serve some medium-sized files. There
> doesn't seem to be built-in support for this, although the soc2009/
> http-wsgi-improvements branch looks like somebody attempted to
> implement HttpResponseSendFile (although it seems to use the slow
> path).
>
> Building a proper response manually is not the problem (since the
> apache extension does the content-type guessing and length
> calculation) but I would like an automatic fall-back so `manage.py
> runserver` doesn't put out an empty response leaking the header but
> serves the file like a proper server should.
>
> So: is there a class that detects the server and uses the X-Sendfile
> header if supported, or a branch where core.handlers.wsgi.WSGIHandler
> supports X-Sendfile?

At present, there isn't any support for X-Sendfile. It is a feature
that has been discussed at many times in the past, including the
GSOC2009; however, nobody has ever carried it the last mile and got it
into trunk.

So; if you're interested, this is an area where you could make a contribution.

Yours,
Russ Magee %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

State of X-Sendfile support?

[Pascal Germroth]

Hi,

I want to use X-Sendfile to serve some medium-sized files. There
doesn't seem to be built-in support for this, although the soc2009/
http-wsgi-improvements branch looks like somebody attempted to
implement HttpResponseSendFile (although it seems to use the slow
path).

Building a proper response manually is not the problem (since the
apache extension does the content-type guessing and length
calculation) but I would like an automatic fall-back so `manage.py
runserver` doesn't put out an empty response leaking the header but
serves the file like a proper server should.

So: is there a class that detects the server and uses the X-Sendfile
header if supported, or a branch where core.handlers.wsgi.WSGIHandler
supports X-Sendfile?


cheers,

--
pascal

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.