#13960: abstract file upload/download handling
-------------------------------------------+--------------------------------
Reporter: wkornewald | Owner: nobody
Status: new | Milestone: 1.3
Component: File uploads/storage | Version: SVN
Resolution: | Keywords:
Stage: Unreviewed | Has_patch: 1
Needs_docs: 0 | Needs_tests: 0
Needs_better_patch: 0 |
-------------------------------------------+--------------------------------
Comment (by wkornewald):
OK, I think your suggestion does have valid points, so here are a few
thoughts:
The upload preparation process (see prepare_upload() function) is most
probably tightly 1:1-coupled with FileUploadHandler. IOW, two upload
backends will never share the same upload preparation process (unless it
needs no preparation, at all). So, as a first step, what do you think
about moving prepare_upload() to FileUploadHandler?
The "post-upload -> storage" process has an m:n mapping from
FileUploadHandler to Storage. At least the tempfile and the in-memory
upload handlers are both used with different storage handlers (file
system, MongoDB, S3, etc.). Merging those two backends into one backend
wouldn't make sense because it would lead to an unnecessary explosion in
the number of backends (m*n combinations), so FileUploadHandler and
Storage would stay two different backends. This already separates the file
handling process into two parts.
The download process consists of these parts:
1. serving the file via a Django view
1. generating a download URL which can either
1. point to a Django view (always the case for non-public downloads)
1. point to a direct publicly accessible download URL (not all backends
support this, so a fallback to 2.1 must be supported)
Part (1) might actually be handled by the storage backend, but there's no
tight 1:1 coupling between those two. Obviously, one storage mechanism can
have different file serving mechanisms which means it's at least 1:n. The
important question is: Can one file serving mechanism also apply to
multiple storage mechanisms (i.e.: m:n)? A download mechanism that streams
the download via HttpResponse might be shared between multiple storage
mechanisms which means that many backends would have to be provided in two
or more variations (FileStorageStreaming, FileStorageXSendfile,
S3Streaming, S3Redirect, etc.) and hopefully those variations are
compatible with each other, so you can replace your streaming download
with XSendfile without having to migrate the data stored in your
FileFields. Also, while Django could provide streaming downloads in the
base backend it would lead to unnecessary code duplication if another
download mechanism is shared between multiple storage backends (e.g.,
redirect to MEDIA_URL, if that makes sense). I have strong doubts that
it's a good idea to combine the file serving mechanism with Storage.
What's the benefit of combining those backends, anyway? Removing one entry
from settings.py? The risk is that we're overlooking something and at some
point the combination of both backends has more disadvantages than the
advantages that come with a *minor* settings simplification.
Case (2) is about handling public and private download URLs. Many projects
will have both types of downloads. Some files are publicly accessible and
thus should be served directly and efficiently without bothering a Django
instance. Other files are are only accessible by certain users, so those
downloads have to be handled by a Django view which can check permissions
and do other fancy stuff before passing on to the backend (X-Sendfile or a
streaming HttpResponse, etc.). Since in many cases both types of downloads
(public and private) need to be supported we'll need to specify different
download_url() and prepare_upload()/FileUploadHandler backend combinations
for different apps or models. Almost every storage option provides one or
more solutions for private downloads and one or more solutions for public
downloads. Again, this will lead to an explosion of combinations.
There is also the possibility to combine upload and download backends, but
since almost every storage solution will work with at least two upload
mechanisms and sometimes three or more download mechanisms I'm sure nobody
wants to have six or more backends per storage mechanism to handle all
possible combinations.
Finally, the TransferBackend API also provides a mechanism for selecting
the Storage backend for a FileField without hard-coding it in your
models.py. Specifying only one backend isn't sufficient because we need to
support public and private uploads and in some cases these are even on
different hosting solutions (e.g. for performance reasons or whatever).
This mechanism can't be added to Storage because it doesn't really make
sense for Storage to be responsible for selecting which Storage should be
used. So, even if we combine all of the other APIs into Storage we're
still left with a TransferBackend which selects the Storage.
That's why I don't really see how we can combine TransferBackend with
Storage. Only the prepare_upload() function might need to be moved from
TransferBackend to FileUploadHandler. Well, I'm open for suggestions if
you know a better design. Also, apart from this adding a few more
*optional* entries to settings.py (in simple cases you won't need a
TransferBackend), what's the big issue with this approach, anyway?
--
Ticket URL: <http://code.djangoproject.com/ticket/13960#comment:6>
Django <http://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To post to this group, send email to django-upda...@googlegroups.com.
To unsubscribe from this group, send email to
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-updates?hl=en.
