Bug#655154: woof: No space left on device when /tmp is full
tags 655154 + upstream
forwarded 655154 si...@budig.de
thanks
Hello,
The attached patch fixes this problem. I have forwarded it to the
upstream maintainer. Here follows the patch description:
,
| Improve handling of files received with option -U
|
| Instead of storing the downloaded contents into a temporary file
| that is copied by woof to the final destination (current directory),
| the uploaded file is directly written to the current directory, with
| precautions to avoid overwriting existing files.
|
| This is done by subclassing cgi.FieldStorage to override make_file() so
| that it does not create a temporary file when encountering an atomic
| part whose name is "upfile" in the MIME data.
|
| This improvement is particularly welcome when receiving large files,
| since it avoids:
| - a possible failure due to the temporary directory being too small to
| hold the file;
| - a useless copy of the whole file;
| - a possible loss of the whole download in case a failure occurs when
| woof copies the (complete) temporary file to its final destination.
`
For people who don't want to apply the patch, setting TMPDIR allows a
partial workaround: you may indicate a directory with more space than
/tmp this way, but you will still have to hold two copies of the
downloaded file at the same time (one in TMPDIR and the other in the
directory from which 'woof -U' was started), until woof deletes the
temporary file through garbage collection. Depending on the file size
and the available space, this may or may not be possible.
--
Florent
diff --git a/woof b/woof
index 2abedf0..fde8de1 100755
--- a/woof
+++ b/woof
@@ -124,6 +124,64 @@ class ForkingHTTPServer (BaseHTTPServer.HTTPServer):
self.close_request (request)
+class MyFieldStorage (cgi.FieldStorage):
+ """Subclass of FieldStorage with efficient handling of uploaded files.
+
+ This class behaves the same as FieldStorage except that it handles uploaded
+ files more efficiently: instead of storing the data into a temporary file
+ that is copied by woof to the final destination (current directory),
+ the uploaded file is directly written to the current directory, with
+ precautions to avoid overwriting existing files.
+
+ """
+ # The function signature changed between Python 2 and Python 3; this will
+ # work in all cases.
+ def make_file(self, *args, **kwargs):
+ # The "self.list is None" check is here to prevent misbehaving in case we
+ # receive a multipart whose name happens to be "upfile".
+ if self.list is None and self.name == "upfile":
+ # make_file() was called for an uploaded file. No need to create a
+ # potentially huge temporary file only to copy it to the final
+ # destination. Just create the file in a way that avoids overwriting
+ # existing files.
+ upfilename = self.filename
+
+ if "\\" in upfilename:
+upfilename = upfilename.split ("\\")[-1]
+
+ upfilename = os.path.basename (self.filename)
+
+ destfile = None
+ for suffix in ["", ".1", ".2", ".3", ".4", ".5", ".6", ".7", ".8",
+".9"]:
+destfilename = os.path.join (upfilename + suffix)
+try:
+ destfile = os.open (destfilename,
+ os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o644)
+ break
+except OSError, e:
+ if e.errno == errno.EEXIST:
+ continue
+ raise
+
+ # Fallback to tempfile.mkstemp() in case the above did not manage to
+ # create the file.
+ if destfile is None:
+destfile, destfilepath = tempfile.mkstemp (
+ prefix = upfilename + ".", dir = ".")
+# 'destfilepath' is a full path, be consistent with the normal case
+destfilename = os.path.basename (destfilepath)
+
+ msg = "Accepting uploaded file: '%s'" % upfilename
+ if upfilename != destfilename:
+msg += ", stored as '%s'" % destfilename
+ sys.stderr.write (msg + "...\n")
+
+ return os.fdopen (destfile, "wb")
+ else:
+ return cgi.FieldStorage.make_file (self, *args, **kwargs)
+
+
# Main class implementing an HTTP-Requesthandler, that serves just a single
# file and redirects all other requests to this file (this passes the actual
# filename to the client).
@@ -152,49 +210,26 @@ class FileServHTTPRequestHandler (BaseHTTPServer.BaseHTTPRequestHandler):
# http://mail.python.org/pipermail/python-list/2006-September/402441.html
ctype, pdict = cgi.parse_header (self.headers.getheader ('Content-Type'))
- form = cgi.FieldStorage (fp = self.rfile,
- headers = self.headers,
- environ = {'REQUEST_METHOD' : 'POST'},
- keep_blank_values = 1,
- strict_parsing = 1)
+ fo
Bug#655154: woof: No space left on device when /tmp is full
Hi Arne, thanks for the report, and sorry for delay as well. The problem you raise, is matter of how the python standard library works. I guess the problem can be ignored at all (probably writing something smarter instead of the complete traceback), or an option can be added to specify the temp dir while uploading. I'll bring this to the upstream (not responsive at all, frankly) and to a couple of other people already helped with woof. Thanks again. -- - Andrea Gasparini - -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Bug#655154: woof: No space left on device when /tmp is full
Package: woof
Version: 20091227-2
Severity: normal
Dear Maintainer,
today woof threw exceptions while receiving an uploading file.
* What led up to the situation?
While using the upload-function, uploading to a machine with a 50MB
/tmp (my router) i encountered "No space left on device" exception.
woof was started in a directory where enough space was provided for
the expected data and (smaller) testuploads succeeded.
* What was the outcome of this action?
Exception happened during processing of request from ('__.__.__.__', 49201)
Traceback (most recent call last):
File "/usr/lib/python2.6/SocketServer.py", line 560, in process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python2.6/SocketServer.py", line 322, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.6/SocketServer.py", line 617, in __init__
self.handle()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 329, in handle
self.handle_one_request()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 323, in handle_one_request
method()
File "/usr/bin/woof", line 153, in do_POST
strict_parsing = 1)
File "/usr/lib/python2.6/cgi.py", line 508, in __init__
self.read_multi(environ, keep_blank_values, strict_parsing)
File "/usr/lib/python2.6/cgi.py", line 637, in read_multi
environ, keep_blank_values, strict_parsing)
File "/usr/lib/python2.6/cgi.py", line 510, in __init__
self.read_single()
File "/usr/lib/python2.6/cgi.py", line 647, in read_single
self.read_lines()
File "/usr/lib/python2.6/cgi.py", line 669, in read_lines
self.read_lines_to_outerboundary()
File "/usr/lib/python2.6/cgi.py", line 720, in read_lines_to_outerboundary
self.__write(odelim + line)
File "/usr/lib/python2.6/cgi.py", line 679, in __write
self.file.write(line)
IOError: [Errno 28] No space left on device
* What outcome did you expect instead?
I would have expected that woof does not use /tmp to cache large files. A cache
in the destination directory seems more appropriate.
Yours,
arne wichmann
-- System Information:
Debian Release: wheezy/sid
APT prefers testing
APT policy: (990, 'testing'), (500, 'unstable'), (500, 'stable'), (500,
'oldstable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Kernel: Linux 3.1.0-1-amd64 (SMP w/2 CPU cores)
Locale: LANG=C, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Versions of packages woof depends on:
ii python 2.6.7-3
woof recommends no packages.
woof suggests no packages.
-- no debconf information
--
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
