I have been building an "ExternalFile" class which stores the body of
the file in an external file, mirroring the Zope path/hierarchy. This
will allow easy integration with servers that can mount the external
representation of the content and serve it with a consistent namespace.
To make life zimple, I tried to move all file manipulation to Zope,
including upload/download/copy/cut/paste/delete and permissions. These
external files are transaction aware, blah blah..
Working with files > 20MB I notices some serious performance/scalability
issues and investigated. Here are the results.
A diff with my changes against version 2.2.2 is available at
<http://www.superchannel.org/Playground/large_file_zope2.2.2_200010241.diff>
Concerns:
Zope objects like File require data as a seekable file or as a
coherent block, rather than as a stream. Initializing/updating
these objects *may* require loading the entire file into memory.
In memory buffering of request or response data could cause
excessive swapping of the working set.
Multi-service architecture (ZServer->ZPublisher) could limit the
reuse of stream handles.
Creating temporary files as FIFOs buffers between the services
causes signficant swapping.
Modifications:
Using pipes I found that FTPServer.ContentCollector was using a
StringIO to buffer the uploads from FTP clients. I changed this
into a TemporaryFile for a while which revealed the leaked file
descriptor bug (see below). This intermediary temp file caused 1
extra file copy for each request. The goal is to not have any
intermediary files at all, and pipeline the content directly into
the Zope objects.
To remove this FTP upload file buffer, I converted the FTP collector
again from a TemporaryFile into a pipe with a reader and writer file
objects. The FTPRequest receives the reader from which it can
process the input on the publish thread in processInputs.
Since we are dealing with blocking pipes it is OK to have a reader
on the publish thread and a writer on the ZServer thread. The major
considerations were regarding the proper way to read from a pipe
through the chain of control, especially in cgi.FieldStorage.
Stdin is treated as the reader of the pipe throughout the code. All
seek()s and tell()s on sys.stdin type objects (a tty not a seekable
file) should be considered illegal and removed.
Usage of FieldStorage from FTP (Unknown content-length)
To gain access to the body of a request, one typically calls
REQUEST['BODY'] or REQUEST['BODYFILE']. This returns the file
object the FieldStorage copied from stdin.
To prevent FieldStorage from copying the file from stdin to a
temporary file, we can set the CONTENT_LENGTH header to '0' in the
FTP _get_env for a STOR.
In this case, FieldStorage creates a temporary file but doesn't read
any data from stdin so we can return stdin directly when BODYFILE is
requested and 'content-length' is '0'. However, BODYFILE could be a
pipe which doesn't support 'seek' or 'tell'. The code used to suck
the data off the BODYFILE needs to be modified to adapt to the
possibly of being passed a pipe.
Updating Image.File to play with pipes
The _read_data method of Image.File pulls the data out of the
BODYFILE and sticks it in the instance as a string, pdata object, or
a linked list of pdata objects. The existing code reads and builds
the list in one clean sweep back-to-front. I belive this keeps the
pdata.data chunks out of memory, quickly (sub)committing then
deactivating (_p_changed = None) them.
Since we can no longer safely assume 'seek' is valid for BODYFILE, I
tried to read and build the list front-to-back. This kept the data
in memory, even though I tried to deactivate the objects quickly.
As a tradeoff, I read the data front-to-back then built the list
back-to-front taking another pass to reverse the list so it is in
the correct order.
Memory usage appears to be steady, meaning the whole file is not
loaded into the working set. This also prevents unecessary reading
into a temporary FieldStorage file during an FTP upload.
Web based uploads...
...suck. I do not recommend doing a web based upload for files
> 1mb. First, a content-length is known, so we don't get the
advantage of pipelining the data directly from the socket, a
temporary file must be created, written and read. Second, I believe
the content is encoded so the transferred bitcount is much higher
than using FTP.
Plus, most browsers today do not support a progress bar for posts,
so there is no indication of status, causing most people to click
'Upload' multiple times.
I haven't done any optimization for this case, but have tested that
is still works properly.
Cleaning up (leaked file descriptor bug)
I noticed that when uploading 20+ MB a couple of times, I ran out of
hard drive space. This didn't make sense and I looked into what
files were open by Zope. Doing an 'lsof' I found that the temporary
files which are immediatly unlinked after creation, were still open
until the end of the Zope process. These files (created by
tempfile.TemporaryFile) needed to be closed after the end of the
REQUEST and RESPONSE, rather than at the end of the Zope process.
After publishing, the close method of the REQUEST gets called. Here
I added closing of stdin and the FieldStorage created TemporaryFile
'_file'.
Output producers
The ZServer.HTTPResponse object makes a good attempt of keeping
large results out of memory but does so by creating a temporary file
and copying any written data to it then pushing a file_part_producer
onto the channel output queue.
If the Zope object knows how to produce the data themselves, they
could push producer(s) directly to the channel. I added a single
check in ZServer.HTTPResponse(256) where a temporary file is only
created if the data is larger than the in-memory buffer *and*
doesn't already look like a producer with 'more' as a method.
If the temporary file doesn't exist the rest of the code simply
writes the data to the channel and the channel produces the output
directly from the producer created by the Zope object.
Using a file producer from my Zope object cuts out a file copy, and
those get expensive when one is dealing with 20+ MB files. The
response time is also dramatically reduced because the file copy
step before streaming to the client was removed.
I would like to apply the same concept to Image.File.index_html
where rather than creating a temporary file in the RESPONSE to queue
the contents, create a producer to pull the data directly out of the
backend when it is ready to write. I am experiencing a 10 second
latency (233Mhz laptop) between requesting a 10MB file and receiving
the first byte with the current code. If an output producer is
used, this latency would drop < 1 sec.
I made an attempt to create a pdata_producer but failed because of
ZODB errors reloading the object. I get a traceback like:
2000-10-24T09:19:08 ERROR(200) ZODB Couldn't load state for
'\000\000\000\000\000\000&\370' Traceback (innermost last): File
/usr/local/zope/lib/python/ZODB/Connection.py, line 442, in setstate
AttributeError: 'None' object has no attribute 'load'
My hunch is that the Image, pdata_producer or pdata object gets
deactivated and can't find its DB to load itself. I tried setting a
_p_jar on the pdata_producer, but I don't really know what happens
when the object context leaves publish_module. Since the object
activation happens in the ZServer thread, some voodoo may be needed
to get the proper state in the pdata_producer.... any takers?
Testing...
I have only tested these changes with FTPServer and HTTPServer, not
PCGIServer or FCGIServer.
I have tested round-trip coherency because of the change in
Image.File._read_data.
I haven't completely tested boundary conditions, where
Image.File._read_data makes descisions. The extent has been large
files 10+ MB and small files < 64K.
I haven't tested HTTPRequest.retry which will probably fail because
HTTPRequest.stdin now may be a pipe.
3rd party products which treat BODYFILE as a seekable file object
may fail during FTP uploads.
Summary:
Most of these efforts are geared towards FTP, as HTTP form uploads
don't seem to be worth the effort.
I haven't taken a look at HTTP PUT, for webdav clients etc...
Similar pipelining could be used, however I doubt they would be
possible without modifing cgi.FieldStorage.
Zope seems to be doing a lot with TempStorage and other ZODB magic I
didn't care about checking out. Some performance improvements could
be included here.
FTP I/O with my changes including my ExternalFile custom output
producer dramatically increases Zopes performance and scalability.
-Sean
_______________________________________________
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
