On Fri, Jun 10, 2011 at 07:17, Shane Hathaway <sh...@hathawaymix.org> wrote: > I see your thinking now. RelStorage will download multiple chunks from > Oracle but will now only upload one chunk per blob to Oracle. If others > want to do the same for PostgreSQL or MySQL, you've set the example of > how to do it without disruption. > > Thank you for not changing the schema. I get a lot of complaints these > days anytime the schema changes. I ask just one favor: please run some > manual tests with multi-gigabyte blobs that don't fit in RAM.
I've committed my refactor now. In testing I ran into some limitations of cx_Oracle; it uses the older (pre 10.1) OCI APIs that only allow offsets and sizes of up to 4GB (e.g. using 32-bit unsigned integers). Moreover, it's naive use of signed integers instead of unsigned ones means that on 32-bit platforms the largest *LOB write and read offsets it can handle is 2GB - 1 byte (sys.maxint). I've seen that the author is already aware that there is a newer API that upgrades the offset and size types to 64-bit unsigned integers, and I've asked him what the status is of cx_Oracle supporting that. In the meantime, to answer your request for a favour: Uploading and downloading blobs within these limits worked great and memory usage wasn't impacted at all. For us these limits are absolutely not a problem. Although our Oracle setup is performing very nicely when it comes to BLOB reading and writing (multiple MBs per second), pulling a BLOB of several GB to a file in your local non-shared blob cache before serving is going to take too long anyway, and you really want to use a shared blob storage for that instead. I've only tested the Oracle blob upload and download methods in the mover, since that's what changed in this commit. I didn't run the full test suite as I lacked the required SYS access on the oracle cluster and didn't have time to set up a local Oracle database. If you could run the test suite against your Oracle setup that'd be great. :-) Next I'll look into adding PostgreSQL support too; it has a nicer API still in that it let's you specify files directly to upload from or download to. -- Martijn Pieters _______________________________________________ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
