On Sat, Feb 21, 2009 at 12:45 PM, Chris Anderson <[email protected]> wrote:
> On Sat, Feb 21, 2009 at 10:10 AM, Jeff Hinrichs - DM&T > <[email protected]> wrote: > > > > Google can't seem to help me locate an example of doing a PUT w/ chunked > > transfer for httplib2 -- does anyone have any pointers? > > > > Any luck PUTing attachments with chunked encoding? You'll need to > create the document first, with a PUT, and then create the attachment, > with a chunked PUT. No not yet, I have limited experience with http underpinnings so I am looking for examples on the interweb. Creating the document and then putting the attachments would create an 'unwanted'?? revision during a dump/load cycle -- I am not sure if that is a bad thing or not -- since I was hoping to get it loaded back into the db in one shot. > > It should be possible to avoid non-buffered non-chunked standalone > attachment PUTs now as well, based on my recent patch to Mochiweb. > Implementing that would be pretty simple, just a matter of adapting > the attachment writing code to handle it. I haven't experienced any difficulties with put_attachment() and some of my attachments are over 8MB. Still trying to figure out how put_attachment() which uses self.request.put() and the normal __setitem__ which also uses self.request.put() method differ since put_attachment() can deal with mochiweb's default limit of 1MB while __setitem__ can not. > > > There's not much that can be done about memory usage for inlined > base64 attachments. It sounds like that is what you are using for > load. Yep, I am trying to modify the existing code for couchdb-dump/load to output each as an individual mime object then I am marshalling the individual objects into a zip file -- that part is working. The problem is when I try to db[docid]=doc a doc that exceeds mochiweb's body limit. > > In the long run, I'm hoping that a form of the all_docs view can be > useful for dump and load. Eg, something like > > curl http://localhost:5984/olddb/_all_docs?include_everything=true > > backup.json > > followed by > > curl -X POST -T backup.json http://localhost:5984/newdb/_bulk_docs > > Would be all that is needed for dump and load. I'm not sure how close > we are to this yet, but it sure seems like the simplest way. That would be outstanding. I can't +1 it enough. I would suggest that it should be available before the next release that causes a dump/reload cycle due to changes in rev or other db incompatible changes. Kind of surprised that there isn't something like this already. Otherwise there is no way to migrate an existing database with attachments of more than modest size between machines -- replication of documents with large attachments fails. As I am sure that they new _rev format is going to make it into .9 which is going to require a dump/load. Kind of a rock/hard place for me. Currently, it looks as if hacking mochiweb's default body limit is the only viable solution until I can get chunked transfer working in the couchdb-python module. ;( yick. I am confused on the settings in couchdb for max_document_size and max_attachment_size -- those are in effect after mochiweb has received the data, correct? so neither is a solution for the dilemma at hand. Any possible way to make src/mochiweb/mochiweb_request.erl > -define(MAX_RECV_BODY, (1024*1024)) modifiable via futon's configuration panel? -- I don't know if this is a good idea -- chances are someone has thought hard about it and it is a good setting already. hopefully this will interest cmlenz -- and he will be able to add to the conversation ;) > > Chris > > -- > Chris Anderson > http://jchris.mfdz.com >
