One little trick we've found useful on these kind of loosely coupled file transfer architectures is to transmit a small file AFTER the big file transfer is complete to indicate that file transfer of the big file(s) is complete. Lots of ftp servers will create the file in the target directory and start filling it with bytes. This partially transferred file is then visible to other applications that are monitoring the directory. Lots of opportunities for race conditions between your code that is monitoring the directory for new files to ingest and the (s)ftp server process transferring the contents of the file, so checking on just the existence of the transfer file is not a good approach.
thx, g --- George Florentine VP, Engineering +1 (303) 542-2173 | Office +1 (303) 669-8628 | Cell +1 (303) 544-0522 | Fax george.florent...@flatironssolutions.com http://www.flatironssolutions.com -----Original Message----- From: general-boun...@developer.marklogic.com [mailto:general-boun...@developer.marklogic.com] On Behalf Of Demian Hess Sent: Tuesday, February 07, 2012 1:59 PM To: General MarkLogic Developer Discussion Subject: Re: [MarkLogic Dev General] Can app server handle file uploads >2GB? I think FTP is the best choice for this. In regard to monitoring a directory: is there an xdmp function for getting a listing of files and directories from the file system? I didn't see that in the API documents. Thanks, Demian On Tue, Feb 7, 2012 at 2:04 PM, Geert Josten <geert.jos...@dayon.nl> wrote: > Hi Demian, > > My guess at it is that MarkLogic would try to maintain the binary file as > request param within memory, which is limited to some size for good > reason. With 10+ concurrent requests, all uploading 2 Gb, memory would > build up very rapidly. I'm guessing the limits are related to the limits > within xdmp:document-get, which does not allow reading files larger than > 16Mb or 64Mb (depending on 32/64-bit install). > > I think I wouldn't advice to use form upload to send large files across. > Other protocols like (S)FTP are better suited for that. You could allow > such files to be uploaded to file locations through a separate (S)FTP > server on the same host. You can let MarkLogic monitor that directory > using a Scheduled Task, and perhaps leveraging Information Studio flows. > In these processes you can use different function to load such large > files, though I am not sure they are able to handle 2+Gb files.. > > Kind regards, > Geert > >> -----Oorspronkelijk bericht----- >> Van: general-boun...@developer.marklogic.com [mailto:general- >> boun...@developer.marklogic.com] Namens Demian Hess >> Verzonden: dinsdag 7 februari 2012 19:45 >> Aan: general@developer.marklogic.com >> Onderwerp: [MarkLogic Dev General] Can app server handle file uploads >>2GB? >> >> We would like to store large binary files (many EPS and high >> resolution PDFs) in MarkLogic. It would be easiest for users to create >> zips with all their files and then upload via an HTML form >> (enctype=multipart/form-data), but the size of the zip archives would >> be more than 2GB. Indeed, some individual files can get to 2GB by >> themselves. >> >> I tried POSTing a 2GB file to an app server and got a "413 Request >> Entity Too Large" error. I don't see any settings in MarkLogic to >> control max size of uploads for HTTP app servers. >> >> How efficient would ML be at handling large posts of 100MB and larger? >> Would it buffer everything in memory or would it stream to a temporary >> file on disk? Would ML 5.0 be more efficient than 4.0 at handling >> large file uploads? >> _______________________________________________ >> General mailing list >> General@developer.marklogic.com >> http://developer.marklogic.com/mailman/listinfo/general > _______________________________________________ > General mailing list > General@developer.marklogic.com > http://developer.marklogic.com/mailman/listinfo/general -- Demian Hess Avalon Consulting, LLC 527 Maple Avenue East, Suite 200, Vienna, VA 22180 Mobile: 301-943-8307 Fax: 845-367-5496 he...@avalonconsult.com _______________________________________________ General mailing list General@developer.marklogic.com http://developer.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list General@developer.marklogic.com http://developer.marklogic.com/mailman/listinfo/general
