+1 for deposit by reference. It is almost like giving a metadata receipt
for a deposit not happening via the http route.
I would also highly recommend looking at High-Performance SSH or HPN-SSH.
On comparable hardware, I have been shown that it outpaces even grid-ftp
for file transfer speeds, but is a backward compatible patch for the
openssh library.
This means that if the server and client are both patched, the transfer is
multithreaded and otherwise highly optimized. It means that Unix tools
which use SSH benefit as well - rsync, ssh -X, and so on.
Ben
On Nov 15, 2011 10:37 AM, "Stuart Lewis" <s.le...@auckland.ac.nz> wrote:
> Hi Jesús,
>
> The method that seems to work in this setting is to use 'deposit by
> reference'. That is, you deposit a description of the item, including
> details of where the item can be found. It is then up to ingesting system
> to pull the data - perhaps via an offline queue process, using some other
> method (ftp, scp, nfs, etc).
>
> SWORD v2 might be useful here too, because the SWORD statement could be
> requested to find out the status of the file upload (for example, using a
> status such as pending, in-process, complete, failed, etc). This would
> allow the sender/depositor to be able to find out the status of the item.
>
> Let us know how you get on - it is interesting to see the protocol being
> pushed to its limits with use cases such as yours.
>
> Thanks,
>
>
> Stuart Lewis
> Digital Development Manager
> Te Tumu Herenga The University of Auckland Library
> Auckland Mail Centre, Private Bag 92019, Auckland 1142, New Zealand
> Ph: +64 (0)9 373 7599 x81928
>
>
>
> On 15/11/2011, at 11:27 AM, Jesús García Crespo wrote:
>
> > Hi everyone,
> >
> > We are using SWORD to deposit DIPs in ICA-AtoM from Archivematica, but
> we have encountered some problems with large files (8GB). HTTP requests can
> be very resource intensive and unmanageable when the contents are very big.
> Does anyone have any recommendations, for depositing large files via the
> SWORD protocol? Like maybe sending related files using SFTP and then
> indicating the local file route? (That is something we are considering.)
> >
> > Thank you in advance,
> >
> > --
> > Jesús García Crespo
> >
> ------------------------------------------------------------------------------
> > RSA(R) Conference 2012
> > Save $700 by Nov 18
> > Register now
> >
> http://p.sf.net/sfu/rsa-sfdev2dev1_______________________________________________
> > sword-app-tech mailing list
> > sword-app-tech@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/sword-app-tech
>
>
>
>
>
> ------------------------------------------------------------------------------
> RSA(R) Conference 2012
> Save $700 by Nov 18
> Register now
> http://p.sf.net/sfu/rsa-sfdev2dev1
> _______________________________________________
> sword-app-tech mailing list
> sword-app-tech@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/sword-app-tech
>
------------------------------------------------------------------------------
RSA(R) Conference 2012
Save $700 by Nov 18
Register now
http://p.sf.net/sfu/rsa-sfdev2dev1
_______________________________________________
sword-app-tech mailing list
sword-app-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sword-app-tech
