Stuart,
Can you provide any links to examples of using 'deposit by reference' ?
I am looking at feasibility of depositing very large items (tar.gz or zip'd
data files), say up to 16TB, into Dspace 1.6.x with the obvious problems of
doing this using a web interface.
Wondering if EasyDeposit can be adapted to do 'deposit by reference' with
either a utility of some kind on the dspace server looking for large items to
injest or a client pushing the data onto a directory on the dspace server from
where it can be injested. Ideally want to minimise any copies of the data.
Really want to avoid copying the item once it's on the Dspace server. Could
item be uploaded directly into asset store maybe ?
The other problem is how anyone could download the item once it's in Dspace ?
Anyone else doing this sort of very large item ( i.e. TB's ) injest ?
Thank you,
Pete
From: David FLANDERS [mailto:d.fland...@jisc.ac.uk]
Sent: 21 November 2011 18:10
To: Ben O'Steen; Stuart Lewis
Cc: <, sword-app-tech@lists.sourceforge.net>,
Subject: Re: [sword-app-tech] How to send large fiels
To second that, some amazing things being done down here in Australia as they
take *large* data off of scientific instruments. /dff
From: Ben O'Steen [mailto:bost...@gmail.com]<mailto:[mailto:bost...@gmail.com]>
Sent: 14 November 2011 23:54
To: Stuart Lewis
Cc: <,
sword-app-tech@lists.sourceforge.net><mailto:sword-app-tech@lists.sourceforge.net>>,
Subject: Re: [sword-app-tech] How to send large fiels
+1 for deposit by reference. It is almost like giving a metadata receipt for a
deposit not happening via the http route.
I would also highly recommend looking at High-Performance SSH or HPN-SSH. On
comparable hardware, I have been shown that it outpaces even grid-ftp for file
transfer speeds, but is a backward compatible patch for the openssh library.
This means that if the server and client are both patched, the transfer is
multithreaded and otherwise highly optimized. It means that Unix tools which
use SSH benefit as well - rsync, ssh -X, and so on.
Ben
On Nov 15, 2011 10:37 AM, "Stuart Lewis"
<s.le...@auckland.ac.nz<mailto:s.le...@auckland.ac.nz>> wrote:
Hi Jesús,
The method that seems to work in this setting is to use 'deposit by reference'.
That is, you deposit a description of the item, including details of where the
item can be found. It is then up to ingesting system to pull the data -
perhaps via an offline queue process, using some other method (ftp, scp, nfs,
etc).
SWORD v2 might be useful here too, because the SWORD statement could be
requested to find out the status of the file upload (for example, using a
status such as pending, in-process, complete, failed, etc). This would allow
the sender/depositor to be able to find out the status of the item.
Let us know how you get on - it is interesting to see the protocol being pushed
to its limits with use cases such as yours.
Thanks,
Stuart Lewis
Digital Development Manager
Te Tumu Herenga The University of Auckland Library
Auckland Mail Centre, Private Bag 92019, Auckland 1142, New Zealand
Ph: +64 (0)9 373 7599 x81928<tel:%2B64%20%280%299%20373%207599%20x81928>
On 15/11/2011, at 11:27 AM, Jesús García Crespo wrote:
> Hi everyone,
>
> We are using SWORD to deposit DIPs in ICA-AtoM from Archivematica, but we
> have encountered some problems with large files (8GB). HTTP requests can be
> very resource intensive and unmanageable when the contents are very big. Does
> anyone have any recommendations, for depositing large files via the SWORD
> protocol? Like maybe sending related files using SFTP and then indicating the
> local file route? (That is something we are considering.)
>
> Thank you in advance,
>
> --
> Jesús García Crespo
> ------------------------------------------------------------------------------
> RSA(R) Conference 2012
> Save $700 by Nov 18
> Register now
> http://p.sf.net/sfu/rsa-sfdev2dev1_______________________________________________
> sword-app-tech mailing list
> sword-app-tech@lists.sourceforge.net<mailto:sword-app-tech@lists.sourceforge.net>
> https://lists.sourceforge.net/lists/listinfo/sword-app-tech
------------------------------------------------------------------------------
RSA(R) Conference 2012
Save $700 by Nov 18
Register now
http://p.sf.net/sfu/rsa-sfdev2dev1
_______________________________________________
sword-app-tech mailing list
sword-app-tech@lists.sourceforge.net<mailto:sword-app-tech@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/sword-app-tech
------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
sword-app-tech mailing list
sword-app-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sword-app-tech
