Hi Adam, I can't answer the question regarding threading in any technical detail, but I gather that in general Fedora is multi-threaded, so multiple ingest requests will be handled separately. I think it's the case though that when ingesting multiple objects simultaneously, the shell utilities provided with Fedora (e.g. passing a 'd' flag to fedora-ingest.sh ) work sequentially - so you may want to make API requests directly rather than using these if you want to do things in parallel.
We're using Fedora for storage of video files - mostly compressed MPEG-2, so in the 800MB - 4GB file size range. We use SHA-512 checksums without any issues. I generally monitor ingests through a combination of Java VisualVM and iotop to get an impression of what's IO-centric and what's CPU-centric: on ingest of 2 - 4GB files, checksumming is pretty quick - 1 to 5 seconds at the end of an ingest cycle. That's on a Xen VM running on a dual-quad core Xeon physical machine. Just to note, Fedora 3.4 fixes a bug in the REST interface in relation to files over 2GB [1]. Also, versions prior to 3.2 (I think) will generally attempt to read entire datastreams into RAM on ingest, which isn't a great idea with large files. There is also an outstanding JIRA issue relating to streaming files directly into the back-end filestore, which if implemented would speed up large ingests considerably [2]. We have tested 3.4 ingesting up to 45GB files with no particular performance issues, and these ingests included checksumming. Regards, Graeme [1] https://jira.duraspace.org/browse/FCREPO-704 [2] https://jira.duraspace.org/browse/FCREPO-495 On 22 Oct 2010, at 19:02, Adam Wead wrote: Hi all, How does the checksum generation process for very large files, like video files of 50 GB, affect Fedora's performance? The scenario I'm thinking about is if you're ingesting a batch of large files into Fedora, will it fork a new process to generate the checksum in the background while proceeding on to the next file, or will it only proceed to the next file after the checksum for the first file has been generated. Anyone care to chime in on checksums in general for video files? I've used them before on audio files without too much trouble, but the size of video files might present problems. thanks, ...adam Email has been scanned for viruses by Altman Technologies' email management service<http://www.altman.co.uk/emailsystems> ------------------------------------------------------------------------------ Nokia and AT&T present the 2010 Calling All Innovators-North America contest Create new apps & games for the Nokia N8 for consumers in U.S. and Canada $10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store http://p.sf.net/sfu/nokia-dev2dev Email has been scanned for viruses by Altman Technologies' email management service - www.altman.co.uk/emailsystems _______________________________________________ Fedora-commons-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/fedora-commons-users Email has been scanned for viruses by Altman Technologies' email management service - www.altman.co.uk/emailsystems Glasgow Caledonian University is a registered Scottish charity, number SC021474 Winner: Times Higher Education's Widening Participation Initiative of the Year 2009 and Herald Society's Education Initiative of the Year 2009 http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,6219,en.html ------------------------------------------------------------------------------ Nokia and AT&T present the 2010 Calling All Innovators-North America contest Create new apps & games for the Nokia N8 for consumers in U.S. and Canada $10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store http://p.sf.net/sfu/nokia-dev2dev _______________________________________________ Fedora-commons-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
