The TransferManager code just made the implementation simpler. Otherwise you have to write code that determines how large the bitstream is, and if its small, transfer the file to somewhere in memory, and then stream that, or if its larger, you could store it to temp file, and then stream that. You also have to ensure that you delete the temp file when complete. If its very large, larger than 5GB, you can't transfer that in a single GET. The TransferManager code handles the multipart uploads and download. Also, it can give a performance benefit, where GET and PUT can be done multi-thread multi-part, instead of a single long running GET or PUT to S3. The transfermanager also seemed to do a good job of closing resources when the transfer was complete.
http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/transfer/TransferManager.html High level utility for managing transfers to Amazon S3. TransferManager provides a simple API for uploading content to Amazon S3, and makes extensive use of Amazon S3 multipart uploads to achieve enhanced throughput, performance and reliability. When possible, TransferManager attempts to use multiple threads to upload multiple parts of a single upload at once. When dealing with large content sizes and high bandwidth, this can have a significant increase on throughput. TransferManager is responsible for managing resources such as connections and threads; share a single instance of TransferManager whenever possible. TransferManager, like all the client classes in the AWS SDK for Java, is thread safe. Call TransferManager.shutdownNow() to release the resources once the transfer is complete. ________________ Peter Dietz Longsight www.longsight.com pe...@longsight.com p: 740-599-5005 x809 On Fri, May 26, 2017 at 9:18 AM, <jcdal...@email.wm.edu> wrote: > Thanks Peter, > > I have implemented most of your changes on the DSpace 6 code and now I'm > not running out of connections. I haven't updated to use the Transfer > Manager yet and I was wondering what advantage you found using the Transfer > Manager over the original implementation. > > Thanks, > Justin > > On Thursday, May 18, 2017 at 11:02:22 AM UTC-4, Peter Dietz wrote: >> >> Hi Justin, >> >> With production use, and encountering similar issues to you, I have made >> fixes to our code to protect against http connection leaking with AWS S3 >> assetstore, and also to handle larger uploads/downloads. Here is some >> commit history to the (5.x) S3BitStore: https://github.com >> /LongsightGroup/DSpace/commits/longsight-5.6/dspace-api/src/ >> main/java/org/dspace/storage/bitstore/impl/S3BitStore.java >> >> >> ________________ >> Peter Dietz >> Longsight >> www.longsight.com >> pe...@longsight.com >> p: 740-599-5005 x809 <(740)%20599-5005> >> >> On Thu, May 18, 2017 at 10:57 AM, Claudia Jürgen < >> claudia...@tu-dortmund.de> wrote: >> >>> Hello jcdalton, >>> >>> which UI are you using and are you using google analytics? >>> >>> For the time being you may set the config parameter db.maxidle to >>> something else than -1 >>> see >>> https://jira.duraspace.org/browse/DS-3564?jql=text%20~%20%22maxidle%22 >>> >>> Hope this helps >>> >>> Claudia Jürgen >>> >>> >>> Am 18.05.2017 um 16:25 schrieb jcda...@email.wm.edu: >>> >>>> I'm using S3 as a bitstore on DSpace 6 and I'm coming up with a lot of >>>> errors like the following: >>>> >>>> com.amazonaws.http.AmazonHttpClient @ Unable to execute HTTP request: >>>> Timeout waiting for connection from pool >>>> org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting >>>> for >>>> connection from pool >>>> >>>> It looks to me like either DSpace isn't closing the connections as it >>>> should. Anyone else had this issue? >>>> >>>> Thanks >>>> >>>> >>> -- >>> Claudia Juergen >>> Eldorado >>> >>> Technische Universität Dortmund >>> Universitätsbibliothek >>> Vogelpothsweg 76 >>> 44227 Dortmund >>> >>> Tel.: +49 231-755 40 43 >>> Fax: +49 231-755 40 32 >>> claudia...@tu-dortmund.de >>> www.ub.tu-dortmund.de >>> >>> Wichtiger Hinweis: Die Information in dieser E-Mail ist vertraulich. Sie >>> ist ausschließlich für den Adressaten bestimmt. Sollten Sie nicht der für >>> diese E-Mail bestimmte Adressat sein, unterrichten Sie bitte den Absender >>> und vernichten Sie diese Mail. Vielen Dank. >>> Unbeschadet der Korrespondenz per E-Mail, sind unsere Erklärungen >>> ausschließlich final rechtsverbindlich, wenn sie in herkömmlicher >>> Schriftform (mit eigenhändiger Unterschrift) oder durch Übermittlung eines >>> solchen Schriftstücks per Telefax erfolgen. >>> >>> Important note: The information included in this e-mail is confidential. >>> It is solely intended for the recipient. If you are not the intended >>> recipient of this e-mail please contact the sender and delete this message. >>> Thank you. Without prejudice of e-mail correspondence, our statements are >>> only legally binding when they are made in the conventional written form >>> (with personal signature) or when such documents are sent by fax. >>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "DSpace Technical Support" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to dspace-tech...@googlegroups.com. >>> To post to this group, send email to dspac...@googlegroups.com. >>> Visit this group at https://groups.google.com/group/dspace-tech. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "DSpace Technical Support" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to dspace-tech+unsubscr...@googlegroups.com. > To post to this group, send email to dspace-tech@googlegroups.com. > Visit this group at https://groups.google.com/group/dspace-tech. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group. To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech+unsubscr...@googlegroups.com. To post to this group, send email to dspace-tech@googlegroups.com. Visit this group at https://groups.google.com/group/dspace-tech. For more options, visit https://groups.google.com/d/optout.
