Hi John, I tested Multipart Upload with aws-sdk-js with the patch you mentioned, against riak_cs (s2), upload finished without errors up to 1GB object. The environment is all local on my laptop, so latency is small. The script I used is in [1].
As Luke mentioned, HAProxy would be the point to be investigated first. Or, if it's possible to get packet capture, by finding some wrong TCP packet, like RST or premature (from the point of HTTP) FIN, one can identify who closes TCP connection actively. In this case, packet should be captured on client box, HAProxy box and riak cs box. [1] https://gist.github.com/shino/ac7d56398557fb936899 Thanks, Shino 2016-01-14 8:51 GMT+09:00 Luke Bakken <lbak...@basho.com>: > Hi John, > > Thanks for the info. I'm very curious to see what's in the haproxy > logs with regard to TCP. > -- > Luke Bakken > Engineer > lbak...@basho.com > > > On Wed, Jan 13, 2016 at 3:50 PM, John Fanjoy <jfan...@inetu.net> wrote: >> Luke, >> >> As a test I’ve already increased all timeouts to 5 minutes but the failure >> occurs within under 1 minute so it doesn’t appear to be timeout related. I >> change the logs to tcplog tomorrow and let you know if I find anything. >> >> Thanks >> >> John Fanjoy >> Systems Engineer >> jfan...@inetu.net >> >> >> >> >> >> On 1/13/16, 6:05 PM, "Luke Bakken" <lbak...@basho.com> wrote: >> >>>haproxy ships with some "short" default timeouts. If CyberDuck is able >>>to upload these files faster than aws-sdk, it may be doing so within >>>the default haproxy timeouts. >>> >>>You can also look at haproxy's log to see if you find any TCP >>>connections that it has closed. >>>-- >>>Luke Bakken >>>Engineer >>>lbak...@basho.com >>> >>> >>>On Wed, Jan 13, 2016 at 3:02 PM, John Fanjoy <jfan...@inetu.net> wrote: >>>> Luke, >>>> >>>> I may be able to do that. The only problem is without haproxy I have no >>>> way to inject CORS headers which the browser requires, but I may be able >>>> to write up small nodejs app to get past that and see if it is somehow >>>> related to haproxy. The fact that these errors are not present when using >>>> Cyberduck which is also talking to haproxy leads me to believe that’s not >>>> the cause, but it’s definitely worth testing. >>>> >>>> -- >>>> John Fanjoy >>>> Systems Engineer >>>> jfan...@inetu.net >>>> >>>> >>>> >>>> >>>> >>>> On 1/13/16, 5:55 PM, "Luke Bakken" <lbak...@basho.com> wrote: >>>> >>>>>John - >>>>> >>>>>The following error indicates that the connection was unexpectedly >>>>>closed by something outside of Riak while the chunk is uploading: >>>>> >>>>>{badmatch,{error,closed}} >>>>> >>>>>Is it possible to remove haproxy to test using the the aws-sdk? >>>>> >>>>>That is my first thought as to the cause of this issue, especially >>>>>since writing to S3 works with the same code. >>>>> >>>>>-- >>>>>Luke Bakken >>>>>Engineer >>>>>lbak...@basho.com >>>>> >>>>>On Wed, Jan 13, 2016 at 2:46 PM, John Fanjoy <jfan...@inetu.net> wrote: >>>>>> Luke, >>>>>> >>>>>> Yes on both parts. To confirm cyberduck was using multi-part I actually >>>>>> tailed the console.log while it was uploading the file, and it uploaded >>>>>> the file in approx. 40 parts. Afterwards the parts were reassembled as >>>>>> you would expect. The AWS-SDK for javascript has an object called >>>>>> ManagedUpload which automatically switches to multi-part when the input >>>>>> is larger than the maxpartsize (default 5mb). I have confirmed that it >>>>>> is splitting the files up, but so far I’ve only ever seen one part get >>>>>> successfully uploaded before the others failed at which point it removes >>>>>> the upload (DELETE call) automatically. I also verified that the >>>>>> javascript I have in place does work with an actual AWS S3 bucket to >>>>>> rule out coding issues on my end and the same >400mb file was >>>>>> successfully uploaded to the bucket I created there without issue. >>>>>> >>>>>> A few things worth mentioning that I missed before. I am running riak-s2 >>>>>> behind haproxy. Haproxy is handling ssl and enabling CORS for browser >>>>>> based requests. I have tested smaller files (~4-5mb) and GET requests >>>>>> using the browser client and everything works with my current haproxy >>>>>> configuration, but the larger files are failing, usually after 1 part is >>>>>> successfully uploaded. I can also list bucket contents and delete >>>>>> existing contents. The only feature that is not working appears to be >>>>>> the multi-part uploads. We are running centOS 7 (kernel version >>>>>> 3.10.0-327.4.4.el7.x86_64). Please let me know if you have any further >>>>>> questions. >>>>>> >>>>>> -- >>>>>> John Fanjoy >>>>>> Systems Engineer >>>>>> jfan...@inetu.net > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
