Hi David,
I'd expect MarkLogic and S3 to either ignore the case, or be able to handle it. I can't really comment on the third slash, but it is pretty common on the file protocol. I recommend filing a bug if you haven't done so already. That will give our engineers something to look at.. Cheers, Geert Van: general-boun...@developer.marklogic.com [mailto:general-boun...@developer.marklogic.com] Namens David Ennis Verzonden: woensdag 21 mei 2014 10:01 Aan: MarkLogic Developer Discussion Onderwerp: Re: [MarkLogic Dev General] S3 and xdmp:external-binary Hi David and Danny. @Danny: The version is 7.0-2.3 as launched from the AMI at Amazon. @David: I'll walk through all of the steps below. IN Short, you are correct - they work if I use a lowercased bucket.. But see the last bit where I came to the conclusion that I needed the third slash and some fun oddities related to odd behaviour if I use a capital. 1) Sample files uploaded via local aws - shows bucket is set up and files in place: david@davidennis:$ aws s3 cp /tmp/a.jpg s3://FT-Acceptance-test-bucket/a.jpg upload: ../../../../../../../../../tmp/a.jpg to s3://FT-Acceptance-test-bucket/a.jpg david@davidennis:$ aws s3 cp /tmp/b.jpg s3://FT-Acceptance-test-bucket/b.jpg upload: ../../../../../../../../../tmp/b.jpg to s3://FT-Acceptance-test-bucket/b.jpg david@davidennis:$ aws s3 ls s3://FT-Acceptance-test-bucket 2014-05-21 09:06:48 1998228 a.jpg 2014-05-21 09:06:58 1998228 b.jpg [ 2 files uploaded locally and listed back ] 2) Test - List Directory xdmp:filesystem-directory( "s3://FT-Acceptance-test-bucket/" )//*:filename/text() Fails.. Se the error below: SVC-S3ERR in another test But (3rd slash): xdmp:filesystem-directory( "s3:///FT-Acceptance-test-bucket/" )//*:filename/text() Expected result was returned: a.jpg b.jpg 3) Test - get file: let $bin := xdmp:external-binary( "s3://FT-Acceptance-test-bucket/a.jpg" ) return xdmp:binary-size($bin) Fails.. Se the error below: SVC-S3ERR in another test But (3rd slash): let $bin := xdmp:external-binary( "s3:///FT-Acceptance-test-bucket/a.jpg" ) return xdmp:binary-size($bin) Fails.. In the way I originally described [1.0-ml] XDMP-MISSINGFILE: xdmp:external-binary("s3:///FT-Acceptance-test-bucket/a.jpg") -- Missing file s3:///FT-Acceptance-test-bucket/a.jpg However, see (2) above.. it's not missing.. +++++++++++++++++++++++++ The Third Slash came in from an early test xdmp:document-get("s3://FT-Acceptance-test-bucket/a.jpg") Returns: 1.0-ml] SVC-S3ERR: xdmp:document-get("s3://FT-Acceptance-test-bucket/a.jpg") -- S3 error: GET http://FT-Acceptance-test-bucket.s3.amazonaws.com/ 403 Forbidden Code=SignatureDoesNotMatch Message="The request signature we calculated does not match the signature you provided. Check your key and signing method." StringToSignBytes="47 45 54 0a 0a 0a 57 65 64 2c 20 32 31 20 4d 61 79 20 32 30 31 34 20 30 37 3a 32 36 3a 31 30 20 47 4d 54 0a 2f 66 74 2d 61 63 63 65 70 74 61 6e 63 65 2d 74 65 73 74 2d 62 75 63 6b 65 74 2f" RequestId=856A04B796D59029 HostId=5hRlNGvYu5MReGgPtrAlMbPEcg34ztDo05LD5aqYaYc9pNzK0LqgDyk/zpUGaGQG SignatureProvided=ubNAsl3tx2TqcT4es2ZaD74tY/s= StringToSign="GET Wed, 21 May 2014 07:26:10 GMT /ft-acceptance-test-bucket/" AWSAccessKeyId=AKIAIP2..... But (3rd slash) xdmp:document-get("s3:///FT-Acceptance-test-bucket/a.jpg") Works! That is where I went astray on my early test But wait - it gets more fun - There is something with capital letters that is inconsistent amoung xdmp functions and S3. Take these 2 buckets: M-david-ennis-test-bucket (fails the same as the above and works in the same cases as the ///) david-ennis-test-bucket (all lowercase) works as expected in all cases. So, The hiccup seems that S3 stuff in MarkLogic is that it forces lowercase on the bucketname - yet (for some functions) allows a third slash to find the Case-sensitive file. Bets practice is to use all lowercase for bucket names - so I will do that. However, other tools do not have the same issue (work seamlessly with any case mix). MarkLogic, at best, is inconsistent in some ways of handling the bucket name since I see different results between external-binary and document-get when calling a bucket with a capital letter. Perhaps I missed the requirement for all lowercase in the MarkLogic docs. Thanks for the help! David On 21 May 2014 03:43, David Lee <david....@marklogic.com <mailto:david....@marklogic.com> > wrote: For one you have too many slashes , you should use 2 not 3 slashes Wrong: xdmp:external-binary("s3:///my-bucket/my-file.jpg") Right: xdmp:external-binary("s3://my-bucket/my-file.jpg") Also verify if you have permissions to this bucket setup properly. Are you using explicit AWS credentials or IAM credentials ? Try this from QC and see what you get --- List a directory xdmp:filesystem-directory( "s3://my-bucket/" ) Get a binary file let $bin := xdmp:external-binary( "s3://my-bucket/big-file.bin" ) return xdmp:binary-size($bin) Storing a binary file let $bin := xdmp:external-binary( "s3:// big-file.bin " ) return xdmp:save( "s3://your-bucket/bigfile.bin" , $bin ) From: general-boun...@developer.marklogic.com <mailto:general-boun...@developer.marklogic.com> [mailto:general-boun...@developer.marklogic.com <mailto:general-boun...@developer.marklogic.com> ] On Behalf Of Danny Sokolsky Sent: Tuesday, May 20, 2014 8:27 PM To: MarkLogic Developer Discussion Subject: Re: [MarkLogic Dev General] S3 and xdmp:external-binary Hi David, What version are you running (the output of xdmp:version() )? And what is the exact error you are seeing? Thanks, -Danny From: general-boun...@developer.marklogic.com <mailto:general-boun...@developer.marklogic.com> [mailto:general-boun...@developer.marklogic.com] On Behalf Of David Ennis Sent: Tuesday, May 20, 2014 3:54 PM To: MarkLogic Developer Discussion Subject: [MarkLogic Dev General] S3 and xdmp:external-binary Hi All. I'm at a loss. I understood that any place where a filesystem path is needed we could use S3 uris. In general, this works. load files, save files, etc. However, my biggest need was to register external binaries.. xdmp:external-binary("s3:///my-bucket/my-file.jpg") This throws an error: Missing file but, the file is there and available to MarkLogic in other ways. It is just the use of external-binary that gives the issue.. Any ideas on how to handle this? I have a need to register a few million external files and this just seemed like it should have worked.. I can work around it through some elbow-grease, but this was just such a natural-feeling solution. Kind Regards, David _______________________________________________ General mailing list General@developer.marklogic.com <mailto:General@developer.marklogic.com> http://developer.marklogic.com/mailman/listinfo/general
_______________________________________________ General mailing list General@developer.marklogic.com http://developer.marklogic.com/mailman/listinfo/general
