[jira] [Updated] (JCLOUDS-1582) ETag of the object uploaded via multipart upload does not match the CompleteMPU response from the transient blobstore
[ https://issues.apache.org/jira/browse/JCLOUDS-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timur Alperovich updated JCLOUDS-1582: -- Description: The local transient blobstore returns a correctly computed S3 MPU ETag when completing the upload, but a subsequent GET/HEAD request returns the MD5 of the whole object instead (not matching the ETag in the CompleteMPU response). Ideally, jclouds should store the MPU ETag in the extended attributes along with the object and use its value for GET/HEAD requests. (was: The local filesystem blobstore returns a correctly computed S3 MPU ETag when completing the upload, but a subsequent GET/HEAD request returns the MD5 of the whole object instead (not matching the ETag in the CompleteMPU response). Ideally, jclouds should store the MPU ETag in the extended attributes along with the object and use its value for GET/HEAD requests.) > ETag of the object uploaded via multipart upload does not match the > CompleteMPU response from the transient blobstore > - > > Key: JCLOUDS-1582 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1582 > Project: jclouds > Issue Type: Bug > Components: jclouds-blobstore >Affects Versions: 2.3.0 > Reporter: Timur Alperovich >Priority: Minor > Labels: filesystem > > The local transient blobstore returns a correctly computed S3 MPU ETag when > completing the upload, but a subsequent GET/HEAD request returns the MD5 of > the whole object instead (not matching the ETag in the CompleteMPU response). > Ideally, jclouds should store the MPU ETag in the extended attributes along > with the object and use its value for GET/HEAD requests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (JCLOUDS-1582) ETag of the object uploaded via multipart upload does not match the CompleteMPU response from the transient blobstore
[ https://issues.apache.org/jira/browse/JCLOUDS-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timur Alperovich updated JCLOUDS-1582: -- Description: The local transient blobstore returns a correctly computed S3 MPU ETag when completing the upload, but a subsequent GET/HEAD request returns the MD5 of the whole object instead (not matching the ETag in the CompleteMPU response). Ideally, jclouds should store the MPU ETag and return it on GET/HEAD requests. This behavior also manifests when using the filesystem blobstore if the extended attributes are not supported. In that case, there is nothing jclouds can do. was:The local transient blobstore returns a correctly computed S3 MPU ETag when completing the upload, but a subsequent GET/HEAD request returns the MD5 of the whole object instead (not matching the ETag in the CompleteMPU response). Ideally, jclouds should store the MPU ETag in the extended attributes along with the object and use its value for GET/HEAD requests. > ETag of the object uploaded via multipart upload does not match the > CompleteMPU response from the transient blobstore > - > > Key: JCLOUDS-1582 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1582 > Project: jclouds > Issue Type: Bug > Components: jclouds-blobstore >Affects Versions: 2.3.0 > Reporter: Timur Alperovich >Priority: Minor > Labels: filesystem > > The local transient blobstore returns a correctly computed S3 MPU ETag when > completing the upload, but a subsequent GET/HEAD request returns the MD5 of > the whole object instead (not matching the ETag in the CompleteMPU response). > Ideally, jclouds should store the MPU ETag and return it on GET/HEAD requests. > This behavior also manifests when using the filesystem blobstore if the > extended attributes are not supported. In that case, there is nothing jclouds > can do. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (JCLOUDS-1582) ETag of the object uploaded via multipart upload does not match the CompleteMPU response from the transient blobstore
[ https://issues.apache.org/jira/browse/JCLOUDS-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timur Alperovich updated JCLOUDS-1582: -- Summary: ETag of the object uploaded via multipart upload does not match the CompleteMPU response from the transient blobstore (was: ETag of the object uploaded via multipart upload does not match the CompleteMPU response) > ETag of the object uploaded via multipart upload does not match the > CompleteMPU response from the transient blobstore > - > > Key: JCLOUDS-1582 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1582 > Project: jclouds > Issue Type: Bug > Components: jclouds-blobstore >Affects Versions: 2.3.0 > Reporter: Timur Alperovich >Priority: Minor > Labels: filesystem > > The local filesystem blobstore returns a correctly computed S3 MPU ETag when > completing the upload, but a subsequent GET/HEAD request returns the MD5 of > the whole object instead (not matching the ETag in the CompleteMPU response). > Ideally, jclouds should store the MPU ETag in the extended attributes along > with the object and use its value for GET/HEAD requests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (JCLOUDS-1582) ETag of the object uploaded via multipart upload does not match the CompleteMPU response
Timur Alperovich created JCLOUDS-1582: - Summary: ETag of the object uploaded via multipart upload does not match the CompleteMPU response Key: JCLOUDS-1582 URL: https://issues.apache.org/jira/browse/JCLOUDS-1582 Project: jclouds Issue Type: Bug Components: jclouds-blobstore Reporter: Timur Alperovich The local filesystem blobstore returns a correctly computed S3 MPU ETag when completing the upload, but a subsequent GET/HEAD request returns the MD5 of the whole object instead (not matching the ETag in the CompleteMPU response). Ideally, jclouds should store the MPU ETag in the extended attributes along with the object and use its value for GET/HEAD requests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [jclouds/jclouds] Hash the content for fs MPU ETag if no xattr. (#1285)
Updated to return an empty-string MD5 if the ETag attribute of a blob is null. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1285#issuecomment-608569542
Re: [jclouds/jclouds] Hash the content for fs MPU ETag if no xattr. (#1285)
@timuralp pushed 1 commit. 7970cf43fad93571b9ff4bd8113a6d1d989a906a Hash the content for fs MPU ETag if no xattr. -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/jclouds/jclouds/pull/1285/files/f7dccfd9f2353a666456cd9a8fb667d47de9affd..7970cf43fad93571b9ff4bd8113a6d1d989a906a
Re: [jclouds/jclouds] Hash the content for fs MPU ETag if no xattr. (#1285)
timuralp commented on this pull request.
>
for (MultipartPart part : parts) {
Blob blobPart = getBlob(mpu.containerName(), MULTIPART_PREFIX +
mpu.id() + "-" + mpu.blobName() + "-" + part.partNumber());
contentLength +=
blobPart.getMetadata().getContentMetadata().getContentLength();
blobs.add(blobPart);
-
md5Hasher.putBytes(BaseEncoding.base16().lowerCase().decode(blobPart.getMetadata().getETag()));
+ if (blobPart.getMetadata().getETag() != null) {
+
md5Hasher.putBytes(BaseEncoding.base16().lowerCase().decode(blobPart.getMetadata().getETag()));
I did consider that and thought that previously the blob store returned the
hash of the entire content on PUT. Looking back, it looks like it actually
returned the hash of an empty string (as the blob part ETag didn't exist). I'll
change this patch to the same behavior, keeping the `-{number of parts}` suffix
for consistency.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1285#discussion_r388117257
[jclouds/jclouds] Hash the content for fs MPU ETag if no xattr. (#1285)
If there is no extended attribute support in the file system, the blobs will not have their associated ETags available. In that case, the file system blob store should rehash the content while producing the combined blob and return the expected S3-style ETag. You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds/pull/1285 -- Commit Summary -- * Hash the content for fs MPU ETag if no xattr. -- File Changes -- M blobstore/src/main/java/org/jclouds/blobstore/config/LocalBlobStore.java (62) -- Patch Links -- https://github.com/jclouds/jclouds/pull/1285.patch https://github.com/jclouds/jclouds/pull/1285.diff -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1285
Re: [jclouds/jclouds] JCLOUDS-1371: JCLOUDS-1488: list optimize prefix (#1268)
timuralp approved this pull request.
Looks good to me!
> } else if (child.isDirectory()) {
-blobNames.add(function.apply(child.getAbsolutePath()) +
File.separator); // TODO: undo if failures
-populateBlobKeysInContainer(child, blobNames, function);
+// Consider a prefix /a/b/c but we have only descended to path /a.
+// We need to match the path against the prefix to continue
I was confused by the additional check here, but after removing it and running
the tests, it makes sense: the directories will not include the trailing "/",
so if the prefix is "dir/", then it wouldn't match the directory "dir"
(excluding it from the list and not listing its contents). I think this comment
helps, but maybe it could more explicitly call out this specific issues. As far
as I can tell, this only affects prefix listings that end with the file
separator.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1268#pullrequestreview-197775752
Re: [jclouds/jclouds] Lazily open InputStream during complete MPU (#1269)
timuralp approved this pull request. Looks good to me! -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1269#pullrequestreview-197776920
Re: [jclouds/jclouds] Filesystem: Fix the MPU ETags to match S3. (#1252)
timuralp commented on this pull request.
> .append("-")
.append(partsList.size())
.append("\"")
.toString();
assertThat(blob.getMetadata().getETag()).isEqualTo(expectedETag);
}
+ @Test(groups = { "integration", "live" })
+ public void testMultipartUploadMultiplePartsKnownETag() throws Exception {
+ BlobStore blobStore = view.getBlobStore();
+ String container = getContainerName();
+ // Pre-computed ETag returned by AWS S3 for the MPU consisting of two
5MB parts filled with 'b'
+ String expectedETag = "\"84462a16f6a60478d50148808aa609c1-2\"";
+ int partSize = 5 * (int) Math.pow(2, 20);
Done.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1252#discussion_r230231474
Re: [jclouds/jclouds] Filesystem: Fix the MPU ETags to match S3. (#1252)
timuralp commented on this pull request.
> + @Test(groups = { "integration", "live" })
+ public void testMultipartUploadMultiplePartsKnownETag() throws Exception {
+ BlobStore blobStore = view.getBlobStore();
+ String container = getContainerName();
+ // Pre-computed ETag returned by AWS S3 for the MPU consisting of two
5MB parts filled with 'b'
+ String expectedETag = "\"84462a16f6a60478d50148808aa609c1-2\"";
+ int partSize = 5 * (int) Math.pow(2, 20);
+ try {
+ String name = "blob-name";
+ BlobBuilder blobBuilder = blobStore.blobBuilder(name);
+ Blob blob = blobBuilder.build();
+ MultipartUpload mpu = blobStore.initiateMultipartUpload(container,
blob.getMetadata(), new PutOptions());
+
+ byte[] content = new byte[partSize];
+ Arrays.fill(content, (byte) 'b');
+ Payload payload = Payloads.newByteArrayPayload(content);
Most likely -- the challenging part is that I'd need to pre-compute the ETag
for that output. I picked the easy content of "b"s so I can easily test what
AWS S3 returns and used it as a test value (otherwise, like the original patch
pointed out, we have a problem of the test itself being buggy in the same way).
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1252#discussion_r230231453
Re: [jclouds/jclouds] Filesystem: Fix the MPU ETags to match S3. (#1252)
timuralp commented on this pull request. > @@ -838,10 +839,10 @@ public String completeMultipartUpload(MultipartUpload > mpu, List p throw propagate(ioe); } streams.add(is); - partHashes.append(blobPart.getMetadata().getETag()); + md5Hasher.putBytes(new BigInteger(blobPart.getMetadata().getETag(), 16).toByteArray()); Done. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1252#discussion_r230231187
Re: [jclouds/jclouds] Filesystem: Fix the MPU ETags to match S3. (#1252)
timuralp commented on this pull request. > + MultipartPart part1 = blobStore.uploadMultipartPart(mpu, 1, > payload); + MultipartPart part2 = blobStore.uploadMultipartPart(mpu, 2, payload); + + List parts = blobStore.listMultipartUpload(mpu); + assertThat(parts).hasSize(2); + assertThat(parts.get(0).partNumber()).isEqualTo(part1.partNumber()); + assertThat(parts.get(0).partSize()).isEqualTo(part1.partSize()); + assertThat(parts.get(0).partETag()).isEqualTo(part1.partETag()); + assertThat(parts.get(1).partNumber()).isEqualTo(part2.partNumber()); + assertThat(parts.get(1).partSize()).isEqualTo(part2.partSize()); + assertThat(parts.get(1).partETag()).isEqualTo(part2.partETag()); + + blobStore.completeMultipartUpload(mpu, ImmutableList.of(part1, part2)); + + BlobMetadata newBlobMetadata = blobStore.blobMetadata(container, name); + assertThat(newBlobMetadata.getETag()).isEqualTo(expectedETag); Gotcha! Will remove the extraneous checks. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1252#discussion_r228365019
Re: [jclouds/jclouds] Filesystem: Fix the MPU ETags to match S3. (#1252)
timuralp commented on this pull request.
> .append("-")
.append(partsList.size())
.append("\"")
.toString();
assertThat(blob.getMetadata().getETag()).isEqualTo(expectedETag);
}
+ @Test(groups = { "integration", "live" })
+ public void testMultipartUploadMultiplePartsKnownETag() throws Exception {
+ BlobStore blobStore = view.getBlobStore();
+ String container = getContainerName();
+ // Pre-computed ETag returned by AWS S3 for the MPU consisting of two
5MB parts filled with 'b'
+ String expectedETag = "\"84462a16f6a60478d50148808aa609c1-2\"";
Good question. Some providers return a similar value, like Swift for static
large objects. Others have a different ETag, like Azure, which uses an opaque
identifier. The initial proposal I had was to use AWS S3-style ETags because it
allows for an easy way to emulate S3 using jclouds (like with the S3Proxy
project). That's the motivation behind this PR and the prior one related to it:
https://github.com/jclouds/jclouds/pull/1251.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1252#discussion_r228363312
Re: [jclouds/jclouds] Filesystem: Fix the MPU ETags to match S3. (#1252)
timuralp commented on this pull request. > + MultipartPart part1 = blobStore.uploadMultipartPart(mpu, 1, > payload); + MultipartPart part2 = blobStore.uploadMultipartPart(mpu, 2, payload); + + List parts = blobStore.listMultipartUpload(mpu); + assertThat(parts).hasSize(2); + assertThat(parts.get(0).partNumber()).isEqualTo(part1.partNumber()); + assertThat(parts.get(0).partSize()).isEqualTo(part1.partSize()); + assertThat(parts.get(0).partETag()).isEqualTo(part1.partETag()); + assertThat(parts.get(1).partNumber()).isEqualTo(part2.partNumber()); + assertThat(parts.get(1).partSize()).isEqualTo(part2.partSize()); + assertThat(parts.get(1).partETag()).isEqualTo(part2.partETag()); + + blobStore.completeMultipartUpload(mpu, ImmutableList.of(part1, part2)); + + BlobMetadata newBlobMetadata = blobStore.blobMetadata(container, name); + assertThat(newBlobMetadata.getETag()).isEqualTo(expectedETag); No, there isn't. I considered validating the content itself, but there are existing tests that do that. I can add the content validation if you see value to it. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1252#discussion_r228362221
Re: [jclouds/jclouds] Filesystem: Fix the MPU ETags to match S3. (#1252)
rebuild please -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1252#issuecomment-433188506
[jclouds/jclouds] Filesystem: Fix the MPU ETags to match S3. (#1252)
Prior commit introduced a bug in the computation of the MPU ETag value, where it was concatenating strings, rather than operating on the bytes of the integer value. You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds/pull/1252 -- Commit Summary -- * Filesystem: Fix the MPU ETags to match S3. -- File Changes -- M apis/filesystem/src/test/java/org/jclouds/filesystem/integration/FilesystemBlobIntegrationTest.java (47) M blobstore/src/main/java/org/jclouds/blobstore/config/LocalBlobStore.java (9) -- Patch Links -- https://github.com/jclouds/jclouds/pull/1252.patch https://github.com/jclouds/jclouds/pull/1252.diff -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1252
Re: [jclouds/jclouds] JCLOUDS-1450: Use S3-style ETags for MPUs. (#1251)
timuralp commented on this pull request.
> @@ -373,7 +376,13 @@ public Blob getBlob(final String container, final String
> key) {
if (attributes.contains(XATTR_CONTENT_MD5)) {
ByteBuffer buf =
ByteBuffer.allocate(view.size(XATTR_CONTENT_MD5));
view.read(XATTR_CONTENT_MD5, buf);
- hashCode = HashCode.fromBytes(buf.array());
+ byte [] etagBytes = buf.array();
+ if (etagBytes.length == 16) {
Done
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1251#discussion_r227923384
Re: [jclouds/jclouds] JCLOUDS-1450: Use S3-style ETags for MPUs. (#1251)
timuralp commented on this pull request. > @@ -488,23 +498,37 @@ public String putBlob(final String containerName, final > Blob blob) throws IOExce String tmpBlobName = blobKey + "-" + UUID.randomUUID(); File tmpFile = getFileForBlobKey(containerName, tmpBlobName); Path tmpPath = tmpFile.toPath(); - HashingInputStream his = null; + boolean isMpu = false; + if (blob.getMetadata() != null && blob.getMetadata().getETag() != null) + isMpu = blob.getMetadata().getETag().matches(MPU_ETAG_FORMAT); + InputStream inputStream = null; + byte [] eTag = null; Done -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1251#discussion_r227910332
Re: [jclouds/jclouds] JCLOUDS-1450: Use S3-style ETags for MPUs. (#1251)
timuralp commented on this pull request.
> @@ -115,6 +116,7 @@
private static final String XATTR_USER_METADATA_PREFIX =
"user.user-metadata.";
private static final byte[] DIRECTORY_MD5 =
Hashing.md5().hashBytes(new byte[0]).asBytes();
+ private static final String MPU_ETAG_FORMAT = "^\"[a-f0-9]{32}-\\d+\"$";
Done
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1251#discussion_r227910358
Re: [jclouds/jclouds] JCLOUDS-1450: Use S3-style ETags for MPUs. (#1251)
timuralp commented on this pull request. > @@ -878,7 +888,7 @@ public String completeMultipartUpload(MultipartUpload > mpu, List p setBlobAccess(mpu.containerName(), mpu.blobName(), mpu.putOptions().getBlobAccess()); - return eTag; + return mpuETag; This will return an MPU ETag for any multi-part uploads, even if they are composed of one part. This matches the AWS S3 behavior. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1251#discussion_r227910056
Re: [jclouds/jclouds] JCLOUDS-1450: Use S3-style ETags for MPUs. (#1251)
timuralp commented on this pull request.
>try {
Files.createParentDirs(tmpFile);
- his = new HashingInputStream(Hashing.md5(), payload.openStream());
- long actualSize = Files.asByteSink(tmpFile).writeFrom(his);
+ if (isMpu) {
+inputStream = payload.openStream();
+eTag = blob.getMetadata().getETag().getBytes();
+ }
+ else {
Done
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1251#discussion_r227910267
Re: [jclouds/jclouds] JCLOUDS-1450: Use S3-style ETags for MPUs. (#1251)
timuralp commented on this pull request.
> @@ -373,7 +376,13 @@ public Blob getBlob(final String container, final String
> key) {
if (attributes.contains(XATTR_CONTENT_MD5)) {
ByteBuffer buf =
ByteBuffer.allocate(view.size(XATTR_CONTENT_MD5));
view.read(XATTR_CONTENT_MD5, buf);
- hashCode = HashCode.fromBytes(buf.array());
+ byte [] etagBytes = buf.array();
+ if (etagBytes.length == 16) {
+ hashCode = HashCode.fromBytes(buf.array());
+ eTag = String.format("\"%s\"", hashCode.toString());
Done
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1251#discussion_r227910213
Re: [jclouds/jclouds] JCLOUDS-1450: Use S3-style ETags for MPUs. (#1251)
timuralp commented on this pull request.
>}
+ StringBuilder mpuETagBuilder = new StringBuilder("\"");
+ mpuETagBuilder.append(Hashing.md5().hashString(partHashes.toString(),
US_ASCII).toString());
+ mpuETagBuilder.append(String.format("-%d\"", parts.size()));
+ String mpuETag = mpuETagBuilder.toString();
Done
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1251#discussion_r227910241
Re: [jclouds/jclouds] JCLOUDS-1450: Use S3-style ETags for MPUs. (#1251)
timuralp commented on this pull request.
> @@ -373,7 +376,13 @@ public Blob getBlob(final String container, final String
> key) {
if (attributes.contains(XATTR_CONTENT_MD5)) {
ByteBuffer buf =
ByteBuffer.allocate(view.size(XATTR_CONTENT_MD5));
view.read(XATTR_CONTENT_MD5, buf);
- hashCode = HashCode.fromBytes(buf.array());
+ byte [] etagBytes = buf.array();
+ if (etagBytes.length == 16) {
+ hashCode = HashCode.fromBytes(buf.array());
+ eTag = String.format("\"%s\"", hashCode.toString());
+ } else {
+ eTag = new String(etagBytes);
Done
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1251#discussion_r227910171
[jclouds/jclouds] JCLOUDS-1450: Use S3-style ETags for MPUs. (#1251)
S3 uses a different ETag for multipart uploads (MPUs) than regular objects. The ETag consists of the md5 hash of the concatenated ETags of individual parts followed by the number of parts (separated by "-"). The patch changes the LocalBlobStore's implementation of CompleteMultipartUpload to set the S3-style ETag before calling putBlob() and return that ETag to the caller. To simplify testing, a new protected method with a default NOOP implementation is added to the BaseBlobIntegrationTest. It allows providers to further verify MPUs (i.e. ensuring the correct ETag has been stored alongside the object). You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds/pull/1251 -- Commit Summary -- * JCLOUDS-1450: Use S3-style ETags for MPUs. -- File Changes -- M apis/filesystem/src/main/java/org/jclouds/filesystem/strategy/internal/FilesystemStorageStrategyImpl.java (50) M apis/filesystem/src/test/java/org/jclouds/filesystem/integration/FilesystemBlobIntegrationTest.java (19) M blobstore/src/main/java/org/jclouds/blobstore/config/LocalBlobStore.java (16) M blobstore/src/main/java/org/jclouds/blobstore/domain/BlobBuilder.java (8) M blobstore/src/main/java/org/jclouds/blobstore/domain/internal/BlobBuilderImpl.java (14) M blobstore/src/test/java/org/jclouds/blobstore/integration/internal/BaseBlobIntegrationTest.java (5) -- Patch Links -- https://github.com/jclouds/jclouds/pull/1251.patch https://github.com/jclouds/jclouds/pull/1251.diff -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1251
[jira] [Created] (JCLOUDS-1450) Multi-part upload against the filesystem provider should return ETag similar to S3
Timur Alperovich created JCLOUDS-1450: - Summary: Multi-part upload against the filesystem provider should return ETag similar to S3 Key: JCLOUDS-1450 URL: https://issues.apache.org/jira/browse/JCLOUDS-1450 Project: jclouds Issue Type: Bug Reporter: Timur Alperovich When performing a multi-part upload in S3, the resulting ETag is not a content hash, but rather a hash of the content hashes of the parts, followed by "-". It would be great for the jclouds filesystem provider to have the same behavior if it's trying to be an emulation for S3 (or Swift, which has a similar behavior for its Static Large Objects). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [jclouds/jclouds] JCLOUDS-1263: URL encode object names from list object response before creating URIs with them (#1084)
I'm able to successfully run the tests locally. I'm not sure what the Travis failure was at the time, but I believe we should be able to rebase and merge this PR. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1084#issuecomment-390796709
Re: [jclouds/jclouds] JCLOUDS-1263: URL encode object names from list object response before creating URIs with them (#1084)
Checked out the container listings. We don't have the same issue, as we don't attempt to build URIs from them. We should merge this PR to fix this issue in Swift. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1084#issuecomment-390530613
Re: [jclouds/jclouds] JCLOUDS-1263: URL encode object names from list object response before creating URIs with them (#1084)
@gaul I'd like to resurrect this PR and apologies for the delay. I think this is the right thing to do for Swift, as listings will return non-ASCII UTF-8 characters not URL encoded, but escaped as "\u". I'm curious if the same issue exists when listing the containers -- I'll look into that tomorrow. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1084#issuecomment-388724490
[jclouds/jclouds] Use different content on overwrite. (#1154)
When testing blobstore overwrite behavior, jclouds should use a blob with different content (but same name). You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds/pull/1154 -- Commit Summary -- * Use different content on overwrite. -- File Changes -- M blobstore/src/test/java/org/jclouds/blobstore/integration/internal/BaseBlobIntegrationTest.java (5) -- Patch Links -- https://github.com/jclouds/jclouds/pull/1154.patch https://github.com/jclouds/jclouds/pull/1154.diff -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1154
Re: [jclouds/jclouds] JCLOUDS-1263: URL encode object names from list object response before creating URIs with them (#1084)
@andrewgaul I will try to looking into it this weekend and report back. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1084#issuecomment-332659235
[jclouds/jclouds] Fixes for jclouds openstack swift tests. (#1144)
The ContainerApiLiveTest class has tests that mostly assume that a single test container always exists. They collide on changes to the state of the container. Some of the tests, though, assume the container does _not_ exist and attempt to create it and subsequently delete it. The change in this patch makes their behavior uniform with respect to that test container: assume it exists and if the test tries to delete it or mutate it, delete the container and create it at the end of the test. The fix in ObjectApi is for the Object GET requests. Currently, jclouds passes "format=json", which is not a supported query parameter for object GET. Lastly, ObjectApiLiveTest that test server-side copy should ignore the X-Openstack-Request-ID header, as its value will change between the two requests. You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds/pull/1144 -- Commit Summary -- * Fixes for jclouds openstack swift tests. -- File Changes -- M apis/openstack-swift/src/main/java/org/jclouds/openstack/swift/v1/features/ObjectApi.java (1) M apis/openstack-swift/src/test/java/org/jclouds/openstack/swift/v1/features/ContainerApiLiveTest.java (8) M apis/openstack-swift/src/test/java/org/jclouds/openstack/swift/v1/features/ObjectApiLiveTest.java (1) -- Patch Links -- https://github.com/jclouds/jclouds/pull/1144.patch https://github.com/jclouds/jclouds/pull/1144.diff -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1144
Re: [jclouds/jclouds] Handle empty delimiter/prefix in FS store. (#1121)
timuralp commented on this pull request.
> + /** Test that listing with an empty string for prefix and delimiter
> returns all of the keys. */
+ @Test(groups = {"integration", "live"})
+ public void testListEmptyPrefixDelimiter() throws Exception {
+ final String container = getContainerName();
+ BlobStore blobStore = view.getBlobStore();
+
+ try {
+ blobStore.createContainerInLocation(null, container);
+ ImmutableList blobs = ImmutableList.of("a", "b", "c");
+ for (String blob : blobs) {
+blobStore.putBlob(container,
blobStore.blobBuilder(blob).payload("").build());
+ }
+ ListContainerOptions options =
ListContainerOptions.Builder.delimiter("")
+ .prefix("").afterMarker("");
+ PageSet rs = blobStore.list(container,
options);
+ assertEquals(rs.size(), 3);
Done
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1121#discussion_r131264676
Re: [jclouds/jclouds] Handle empty delimiter/prefix in FS store. (#1121)
timuralp commented on this pull request.
> @@ -595,6 +596,32 @@ public void testListMarkerPrefix() throws Exception {
}
}
+ /** Test that listing with an empty string for prefix and delimiter returns
all of the keys. */
+ @Test(groups = {"integration", "live"})
+ public void testListEmptyPrefixDelimiter() throws Exception {
+ final String container = getContainerName();
+ BlobStore blobStore = view.getBlobStore();
+
+ try {
+ blobStore.createContainerInLocation(null, container);
Done
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1121#discussion_r131264644
Re: [jclouds/jclouds] Handle empty delimiter/prefix in FS store. (#1121)
timuralp commented on this pull request.
Thanks for the review @andrewgaul! I reworked the tests to avoid code
duplication -- thanks for the suggestions.
> @@ -182,6 +182,25 @@ public void testList_NoOptionSingleContainer() throws
> IOException {
checkForContainerContent(CONTAINER_NAME, blobsExpected);
}
+public void testList_EmptyOptionSingleContainer() throws IOException {
+ blobStore.createContainerInLocation(null, CONTAINER_NAME);
+
+ checkForContainerContent(CONTAINER_NAME, null);
+
+ TestUtils.createBlobsInContainer(CONTAINER_NAME, "a", "b", "c");
+ // Test listing where we set the prefix and delimiter to empty string
Done
> @@ -182,6 +182,25 @@ public void testList_NoOptionSingleContainer() throws
> IOException {
checkForContainerContent(CONTAINER_NAME, blobsExpected);
}
+public void testList_EmptyOptionSingleContainer() throws IOException {
+ blobStore.createContainerInLocation(null, CONTAINER_NAME);
+
+ checkForContainerContent(CONTAINER_NAME, null);
+
+ TestUtils.createBlobsInContainer(CONTAINER_NAME, "a", "b", "c");
+ // Test listing where we set the prefix and delimiter to empty string
+ ListContainerOptions options =
ListContainerOptions.Builder.delimiter("")
+ .prefix("").afterMarker("");
+ PageSet rs = blobStore.list(CONTAINER_NAME,
options);
+ assertEquals(rs.size(), 3);
+ Set expected = Sets.newHashSet("a", "b", "c");
+ for (StorageMetadata sm : rs) {
+ assertTrue(expected.contains(sm.getName()));
+ expected.remove(sm.getName());
I'm not mutating the iterated container, though -- `rs` is the result set.
Regardless, will rework to the suggested approach.
> @@ -688,4 +691,26 @@ public void testUpdateObjectCannedACL() throws Exception
> {
returnContainer(containerName);
}
}
+
+ public void testList_EmptyOptionSingleContainer() throws Exception {
+ String containerName = getContainerName();
+ try {
+ S3Object object = getApi().newS3Object();
+ object.getMetadata().setKey("a");
+ object.setPayload(TEST_STRING);
+ getApi().putObject(containerName, object);
+ // Test listing where we set the prefix and delimiter to empty string
+ ListBucketResponse rs = getApi().listBucket(containerName,
+
ListBucketOptions.Builder.delimiter("").withPrefix("").afterMarker(""));
+ assertEquals(rs.size(), 3);
Sorry about that -- the test surely fails and I must have failed to run it and
didn't catch this.
> @@ -688,4 +691,26 @@ public void testUpdateObjectCannedACL() throws Exception
> {
returnContainer(containerName);
}
}
+
+ public void testList_EmptyOptionSingleContainer() throws Exception {
+ String containerName = getContainerName();
+ try {
+ S3Object object = getApi().newS3Object();
+ object.getMetadata().setKey("a");
+ object.setPayload(TEST_STRING);
+ getApi().putObject(containerName, object);
+ // Test listing where we set the prefix and delimiter to empty string
+ ListBucketResponse rs = getApi().listBucket(containerName,
+
ListBucketOptions.Builder.delimiter("").withPrefix("").afterMarker(""));
+ assertEquals(rs.size(), 3);
+ Set expected = Sets.newHashSet("a");
Yes -- will combine these. I wasn't sure about other providers' behavior, but I
suspect it should either match or ignore the test.
> @@ -266,7 +266,8 @@ public StorageMetadata apply(String key) {
if (options != null) {
if (options.getDir() != null && !options.getDir().isEmpty()) {
contents = filterDirectory(contents, options);
- } else if (options.getPrefix() != null) {
+ } else if (options.getPrefix() != null &&
Done
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1121#pullrequestreview-53887958
Re: [jclouds/jclouds] Handle empty delimiter/prefix in FS store. (#1121)
Removed the spurious white space and import ordering changes. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1121#issuecomment-319578968
[jclouds/jclouds] Handle empty delimiter/prefix in FS store. (#1121)
When delimiter/prefix is an empty string, jclouds filesystem blobstore should treat them as not being set. You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds/pull/1121 -- Commit Summary -- * Handle empty delimiter/prefix in FS store. -- File Changes -- M apis/filesystem/src/test/java/org/jclouds/filesystem/FilesystemBlobStoreTest.java (19) M apis/s3/src/test/java/org/jclouds/s3/S3ClientLiveTest.java (29) M blobstore/src/main/java/org/jclouds/blobstore/config/LocalBlobStore.java (6) -- Patch Links -- https://github.com/jclouds/jclouds/pull/1121.patch https://github.com/jclouds/jclouds/pull/1121.diff -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1121
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
timuralp commented on this pull request.
Here are the ones I spotted. If you can configure your editor to remove
trailing white space, please clean them up. If you can't that's fine. Maybe
that will get cleaned up before merging.
> + // configure the blobstore to use multipart uploading of the file
+ for (int partNumber = 0; partNumber < 3; partNumber++) {
+ String objName = String.format("%s/%s/%s", objectName, "dlo",
partNumber);
+ String data = String.format("%s%s", "data", partNumber);
+ ByteSource payload = ByteSource.wrap(data.getBytes(Charsets.UTF_8));
+ Blob blob = blobStore.blobBuilder(objName)
+ .payload(payload)
+ .build();
+ String etag = blobStore.putBlob(defaultContainerName, blob);
+ assertNotNull(etag);
+ total_size += data.length();
+ }
+
+ getApi().getDynamicLargeObjectApi(regionId,
defaultContainerName).putManifest(objectName,
+ImmutableMap.of("myfoo", "Bar"));
+
There is trailing white space on this line. I don't know if there is a way to
get github to highlight it.
> + SwiftObject object1s = getApi().getObjectApi(regionId,
> containerName).get(name);
+ assertThat(object1s.getETag()).isEqualTo(etag1s);
+
assertThat(object1s.getPayload().getContentMetadata().getContentLength()).isEqualTo(Long.valueOf(1024L
* 1024L));
+ }
+
+ protected void deleteAllObjectsInContainerDLO(String regionId, final String
containerName) {
+ ObjectList objects = getApi().getObjectApi(regionId,
containerName).list(new ListContainerOptions());
+ if (objects == null) {
+ return;
+ }
+ List pathsToDelete = Lists.transform(objects, new
Function() {
+ public String apply(SwiftObject input) {
+return containerName + "/" + input.getName();
+ }
+ });
+
Trailing white space here and line 134.
> + @SuppressWarnings("deprecation")
+ @Test
+ public void uploadLargeFile() throws Exception {
+ MockWebServer server = mockOpenStackServer();
+ server.enqueue(addCommonHeaders(new
MockResponse().setBody(stringFromResource("/access.json";
+ server.enqueue(new MockResponse().setBody("").addHeader(ETAG,
"89d903bc35dede724fd52c51437ff5fd"));
+ server.enqueue(new MockResponse().setBody("").addHeader(ETAG,
"d41d8cd98f00b204e9800998ecf8427e"));
+ server.enqueue(addCommonHeaders(new
MockResponse().addHeader("X-Object-Manifest", "myContainer/myObject")));
+
+ try {
+ SwiftApi api = api(server.getUrl("/").toString(), "openstack-swift");
+ assertEquals(api.getObjectApi("DFW",
containerName).put(objectName.concat("1"), Payloads.newPayload("data1")),
+ "89d903bc35dede724fd52c51437ff5fd");
+ assertEquals(api.getDynamicLargeObjectApi("DFW",
containerName).putManifest(objectName,
+ ImmutableMap.of("MyFoo", "Bar"), ImmutableMap.of("MyFoo",
"Bar")), "d41d8cd98f00b204e9800998ecf8427e");
+
Trailing white space here.
> + MockWebServer server = mockOpenStackServer();
+ server.enqueue(addCommonHeaders(new
MockResponse().setBody(stringFromResource("/access.json";
+ server.enqueue(new MockResponse().setBody("").addHeader(ETAG,
"89d903bc35dede724fd52c51437ff5fd"));
+ server.enqueue(new MockResponse().setBody("").addHeader(ETAG,
"d41d8cd98f00b204e9800998ecf8427e"));
+ server.enqueue(addCommonHeaders(new
MockResponse().addHeader("X-Object-Manifest", "myContainer/myObject")));
+
+ try {
+ SwiftApi api = api(server.getUrl("/").toString(), "openstack-swift");
+ assertEquals(api.getObjectApi("DFW",
containerName).put(objectName.concat("1"), Payloads.newPayload("data1")),
+ "89d903bc35dede724fd52c51437ff5fd");
+ assertEquals(api.getDynamicLargeObjectApi("DFW",
containerName).putManifest(objectName,
+ ImmutableMap.of("MyFoo", "Bar"), ImmutableMap.of("MyFoo",
"Bar")), "d41d8cd98f00b204e9800998ecf8427e");
+
+ assertEquals(server.getRequestCount(), 3);
+ assertAuthentication(server);
+
Trailing white space.
> +
+ try {
+ SwiftApi api = api(server.getUrl("/").toString(), "openstack-swift");
+ assertEquals(api.getObjectApi("DFW",
containerName).put(objectName.concat("1"), Payloads.newPayload("data1")),
+ "89d903bc35dede724fd52c51437ff5fd");
+ assertEquals(api.getDynamicLargeObjectApi("DFW",
containerName).putManifest(objectName,
+ ImmutableMap.of("MyFoo", "Bar"), ImmutableMap.of("MyFoo",
"Bar")), "d41d8cd98f00b204e9800998ecf8427e");
+
+ assertEquals(server.getRequestCount(), 3);
+ assertAuthentication(server);
+
+ RecordedRequest uploadRequest = server.takeRequest();
+ assertEquals(uploadRequest.getRequestLine(),
+ "PUT
/v1/MossoCloudFS_5bcf396e-39dd-45ff-93a1-712b9aba90a9/myContainer/myObjectT
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Great! Tests pass for me. Unfortunately, they take a long time, but I think that's related to the jclouds handling of the containers to be created for testing. @archupsg03 there are some small nits with white space changes in the commits. Before merging, can you make sure to remove these? I'll let @andrewgaul comment on any other changes he'd like to see. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1105#issuecomment-313310088
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
timuralp commented on this pull request.
@archupsg03 please try running the live tests. I encountered a few issues when
try to do that.
> + public void testReplaceManifest() throws Exception {
+ for (String regionId : regions) {
+ assertReplaceManifest(regionId, defaultContainerName, defaultName);
+ uploadLargeFile(regionId);
+ }
+ }
+
+ @SuppressWarnings("deprecation")
+ @Test
+ public void uploadLargeFile(String regionId) throws IOException,
InterruptedException {
+ int partNumber = 1;
+ int total_size = 0;
+ RegionScopedBlobStoreContext ctx =
RegionScopedBlobStoreContext.class.cast(view);
+ BlobStore blobStore = ctx.getBlobStore();
+ // configure the blobstore to use multipart uploading of the file
+ for (int i = partNumber; i <= 3; partNumber++) {
This looks like a bug. Wouldn't `i` be incremented? As is, I have a test that
never completes, but creates 1000s of parts.
> + public String apply(SwiftObject input) {
+return containerName + "/" + input.getName();
+ }
+ });
+
+ for (String name : pathsToDelete)
+ getApi().getObjectApi(regionId, containerName).delete(name);
+
+ }
+
+ @Override
+ @BeforeClass(groups = "live")
+ public void setup() {
+ super.setup();
+ for (String regionId : regions) {
+ boolean created =
getApi().getContainerApi(regionId).create(defaultContainerName);
You shouldn't have to do this. The jclouds tests provide containers. Check out
this method and the S3 provider on how to use them:
`blobstore/src/test/java/[...]/integration/internal/BaseBlobStoreIntegrationTest.java:434`.
This a copy/paste from the SLO test, but we shouldn't use the same pattern.
> + RegionScopedBlobStoreContext ctx =
> RegionScopedBlobStoreContext.class.cast(view);
+ BlobStore blobStore = ctx.getBlobStore();
+ // configure the blobstore to use multipart uploading of the file
+ for (int i = partNumber; i <= 3; partNumber++) {
+ String objName = String.format("%s/%s/%s", objectName, "dlo",
partNumber);
+ String data = String.format("%s%s", "data", partNumber);
+ ByteSource payload = ByteSource.wrap(data.getBytes(Charsets.UTF_8));
+ Blob blob = blobStore.blobBuilder(objName)
+ .payload(payload)
+ .build();
+ String etag = blobStore.putBlob(defaultContainerName, blob);
+ assertNotNull(etag);
+ total_size += data.length();
+ }
+
+ SwiftObject bigObject = getApi().getObjectApi(regionId,
defaultContainerName).get(objectName);
After fixing the above bug, I get a null pointer exception here.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#pullrequestreview-47052241
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
@archupsg03 thanks! This looks pretty good. I will try to run the tests this weekend to verify. @andrewgaul is still traveling for next couple of weeks, so we'll have to wait until the first week of July or so to merge this and for his comments on the PR. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1105#issuecomment-310580163
[jira] [Created] (JCLOUDS-1313) Filesystem blob store does not handle empty string LIST options properly
Timur Alperovich created JCLOUDS-1313:
-
Summary: Filesystem blob store does not handle empty string LIST
options properly
Key: JCLOUDS-1313
URL: https://issues.apache.org/jira/browse/JCLOUDS-1313
Project: jclouds
Issue Type: Bug
Components: jclouds-blobstore
Reporter: Timur Alperovich
When attempting to list a container and setting prefix. delimiter, and marker
to "", the filesystem blobstore does not return any entries. Against AWS,
setting the values to an empty is tantamount to not setting them at all.
Here is an example that fails. A similar test against AWS succeeds.
{code}
public void testList_EmptyOptionSingleContainer() throws IOException {
blobStore.createContainerInLocation(null, CONTAINER_NAME);
checkForContainerContent(CONTAINER_NAME, null);
TestUtils.createBlobsInContainer(CONTAINER_NAME, "a", "b", "c");
// Test listing where we set the prefix and delimiter to empty string
ListContainerOptions options = ListContainerOptions.Builder.delimiter("")
.prefix("").afterMarker("");
PageSet rs = blobStore.list(CONTAINER_NAME,
options);
assertEquals(rs.size(), 3);
Set expected = Sets.newHashSet("a", "b", "c");
for (StorageMetadata sm : rs) {
assertTrue(expected.contains(sm.getName()));
expected.remove(sm.getName());
}
assertTrue(expected.isEmpty());
}
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
timuralp commented on this pull request.
> + SwiftObject object1s = getApi().getObjectApi(regionId,
> containerName).get(name);
+ assertThat(object1s.getETag().equals(etag1s));
+
assertThat(object1s.getPayload().getContentMetadata().getContentLength().equals(Long.valueOf(1024
* 1024)));
+ }
+
+ protected void deleteAllObjectsInContainerDLO(String regionId, final String
containerName) {
+ ObjectList objects = getApi().getObjectApi(regionId,
containerName).list(new ListContainerOptions());
+ if (objects == null) {
+ return;
+ }
+ List pathsToDelete = Lists.transform(objects, new
Function() {
+ public String apply(SwiftObject input) {
+return containerName + "/" + input.getName();
+ }
+ });
+ if (!pathsToDelete.isEmpty()) {
@archupsg03 can you remove this `if` clause? It's not a useful check. If the
list is empty, the loop is a NOOP.
> @@ -37,7 +37,7 @@
@Named("size_bytes")
private final long sizeBytes;
- private Segment(String path, String etag, long sizeBytes) {
+ public Segment(String path, String etag, long sizeBytes) {
@archupsg03 I think we can remove this change?
> +* @param metadata
+* corresponds to {@link SwiftObject#getMetadata()}.
+* @param headers
+* Binds the map to headers, without prefixing/escaping the header
+* name/key.
+*
+* @return {@link SwiftObject#getEtag()} of the object, which is the MD5
+* checksum of the concatenated ETag values of the {@code segments}.
+*/
+
+ @Deprecated
+ @Named("dynamicLargeObject:uploadLargeFile")
+ @PUT
+ @Headers(keys = EXPECT, values = "100-continue")
+ @ResponseParser(ETagHeader.class)
+ String uploadPart(@PathParam("containerName") String container,
@PathParam("objectName") String objectName,
Sorry -- I called the method wrong. It's put() in ObjectApi or putBlob() in the
RegionScopedSwiftBlobStore class. I suspect consumers should call putBlob() and
then upload the manifest through the API you're adding. You can probably call
either one in the tests.
> +
> assertThat(object1s.getPayload().getContentMetadata().getContentLength()).isEqualTo(Long.valueOf(1024L
> * 1024L));
+ }
+
+ protected void deleteAllObjectsInContainerDLO(String regionId, final String
containerName) {
+ ObjectList objects = getApi().getObjectApi(regionId,
containerName).list(new ListContainerOptions());
+ if (objects == null) {
+ return;
+ }
+ List pathsToDelete = Lists.transform(objects, new
Function() {
+ public String apply(SwiftObject input) {
+return containerName + "/" + input.getName();
+ }
+ });
+ if (!pathsToDelete.isEmpty()) {
+ for (String name : pathsToDelete)
+getApi().getObjectApi(regionId, containerName).delete(name);
The object API takes in the containerName. Does the containerName still need to
be in the object Name, as well? In the above List transformation, we prepend
containerName to every object -- that seems wrong or I'm missing something.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#pullrequestreview-42796224
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
timuralp commented on this pull request.
> +* @param metadata
+* corresponds to {@link SwiftObject#getMetadata()}.
+* @param headers
+* Binds the map to headers, without prefixing/escaping the header
+* name/key.
+*
+* @return {@link SwiftObject#getEtag()} of the object, which is the MD5
+* checksum of the concatenated ETag values of the {@code segments}.
+*/
+
+ @Deprecated
+ @Named("dynamicLargeObject:uploadLargeFile")
+ @PUT
+ @Headers(keys = EXPECT, values = "100-continue")
+ @ResponseParser(ETagHeader.class)
+ String uploadPart(@PathParam("containerName") String container,
@PathParam("objectName") String objectName,
Sorry -- I called the method wrong. It's `put()` in `ObjectApi` or `putBlob()`
in the `RegionScopedSwiftBlobStore` class. I suspect consumers should call
`putBlob()` and then upload the manifest through the API you're adding. You can
probably call either one in the tests.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r120803915
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
timuralp commented on this pull request.
I think this is getting better. Please addres the outstanding comments.
> @@ -37,7 +37,7 @@
@Named("size_bytes")
private final long sizeBytes;
- private Segment(String path, String etag, long sizeBytes) {
+ public Segment(String path, String etag, long sizeBytes) {
Is this change still necessary? Is it only used in tests?
> +@RequestFilters(AuthenticateRequest.class)
+@Consumes(APPLICATION_JSON)
+@Path("/{objectName}")
+public interface DynamicLargeObjectApi {
+ /**
+* Creates or updates a dynamic large object's manifest.
+*
+* @param objectName
+* corresponds to {@link SwiftObject#getName()}.
+* @param metadata
+* corresponds to {@link SwiftObject#getMetadata()}.
+* @param headers
+* Binds the map to headers, without prefixing/escaping the header
+* name/key.
+*
+* @return {@link SwiftObject#getEtag()} of the object, which is the MD5
As a result of a PUT of a 0-sized object with X-Object-Manifest header set, I'm
observing that Swift returns an ETag of the 0-sized object. Are you observing a
different behavior in your tests?
> + @PUT
+ @ResponseParser(ETagHeader.class)
+ @Headers(keys = "X-Object-Manifest", values =
"{containerName}/{objectName}/")
+ String putManifest(@PathParam("objectName") String objectName,
+ @BinderParam(BindObjectMetadataToHeaders.class) Map
metadata,
+ @BinderParam(BindToHeaders.class) Map headers);
+
+ /**
+* Creates or updates a dynamic large object's manifest.
+*
+* @param objectName
+* corresponds to {@link SwiftObject#getName()}.
+* @param metadata
+* corresponds to {@link SwiftObject#getMetadata()}.
+*
+* @return {@link SwiftObject#getEtag()} of the object, which is the MD5
Same comment as above about the ETag applies here.
> +* @param metadata
+* corresponds to {@link SwiftObject#getMetadata()}.
+* @param headers
+* Binds the map to headers, without prefixing/escaping the header
+* name/key.
+*
+* @return {@link SwiftObject#getEtag()} of the object, which is the MD5
+* checksum of the concatenated ETag values of the {@code segments}.
+*/
+
+ @Deprecated
+ @Named("dynamicLargeObject:uploadLargeFile")
+ @PUT
+ @Headers(keys = EXPECT, values = "100-continue")
+ @ResponseParser(ETagHeader.class)
+ String uploadPart(@PathParam("containerName") String container,
@PathParam("objectName") String objectName,
@archupsg03 can you address the questions above? I'm still unclear on why this
method is required. The point is that callers can and should call `putObject()`
to upload new or replace existing parts of the DLO. Could you explain how you
expect this method to be used and why it would be preferred to calling
`putObject()`?
> + private static final ByteSource megOf1s =
> TestUtils.randomByteSource().slice(0, 1024 * 1024);;
+ private static final ByteSource megOf2s =
TestUtils.randomByteSource().slice(0, 1024 * 1024);;
+ private String objectName = "myObject";
+
+ @Test
+ public void testReplaceManifest() throws Exception {
+ for (String regionId : regions) {
+ assertReplaceManifest(regionId, defaultContainerName, defaultName);
+ uploadLargeFile(regionId);
+ }
+ }
+
+ @SuppressWarnings("deprecation")
+ @Test
+ public void uploadLargeFile(String regionId) throws IOException,
InterruptedException {
+ List segmentList = new ArrayList();
This is no longer used in the test. Can we remove this code entirely? On the
other hand, it would be useful to upload some blobs and verify that the ETag is
returned as expected from a HEAD.
> +
> assertThat(getApi().getContainerApi(regionId).get(defaultContainerName).getObjectCount()).isEqualTo(Long.valueOf(3));
+ }
+
+ @SuppressWarnings("deprecation")
+ protected void assertReplaceManifest(String regionId, String containerName,
String name) {
+ ObjectApi objectApi = getApi().getObjectApi(regionId, containerName);
+
+ String etag1s = objectApi.put(name + "/1",
newByteSourcePayload(megOf1s));
+ awaitConsistency();
+ assertMegabyteAndETagMatches(regionId, containerName, name + "/1",
etag1s);
+
+ String etag2s = objectApi.put(name + "/2",
newByteSourcePayload(megOf2s));
+ awaitConsistency();
+ assertMegabyteAndETagMatches(regionId, containerName, name + "/2",
etag2s);
+
+ List segments = ImmutableList. builder()
I don't think these segments are ever uploaded?
> + SwiftObject object1s = getApi().getObjectApi(regionId,
> containerName).get(name);
+ assertThat(object1s.getETag().equals(etag1s));
+
assertThat(object1s.getPayload().getContentMetadata().getContentLength().equals(Long.valueOf(1024
* 1024)));
+ }
+
+ protected void deleteAllObjectsInContainerDLO(String regionId, final String
co
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
timuralp commented on this pull request.
> +* @param metadata
+* corresponds to {@link SwiftObject#getMetadata()}.
+* @param headers
+* Binds the map to headers, without prefixing/escaping the header
+* name/key.
+*
+* @return {@link SwiftObject#getEtag()} of the object, which is the MD5
+* checksum of the concatenated ETag values of the {@code segments}.
+*/
+ @Deprecated
+ @Named("dynamicLargeObject:replaceManifest")
+ @PUT
+ @ResponseParser(ETagHeader.class)
+ @Headers(keys = "X-Object-Manifest", values =
"{containerName}/{objectName}/")
+ String replaceManifest(@PathParam("objectName") String objectName,
+ @BinderParam(BindToJsonPayload.class) Collection segments,
Swift does not return the DLO ETag as a response to PUT. At least I do not
observe this with docker-swift and Swift 2.14 (latest release):
```
DEBUG:swiftclient:REQ: curl -i
http://localhost:32768/v1/AUTH_test/test/s3proxy.jar -X PUT -H
"x-object-meta-mtime: 1440388916.150022" -H "Content-Length: 0" -H
"x-object-manifest:
test_segments/s3proxy.jar/1440388916.150022/7663104/100/" -H "Content-Type:
" -H "X-Auth-Token: AUTH_tkadf8c03346214185ba387e393305"
DEBUG:swiftclient:RESP STATUS: 201 Created
DEBUG:swiftclient:RESP HEADERS: {u'Content-Length': u'0', u'Last-Modified':
u'Tue, 06 Jun 2017 06:09:31 GMT', u'Etag': u'd41d8cd98f00b204e9800998ecf8427e',
u'X-Trans-Id': u'tx9825d3a9fadb4c50bf4f8-005936471a', u'Date': u'Tue, 06 Jun
2017 06:09:30 GMT', u'Content-Type': u'text/html; charset=UTF-8',
u'X-Openstack-Request-Id': u'tx9825d3a9fadb4c50bf4f8-005936471a'}
```
The ETag above is of the empty string and jclouds should expose exactly that as
a response to a putManifest request.
In jclouds 1.9, the interaction you're describing is the end of performing an
MPU where a 0-sized DLO manifest is uploaded (0-sized object with the
X-Object-Manifest header). The two methods that are implemented are for the
case of specifying manifest metadata through an existing Blob or through the
object name. I don't think jclouds 2.0 should replicate either of the
approaches.
In jclouds 2.0, you could introduce two DLO methods, akin to the SLO setup,
where one has only the metadata parameter
(```@BinderParam(BindObjectMetadataToHeaders.class) Map
metadata```), while the other one includes a headers parameter
(```@BinderParam(BindToHeaders.class) Map headers```). That
would be inline with the current code.
We should not attempt to satisfy old clients, as the prior API has been marked
deprecated and jclouds should prefer to use SLO whenever possible. DLO is an
option only for clients who explicitly have to use it to interact with old data
or if they have some other special reason.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r120275386
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
timuralp commented on this pull request.
> +* @param metadata
+* corresponds to {@link SwiftObject#getMetadata()}.
+* @param headers
+* Binds the map to headers, without prefixing/escaping the header
+* name/key.
+*
+* @return {@link SwiftObject#getEtag()} of the object, which is the MD5
+* checksum of the concatenated ETag values of the {@code segments}.
+*/
+ @Deprecated
+ @Named("dynamicLargeObject:replaceManifest")
+ @PUT
+ @ResponseParser(ETagHeader.class)
+ @Headers(keys = "X-Object-Manifest", values =
"{containerName}/{objectName}/")
+ String replaceManifest(@PathParam("objectName") String objectName,
+ @BinderParam(BindToJsonPayload.class) Collection segments,
@archupsg03 I probably was not clear on this. I was suggesting that using the
Swift PutOptions class and setting the correct header (X-Object-Manifest) would
be sufficient with the existing PutObject API. However, talking to @andrewgaul,
I understand the preference for the separate API class, as it makes the DLO
separation from the regular object API obvious.
The X-Object-Manifest header is mandatory in the API, so that header must be
set -- that is correct.
In terms of your question on behavior, for DLO, Swift's behavior depends on the
number of segments. If the number of segments is less than the full listing,
Swift computes DLO ETag as an md5 sum of segment ETags. Otherwise, the ETag is
not included. [1]
I'll try to look into the prior jclouds DLO documentation to try to understand
your question about the 1.9 implementation.
1.
https://github.com/openstack/swift/blob/master/swift/common/middleware/dlo.py#L307
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r120170979
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
timuralp commented on this pull request.
It's a little difficult for me to review this in isolation, without the DLO
abstraction that is using this PR. I would expect very few new methods added
here -- specifically, only replaceManifest (as you have it, or putManifest that
I suggested), which has the DLO header. Outside of that, the other methods
don't make sense to me as-is (e.g. uploadPart is the same putObject, as far as
I can tell).
> +* @param metadata
+* corresponds to {@link SwiftObject#getMetadata()}.
+* @param headers
+* Binds the map to headers, without prefixing/escaping the header
+* name/key.
+*
+* @return {@link SwiftObject#getEtag()} of the object, which is the MD5
+* checksum of the concatenated ETag values of the {@code segments}.
+*/
+ @Deprecated
+ @Named("dynamicLargeObject:replaceManifest")
+ @PUT
+ @ResponseParser(ETagHeader.class)
+ @Headers(keys = "X-Object-Manifest", values =
"{containerName}/{objectName}/")
+ String replaceManifest(@PathParam("objectName") String objectName,
+ @BinderParam(BindToJsonPayload.class) Collection segments,
With DLO, manifest header specifies a segment prefix. How does the segments
collection get used? This seems like a copy of the SLO API, but I don't think
it actually makes sense here -- in the SLO case, the actual JSON manifest
object is constructed from the segments.
> +* ordered parts which will be concatenated upon download.
+* @param metadata
+* corresponds to {@link SwiftObject#getMetadata()}.
+* @param headers
+* Binds the map to headers, without prefixing/escaping the header
+* name/key.
+*
+* @return {@link SwiftObject#getEtag()} of the object, which is the MD5
+* checksum of the concatenated ETag values of the {@code segments}.
+*/
+ @Deprecated
+ @Named("dynamicLargeObject:replaceManifest")
+ @PUT
+ @ResponseParser(ETagHeader.class)
+ @Headers(keys = "X-Object-Manifest", values =
"{containerName}/{objectName}/")
+ String replaceManifest(@PathParam("objectName") String objectName,
I understand that this is the same as the SLO API, but the replaceManifest name
seems like a misnomer. Maybe `putManifest` would make more sense? Don't have a
strong preference on this, though.
> + * use it and give us feedback. Based on that feedback, minor changes to the
+ * interfaces may happen. This code will replace
+ * org.jclouds.openstack.swift.SwiftClient in jclouds 2.0 and it is recommended
+ * you adopt it sooner than later.
+ */
+@Beta
+@RequestFilters(AuthenticateRequest.class)
+@Consumes(APPLICATION_JSON)
+@Path("/{objectName}")
+public interface DynamicLargeObjectApi {
+ /**
+* Creates or updates a dynamic large object's manifest.
+*
+* @param objectName
+* corresponds to {@link SwiftObject#getName()}.
+* @param segments
This comment does not make sense in the DLO context. See a comment below about
the segments parameter.
> +* @param metadata
+* corresponds to {@link SwiftObject#getMetadata()}.
+* @param headers
+* Binds the map to headers, without prefixing/escaping the header
+* name/key.
+*
+* @return {@link SwiftObject#getEtag()} of the object, which is the MD5
+* checksum of the concatenated ETag values of the {@code segments}.
+*/
+
+ @Deprecated
+ @Named("dynamicLargeObject:uploadLargeFile")
+ @PUT
+ @Headers(keys = EXPECT, values = "100-continue")
+ @ResponseParser(ETagHeader.class)
+ String uploadPart(@PathParam("containerName") String container,
@PathParam("objectName") String objectName,
Is this any different from `PutObject`? I think a higher level wrapper is
required to make DLO operations make sense, but I don't think this API method
is required. Maybe you could explain how you envision `uploadPart` to fit with
the higher level primitives.
> + @Deprecated
+ @Named("dynamicLargeObject:replaceManifest")
+ @PUT
+ @ResponseParser(ETagHeader.class)
+ @Headers(keys = "X-Object-Manifest", values =
"{containerName}/{objectName}/")
+ String replaceManifest(@PathParam("objectName") String objectName,
+ @BinderParam(BindToJsonPayload.class) Collection segments,
+ @BinderParam(BindObjectMetadataToHeaders.class) Map
metadata,
+ @BinderParam(BindToHeaders.class) Map headers);
+
+ /**
+* Creates object segments.
+*
+* @param objectName
+* corresponds to {@link SwiftObject#getName()}.
+* @param segments
Copy/paste comment error? I don't see a segments parameter.
> +* checksum of the concatenated ETag values of the {@code
> segments}.
+*/
+ @Deprecated
+ @Named("dynamicLargeObject:replaceManifest")
+ @PUT
+ @ResponseParser(ETagHeader.class)
+ @Headers(keys = "X-Object-Manifest", values =
"{containerName}/{objectName
Re: [jclouds/jclouds] JCLOUDS-1299: Handle % in names during Swift list (#1101)
Looks good to me! :+1: -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1101#issuecomment-302011333
[jira] [Created] (JCLOUDS-1285) Store large object segments in a separate container
Timur Alperovich created JCLOUDS-1285: - Summary: Store large object segments in a separate container Key: JCLOUDS-1285 URL: https://issues.apache.org/jira/browse/JCLOUDS-1285 Project: jclouds Issue Type: Improvement Components: jclouds-blobstore Affects Versions: 2.0.1 Environment: OpenStack Swift Reporter: Timur Alperovich Swift recommends storing the large object segments in a separate container [1]. I propose changing jclouds to do the same. The Swift convention -- in their python library -- is to create a "_segments" container for the segments. We need to consider ways to reduce the number of HEAD requests jclouds would make for container existence. One possible way could be to assume the container exists for the segments and create it if the PUT requests return 404. I will need to look through the existing MPU abstraction to figure out the other caveats to work out in this improvement. 1. https://docs.openstack.org/developer/swift/overview_large_objects.html -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jclouds/jclouds] GetBucketLocation should use path-style requests. (#1052)
When making a GetBucketLocation request, Amazon may route the request
to the bucket region. When making it with v4 signer, the request may
fail because of the region mismatch. Concretely, a request to
test.s3.amazonaws.com may resolve to s3-us-west-2-w.amazonaws.com. The
request itself is prepared for the us-east-1 region (s3.amazonaws.com
endpoint), but then fails when the DNS resolution points to a
us-west-2 endpoint.
Bucket-in-path works around this for the GetBucketLocation requests.
That means that every GetBucketLocation request will be of the form:
https://s3.amazonaws.com/{bucket}?location. This ensures that jclouds
requests will not be subjected to Amazon's routing/DNS pointers.
Fixes: JCLOUDS-1213
You can view, comment on, or merge this pull request online at:
https://github.com/jclouds/jclouds/pull/1052
-- Commit Summary --
* GetBucketLocation should use path-style requests.
-- File Changes --
M apis/s3/src/main/java/org/jclouds/s3/S3Client.java (4)
M
providers/aws-s3/src/test/java/org/jclouds/aws/s3/AWSS3ClientExpectTest.java (5)
M providers/aws-s3/src/test/java/org/jclouds/aws/s3/AWSS3ClientTest.java
(10)
-- Patch Links --
https://github.com/jclouds/jclouds/pull/1052.patch
https://github.com/jclouds/jclouds/pull/1052.diff
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1052
[jira] [Assigned] (JCLOUDS-1213) S3: Region is not respected
[
https://issues.apache.org/jira/browse/JCLOUDS-1213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timur Alperovich reassigned JCLOUDS-1213:
-
Assignee: Timur Alperovich
> S3: Region is not respected
> ---
>
> Key: JCLOUDS-1213
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1213
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
>Affects Versions: 2.0.0
>Reporter: Halvdan Hoem Grelland
> Assignee: Timur Alperovich
> Labels: aws-s3
>
> When using a BlobStore configured for the 'aws-s3' provider and with an
> explicit region ('eu-west-1' in this case),
> _BlobStore.createContainerInLocation(Location location, String container)_
> fails with a 400 when attempting to create an already existing bucket on that
> location.
> The error message from AWS claims the cause to be
> _AuthorizationHeaderMalformed_, and says that 'eu-west-1' was expected, but
> that 'us-east-1' was supplied in the PUT request.
> I have confirmed that 'eu-west-1' is, in fact, given to the method call, but
> it does not seem to be respected when forming and signing the request.
> This does not happen on 1.9.x.
> From the debugging i have done a likely culprit seems to be
> _Aws4SignerForAuthorizationHeader_ which resolves a _ServiceAndRegion_
> instance, which in turn seem to attempt parsing the region from the hostname
> URL (_AWSHostNameUtils.parseRegionName(..)_). I cannot see the region set for
> the provider being respected in this case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (JCLOUDS-1213) S3: Region is not respected
[
https://issues.apache.org/jira/browse/JCLOUDS-1213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15819686#comment-15819686
]
Timur Alperovich commented on JCLOUDS-1213:
---
Reading more about this, I suspect I'm bumping into an issue similar to the one
reported here: https://github.com/aws/aws-sdk-js/issues/462.
Specifically, when I attempt to resolve the bucket in question, I get back the
following for a bucket in US-West-2:
{code}
timur-test.s3.amazonaws.com is an alias for s3-us-west-2-w.amazonaws.com.
s3-us-west-2-w.amazonaws.com has address 54.231.184.246
{code}
At this point the GetBucketLocation request, which is meant to go to
timur-test.s3.amazonaws.com in us-east-1 gets re-routed to a us-west-2 endpoint
and results in the error I'm seeing. I investigated how the boto3 library
handles this and they use bucket in-path requests. I'm going to submit a pull
request for jclouds to do the same. For the particular failure, I suspect the
difference in networking resulted in my issue, which was not reproducible by
Andrew.
> S3: Region is not respected
> ---
>
> Key: JCLOUDS-1213
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1213
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
>Affects Versions: 2.0.0
>Reporter: Halvdan Hoem Grelland
> Labels: aws-s3
>
> When using a BlobStore configured for the 'aws-s3' provider and with an
> explicit region ('eu-west-1' in this case),
> _BlobStore.createContainerInLocation(Location location, String container)_
> fails with a 400 when attempting to create an already existing bucket on that
> location.
> The error message from AWS claims the cause to be
> _AuthorizationHeaderMalformed_, and says that 'eu-west-1' was expected, but
> that 'us-east-1' was supplied in the PUT request.
> I have confirmed that 'eu-west-1' is, in fact, given to the method call, but
> it does not seem to be respected when forming and signing the request.
> This does not happen on 1.9.x.
> From the debugging i have done a likely culprit seems to be
> _Aws4SignerForAuthorizationHeader_ which resolves a _ServiceAndRegion_
> instance, which in turn seem to attempt parsing the region from the hostname
> URL (_AWSHostNameUtils.parseRegionName(..)_). I cannot see the region set for
> the provider being respected in this case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (JCLOUDS-1213) S3: Region is not respected
[
https://issues.apache.org/jira/browse/JCLOUDS-1213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15819568#comment-15819568
]
Timur Alperovich commented on JCLOUDS-1213:
---
Thanks for checking. I can reproduce this consistently with 2.1.0-SNAPSHOT, as
well. I'll attempt to debug this locally. The specific difference that I'm
observing is the response to the bucket location request against the EU bucket:
{code}
>> GET https://timur-test-ireland.s3.amazonaws.com/?location HTTP/1.1
>> Host: timur-test-ireland.s3.amazonaws.com
>> x-amz-content-sha256:
>> e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
>> X-Amz-Date: 20170111T233842Z
>> Authorization: AWS4-HMAC-SHA256
>> Credential=AKIAIT4YWLZX26RHCRKQ/20170111/us-east-1/s3/aws4_request,
>> SignedHeaders=host;x-amz-content-sha256;x-amz-date,
>> Signature=2ca401371c162bd8e08fd161c2c330d5f887d08ae4f5996d3d2040e9032be86d
<< HTTP/1.1 400 Bad Request
<< Transfer-Encoding: chunked
<< Server: AmazonS3
<< Connection: close
<< x-amz-request-id: 580278C04C186502
<< x-amz-id-2:
1lEvRCBWBmb3Tt/xe8T7u7HVuUNCT3hNWfzfd3VwKFWhmE56rf2y2iaRk1UCT1YkMK78Q+hpKlI=
<< Date: Wed, 11 Jan 2017 23:37:57 GMT
<< Content-Type: application/xml
org.jclouds.aws.AWSResponseException: request GET
https://timur-test-ireland.s3.amazonaws.com/?location HTTP/1.1 failed with code
400, error: AWSError{requestId='580278C04C186502',
requestToken='1lEvRCBWBmb3Tt/xe8T7u7HVuUNCT3hNWfzfd3VwKFWhmE56rf2y2iaRk1UCT1YkMK78Q+hpKlI=',
code='AuthorizationHeaderMalformed', message='The authorization header is
malformed; the region 'us-east-1' is wrong; expecting 'eu-west-1'',
context='{Region=eu-west-1,
HostId=1lEvRCBWBmb3Tt/xe8T7u7HVuUNCT3hNWfzfd3VwKFWhmE56rf2y2iaRk1UCT1YkMK78Q+hpKlI=}'}
{code}
> S3: Region is not respected
> ---
>
> Key: JCLOUDS-1213
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1213
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
>Affects Versions: 2.0.0
>Reporter: Halvdan Hoem Grelland
> Labels: aws-s3
>
> When using a BlobStore configured for the 'aws-s3' provider and with an
> explicit region ('eu-west-1' in this case),
> _BlobStore.createContainerInLocation(Location location, String container)_
> fails with a 400 when attempting to create an already existing bucket on that
> location.
> The error message from AWS claims the cause to be
> _AuthorizationHeaderMalformed_, and says that 'eu-west-1' was expected, but
> that 'us-east-1' was supplied in the PUT request.
> I have confirmed that 'eu-west-1' is, in fact, given to the method call, but
> it does not seem to be respected when forming and signing the request.
> This does not happen on 1.9.x.
> From the debugging i have done a likely culprit seems to be
> _Aws4SignerForAuthorizationHeader_ which resolves a _ServiceAndRegion_
> instance, which in turn seem to attempt parsing the region from the hostname
> URL (_AWSHostNameUtils.parseRegionName(..)_). I cannot see the region set for
> the provider being respected in this case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (JCLOUDS-1213) S3: Region is not respected
[
https://issues.apache.org/jira/browse/JCLOUDS-1213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15819472#comment-15819472
]
Timur Alperovich commented on JCLOUDS-1213:
---
I'm running into a similar problem when attempting to list contents of a
non-US-Standard bucket.
Here is the simple failing example (fails with jclouds 2.0.0):
{code}
public class TestMain {
public static void main(String[] args) throws Exception {
BlobStoreContext context = ContextBuilder.newBuilder("aws-s3")
.credentials("AWS-ACCESS-KEY-ID", "SECRET-ACCESS-KEY")
.buildView(BlobStoreContext.class);
BlobStore blobStore = context.getBlobStore();
for (StorageMetadata meta: blobStore.list("EU-West-1-Bucket")) {
System.out.println(meta.getName());
}
}
}
{code}
The resulting error is:
{{message='The authorization header is malformed; the region 'us-east-1' is
wrong; expecting 'eu-west-1''}}
I suspect jclouds needs to check the bucket location before submitting the
request. I haven't figured out how to express that.
> S3: Region is not respected
> ---
>
> Key: JCLOUDS-1213
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1213
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
>Affects Versions: 2.0.0
>Reporter: Halvdan Hoem Grelland
> Labels: aws-s3
>
> When using a BlobStore configured for the 'aws-s3' provider and with an
> explicit region ('eu-west-1' in this case),
> _BlobStore.createContainerInLocation(Location location, String container)_
> fails with a 400 when attempting to create an already existing bucket on that
> location.
> The error message from AWS claims the cause to be
> _AuthorizationHeaderMalformed_, and says that 'eu-west-1' was expected, but
> that 'us-east-1' was supplied in the PUT request.
> I have confirmed that 'eu-west-1' is, in fact, given to the method call, but
> it does not seem to be respected when forming and signing the request.
> This does not happen on 1.9.x.
> From the debugging i have done a likely culprit seems to be
> _Aws4SignerForAuthorizationHeader_ which resolves a _ServiceAndRegion_
> instance, which in turn seem to attempt parsing the region from the hostname
> URL (_AWSHostNameUtils.parseRegionName(..)_). I cannot see the region set for
> the provider being respected in this case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jclouds/jclouds-labs-aws] Patches to be able to use the Glacier Proxy (#69)
I'm working on a Glacier proxy/emulator and jclouds seems like a good use case for it (so that one could more quickly run the integration tests, for example). There two changes that I've had to make: 1. the tests shouldn't assume an AWS endpoint (otherwise they won't pass with glacier proxy) 2. the `initial-wait` setting -- the time to wait before checking a job completed -- is a system property After these changes, I was able to pass all of the jclouds tests with the glacier proxy. You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds-labs-aws/pull/69 -- Commit Summary -- * Do not assume an endpoint in tests. * Add a system setting for Glacier's initial wait. -- File Changes -- M glacier/src/main/java/org/jclouds/glacier/blobstore/strategy/internal/BasePollingStrategy.java (3) M glacier/src/test/java/org/jclouds/glacier/GlacierClientLiveTest.java (8) -- Patch Links -- https://github.com/jclouds/jclouds-labs-aws/pull/69.patch https://github.com/jclouds/jclouds-labs-aws/pull/69.diff --- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-labs-aws/pull/69
Re: [jclouds] JCLOUDS-1065: Deprecate inDirectory option (#903)
I like this change and believe we should be able to remove the directory notions out of jclouds! Having said that, there are some providers where the behavior between delimiter/prefix and directory notions is not quite identical (I won't mention the file system provider, because it's pretty convoluted there anyway). In swift, the path parameter (what is used for directories) may return different results than when using delimiter/prefix. It supports the notion of directory blobs, which prefix and delimiter would not actually implicitly create. However, I don't know how many people actually rely on that functionality. I do think jclouds should deprecate support for directory blobs and move away from using the path parameter. None of the above precludes deprecating the support, but to actually switch over, consumers may need to do additional work on top of changing the parameters. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/903#issuecomment-174037917
Re: [jclouds] Move QueryParam encoding to a separate class. (#872)
@nacx is this in line with what you had in mind? --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/872#issuecomment-151341254
[jclouds] Move QueryParam encoding to a separate class. (#872)
The patch implements a QueryValue class, which encodes the underlying value based on whether the "encoded" flag is set. This class is used by the RestAnnotationProcessor to propagate the @Encoded value set on any parameters. Since the encoding is now handled by the QueryValue instances, we should no longer call encodeQueryLine() in the URI builder and instead call buildQueryLine(). The caveat is that we need to make sure all of the parameters that may need to be encoded are converted to QueryValue objects. This is done by converting Object instances to QueryValue by an instance of the TransformObjectToQueryValue when adding any query parameters to the URI. You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds/pull/872 -- Commit Summary -- * Move QueryParam encoding to a separate class. -- File Changes -- M core/src/main/java/org/jclouds/http/Uris.java (50) A core/src/main/java/org/jclouds/http/utils/QueryValue.java (62) M core/src/main/java/org/jclouds/rest/internal/RestAnnotationProcessor.java (33) M core/src/test/java/org/jclouds/http/UrisTest.java (39) -- Patch Links -- https://github.com/jclouds/jclouds/pull/872.patch https://github.com/jclouds/jclouds/pull/872.diff --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/872
Re: [jclouds-labs-google] JCLOUDS-1008: Use @Encoded with GCS. (#167)
Sorry -- my fault. I forgot to check that file in. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-labs-google/pull/167#issuecomment-149724922
Re: [jclouds-labs-google] JCLOUDS-1008: Use @Encoded with GCS. (#167)
I do not believe so. This should test and merge cleanly (at least I did run the tests locally before uploading with that commit in place). --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-labs-google/pull/167#issuecomment-149721410
Re: [jclouds] JCLOUDS-1008: Add @Encoded annotation. (#861)
I'd like to keep the ```UriBuilder``` as dumb as possible. These encoding issues arose from the fact that the encoding logic got baked into the ```UriBuilder```, making it hard to maintain a clean abstraction. Moving encoding out of it makes the most sense. The only question is what makes more sense: annotation processor or an auxiliary class, such as ```QueryValue```. Using ```QueryValue``` will allow us to sink the conditional around encoding and out of the annotation processor itself, so that seems like a win. On the other hand, the code around extracting the ```@Encoded``` value remains. Let me give it a shot and see what this might look like. Also, should we split off this work into another PR? It's becoming a bit more involved and blocks fixing the actual bug in GCS (addressed by the first commit in this PR). --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/861#issuecomment-149387151
Re: [jclouds-labs-google] JCLOUDS-1008: Use @Encoded with GCS. (#167)
I updated the PR inline with https://github.com/jclouds/jclouds/pull/861, however this PR will continue to fail until it is merged. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-labs-google/pull/167#issuecomment-148863921
Re: [jclouds] JCLOUDS-1008: Add @Encoded annotation. (#861)
Added ```@Encoded``` to the ```QueryParam```. In the process, all of the encoding handling for ```QueryParam``` and ```QueryParams``` was moved into the ```RestAnnotationProcessor```. I tried to explain the reasoning and give some context in the commit comment. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/861#issuecomment-148863454
Re: [jclouds] JCLOUDS-1008: Add @Encoded annotation. (#861)
If I'm reading ```http/utils/Queries.java``` correctly, the parameters are individually encoded, since the ```UriBuilder``` class will call ```encodeQueryLine()```, which in turn encodes the string before appending it to the URI. In that sense, I think you're right that maybe adding ```@Encoded``` to the query parameters may make sense to indicate that jclouds should skip encoding the parameter value. Do you think this is something that should be added now or deferred until there is a use case for it? --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/861#issuecomment-147006237
Re: [jclouds] JCLOUDS-1008: Add @Encoded annotation. (#861)
No problem -- the original approach was still not quite right, as I used @encoded to indicate that a parameter should be encoded, rather than it was previously encoded. I reworked it to be inline with the ```javax.ws.rs.Encoded``` documentation above. Let me know if it still doesn't make sense! I'll push an update to google labs to use this annotation. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/861#issuecomment-146660139
Re: [jclouds] JCLOUDS-1008: Add @Encoded annotation. (#861)
@nacx I think you're right -- that does make more sense. The contract then is that the user can either use the old style path encoding where jclouds will encode the entire path string once (and not encode "/" in parameters) or encode each parameter individually. Honestly, I considered changing jclouds to the second model altogether from its current behavior, but was concerned about breaking any consumers that may rely on the behavior. For example, in S3 the blob name should include "/" in the request and not encode it (although that could be resolved with ```@SkipEncoding``` on the parameter level). --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/861#issuecomment-144857229
Re: [jclouds] JCLOUDS-1008: Add @Encoded annotation. (#861)
> @@ -147,6 +149,7 @@ public Part apply(Entry from) {
> private final GetAcceptHeaders getAcceptHeaders;
> private final Invocation caller;
> private final boolean stripExpectHeader;
> + private boolean encodedParams;
Ok -- let me rework this and remove the member variable.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/861/files#r40968631
Re: [jclouds-labs-google] FIx issue with blob name / encoding for getBlob method (#166)
Sounds good. Opened PR for jclouds core changes and google-cloud-storage: https://github.com/jclouds/jclouds/pull/861 and https://github.com/jclouds/jclouds-labs-google/pull/167 --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-labs-google/pull/166#issuecomment-144279640
[jclouds-labs-google] JCLOUDS-1008: Use @Encoded with GCS. (#167)
Google cloud storage should use the @Encoded annotation with the object names to make sure that the object is percent-encoded prior to being submitted in the path of the request. This was previously broken because the default path encoding ignores "/" and encodes the entire string. The @Encoded decorator allows jclouds rest annotation processor to encode the values being placed in the path instead. Requires https://github.com/jclouds/jclouds/pull/861 to be merged. You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds-labs-google/pull/167 -- Commit Summary -- * JCLOUDS-1008: Use @Encoded with GCS. -- File Changes -- M google-cloud-storage/src/main/java/org/jclouds/googlecloudstorage/blobstore/GoogleCloudStorageBlobStore.java (21) M google-cloud-storage/src/main/java/org/jclouds/googlecloudstorage/features/ObjectAccessControlsApi.java (30) M google-cloud-storage/src/main/java/org/jclouds/googlecloudstorage/features/ObjectApi.java (65) M google-cloud-storage/src/test/java/org/jclouds/googlecloudstorage/blobstore/integration/GoogleCloudStorageBlobIntegrationLiveTest.java (1) -- Patch Links -- https://github.com/jclouds/jclouds-labs-google/pull/167.patch https://github.com/jclouds/jclouds-labs-google/pull/167.diff --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-labs-google/pull/167
[jclouds-labs-google] GCS delimiter test should not be special cased. (#168)
GCS delimiter implementation now works similarly to the other providers (fixed in a prior commit) and we do not need to special case the test. You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds-labs-google/pull/168 -- Commit Summary -- * GCS delimiter test should not be special cased. -- File Changes -- M google-cloud-storage/src/test/java/org/jclouds/googlecloudstorage/blobstore/integration/GoogleCloudStorageContainerIntegrationLiveTest.java (19) -- Patch Links -- https://github.com/jclouds/jclouds-labs-google/pull/168.patch https://github.com/jclouds/jclouds-labs-google/pull/168.diff --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-labs-google/pull/168
[jclouds] JCLOUDS-1008: Add @Encoded annotation. (#861)
Certain providers (e.g. Google Cloud Storage) place tokens that should be encoded in the request path (e.g. GET http:///b//o/) and expect them to be percent-encoded. In the above example a GET request for "foo/bar" should be translated to http:///b//o/foo%2Fbar. Currently, there is no way to express this in jclouds, as the entire request path is encoded exactly once and there is no control over whether a request parameter should be handled specially. In the example above, "/" are not encoded in the path and the URL is submitted as "http:///b//o/foo/bar", which may be wrong. This patch extends the annotation processor to support @Encoded for the individual parameters of the request. However, this means that the entire path is _NOT_ URL encoded. The caller *must* make sure that the appropriate parameters are encoded. You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds/pull/861 -- Commit Summary -- * JCLOUDS-1008: Add @Encoded annotation. -- File Changes -- M core/src/main/java/org/jclouds/http/Uris.java (21) M core/src/main/java/org/jclouds/rest/internal/RestAnnotationProcessor.java (20) M core/src/test/java/org/jclouds/rest/internal/RestAnnotationProcessorTest.java (14) -- Patch Links -- https://github.com/jclouds/jclouds/pull/861.patch https://github.com/jclouds/jclouds/pull/861.diff --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/861
Re: [jclouds-labs-google] FIx issue with blob name / encoding for getBlob method (#166)
@freddy33 yes, it is strange, but I believe it's ok. For context: in 1.9, jclouds annotation processor will encode the entirety of the path. There are some caveats with how it works and known issues (see https://issues.apache.org/jira/browse/JCLOUDS-217 and associated commits for more details), but that's the gist. The annotation processor, however, will not encode any "/" characters in the path, so a request for "http://host/b/bucket/o/foo/bar"; is not translated to "http://host/b/bucket/o/foo%2Fbar";. The prior behavior was to fix this up in the google provider, but clearly not complete for all places where it is called. :+1: --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-labs-google/pull/166#issuecomment-144139959
Re: [jclouds-labs-google] FIx issue with blob name / encoding for getBlob method (#166)
@freddy33 I believe the same problem exists in the ACL controls. The file to look at is ```ObjectAccessControlsApi.java```. Anytime a path parameter may include "/", we would need to escape it. This is an issue for all requests in that API. I'm going to fix this problem on master (and the upcoming jclouds 2.0 release) by extending the jclouds annotation processor to consider and honor an ```@Encoded``` annotation that may be attached to any request. I will then go through and annotate all of the Google Storage requests appropriately. Filed a JIRA issue here: https://issues.apache.org/jira/browse/JCLOUDS-1008 As for the 1.9 fix-up, it looks fine to me, aside from not using ```Strings2.encode()``` -- could you change the PR to do that? Fixing up the ACL calls would also be preferred. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-labs-google/pull/166#issuecomment-143908824
[jira] [Created] (JCLOUDS-1008) Google storage provider cannot retrieve "foo/bar"
Timur Alperovich created JCLOUDS-1008:
-
Summary: Google storage provider cannot retrieve "foo/bar"
Key: JCLOUDS-1008
URL: https://issues.apache.org/jira/browse/JCLOUDS-1008
Project: jclouds
Issue Type: Bug
Components: jclouds-core, jclouds-labs-google
Affects Versions: 1.9.1, 1.9.0, 2.0.0
Reporter: Timur Alperovich
The Google cloud storage provider can PUT, but not GET a blob containing "/" in
its name. When attempting to retrieve "foo/bar", _null_ is always returned.
The reason for this behavior is twofold. First, Google expects "/" in the path
parameters to be percent-encoded. Secondly, jclouds-core does not have a way to
express that a specific path parameter should be encoded prior to being added
to the path.
Currently, the rest processor does the following for the endpoint: assemble all
of the tokens together and call {code}urlEncode(){code} on the result. When it
does so, it also skips encoding any "/" characters. If an encoded token is
passed to the processor, it ends up being encoded twice, as the "%" character
is encoded as %25.
jclouds-core should add {code}@Encoded{code} parameter annotation, which would
encode each parameter individually (including any "/" characters) before
assembling the URL. In this case, the consumers of the API must make sure that
they do not need the entire path to be encoded.
I plan to submit a PR to add the annotation and to use it in the
google-cloud-storage provider. When the annotation is not used, endpoints will
be processed as is.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: [jclouds-labs-google] FIx issue with blob name / encoding for getBlob method (#166)
Ok, the commit makes more sense to me now with respect to what was changed and why. There is a bit of an inconsistency with the google provider here in two aspects: not using ```Strings2.urlEncode()``` and not encoding parameters of the requests in the HTTP layer -- I'd like to figure out why the HTTP transport doesn't encode the parameters correctly (like S3 does, for example). Will look into this particular bit tomorrow. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-labs-google/pull/166#issuecomment-143528958
Re: [jclouds-labs-google] FIx issue with blob name / encoding for getBlob method (#166)
@freddy33 looking into the Google provider, it's not clear to me that "/" should actually be encoded. Putting an object, such as _foo/bar_ results in _foo/bar_ being placed in the object store (and I observe that this is the object returned when listing the store contents). I see an issue with ```getBlob()```, however. Without any changes, ```getBlob(, "foo/bar")``` fails to retrieve the blob _foo/bar_ and returns ```null``` instead -- clearly wrong. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-labs-google/pull/166#issuecomment-143525992
Re: [jclouds-labs-google] FIx issue with blob name / encoding for getBlob method (#166)
Yep, will look into this. Possibly related to the changes in core. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-labs-google/pull/166#issuecomment-143125905
Re: [jclouds] JCLOUDS-217: do not double decode strings. (#856)
Rebased. I'll look into GCS in labs and see if anything else may be broken. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/856#issuecomment-141215659
[jclouds] JCLOUDS-217: do not double decode strings. (#856)
This PR removes the decoding multi map from URI and makes changes primarily to tests to not work around its behavior. This does not fix an existing issue, but I believe it simplifies the internals of handling parameters in jclouds. Background - Currently, calling ```addQueryParam()``` when constructing an HttpRequest results in inserting the key and the value into a decoding MultiMap. What it means is that the parameter value is decoded prior to being inserted and then encoded again on retrieval. Further, there is some additional logic that checks whether a string is encoded or not in the ```Strings2.urlDecode()``` method. This behavior is also problematic such that providers need to make sure they encode strings before adding them as a query parameter, as they may not get the expected result back (e.g. a string that contains "+" is decoded into " " and encoded into "%20"). The jclouds code currently works around these issues, however, I believe the parameter handling is not clear to developers and leads to confusion. The PR suggests a more straightforward approach: 1. all query parameters will be encoded by ```addQueryParam()``` 2. the query parameters are _never_ decoded by ```addQueryParam()``` I tested the PR with Azure, AWS S3, Swift, and the Filesystem backend. I removed one workaround from AWS S3, but otherwise all providers were functional. The majority of the changes are in fixing up tests to not use encoded strings (as these now become doubly encoded, so a parameter "my%20account" is turned into "my%2520account". You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds/pull/856 -- Commit Summary -- * JCLOUDS-217: Do not decode query strings. * Skip broken Azure test. * JCLOUDS-217: Cloudstack: avoid encoded strings. * JCLOUDS-217: Old Swift should encode blob names. * JCLOUDS-217: AWS-S3: Remove special encoding. -- File Changes -- M apis/cloudstack/src/test/java/org/jclouds/cloudstack/compute/CloudStackComputeServiceAdapterExpectTest.java (22) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/compute/extensions/CloudStackImageExtensionExpectTest.java (24) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/compute/extensions/CloudStackSecurityGroupExtensionExpectTest.java (36) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/features/FirewallApiExpectTest.java (380) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/features/GlobalHostApiExpectTest.java (6) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/features/GlobalUserApiTest.java (2) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/features/GlobalZoneApiExpectTest.java (45) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/features/ISOApiExpectTest.java (18) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/features/SecurityGroupApiTest.java (24) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/features/SnapshotApiTest.java (12) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/features/TagApiExpectTest.java (40) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/features/TemplateApiTest.java (6) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/features/VirtualMachineApiExpectTest.java (19) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/features/ZoneApiExpectTest.java (19) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/internal/BaseCloudStackComputeServiceContextExpectTest.java (52) M apis/swift/src/main/java/org/jclouds/openstack/swift/functions/ParseObjectInfoListFromJsonResponse.java (5) M core/src/main/java/org/jclouds/http/Uris.java (65) M core/src/main/java/org/jclouds/util/Strings2.java (9) M core/src/test/java/org/jclouds/http/HttpRequestTest.java (5) M core/src/test/java/org/jclouds/util/Strings2Test.java (10) M providers/aws-s3/src/main/java/org/jclouds/aws/s3/blobstore/AWSS3BlobRequestSigner.java (11) M providers/aws-s3/src/test/java/org/jclouds/aws/s3/blobstore/AWSS3BlobSignerExpectTest.java (23) M providers/azureblob/src/test/java/org/jclouds/azureblob/blobstore/integration/AzureBlobIntegrationLiveTest.java (6) -- Patch Links -- https://github.com/jclouds/jclouds/pull/856.patch https://github.com/jclouds/jclouds/pull/856.diff --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/856
[jclouds] Skip broken Azure Blob test. (#855)
Azure ```testGetRange()``` has been consistently failing for a while. Disable the test until it is fixed. You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds/pull/855 -- Commit Summary -- * Skip broken Azure test. -- File Changes -- M providers/azureblob/src/test/java/org/jclouds/azureblob/blobstore/integration/AzureBlobIntegrationLiveTest.java (6) -- Patch Links -- https://github.com/jclouds/jclouds/pull/855.patch https://github.com/jclouds/jclouds/pull/855.diff --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/855
[jira] [Commented] (JCLOUDS-94) Keeping % signs in bucket names
[ https://issues.apache.org/jira/browse/JCLOUDS-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14735809#comment-14735809 ] Timur Alperovich commented on JCLOUDS-94: - AWS S3, Azure Blobs, and GCS all prohibit the _%_ character in the bucket names. If the issue actually is creating a bucket with a _%_ character in the name, then this behavior is expected. If the issue is that jclouds does not allow one to create blobs with _%_ characters, then this is a duplicate of JCLOUDS-217. > Keeping % signs in bucket names > --- > > Key: JCLOUDS-94 > URL: https://issues.apache.org/jira/browse/JCLOUDS-94 > Project: jclouds > Issue Type: Bug > Components: jclouds-blobstore >Affects Versions: 1.6.0 >Reporter: Oliver Kopp > Assignee: Timur Alperovich >Priority: Minor > > I'm trying to create a bucket having % signs in the name. Before sending the > bucket name to AWS, the name is decoded three (3) times. I expect the bucket > name not to be decoded at all. > Example: > http%253A%252F%252Fwww.example.org > %252Fwinery%252Ftest%252Fjclouds/test/test.properties > gets > http: / / www.example.org / winery / test / jclouds / test / test.properties > It might have something to do with > unescapeSpecialChars(uri.getPath()) > in line 284 of org.jclouds.http.URI > It is called multiple times before a URI is really used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [jclouds] JCLOUDS-217: Allow for strings with % characters in jclouds (#851)
I've been thinking of where the reliance on *not* encoding the string -- the crux of this change -- could have been successful. No obvious examples come to mind that would still function. For example, putting a key "%24" would actually put "$" -- the user would notice this when listing the bucket. I think the biggest issue is that user's may need to update their code to *not* work around this issue. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/851#issuecomment-138672187
Re: [jclouds] JCLOUDS-217: Allow for strings with % characters in jclouds (#851)
I ran the EC2 live tests. There were some failures (10), but I observed those on master and on the percent-encoding branch. The vast majority of the tests does pass successfully. It appears that one reason for failures is that I did not setup the ssh key to be used with the EC2 instances and I'm not sure what jclouds expects in that regard -- did not investigate. I can try to figure it out if it may be of consequence. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/851#issuecomment-137818359
Re: [jclouds] JCLOUDS-217: Allow for strings with % characters in jclouds (#851)
@nacx the reason for the change in Uris.java is the current behavior of ```appendPath()```: decode the path and then call ```path()``` with the decoded string (possibly, prepending a "/" character). ```path()``` then, in turn, _also_ decodes the string (as you pointed out). That leads to a string that is decoded twice. I couldn't figure out the original intent of whoever introduced such behavior, but it doesn't seem right to me. Here is an example that breaks with this today: appending a path of the form "%24$" (%24 is $), which is encoded "%2524%24". When passing this value to ```appendPath()``` it is then decoded as "%24$", but when we decode it one more time we end up with " $$", which is not the original path. We then generate a URI with the appended path being _%24%24_ I think making this change to ```appendPath()``` makes sense, but I might be missing something. This was turned up when I attempted to put and get a blob named "%20 " (and " %20"). I looked at where ```@EndpointParam``` is used and it looks like EC2 is a pretty heavy user of this. I'm going to run all of the live EC2 tests and report back with results. I looked at what it does and it's not clear to me where there would be a problem with its usage. Could you elaborate further what this PR makes incorrect or difficult? --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/851#issuecomment-137338082
Re: [jclouds] JCLOUDS-217: Allow for strings with % characters in jclouds (#851)
@zack-shoylev could you retry the Windows tests with the latest PR? I tried it on my Windows setup and there were no failures, but wanted to double check. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/851#issuecomment-136900785
Re: [jclouds] JCLOUDS-217: Allow for strings with % characters in jclouds (#851)
@nacx I looked at all the uses of ```SkipEncoding```. We have to keep it in Azure BlobStore, as Azure expects its root container to be named ```$root```. Passing ```%24root``` does not work in that case, as the provider does not decode the name. For GCE, I ran all of the live tests with ```SkipEncoding``` and without it. The same tests fail in both cases (6 of them right now) with the same errors. It seems to me that either we can remove skip encoding OR there are no live tests that verify that it is required. The ```SkipEncoding``` is also used in Chef, OpenStack Cinder, and Trove, however, I don't have an easy way to run those live tests. I can supply a gist removing it from the API if someone could run the live tests. Let's take this on as a separate PR? --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/851#issuecomment-136813022
Re: [jclouds] JCLOUDS-217: Allow for strings with % characters in jclouds (#851)
@zack-shoylev thanks for the update. This is a little surprising to me. I'm going to setup a Windows dev environment to figure out what's going on. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/851#issuecomment-136477865
Re: [jclouds] JCLOUDS-217: Allow for strings with % characters in jclouds (#851)
@zack-shoylev do you know which particular blob this fails on? If it's " ", we may want to explicitly avoid testing that case on Windows (or, probably better, use a different character that would be escaped). --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/851#issuecomment-135885792
Re: [jclouds] JCLOUDS-217: Allow for strings with % characters in jclouds (#851)
@nacx agreed that this may break compatibility with many consumers. I'll look into the uses of SkipEncoding -- did not do so previously as it looked like the BlobStore providers did not rely on this. For the compute side, I will try to check some of the live tests (e.g. EC2), but I don't have credentials for a lot of providers. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/851#issuecomment-135572479
[jira] [Commented] (JCLOUDS-856) openstack-swift provider mangles blob names that end with slash
[
https://issues.apache.org/jira/browse/JCLOUDS-856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14717652#comment-14717652
]
Timur Alperovich commented on JCLOUDS-856:
--
Yep, this is now resolved.
> openstack-swift provider mangles blob names that end with slash
> ---
>
> Key: JCLOUDS-856
> URL: https://issues.apache.org/jira/browse/JCLOUDS-856
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
>Affects Versions: 1.8.1
>Reporter: Ka-Hing Cheung
>Assignee: Zack Shoylev
> Labels: openstack-swift, swift
>
> Swift (both the old swift and the new openstack-swift providers) deletes
> trailing slash on listContainer. deleteContainer() also fails if you have a
> blob with trailing slash.
> The mangling is done in MarkersIfDirectoryReturnNameStrategy. Working around
> that in application is not ideal because metadata.getType() is RELATIVE_PATH
> could mean a slash was deleted from the end, or the blob's content type is
> application/directory. It's not clear how to figure out what the real name of
> the returned objects without probing the store again.
> {code:java}
> @Test
> public void testListObjectWithTrailingSlash() throws InterruptedException {
>BlobStore blobStore = view.getBlobStore();
>String containerName = getContainerName();
>try {
>String key = "a/";
>String etag = blobStore.putBlob(containerName,
>blobStore.blobBuilder(key).payload("content").build());
>assertThat(etag).isNotNull();
>PageSet res =
>blobStore.list(containerName, new
> ListContainerOptions().recursive());
>assertThat(res).hasSize(1);
>StorageMetadata meta = res.iterator().next();
>assertThat(meta.getName()).isEqualTo(key);
>} finally {
>returnContainer(containerName);
>}
>}
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (JCLOUDS-976) Azureblob: listing objects with a space in the name causes URISyntaxException
[
https://issues.apache.org/jira/browse/JCLOUDS-976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14717530#comment-14717530
]
Timur Alperovich commented on JCLOUDS-976:
--
[~gaul] This should be resolved by: https://github.com/jclouds/jclouds/pull/851
> Azureblob: listing objects with a space in the name causes URISyntaxException
> -
>
> Key: JCLOUDS-976
> URL: https://issues.apache.org/jira/browse/JCLOUDS-976
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
>Affects Versions: 1.9.0
>Reporter: Roman Novokhatsky
>
> When having an Azure container with blobs with a space in the name, a call to
> {{BlobStore.list()}} throws the following exception:
> {noformat}
> Exception in thread "pool-1-thread-1" java.lang.IllegalArgumentException:
> Illegal character in path at index 61:
> https://x.blob.core.windows.net/container-name/01 - Artist - Song .mp3
> at java.net.URI.create(URI.java:852)
> at org.jclouds.http.Uris$UriBuilder.(Uris.java:218)
> at org.jclouds.http.Uris$UriBuilder.(Uris.java:99)
> at org.jclouds.http.Uris.uriBuilder(Uris.java:78)
> at
> org.jclouds.azureblob.xml.ContainerNameEnumerationResultsHandler.endElement(ContainerNameEnumerationResultsHandler.java:153)
> at
> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:609)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1782)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2973)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
> at
> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
> at
> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
> at
> com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
> at
> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
> at
> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:649)
> at org.jclouds.http.functions.ParseSax.doParse(ParseSax.java:140)
> at org.jclouds.http.functions.ParseSax.parse(ParseSax.java:129)
> at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:83)
> at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:51)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
> at
> org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
> at
> org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
> at com.sun.proxy.$Proxy66.listBlobs(Unknown Source)
> at
> org.jclouds.azureblob.blobstore.AzureBlobStore.list(AzureBlobStore.java:147)
> at
> org.jclouds.blobstore.internal.BaseBlobStore.list(BaseBlobStore.java:81)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
> at com.sun.proxy.$Proxy50.list(Unknown Source)
> <..>
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.URISyntaxException: Illegal character in path at index
> 61: https://x.blob.core.windows.net/container-name/01 - Artist - Song .mp3
> at java.net.URI$Parser.fail(URI.java:2848)
> at java.net.URI$Parser.checkChars(URI.java:3021)
> at java.net.URI$Parser.parseHierarchical(URI.java:3105)
> at java.net.URI$Parser.parse(URI.java:3053)
> at java.net.URI.(URI.java:588)
> at java.net.URI.create(URI.java:850)
> ... 39 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (JCLOUDS-94) Keeping % signs in bucket names
[ https://issues.apache.org/jira/browse/JCLOUDS-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14717529#comment-14717529 ] Timur Alperovich commented on JCLOUDS-94: - [~gaul] This is most likely related, however, there are no tests for bucket names with % characters. I can look into this, as it's likely easy to resolve. > Keeping % signs in bucket names > --- > > Key: JCLOUDS-94 > URL: https://issues.apache.org/jira/browse/JCLOUDS-94 > Project: jclouds > Issue Type: Bug > Components: jclouds-blobstore >Affects Versions: 1.6.0 >Reporter: Oliver Kopp >Priority: Minor > > I'm trying to create a bucket having % signs in the name. Before sending the > bucket name to AWS, the name is decoded three (3) times. I expect the bucket > name not to be decoded at all. > Example: > http%253A%252F%252Fwww.example.org > %252Fwinery%252Ftest%252Fjclouds/test/test.properties > gets > http: / / www.example.org / winery / test / jclouds / test / test.properties > It might have something to do with > unescapeSpecialChars(uri.getPath()) > in line 284 of org.jclouds.http.URI > It is called multiple times before a URI is really used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (JCLOUDS-548) Azure java.lang.IllegalArgumentException: Illegal character in path at index
[
https://issues.apache.org/jira/browse/JCLOUDS-548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14717527#comment-14717527
]
Timur Alperovich commented on JCLOUDS-548:
--
Yes. This PR fixes the issue in Azure:
https://github.com/jclouds/jclouds/pull/851
> Azure java.lang.IllegalArgumentException: Illegal character in path at index
>
>
> Key: JCLOUDS-548
> URL: https://issues.apache.org/jira/browse/JCLOUDS-548
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
>Affects Versions: 1.7.2
> Environment: All
>Reporter: srini
> Labels: azureblob
>
> hi All,
> I m trying to fetch the Azure blobs. Some of the file names are having spaces.
> This is breaking the blobstorage calls
> Can we get a alternative solution..
> {noformat}
> java.lang.IllegalArgumentException: Illegal character in path at index 58:
> https://cloudfuzetest1.blob.core.windows.net/test/cmisatom (6)
> at java.net.URI.create(Unknown Source)
> at org.jclouds.http.Uris$UriBuilder.(Uris.java:221)
> at org.jclouds.http.Uris$UriBuilder.(Uris.java:102)
> at org.jclouds.http.Uris.uriBuilder(Uris.java:81)
> at
> org.jclouds.azureblob.xml.ContainerNameEnumerationResultsHandler.endElement(ContainerNameEnumerationResultsHandler.java:154)
> at
> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(Unknown
> Source)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(Unknown
> Source)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(Unknown
> Source)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(Unknown
> Source)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown
> Source)
> at
> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown
> Source)
> at
> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown
> Source)
> at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown
> Source)
> at
> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Unknown
> Source)
> at
> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown
> Source)
> at org.jclouds.http.functions.ParseSax.doParse(ParseSax.java:139)
> at org.jclouds.http.functions.ParseSax.parse(ParseSax.java:128)
> at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:84)
> at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:52)
> at
> org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:129)
> at
> org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:95)
> at
> org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:56)
> at
> org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
> at
> org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
> at com.sun.proxy.$Proxy116.listBlobs(Unknown Source)
> at
> org.jclouds.azureblob.blobstore.AzureBlobStore.list(AzureBlobStore.java:148)
> at
> org.jclouds.blobstore.internal.BaseBlobStore.list(BaseBlobStore.java:85)
> at
> com.cloudfuze.connectors.jclouds.ObjectConnector.getFilesInContainer(ObjectConnector.java:220)
> at
> com.cloudfuze.cloud.connectors.ObjectStorageCloudConnector.getFiles(ObjectStorageCloudConnector.java:470)
> at
> com.cloudfuze.cloud.connectors.management.ConnectorLoadTask.run(ConnectorLoadTask.java:175)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecut
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (JCLOUDS-121) Missing trailing slash in blob names
[ https://issues.apache.org/jira/browse/JCLOUDS-121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14717525#comment-14717525 ] Timur Alperovich commented on JCLOUDS-121: -- This was an issue with the jclouds IfDirectoryReturnNameStrategy, which would strip off the "/" characters from the blobs. This was resolved as part of fixing JCLOUDS-992. > Missing trailing slash in blob names > > > Key: JCLOUDS-121 > URL: https://issues.apache.org/jira/browse/JCLOUDS-121 > Project: jclouds > Issue Type: Bug > Components: jclouds-blobstore >Affects Versions: 1.6.0 >Reporter: Timur Sungur >Priority: Minor > > The trailing slash in bucket names are removed during put method into AWS > blobstore. > Whenever the bucket name is retrieved from AWS using blobStore.list() method, > the bucket name does not contain the final slash; moreover, the blob cannot > be retrieved without adding it. > Example: > The item name but into blob store: > http%2525253A%2525252F%2525252Fwww.example.org%2525252Fwinery%2525252FTEST%2525252Fjclouds1/test2/ > The retrieved name using blobStore.list(): > http%3A%2F%2Fwww.example.org%2Fwinery%2FTEST%2Fjclouds1/test2 > (Length is different due to 3 times implicit encoding see: > https://issues.apache.org/jira/browse/JCLOUDS-94) > One needs to encode 3 times and add the trailing slash to retrieve > corresponding blob. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (JCLOUDS-955) jclouds strips off the delimiter from common prefixes (at least in S3)
[
https://issues.apache.org/jira/browse/JCLOUDS-955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14717515#comment-14717515
]
Timur Alperovich commented on JCLOUDS-955:
--
Ended up resolving this issue as part of fixing JCLOUDS-992. Closing.
> jclouds strips off the delimiter from common prefixes (at least in S3)
> --
>
> Key: JCLOUDS-955
> URL: https://issues.apache.org/jira/browse/JCLOUDS-955
> Project: jclouds
> Issue Type: Bug
> Reporter: Timur Alperovich
> Assignee: Timur Alperovich
> Fix For: 2.0.0
>
>
> I noticed that jclouds strips off the delimiter when returning results that
> include common prefixes. For example, if blobs "test/foo" and "test/bar" are
> in the container, listing the container non-recursively (i.e. setting the
> delimiter to "/"), will return "test", whereas the providers commonly return
> "test/".
> AWS S3\[[#1]\], OpenStack Swift\[#2\], Google Cloud Storage\[[#3]\], and
> Azure\[[#4]\] all include the delimiter string (or character, in the case of
> Swift) in the results.
> This may not be a valid issue if this was always the intent of jclouds to
> elide the delimiter parameter. In which case, could someone elaborate about
> the reasons for that here? If this is the expected behavior, I would be happy
> to amend the prefix changes to _not_ include the delimiter in the Local Blob
> Store listings. Otherwise, I'm also happy to expose the results as-is in the
> other providers.
> References:
> {anchor:1} 1. From [AWS S3 List
> documentation|http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGET.html]:
> "In effect, CommonPrefixes lists keys that act like subdirectories in the
> directory specified by Prefix. For example, if prefix is notes/ and delimiter
> is a slash \(/\), in notes/summer/july, the common prefix is notes/summer/."
> {anchor:2} 2. Swift does not explicitly document this, however, using the
> swift client, one can observe the behavior where specifying "\-" as a
> delimiter results in the following json (the container has keys "tmp-foo" and
> "tmp-bar"): \{"subdir": "tmp-"\}.
> {anchor:3} 3. From the [GCS
> API|https://cloud.google.com/storage/docs/json_api/v1/objects/list]: "Objects
> whose names, aside from the prefix, contain delimiter will have their name,
> truncated after the delimiter, returned in prefixes."
> {anchor:4} 4. From the [Azure Blobs
> API|https://msdn.microsoft.com/en-us/library/azure/hh452233.aspx]: "The value
> of the BlobPrefix element is substring+delimiter, where substring is the
> common substring that begins one or more blob names, and delimiter is the
> value of the delimiter parameter."
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (JCLOUDS-955) jclouds strips off the delimiter from common prefixes (at least in S3)
[
https://issues.apache.org/jira/browse/JCLOUDS-955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timur Alperovich resolved JCLOUDS-955.
--
Resolution: Fixed
Fix Version/s: 2.0.0
> jclouds strips off the delimiter from common prefixes (at least in S3)
> --
>
> Key: JCLOUDS-955
> URL: https://issues.apache.org/jira/browse/JCLOUDS-955
> Project: jclouds
> Issue Type: Bug
> Reporter: Timur Alperovich
> Assignee: Timur Alperovich
> Fix For: 2.0.0
>
>
> I noticed that jclouds strips off the delimiter when returning results that
> include common prefixes. For example, if blobs "test/foo" and "test/bar" are
> in the container, listing the container non-recursively (i.e. setting the
> delimiter to "/"), will return "test", whereas the providers commonly return
> "test/".
> AWS S3\[[#1]\], OpenStack Swift\[#2\], Google Cloud Storage\[[#3]\], and
> Azure\[[#4]\] all include the delimiter string (or character, in the case of
> Swift) in the results.
> This may not be a valid issue if this was always the intent of jclouds to
> elide the delimiter parameter. In which case, could someone elaborate about
> the reasons for that here? If this is the expected behavior, I would be happy
> to amend the prefix changes to _not_ include the delimiter in the Local Blob
> Store listings. Otherwise, I'm also happy to expose the results as-is in the
> other providers.
> References:
> {anchor:1} 1. From [AWS S3 List
> documentation|http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGET.html]:
> "In effect, CommonPrefixes lists keys that act like subdirectories in the
> directory specified by Prefix. For example, if prefix is notes/ and delimiter
> is a slash \(/\), in notes/summer/july, the common prefix is notes/summer/."
> {anchor:2} 2. Swift does not explicitly document this, however, using the
> swift client, one can observe the behavior where specifying "\-" as a
> delimiter results in the following json (the container has keys "tmp-foo" and
> "tmp-bar"): \{"subdir": "tmp-"\}.
> {anchor:3} 3. From the [GCS
> API|https://cloud.google.com/storage/docs/json_api/v1/objects/list]: "Objects
> whose names, aside from the prefix, contain delimiter will have their name,
> truncated after the delimiter, returned in prefixes."
> {anchor:4} 4. From the [Azure Blobs
> API|https://msdn.microsoft.com/en-us/library/azure/hh452233.aspx]: "The value
> of the BlobPrefix element is substring+delimiter, where substring is the
> common substring that begins one or more blob names, and delimiter is the
> value of the delimiter parameter."
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jclouds] Allow for strings with % characters in jclouds (#851)
Fixes handling of strings that may be encoded when passed to jclouds. Apache jclouds should not double-encode strings and should not guard against handling encoded strings. This patch makes a change ```jclouds/core``` to remove the check for encoded strings when calling ```Strings2.urlEncode()```. The changes to all of the other components are made to remove the reliance on this behavior. In jclouds blobstore, this further allows supporting strings with "%" characters -- a matter of a number of JIRA issues. You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds/pull/851 -- Commit Summary -- * Percent encoding changes. * Fix encoding handling in S3. * Azure: Handle blobs with % characters correctly. * Do not use encoded strings in STS tests. * Fix a cloudstack test that encoded parameters. * Do not use encoded strings in EC2 tests. * Do not use encoded strings in SQS API tests. * Do not use encoded strings in AWS EC2 tests. * OpenStack Nova tests: do not pass encoded strings. -- File Changes -- M apis/cloudstack/src/test/java/org/jclouds/cloudstack/features/SSHKeyPairApiTest.java (6) M apis/ec2/src/main/java/org/jclouds/ec2/compute/EC2ComputeService.java (4) M apis/ec2/src/test/java/org/jclouds/ec2/features/AMIApiExpectTest.java (4) M apis/ec2/src/test/java/org/jclouds/ec2/features/AMIApiTest.java (56) M apis/ec2/src/test/java/org/jclouds/ec2/features/EC2ElasticBlockStoreApiExpectTest.java (19) M apis/ec2/src/test/java/org/jclouds/ec2/features/ElasticBlockStoreApiTest.java (24) M apis/ec2/src/test/java/org/jclouds/ec2/features/ElasticIPAddressApiTest.java (4) M apis/ec2/src/test/java/org/jclouds/ec2/features/InstanceApiTest.java (26) M apis/ec2/src/test/java/org/jclouds/ec2/features/KeyPairApiExpectTest.java (4) M apis/ec2/src/test/java/org/jclouds/ec2/features/SecurityGroupApiExpectTest.java (4) M apis/ec2/src/test/java/org/jclouds/ec2/features/SecurityGroupApiTest.java (12) M apis/ec2/src/test/java/org/jclouds/ec2/features/SubnetApiExpectTest.java (12) M apis/ec2/src/test/java/org/jclouds/ec2/features/WindowsApiExpectTest.java (4) M apis/ec2/src/test/java/org/jclouds/ec2/features/WindowsApiTest.java (16) M apis/openstack-nova/src/test/java/org/jclouds/openstack/nova/v2_0/features/ImageApiExpectTest.java (21) M apis/openstack-nova/src/test/java/org/jclouds/openstack/nova/v2_0/features/ServerApiExpectTest.java (14) M apis/s3/src/main/java/org/jclouds/s3/binders/BindAsHostPrefixIfConfigured.java (2) M apis/s3/src/main/java/org/jclouds/s3/xml/ListBucketHandler.java (11) M apis/sqs/src/test/java/org/jclouds/sqs/features/MessageApiExpectTest.java (70) M apis/sqs/src/test/java/org/jclouds/sqs/features/PermissionApiExpectTest.java (8) M apis/sqs/src/test/java/org/jclouds/sqs/features/QueueApiExpectTest.java (28) M apis/sts/src/test/java/org/jclouds/sts/STSApiExpectTest.java (24) M blobstore/src/test/java/org/jclouds/blobstore/integration/internal/BaseContainerIntegrationTest.java (17) M core/src/main/java/org/jclouds/http/Uris.java (1) M core/src/main/java/org/jclouds/rest/binders/BindAsHostPrefix.java (5) M core/src/main/java/org/jclouds/util/Strings2.java (9) M core/src/test/java/org/jclouds/util/Strings2Test.java (6) M providers/aws-ec2/src/test/java/org/jclouds/aws/ec2/features/AWSInstanceApiTest.java (2) M providers/aws-ec2/src/test/java/org/jclouds/aws/ec2/features/AWSKeyPairApiTest.java (2) M providers/azureblob/src/main/java/org/jclouds/azureblob/xml/ContainerNameEnumerationResultsHandler.java (49) -- Patch Links -- https://github.com/jclouds/jclouds/pull/851.patch https://github.com/jclouds/jclouds/pull/851.diff --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/851
