Re: [jclouds] add aws s3 signature v4 (#678)

Closed #678. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/678#event-514370147

Re: [jclouds] add aws s3 signature v4 (#678)

@zhaojin0 I made some stylistic modifications, squashed the commits, and pushed to master as 8bddbb496a7b1a52f851e3e22626eaf644072973. Thank you for your contribution and for your patience with this extraordinarily long review! --- Reply to this email directly or view it on GitHub: https://gith

Re: [jclouds] add aws s3 signature v4 (#678)

@demobox I will try to make a run at this early next week when I work on some related V4 issues. Sorry for the delay! --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/678#issuecomment-165925177

Re: [jclouds] add aws s3 signature v4 (#678)

@andrewgaul Ping? Is this something you'd be able to take a look at? --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/678#issuecomment-165160819

Re: [jclouds] add aws s3 signature v4 (#678)

hey guys any progress on it? I hate that we have to use a patch version in production, but we already use it for some time and have load tested it as well. :+1: --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/678#issuecomment-165158363

Re: [jclouds] add aws s3 signature v4 (#678)

AFAIK this is reported against 1.8.x. I've applied this PR to a 1.8.1 fork without a problem. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/678#issuecomment-139468927

Re: [jclouds] add aws s3 signature v4 (#678)

Any reason for this not to have been merged? Is it reported to 2.0 version only? Could it be merge on 1.9 branch? --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/678#issuecomment-139418881

Re: [jclouds] add aws s3 signature v4 (#678)

@pswvg This PR still needs a thorough review -- S3 is our most popular provider and I want to ensure we do not introduce any regressions for our users. @zhaojin0 Sorry about the delay; I will try to get to this soon. --- Reply to this email directly or view it on GitHub: https://github.com/jclou

Re: [jclouds] add aws s3 signature v4 (#678)

The last time I tried Multipart Upload with jclouds aws-s3, it also forced me to specify the content length in advance. See my post here: https://www.mail-archive.com/user@jclouds.apache.org/msg01562.html. But that's another story. What is the state about this PR? What is missing before it coul

Re: [jclouds] add aws s3 signature v4 (#678)

try Multipart Upload. initiate Multipart Upload..temporarily store specific length part, upload all stream part..complete multipart upload... part requires Content-Length. not all a file. but I have not tried :) --- Reply to this email directly or view it on GitHub: https://github.com/jclouds

Re: [jclouds] add aws s3 signature v4 (#678)

I thought about the content-length again. The old V2 implementation wasn't able to handle unknown content-lengths. So does the V4. But I think it wouldn't be so difficult to add this. Because chunking is already implemented nicely and instead of sending the content-length of the complete payload

Re: [jclouds] add aws s3 signature v4 (#678)

> +ChunkedInputStreamEnumeration(InputStream inputStream, int > chunkedBlockSize) { > +this.inputStream = new BufferedInputStream(inputStream, > chunkedBlockSize); > +buffer = new byte[chunkedBlockSize]; > +lastChunked = false; > +} > + > +

Re: [jclouds] add aws s3 signature v4 (#678)

> +ChunkedInputStreamEnumeration(InputStream inputStream, int > chunkedBlockSize) { > +this.inputStream = new BufferedInputStream(inputStream, > chunkedBlockSize); > +buffer = new byte[chunkedBlockSize]; > +lastChunked = false; > +} > + > +

Re: [jclouds] add aws s3 signature v4 (#678)

For our project, we took the PR, applied some minor fixes (https://github.com/zhaojin0/jclouds/pull/1) and backported it to stable 1.8.1. With our Integration tests, it works very well. Tested it against Frankfurt - which supports v4 only. --- Reply to this email directly or view it on GitHub:

Re: [jclouds] add aws s3 signature v4 (#678)

Hi, I impl aws s3 signer v4 chunked upload, use when put object, payload cannot repeatable. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/678#issuecomment-98595506

Re: [jclouds] add aws s3 signature v4 (#678)

It's seem as region eu-central-1 doesn't supported AWS sign V2. 在 2015/4/18 2:46, Andrew Gaul 写道: > > @zhaojin0 I am testing this and see many > errors of the form: > > |org.jclouds.aws.AWSResponseException: request GET > https://gaul-blobstore3760643340725640912-v

Re: [jclouds] add aws s3 signature v4 (#678)

> @@ -29,6 +25,10 @@ > import org.testng.annotations.BeforeClass; > import org.testng.annotations.Test; > > +import java.io.IOException; > + > +import static org.testng.Assert.assertEquals; > + Spurious code movement? --- Reply to this email directly or view it on GitHub: https://github.com/j

Re: [jclouds] add aws s3 signature v4 (#678)

> > - private static final Set SIGNED_PARAMETERS = > ImmutableSet.of("acl", "torrent", "logging", "location", "policy", > +private static final Set SIGNED_PARAMETERS = > ImmutableSet.of("acl", "torrent", "logging", "location", "policy", Right, but can we move this code into its own s

Re: [jclouds] add aws s3 signature v4 (#678)

@zhaojin0 I am testing this and see many errors of the form: ``` org.jclouds.aws.AWSResponseException: request GET https://gaul-blobstore3760643340725640912-v4-only.s3-eu-central-1.amazonaws.com/ HTTP/1.1 failed with code 400, error: AWSError{requestId='0F84681CE6013127', requestToken='THXHcOkp

Re: [jclouds] add aws s3 signature v4 (#678)

AWS Sign V4 use sha256 content hash. If payload can not be reset, aws supported chunked uploads. http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/678#issuecomment-88369850

Re: [jclouds] add aws s3 signature v4 (#678)

Any plans to continue with this pull request? --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/678#issuecomment-88231285

Re: [jclouds] add aws s3 signature v4 (#678)

> +HttpRequest.Builder requestBuilder, > +String method, > +URI endpoint, > +Payload payload > +) { > +InputStream payloadStream; > +try { > +payloadStream = usePayloadForQueryParameters(method, payload) ? > +getQuerySt

Re: [jclouds] add aws s3 signature v4 (#678)

> +HttpRequest.Builder requestBuilder, > +String method, > +URI endpoint, > +Payload payload > +) { > +InputStream payloadStream; > +try { > +payloadStream = usePayloadForQueryParameters(method, payload) ? > +getQuerySt

Re: [jclouds] add aws s3 signature v4 (#678)

it's use for sign a temporary access... I provided testcase AWSS3BlobSignerV4ExpectTest. sorry, this's my first pull request, I format some code --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/678#issuecomment-76351814

Re: [jclouds] add aws s3 signature v4 (#678)

@zhaojin0 I do not understand how this is supposed to work -- you should bind the V4 signer somewhere like `AWSS3BlobStoreContextModule`: ```java bind(BlobRequestSigner.class).to(AWSS3BlobRequestSignerV4.class); ``` I do not understand the overall strategy here, but I believe that jclouds shoul

Re: [jclouds] add aws s3 signature v4 (#678)

I'm sorry for my lazy... i add temporary access signature code, but i dont known how to test it. It could work in aws region cn-north-1. ```java public class AWSS3BlobStoreContextModule extends S3BlobStoreContextModule { //... @Override protected void bindRequestSigner() { bind(Blo

Re: [jclouds] add aws s3 signature v4 (#678)

@zhaojin0 Any updates on this pull request? I would like to clean it up and include it in the upcoming 1.9.0 release. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/678#issuecomment-74991773

Re: [jclouds] add aws s3 signature v4 (#678)

> @@ -56,7 +56,7 @@ > public AWSS3BlobRequestSigner(RestAnnotationProcessor processor, > BlobToObject blobToObject, > BlobToHttpGetOptions blob2HttpGetOptions, Class > interfaceClass, > @org.jclouds.location.Provider Supplier credentials, > - RequestAuthorizeSignat

Re: [jclouds] add aws s3 signature v4 (#678)

Could you tag this commit with [JCLOUDS-480](https://issues.apache.org/jira/browse/JCLOUDS-480) so that JIRA will note it? --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/678#issuecomment-74193650

Re: [jclouds] add aws s3 signature v4 (#678)

> +try { > +String encoded = URLEncoder.encode(value, DEFAULT_ENCODING); > + > +Matcher matcher = ENCODED_CHARACTERS_PATTERN.matcher(encoded); > +StringBuffer buffer = new StringBuffer(encoded.length()); > + > +while (matcher.find()) { > +

Re: [jclouds] add aws s3 signature v4 (#678)

> + > +@Override > +public String service() { > +return service; > +} > + > +@Override > +public String region(String host) { > +return AwsHostNameUtils.parseRegionName(host, service()); > +} > +

Re: [jclouds] add aws s3 signature v4 (#678)

> +} > + > +byte[] hmacSHA256(String toSign, byte[] key) { > +try { > +ByteProcessor hmacSHA256 = > asByteProcessor(crypto.hmacSHA256(key)); > +return readBytes(toInputStream(toSign), hmacSHA256); > +} catch (IOException e) { > +throw new

Re: [jclouds] add aws s3 signature v4 (#678)

> @@ -56,7 +56,7 @@ > public AWSS3BlobRequestSigner(RestAnnotationProcessor processor, > BlobToObject blobToObject, > BlobToHttpGetOptions blob2HttpGetOptions, Class > interfaceClass, > @org.jclouds.location.Provider Supplier credentials, > - RequestAuthorizeSignat

Re: [jclouds] add aws s3 signature v4 (#678)

> +try { > +String encoded = URLEncoder.encode(value, DEFAULT_ENCODING); > + > +Matcher matcher = ENCODED_CHARACTERS_PATTERN.matcher(encoded); > +StringBuffer buffer = new StringBuffer(encoded.length()); > + > +while (matcher.find()) { > +

Re: [jclouds] add aws s3 signature v4 (#678)

> +SortedMap sorted = new TreeMap(); > +if (params == null) { > +return ""; > +} > +Iterator> pairs = > params.entries().iterator(); > +while (pairs.hasNext()) { > +Map.Entry pair = pairs.next(); > +String key = pair.getKe

Re: [jclouds] add aws s3 signature v4 (#678)

> > - private static final Set SIGNED_PARAMETERS = > ImmutableSet.of("acl", "torrent", "logging", "location", "policy", > +private static final Set SIGNED_PARAMETERS = > ImmutableSet.of("acl", "torrent", "logging", "location", "policy", Signature v4 need signed all of query string pa

Re: [jclouds] add aws s3 signature v4 (#678)

@zhaojin0 Please address Checkstyle violations and remove gratuitous reindentation. I will look at this some more afterwards. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/678#issuecomment-73876014

Re: [jclouds] add aws s3 signature v4 (#678)

> + > +@Override > +public String service() { > +return service; > +} > + > +@Override > +public String region(String host) { > +return AwsHostNameUtils.parseRegionName(host, service()); > +} > +

Re: [jclouds] add aws s3 signature v4 (#678)

> @@ -56,7 +56,7 @@ > public AWSS3BlobRequestSigner(RestAnnotationProcessor processor, > BlobToObject blobToObject, > BlobToHttpGetOptions blob2HttpGetOptions, Class > interfaceClass, > @org.jclouds.location.Provider Supplier credentials, > - RequestAuthorizeSignat

Re: [jclouds] add aws s3 signature v4 (#678)

> +try { > +String encoded = URLEncoder.encode(value, DEFAULT_ENCODING); > + > +Matcher matcher = ENCODED_CHARACTERS_PATTERN.matcher(encoded); > +StringBuffer buffer = new StringBuffer(encoded.length()); > + > +while (matcher.find()) { > +

Re: [jclouds] add aws s3 signature v4 (#678)

> +SortedMap sorted = new TreeMap(); > +if (params == null) { > +return ""; > +} > +Iterator> pairs = > params.entries().iterator(); > +while (pairs.hasNext()) { > +Map.Entry pair = pairs.next(); > +String key = pair.getKe

Re: [jclouds] add aws s3 signature v4 (#678)

> > - private static final Set SIGNED_PARAMETERS = > ImmutableSet.of("acl", "torrent", "logging", "location", "policy", > +private static final Set SIGNED_PARAMETERS = > ImmutableSet.of("acl", "torrent", "logging", "location", "policy", Should this be in its own class like the V4 sig

Re: [jclouds] add aws s3 signature v4 (#678)

> + * Unless required by applicable law or agreed to in writing, software > + * distributed under the License is distributed on an "AS IS" BASIS, > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. > + * See the License for the specific language governing permissions and

Re: [jclouds] add aws s3 signature v4 (#678)

> +} > + > +byte[] hmacSHA256(String toSign, byte[] key) { > +try { > +ByteProcessor hmacSHA256 = > asByteProcessor(crypto.hmacSHA256(key)); > +return readBytes(toInputStream(toSign), hmacSHA256); > +} catch (IOException e) { > +throw new

Re: [jclouds] add aws s3 signature v4 (#678)

> - > - @Provides > - @Bucket > - @Singleton > - protected CacheLoader> bucketToRegion(@Region > Supplier> regionSupplier, > -final S3Client client) { > - Set regions = regionSupplier.get(); > - if (regions.size() == 0) { > - return new CacheLoader>() { > - >

Re: [jclouds] add aws s3 signature v4 (#678)

add aws s3 signature v4, plz review. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/678#issuecomment-73835221

[jclouds] add aws s3 signature v4 (#678)

AWS S3 signature v4 impl You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds/pull/678 -- Commit Summary -- * add aws s3 signature v4 -- File Changes -- M apis/s3/src/main/java/org/jclouds/s3/config/S3HttpApiModule.java (308) A apis/s3/