Re: [jclouds] add aws s3 signature v4 (#678)
try Multipart Upload. initiate Multipart Upload..temporarily store specific length part, upload all stream part..complete multipart upload... part requires Content-Length. not all a file. but I have not tried :) --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/678#issuecomment-104135788
Re: [jclouds] add aws s3 signature v4 (#678)
Hi, I impl aws s3 signer v4 chunked upload, use when put object, payload cannot repeatable. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/678#issuecomment-98595506
Re: [jclouds] add aws s3 signature v4 (#678)
It's seem as region eu-central-1 doesn't supported AWS sign V2.
在 2015/4/18 2:46, Andrew Gaul 写道:
>
> @zhaojin0 <https://github.com/zhaojin0> I am testing this and see many
> errors of the form:
>
> |org.jclouds.aws.AWSResponseException: request GET
> https://gaul-blobstore3760643340725640912-v4-only.s3-eu-central-1.amazonaws.com/
> HTTP/1.1 failed with code 400, error: AWSError{requestId='0F84681CE6013127',
> requestToken='THXHcOkpHKTdN7DedbIb8qCj7/MuvvvGyu31O42h6BjkTFSvHdKNggDL/aQ6Mm1IGROwsAO58fw=',
> code='InvalidRequest', message='The authorization mechanism you have
> provided is not supported. Please use AWS4-HMAC-SHA256.',
> context='{HostId=THXHcOkpHKTdN7DedbIb8qCj7/MuvvvGyu31O42h6BjkTFSvHdKNggDL/aQ6Mm1IGROwsAO58fw=}'}
> |
>
> when deleting items in the container between runs. Any suggestions on
> this? Also do all the integration tests pass for you? I see a few errors:
>
> |
> AWSS3ContainerIntegrationLiveTest>BaseContainerIntegrationTest.deleteContainerIfEmptyWithoutContents:315
> expected [true] but found [false]
>
> AWSS3ContainerLiveTest>BaseContainerLiveTest.testPublicAccessInNonDefaultLocationWithBigBlob:112->BaseContainerLiveTest.runCreateContainerInLocation:124->BaseBlobStoreIntegrationTest.assertConsistencyAwareContainerExists:361->BaseBlobStoreIntegrationTest.assertConsistencyAware:248->BaseBlobStoreIntegrationTest.assertConsistencyAware:235
> » HttpResponse
>
> AWSS3ServiceIntegrationLiveTest>BaseServiceIntegrationTest.testAllLocations:52->BaseBlobStoreIntegrationTest.assertConsistencyAware:248->BaseBlobStoreIntegrationTest.assertConsistencyAware:235
> » HttpResponse
>
> AWSS3ServiceIntegrationLiveTest>BaseServiceIntegrationTest.testGetAssignableLocations:93
> {scope=REGION, id=eu-central-1, description=eu-central-1, parent=aws-s3,
> iso3166Codes=[DE-HE]} ||{scope=PROVIDER, id=aws-s3,
> description=https://s3.amazonaws.com, iso3166Codes=[US, US-CA, US-OR, BR-SP,
> IE, SG, AU-NSW, JP-13]}
>
> Tests run: 99, Failures: 4, Errors: 0, Skipped: 1
> |
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/jclouds/jclouds/pull/678#issuecomment-94050351>.
>
--
赵金
Zhao Jin
18610722868
北京优创联动科技有限公司
北京市 海淀区 学清路38号 金码大厦16层
100083
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/678#issuecomment-94248352
Re: [jclouds] add aws s3 signature v4 (#678)
AWS Sign V4 use sha256 content hash. If payload can not be reset, aws supported chunked uploads. http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/678#issuecomment-88369850
Re: [jclouds] add aws s3 signature v4 (#678)
> +HttpRequest.Builder requestBuilder,
> +String method,
> +URI endpoint,
> +Payload payload
> +) {
> +InputStream payloadStream;
> +try {
> +payloadStream = usePayloadForQueryParameters(method, payload) ?
> +getQueryStringContent(endpoint)
> +: getPayloadContentWithoutQueryString(payload);
> +} catch (IOException e) {
> +throw new HttpException("Unable to open stream before calculate
> AWS4 signature", e);
> +}
> +String contentSha256 =
> base16().lowerCase().encode(hash(payloadStream));
> +try {
> +payloadStream.reset();
payload stream use calculate content hash.
if can not be repeatable, the payload cannot append to http request body.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/678/files#r25842533
Re: [jclouds] add aws s3 signature v4 (#678)
it's use for sign a temporary access... I provided testcase AWSS3BlobSignerV4ExpectTest. sorry, this's my first pull request, I format some code --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/678#issuecomment-76351814
Re: [jclouds] add aws s3 signature v4 (#678)
I'm sorry for my lazy...
i add temporary access signature code, but i dont known how to test it. It
could work in aws region cn-north-1.
```java
public class AWSS3BlobStoreContextModule extends S3BlobStoreContextModule {
//...
@Override
protected void bindRequestSigner() {
bind(BlobRequestSigner.class).to(AWSS3BlobRequestSignerV4.class);
}
}
```
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/678#issuecomment-75247425
Re: [jclouds] add aws s3 signature v4 (#678)
> @@ -56,7 +56,7 @@ > public AWSS3BlobRequestSigner(RestAnnotationProcessor processor, > BlobToObject blobToObject, > BlobToHttpGetOptions blob2HttpGetOptions, Class > interfaceClass, > @org.jclouds.location.Provider Supplier credentials, > - RequestAuthorizeSignature authSigner, @TimeStamp Provider > timeStampProvider, > + RequestAuthorizeSignatureV2 authSigner, @TimeStamp Provider > timeStampProvider, sorry, this is an wrong.. BlobRequestSigner use to generating pre-signed URLs, [Share an Object with Others](http://docs.aws.amazon.com/AmazonS3/latest/dev/ShareObjectPreSignedURL.html) AWSS3BlobRequestSigner is use for Signature V2 I'll impl AWSS3BlobRequestSignerV4 for Signature V4 [sigv4-query-string-auth](http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html) --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/678/files#r24713534
Re: [jclouds] add aws s3 signature v4 (#678)
> +try {
> +String encoded = URLEncoder.encode(value, DEFAULT_ENCODING);
> +
> +Matcher matcher = ENCODED_CHARACTERS_PATTERN.matcher(encoded);
> +StringBuffer buffer = new StringBuffer(encoded.length());
> +
> +while (matcher.find()) {
> +String replacement = matcher.group(0);
> +
> +if ("+".equals(replacement)) {
> +replacement = "%20";
> +} else if ("*".equals(replacement)) {
> +replacement = "%2A";
> +} else if ("%7E".equals(replacement)) {
> +replacement = "~";
> +}
URLEncoder.encode can be replace with UrlEscapers.urlFormParameterEscaper.escape
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/678/files#r24557031
Re: [jclouds] add aws s3 signature v4 (#678)
> +
> +@Override
> +public String service() {
> +return service;
> +}
> +
> +@Override
> +public String region(String host) {
> +return AwsHostNameUtils.parseRegionName(host, service());
> +}
> +}
> +}
> +
> +private final SignatureWire signatureWire;
> +private final String headerTag;
> +//private final String apiVersion;
copy from ```org.jclouds.aws.filters.FormSignerV4```
current, s3 api havn't any api version parameters, it's use in ec2 or other aws
api...
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/678/files#r24556779
Re: [jclouds] add aws s3 signature v4 (#678)
> +}
> +
> +byte[] hmacSHA256(String toSign, byte[] key) {
> +try {
> +ByteProcessor hmacSHA256 =
> asByteProcessor(crypto.hmacSHA256(key));
> +return readBytes(toInputStream(toSign), hmacSHA256);
> +} catch (IOException e) {
> +throw new HttpException("read bytes error", e);
> +} catch (InvalidKeyException e) {
> +throw new HttpException("invalid key", e);
> +}
> +}
> +
> +protected byte[] hash(InputStream input) throws HTTPException {
> +try {
> +MessageDigest md = MessageDigest.getInstance("SHA-256");
ok. Hashing.sha256 is same as MessageDigest
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/678/files#r24555082
Re: [jclouds] add aws s3 signature v4 (#678)
> @@ -56,7 +56,7 @@ > public AWSS3BlobRequestSigner(RestAnnotationProcessor processor, > BlobToObject blobToObject, > BlobToHttpGetOptions blob2HttpGetOptions, Class > interfaceClass, > @org.jclouds.location.Provider Supplier credentials, > - RequestAuthorizeSignature authSigner, @TimeStamp Provider > timeStampProvider, > + RequestAuthorizeSignatureV2 authSigner, @TimeStamp Provider > timeStampProvider, BlobRequestSigner Generates signed requests for blobs. useful in other tools such as backup utilities. keep AWSS3BlobRequestSigner use RequestAuthorizeSignatureV2 to signed requests for blobs as before. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/678/files#r24556033
Re: [jclouds] add aws s3 signature v4 (#678)
> +try {
> +String encoded = URLEncoder.encode(value, DEFAULT_ENCODING);
> +
> +Matcher matcher = ENCODED_CHARACTERS_PATTERN.matcher(encoded);
> +StringBuffer buffer = new StringBuffer(encoded.length());
> +
> +while (matcher.find()) {
> +String replacement = matcher.group(0);
> +
> +if ("+".equals(replacement)) {
> +replacement = "%20";
> +} else if ("*".equals(replacement)) {
> +replacement = "%2A";
> +} else if ("%7E".equals(replacement)) {
> +replacement = "~";
> +}
UrlEscapers.urlFormParameterEscaper isn's same as this urlEncode.
* URI encode every byte except the unreserved characters: 'A'-'Z', 'a'-'z',
'0'-'9', '-', '.', '_', and '~'.
* The space character is a reserved character and must be encoded as "%20" (and
not as "+").
* Each Uri-encoded byte is formed by a '%' and the two-digit hexadecimal value
of the byte.
* Letters in the hexadecimal value must be uppercase, for example "%1A".
* Encode the forward slash character, '/', everywhere except in the object key
name. For example, if the object key name is photos/Jan/sample.jpg, the forward
slash in the key name is not encoded.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/678/files#r24555634
Re: [jclouds] add aws s3 signature v4 (#678)
> +SortedMap sorted = new TreeMap();
> +if (params == null) {
> +return "";
> +}
> +Iterator> pairs =
> params.entries().iterator();
> +while (pairs.hasNext()) {
> +Map.Entry pair = pairs.next();
> +String key = pair.getKey();
> +String value = pair.getValue();
> +sorted.put(urlEncode(key), urlEncode(value));
> +}
> +
> +StringBuilder builder = new StringBuilder();
> +pairs = sorted.entrySet().iterator();
> +while (pairs.hasNext()) {
> +Map.Entry pair = pairs.next();
ok
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/678/files#r24555334
Re: [jclouds] add aws s3 signature v4 (#678)
>
> - private static final Set SIGNED_PARAMETERS =
> ImmutableSet.of("acl", "torrent", "logging", "location", "policy",
> +private static final Set SIGNED_PARAMETERS =
> ImmutableSet.of("acl", "torrent", "logging", "location", "policy",
Signature v4 need signed all of query string parameters.
_CanonicalQueryString specifies the URI-encoded query string parameters. You
URI-encode name and values individually. You must also sort the parameters in
the canonical query string alphabetically by key name. The sorting occurs after
encoding._
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/678/files#r24555329
Re: [jclouds] add aws s3 signature v4 (#678)
add aws s3 signature v4, plz review. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/678#issuecomment-73835221
[jclouds] add aws s3 signature v4 (#678)
AWS S3 signature v4 impl You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds/pull/678 -- Commit Summary -- * add aws s3 signature v4 -- File Changes -- M apis/s3/src/main/java/org/jclouds/s3/config/S3HttpApiModule.java (308) A apis/s3/src/main/java/org/jclouds/s3/filters/AwsHostNameUtils.java (185) M apis/s3/src/main/java/org/jclouds/s3/filters/RequestAuthorizeSignature.java (319) A apis/s3/src/main/java/org/jclouds/s3/filters/RequestAuthorizeSignatureV4.java (461) A apis/s3/src/main/java/org/jclouds/s3/filters/S3RequestAuthorizeSignatureV4.java (70) M apis/s3/src/test/java/org/jclouds/s3/S3ClientMockTest.java (33) A apis/s3/src/test/java/org/jclouds/s3/filters/AwsHostNameUtilsTest.java (61) D apis/s3/src/test/java/org/jclouds/s3/filters/RequestAuthorizeSignatureTest.java (157) A apis/s3/src/test/java/org/jclouds/s3/filters/RequestAuthorizeSignatureV2Test.java (157) A apis/s3/src/test/java/org/jclouds/s3/filters/RequestAuthorizeSignatureV4Test.java (118) M apis/s3/src/test/java/org/jclouds/s3/internal/BaseS3ClientTest.java (10) M providers/aws-s3/src/main/java/org/jclouds/aws/s3/blobstore/AWSS3BlobRequestSigner.java (6) M providers/aws-s3/src/main/java/org/jclouds/aws/s3/filters/AWSRequestAuthorizeSignature.java (5) A providers/aws-s3/src/main/java/org/jclouds/aws/s3/filters/AWSRequestAuthorizeSignatureV4.java (58) -- Patch Links -- https://github.com/jclouds/jclouds/pull/678.patch https://github.com/jclouds/jclouds/pull/678.diff --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/678
