Re: [jclouds] add aws s3 signature v4 (#678)
try Multipart Upload. initiate Multipart Upload..temporarily store specific length part, upload all stream part..complete multipart upload... part requires Content-Length. not all a file. but I have not tried :) --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/678#issuecomment-104135788
Re: [jclouds] add aws s3 signature v4 (#678)
Hi, I impl aws s3 signer v4 chunked upload, use when put object, payload cannot repeatable. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/678#issuecomment-98595506
Re: [jclouds] add aws s3 signature v4 (#678)
It's seem as region eu-central-1 doesn't supported AWS sign V2.
在 2015/4/18 2:46, Andrew Gaul 写道:
@zhaojin0 https://github.com/zhaojin0 I am testing this and see many
errors of the form:
|org.jclouds.aws.AWSResponseException: request GET
https://gaul-blobstore3760643340725640912-v4-only.s3-eu-central-1.amazonaws.com/
HTTP/1.1 failed with code 400, error: AWSError{requestId='0F84681CE6013127',
requestToken='THXHcOkpHKTdN7DedbIb8qCj7/MuvvvGyu31O42h6BjkTFSvHdKNggDL/aQ6Mm1IGROwsAO58fw=',
code='InvalidRequest', message='The authorization mechanism you have
provided is not supported. Please use AWS4-HMAC-SHA256.',
context='{HostId=THXHcOkpHKTdN7DedbIb8qCj7/MuvvvGyu31O42h6BjkTFSvHdKNggDL/aQ6Mm1IGROwsAO58fw=}'}
|
when deleting items in the container between runs. Any suggestions on
this? Also do all the integration tests pass for you? I see a few errors:
|
AWSS3ContainerIntegrationLiveTestBaseContainerIntegrationTest.deleteContainerIfEmptyWithoutContents:315
expected [true] but found [false]
AWSS3ContainerLiveTestBaseContainerLiveTest.testPublicAccessInNonDefaultLocationWithBigBlob:112-BaseContainerLiveTest.runCreateContainerInLocation:124-BaseBlobStoreIntegrationTest.assertConsistencyAwareContainerExists:361-BaseBlobStoreIntegrationTest.assertConsistencyAware:248-BaseBlobStoreIntegrationTest.assertConsistencyAware:235
» HttpResponse
AWSS3ServiceIntegrationLiveTestBaseServiceIntegrationTest.testAllLocations:52-BaseBlobStoreIntegrationTest.assertConsistencyAware:248-BaseBlobStoreIntegrationTest.assertConsistencyAware:235
» HttpResponse
AWSS3ServiceIntegrationLiveTestBaseServiceIntegrationTest.testGetAssignableLocations:93
{scope=REGION, id=eu-central-1, description=eu-central-1, parent=aws-s3,
iso3166Codes=[DE-HE]} ||{scope=PROVIDER, id=aws-s3,
description=https://s3.amazonaws.com, iso3166Codes=[US, US-CA, US-OR, BR-SP,
IE, SG, AU-NSW, JP-13]}
Tests run: 99, Failures: 4, Errors: 0, Skipped: 1
|
—
Reply to this email directly or view it on GitHub
https://github.com/jclouds/jclouds/pull/678#issuecomment-94050351.
--
赵金
Zhao Jin
18610722868
北京优创联动科技有限公司
北京市 海淀区 学清路38号 金码大厦16层
100083
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/678#issuecomment-94248352
Re: [jclouds] add aws s3 signature v4 (#678)
AWS Sign V4 use sha256 content hash. If payload can not be reset, aws supported chunked uploads. http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/678#issuecomment-88369850
Re: [jclouds] add aws s3 signature v4 (#678)
+HttpRequest.Builder requestBuilder,
+String method,
+URI endpoint,
+Payload payload
+) {
+InputStream payloadStream;
+try {
+payloadStream = usePayloadForQueryParameters(method, payload) ?
+getQueryStringContent(endpoint)
+: getPayloadContentWithoutQueryString(payload);
+} catch (IOException e) {
+throw new HttpException(Unable to open stream before calculate
AWS4 signature, e);
+}
+String contentSha256 =
base16().lowerCase().encode(hash(payloadStream));
+try {
+payloadStream.reset();
payload stream use calculate content hash.
if can not be repeatable, the payload cannot append to http request body.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/678/files#r25842533
Re: [jclouds] add aws s3 signature v4 (#678)
it's use for sign a temporary access... I provided testcase AWSS3BlobSignerV4ExpectTest. sorry, this's my first pull request, I format some code --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/678#issuecomment-76351814
Re: [jclouds] add aws s3 signature v4 (#678)
I'm sorry for my lazy...
i add temporary access signature code, but i dont known how to test it. It
could work in aws region cn-north-1.
```java
public class AWSS3BlobStoreContextModule extends S3BlobStoreContextModule {
//...
@Override
protected void bindRequestSigner() {
bind(BlobRequestSigner.class).to(AWSS3BlobRequestSignerV4.class);
}
}
```
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/678#issuecomment-75247425
Re: [jclouds] add aws s3 signature v4 (#678)
@@ -56,7 +56,7 @@ public AWSS3BlobRequestSigner(RestAnnotationProcessor processor, BlobToObject blobToObject, BlobToHttpGetOptions blob2HttpGetOptions, ClassAWSS3Client interfaceClass, @org.jclouds.location.Provider SupplierCredentials credentials, - RequestAuthorizeSignature authSigner, @TimeStamp ProviderString timeStampProvider, + RequestAuthorizeSignatureV2 authSigner, @TimeStamp ProviderString timeStampProvider, sorry, this is an wrong.. BlobRequestSigner use to generating pre-signed URLs, [Share an Object with Others](http://docs.aws.amazon.com/AmazonS3/latest/dev/ShareObjectPreSignedURL.html) AWSS3BlobRequestSigner is use for Signature V2 I'll impl AWSS3BlobRequestSignerV4 for Signature V4 [sigv4-query-string-auth](http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html) --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/678/files#r24713534
Re: [jclouds] add aws s3 signature v4 (#678)
- private static final SetString SIGNED_PARAMETERS = ImmutableSet.of(acl, torrent, logging, location, policy, +private static final SetString SIGNED_PARAMETERS = ImmutableSet.of(acl, torrent, logging, location, policy, Signature v4 need signed all of query string parameters. _CanonicalQueryString specifies the URI-encoded query string parameters. You URI-encode name and values individually. You must also sort the parameters in the canonical query string alphabetically by key name. The sorting occurs after encoding._ --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/678/files#r24555329
Re: [jclouds] add aws s3 signature v4 (#678)
+SortedMapString, String sorted = new TreeMapString, String();
+if (params == null) {
+return ;
+}
+IteratorMap.EntryString, String pairs =
params.entries().iterator();
+while (pairs.hasNext()) {
+Map.EntryString, String pair = pairs.next();
+String key = pair.getKey();
+String value = pair.getValue();
+sorted.put(urlEncode(key), urlEncode(value));
+}
+
+StringBuilder builder = new StringBuilder();
+pairs = sorted.entrySet().iterator();
+while (pairs.hasNext()) {
+Map.EntryString, String pair = pairs.next();
ok
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/678/files#r24555334
Re: [jclouds] add aws s3 signature v4 (#678)
+try {
+String encoded = URLEncoder.encode(value, DEFAULT_ENCODING);
+
+Matcher matcher = ENCODED_CHARACTERS_PATTERN.matcher(encoded);
+StringBuffer buffer = new StringBuffer(encoded.length());
+
+while (matcher.find()) {
+String replacement = matcher.group(0);
+
+if (+.equals(replacement)) {
+replacement = %20;
+} else if (*.equals(replacement)) {
+replacement = %2A;
+} else if (%7E.equals(replacement)) {
+replacement = ~;
+}
UrlEscapers.urlFormParameterEscaper isn's same as this urlEncode.
* URI encode every byte except the unreserved characters: 'A'-'Z', 'a'-'z',
'0'-'9', '-', '.', '_', and '~'.
* The space character is a reserved character and must be encoded as %20 (and
not as +).
* Each Uri-encoded byte is formed by a '%' and the two-digit hexadecimal value
of the byte.
* Letters in the hexadecimal value must be uppercase, for example %1A.
* Encode the forward slash character, '/', everywhere except in the object key
name. For example, if the object key name is photos/Jan/sample.jpg, the forward
slash in the key name is not encoded.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/678/files#r24555634
Re: [jclouds] add aws s3 signature v4 (#678)
@@ -56,7 +56,7 @@ public AWSS3BlobRequestSigner(RestAnnotationProcessor processor, BlobToObject blobToObject, BlobToHttpGetOptions blob2HttpGetOptions, ClassAWSS3Client interfaceClass, @org.jclouds.location.Provider SupplierCredentials credentials, - RequestAuthorizeSignature authSigner, @TimeStamp ProviderString timeStampProvider, + RequestAuthorizeSignatureV2 authSigner, @TimeStamp ProviderString timeStampProvider, BlobRequestSigner Generates signed requests for blobs. useful in other tools such as backup utilities. keep AWSS3BlobRequestSigner use RequestAuthorizeSignatureV2 to signed requests for blobs as before. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/678/files#r24556033
Re: [jclouds] add aws s3 signature v4 (#678)
+}
+
+byte[] hmacSHA256(String toSign, byte[] key) {
+try {
+ByteProcessorbyte[] hmacSHA256 =
asByteProcessor(crypto.hmacSHA256(key));
+return readBytes(toInputStream(toSign), hmacSHA256);
+} catch (IOException e) {
+throw new HttpException(read bytes error, e);
+} catch (InvalidKeyException e) {
+throw new HttpException(invalid key, e);
+}
+}
+
+protected byte[] hash(InputStream input) throws HTTPException {
+try {
+MessageDigest md = MessageDigest.getInstance(SHA-256);
ok. Hashing.sha256 is same as MessageDigest
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/678/files#r24555082
Re: [jclouds] add aws s3 signature v4 (#678)
+try {
+String encoded = URLEncoder.encode(value, DEFAULT_ENCODING);
+
+Matcher matcher = ENCODED_CHARACTERS_PATTERN.matcher(encoded);
+StringBuffer buffer = new StringBuffer(encoded.length());
+
+while (matcher.find()) {
+String replacement = matcher.group(0);
+
+if (+.equals(replacement)) {
+replacement = %20;
+} else if (*.equals(replacement)) {
+replacement = %2A;
+} else if (%7E.equals(replacement)) {
+replacement = ~;
+}
URLEncoder.encode can be replace with UrlEscapers.urlFormParameterEscaper.escape
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/678/files#r24557031
Re: [jclouds] add aws s3 signature v4 (#678)
+
+@Override
+public String service() {
+return service;
+}
+
+@Override
+public String region(String host) {
+return AwsHostNameUtils.parseRegionName(host, service());
+}
+}
+}
+
+private final SignatureWire signatureWire;
+private final String headerTag;
+//private final String apiVersion;
copy from ```org.jclouds.aws.filters.FormSignerV4```
current, s3 api havn't any api version parameters, it's use in ec2 or other aws
api...
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/678/files#r24556779
[jclouds] add aws s3 signature v4 (#678)
AWS S3 signature v4 impl You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds/pull/678 -- Commit Summary -- * add aws s3 signature v4 -- File Changes -- M apis/s3/src/main/java/org/jclouds/s3/config/S3HttpApiModule.java (308) A apis/s3/src/main/java/org/jclouds/s3/filters/AwsHostNameUtils.java (185) M apis/s3/src/main/java/org/jclouds/s3/filters/RequestAuthorizeSignature.java (319) A apis/s3/src/main/java/org/jclouds/s3/filters/RequestAuthorizeSignatureV4.java (461) A apis/s3/src/main/java/org/jclouds/s3/filters/S3RequestAuthorizeSignatureV4.java (70) M apis/s3/src/test/java/org/jclouds/s3/S3ClientMockTest.java (33) A apis/s3/src/test/java/org/jclouds/s3/filters/AwsHostNameUtilsTest.java (61) D apis/s3/src/test/java/org/jclouds/s3/filters/RequestAuthorizeSignatureTest.java (157) A apis/s3/src/test/java/org/jclouds/s3/filters/RequestAuthorizeSignatureV2Test.java (157) A apis/s3/src/test/java/org/jclouds/s3/filters/RequestAuthorizeSignatureV4Test.java (118) M apis/s3/src/test/java/org/jclouds/s3/internal/BaseS3ClientTest.java (10) M providers/aws-s3/src/main/java/org/jclouds/aws/s3/blobstore/AWSS3BlobRequestSigner.java (6) M providers/aws-s3/src/main/java/org/jclouds/aws/s3/filters/AWSRequestAuthorizeSignature.java (5) A providers/aws-s3/src/main/java/org/jclouds/aws/s3/filters/AWSRequestAuthorizeSignatureV4.java (58) -- Patch Links -- https://github.com/jclouds/jclouds/pull/678.patch https://github.com/jclouds/jclouds/pull/678.diff --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/678
Re: [jclouds] add aws s3 signature v4 (#678)
add aws s3 signature v4, plz review. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/678#issuecomment-73835221
