Re: AWS v4 Signature algorithm required for generic-S3
Some background, there is a generic s3 api with most of the
functionality and an aws-s3 provider which inherits from the former and
overrides the endpoints. The aws-s3 provider also overrides the signer
that you can see in AWSS3HttpApiModule.bindRequestSigner and
AWSS3BlobStoreContextModule.bindRequestSigner. Instead
S3HttpApiModule.bindRequestSigner and
S3BlobStoreContextModule.bindRequestSigner should set this conditionally
based on a property. The Guice declarative style is confusing but you
can see some examples like @Named(Constants.PROPERTY_SESSION_INTERVAL)
which allow setting values based on a default or a user-provided value.
Fixing this will require moving AWSS3BlobRequestSignerV4 and its tests
from the aws-s3 provider to the s3 api.
jclouds has two styles of tests. You can run the unit tests via:
mvn test -pl :s3,:aws-s3
and the integration tests via:
mvn integration-test -pl :s3 -Plive \
-Dtest.s3.endpoint="${JCLOUDS_ENDPOINT}" \
-Dtest.s3.identity="${JCLOUDS_IDENTITY}" \
-Dtest.s3.credential="${JCLOUDS_CREDENTIAL}"
mvn integration-test -pl :aws-s3 -Plive \
-Dtest.aws-s3.identity="${JCLOUDS_IDENTITY}" \
-Dtest.aws-s3.credential="${JCLOUDS_CREDENTIAL}"
The ideal initial state would be to allow the s3 api to default to V2
and opt into V4 while the aws-s3 provider continues to default to V4.
There is a reasonable question about what the s3 api default should be
which we could address in a subsequent commit.
If you get stuck it might make sense to share a work-in-progress PR on
GitHub. I hope this helps!
On Sat, Dec 12, 2020 at 06:43:45AM -, John Calcote wrote:
> Andrew,
>
> I would love to submit this PR. But I'm going to need some guidance. I'm a
> pretty good coder, but I must tell you that there are big gaps in my
> understanding of how the jclouds code works.
>
> Can you please point me in the right direction? Tell me which files to look
> at and I'll try to decipher what needs to be done. And any additional
> information you think might be relevant. My hope is that I can do the work,
> saving you that effort, if you will spend just a few minutes jotting down how
> you would approach the patch since you obviously understand the code better
> than most.
>
> Thanks very much for the opportunity to help.
>
>
> On 2020/12/12 03:04:12, Andrew Gaul wrote:
> > 1) and 2) are hacky but will work. However we should really properly
> > support this by giving the generic S3 provider a property. This would
> > require reparenting some code from aws-s3 to s3. Would you be able to
> > submit a PR for this?
> >
> > The unsigned payload is also a desired feature and easy to add after
> > fixing the above.
> >
> > We plan to release 2.3.0 later this month so it could include these
> > features if you can help out!
> >
> > On Thu, Dec 10, 2020 at 07:42:22PM -, John Calcote wrote:
> > > I see someone asked this very question a while back (2018):
> > > https://lists.apache.org/thread.html/3553946dcbe7c72fc8f82c6353c9a5c56d7b587da18a4ebe12df1e26%40%3Cuser.jclouds.apache.org%3E
> > >
> > > Did anyone ever solve the double read issue regarding disabling
> > > data-hashing?
> > >
> > > On 2020/12/10 18:03:13, John Calcote wrote:
> > > > Hi all -
> > > >
> > > > We have a client who would like to use jclouds with ONTAP 9.8, which
> > > > requires V4 signatures AND path-based URL naming. I believe the true
> > > > AWS S3 provider requires virtual bucket URL naming. Anyone have any
> > > > ideas on how to either:
> > > >
> > > > 1) coerce jclouds into using the AWS S3 provider with path-based naming
> > > > or
> > > > 2) coerce jclouds into using the generic S3 provider with V4 signing
> > > >
> > > > Thanks in advance.
> > > > John
> > > >
> >
> > --
> > Andrew Gaul
> > http://gaul.org/
> >
--
Andrew Gaul
http://gaul.org/
Re: AWS v4 Signature algorithm required for generic-S3
Andrew, I would love to submit this PR. But I'm going to need some guidance. I'm a pretty good coder, but I must tell you that there are big gaps in my understanding of how the jclouds code works. Can you please point me in the right direction? Tell me which files to look at and I'll try to decipher what needs to be done. And any additional information you think might be relevant. My hope is that I can do the work, saving you that effort, if you will spend just a few minutes jotting down how you would approach the patch since you obviously understand the code better than most. Thanks very much for the opportunity to help. On 2020/12/12 03:04:12, Andrew Gaul wrote: > 1) and 2) are hacky but will work. However we should really properly > support this by giving the generic S3 provider a property. This would > require reparenting some code from aws-s3 to s3. Would you be able to > submit a PR for this? > > The unsigned payload is also a desired feature and easy to add after > fixing the above. > > We plan to release 2.3.0 later this month so it could include these > features if you can help out! > > On Thu, Dec 10, 2020 at 07:42:22PM -, John Calcote wrote: > > I see someone asked this very question a while back (2018): > > https://lists.apache.org/thread.html/3553946dcbe7c72fc8f82c6353c9a5c56d7b587da18a4ebe12df1e26%40%3Cuser.jclouds.apache.org%3E > > > > Did anyone ever solve the double read issue regarding disabling > > data-hashing? > > > > On 2020/12/10 18:03:13, John Calcote wrote: > > > Hi all - > > > > > > We have a client who would like to use jclouds with ONTAP 9.8, which > > > requires V4 signatures AND path-based URL naming. I believe the true AWS > > > S3 provider requires virtual bucket URL naming. Anyone have any ideas on > > > how to either: > > > > > > 1) coerce jclouds into using the AWS S3 provider with path-based naming or > > > 2) coerce jclouds into using the generic S3 provider with V4 signing > > > > > > Thanks in advance. > > > John > > > > > -- > Andrew Gaul > http://gaul.org/ >
Re: AWS v4 Signature algorithm required for generic-S3
1) and 2) are hacky but will work. However we should really properly support this by giving the generic S3 provider a property. This would require reparenting some code from aws-s3 to s3. Would you be able to submit a PR for this? The unsigned payload is also a desired feature and easy to add after fixing the above. We plan to release 2.3.0 later this month so it could include these features if you can help out! On Thu, Dec 10, 2020 at 07:42:22PM -, John Calcote wrote: > I see someone asked this very question a while back (2018): > https://lists.apache.org/thread.html/3553946dcbe7c72fc8f82c6353c9a5c56d7b587da18a4ebe12df1e26%40%3Cuser.jclouds.apache.org%3E > > Did anyone ever solve the double read issue regarding disabling data-hashing? > > On 2020/12/10 18:03:13, John Calcote wrote: > > Hi all - > > > > We have a client who would like to use jclouds with ONTAP 9.8, which > > requires V4 signatures AND path-based URL naming. I believe the true AWS S3 > > provider requires virtual bucket URL naming. Anyone have any ideas on how > > to either: > > > > 1) coerce jclouds into using the AWS S3 provider with path-based naming or > > 2) coerce jclouds into using the generic S3 provider with V4 signing > > > > Thanks in advance. > > John > > -- Andrew Gaul http://gaul.org/
Re: AWS v4 Signature algorithm required for generic-S3
I see someone asked this very question a while back (2018): https://lists.apache.org/thread.html/3553946dcbe7c72fc8f82c6353c9a5c56d7b587da18a4ebe12df1e26%40%3Cuser.jclouds.apache.org%3E Did anyone ever solve the double read issue regarding disabling data-hashing? On 2020/12/10 18:03:13, John Calcote wrote: > Hi all - > > We have a client who would like to use jclouds with ONTAP 9.8, which requires > V4 signatures AND path-based URL naming. I believe the true AWS S3 provider > requires virtual bucket URL naming. Anyone have any ideas on how to either: > > 1) coerce jclouds into using the AWS S3 provider with path-based naming or > 2) coerce jclouds into using the generic S3 provider with V4 signing > > Thanks in advance. > John >
