[
https://issues.apache.org/jira/browse/HADOOP-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16401949#comment-16401949
]
Steve Loughran commented on HADOOP-15314:
-----------------------------------------
I see your goal here: have code which works with EMR switch to ASF Hadoop
without changes,
A concern I have, however, is that the db is shared across all clients, and if
at some point things were ever mixed to really support >1 scheme, then its a
mess. Removing schema checks altogether would be danger, but s3 & s3n could be
viable ish. Or somehow switch to s3a for the s3guard interaction, so the fact
that you've brought s3a up under a different original schema isn't visible.
That would be the best for stability over versions.
Thoughts, [~fabbri]?
> Scheme assertion in S3Guard DynamoDBMetadataStore::checkPath is unnecessarily
> restrictive
> -----------------------------------------------------------------------------------------
>
> Key: HADOOP-15314
> URL: https://issues.apache.org/jira/browse/HADOOP-15314
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.0.0
> Reporter: DJ Hoffman
> Priority: Major
>
> In version 3.0.0, the checkPath method for dealing with paths prevents us
> from using the s3:// scheme when utilizing S3Guard. However, in our
> core-site.xml we have included
> {noformat}
> <property>
> <name>fs.s3.impl</name>
> <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
> </property>{noformat}
> which should enforce that s3 prefixed paths go through s3a and are properly
> compatible with s3guard. We removed the assertion that paths use the s3a
> scheme (some of our paths use the s3 scheme) and our testing thus far with
> S3Guard enabled have been positive. We believe the assertion in checkPath is
> unnecessary and could be expanded to include the s3 and s3n schemes if not
> dropped altogether or altered in some other way. We're happy to develop and
> test a patch if the community is amenable to the change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
