[
https://issues.apache.org/jira/browse/HDDS-4243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195782#comment-17195782
]
Bharat Viswanadham commented on HDDS-4243:
------------------------------------------
With below config changes, able to read file from Ozone S3buckets with S3A
filesystem from OzoneS3
{code:java}
<property>
<name>fs.s3a.change.detection.version.required</name>
<value>false</value>
</property>
<property>
<name>fs.s3a.change.detection.mode</name>
<value>none</value>
</property>
{code}
>From my understanding one reason for this might be OzoneS3 does not support
>ETag feature, in HeadObject we returned ETag as Key Modification time, in
>GetObject ETag is not returned, and this has caused to fail with exception.
*S3ACode:*
{code:java}
default:
// mode == Client or Server; will trigger on version failures
// of getObjectMetadata even on server.
return new ImmutablePair<>(true,
new RemoteFileChangedException(uri,
operation,
String.format("%s "
+ CHANGE_DETECTED
+ " during %s%s."
+ " Expected %s got %s",
getSource(), operation, positionText, revisionId,
newRevisionId)));
{code}
> Get API not working from S3A filesystem with Ozone S3
> -----------------------------------------------------
>
> Key: HDDS-4243
> URL: https://issues.apache.org/jira/browse/HDDS-4243
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
> Reporter: Bharat Viswanadham
> Assignee: Bharat Viswanadham
> Priority: Major
> Labels: OzoneS3, S3A
>
> TroubleShooting S3A mentions S3 compatible servers that donot support Etags
> will see this server
> Refer
> [link|https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/troubleshooting_s3a.html]
> and look for below section content.
> Using a third-party S3 implementation that doesn’t support eTags might result
> in the following error.
> org.apache.hadoop.fs.s3a.NoVersionAttributeException:
> `s3a://my-bucket/test/file.txt':
> Change detection policy requires ETag
> at
> org.apache.hadoop.fs.s3a.impl.ChangeTracker.processResponse(ChangeTracker.java:153)
> at org.apache.hadoop.fs.s3a.S3AInputStream.reopen(S3AInputStream.java:200)
> at
> org.apache.hadoop.fs.s3a.S3AInputStream.lambda$lazySeek$1(S3AInputStream.java:346)
> at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$2(Invoker.java:195)
> at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109)
> at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:265)
> at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:322)
> at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:261)
> at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:193)
> at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:215)
> at org.apache.hadoop.fs.s3a.S3AInputStream.lazySeek(S3AInputStream.java:339)
> at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:372)
> {code:java}
> org.apache.hadoop.fs.s3a.NoVersionAttributeException:
> `s3a://sept14/dir1/dir2/dir3/key1': Change detection policy requires ETag
> at
> org.apache.hadoop.fs.s3a.impl.ChangeTracker.processNewRevision(ChangeTracker.java:275)
> at
> org.apache.hadoop.fs.s3a.impl.ChangeTracker.processMetadata(ChangeTracker.java:261)
> at
> org.apache.hadoop.fs.s3a.impl.ChangeTracker.processResponse(ChangeTracker.java:195)
> at
> org.apache.hadoop.fs.s3a.S3AInputStream.reopen(S3AInputStream.java:208)
> at
> org.apache.hadoop.fs.s3a.S3AInputStream.lambda$lazySeek$1(S3AInputStream.java:359)
> at
> org.apache.hadoop.fs.s3a.Invoker.lambda$maybeRetry$3(Invoker.java:223)
> at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:110)
> at
> org.apache.hadoop.fs.s3a.Invoker.lambda$maybeRetry$5(Invoker.java:347)
> at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:407)
> at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:343)
> at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:221)
> at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:265)
> at
> org.apache.hadoop.fs.s3a.S3AInputStream.lazySeek(S3AInputStream.java:351)
> at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:464)
> at java.io.DataInputStream.read(DataInputStream.java:100)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:94)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:68)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:129)
> at
> org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.writeStreamToFile(CommandWithDestination.java:494)
> at
> org.apache.hadoop.fs.shell.CommandWithDestination.copyStreamToTarget(CommandWithDestination.java:416)
> at
> org.apache.hadoop.fs.shell.CommandWithDestination.copyFileToTarget(CommandWithDestination.java:351)
> at
> org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:286)
> at
> org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:271)
> at
> org.apache.hadoop.fs.shell.Command.processPathInternal(Command.java:367)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:331)
> at
> org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:304)
> at
> org.apache.hadoop.fs.shell.CommandWithDestination.processPathArgument(CommandWithDestination.java:266)
> at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:286)
> at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:270)
> at
> org.apache.hadoop.fs.shell.CommandWithDestination.processArguments(CommandWithDestination.java:237)
> at
> org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:120)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:177)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:328)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:391)
> get: `s3a://sept14/dir1/dir2/dir3/key1': Change detection policy requires ETag
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
