[ https://issues.apache.org/jira/browse/HADOOP-16280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran resolved HADOOP-16280. ------------------------------------- Fix Version/s: 3.3.0 Resolution: Duplicate HADOOP-16490 added exactly this retry logic > S3Guard: Retry failed read with backoff in Authoritative mode when file can > be opened > ------------------------------------------------------------------------------------- > > Key: HADOOP-16280 > URL: https://issues.apache.org/jira/browse/HADOOP-16280 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 > Reporter: Gabor Bota > Priority: Major > Fix For: 3.3.0 > > > When using S3Guard in authoritative mode a file can be reported from AWS S3 > that's missing like it is described in the following exception: > {noformat} > java.io.FileNotFoundException: re-open > s3a://cloudera-dev-gabor-ireland/test/TMCDOR-021df1ad-633f-47b8-97f5-6cd93f0b82d0 > at 0 on > s3a://cloudera-dev-gabor-ireland/test/TMCDOR-021df1ad-633f-47b8-97f5-6cd93f0b82d0: > > com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not > exist. (Service: Amazon S3; Status Code: 404; Error > Code: NoSuchKey; Request ID: E1FF9EA9B5DBBD7E; S3 Extended Request ID: > NzNIL4+dyA89WTnfbcwuYQK+hCfx51TfavwgC3oEvQI0IQ9M/zAspbXOfBIis8/nTolc4tRB9ik=), > S3 Extended Request ID: > NzNIL4+dyA89WTnfbcwuYQK+hCfx51TfavwgC3oEvQI0IQ9M/zAspbXOfBIis8/nTolc4tRB9ik=:NoSuchKey > {noformat} > But the metadata in S3Guard (e.g dynamo db) is there, so it can be opened. > The operation will not fail when it's opened, it will fail when we try to > read it, so the call > {noformat} > FSDataInputStream is = guardedFs.open(testFilePath);{noformat} > won't fail, but the next call > {noformat} > byte[] firstRead = new byte[text.length()]; > is.read(firstRead, 0, firstRead.length); > {noformat} > will fail with the exception message like what's above. > Once Authoritative mode is on, we assume that there is no out of band > operation, so the file will appear eventually. We should re-try in this case. -- This message was sent by Atlassian Jira (v8.3.2#803003) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org