[
https://issues.apache.org/jira/browse/HADOOP-16863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-16863.
-------------------------------------
Resolution: Not A Problem
AWS S3 is now consistent. No more 404 caching.
> Report on S3A cached 404 recovery better
> ----------------------------------------
>
> Key: HADOOP-16863
> URL: https://issues.apache.org/jira/browse/HADOOP-16863
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.0
> Reporter: Steve Loughran
> Priority: Major
>
> A big hadoop -fs copyFromLocal is showing that 404 cacheing is still
> happening.
> {code}
> 20/02/13 01:02:18 WARN s3a.S3AFileSystem: Failed to find file
> s3a://dilbert/dogbert/queries_split_1/catberg.q._COPYING_. Either it is not
> yet visible, or it has been deleted.
> 0/02/13 01:02:18 WARN s3a.S3AFileSystem: Failed to find file
> s3a://dilbert/dogbert/queries_split_1/catberg.q._COPYING_. Either it is not
> yet visible, or it has been deleted.
> {noformat}
> We are recovering (good) but it's (a) got the people running this code
> worried and (b) shouldn't be happening.
> Proposed
> * error message to -> to a wiki link to a (new) doc on the topic.
> * retried clause to increment counter & if count >1 report on #of attempts
> and duration
> * S3A FS.deleteOnExit to avoid all checks
> * and review the copyFromLocal to make sure no other probes are happening'
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
