anujmodi2021 commented on PR #5881: URL: https://github.com/apache/hadoop/pull/5881#issuecomment-1667317736
> There's this whole framework in hadoop for retrying, org.apache.hadoop.io.retry whose RetryPolicy is implemented by lots of things, including one which can map from exception to other policies. for s3a then we have a fairly complex one which tries to do the right thing for different failures (still does backoff for connectivity problems though) org.apache.hadoop.fs.s3a.S3ARetryPolicy > > which is then fed into a lambda-expression wrapper. org.apache.hadoop.fs.s3a.Invoker > > 1. there's a risk that you are re-implementing a lot of this > 2. and by using classes with the same names, causing confusion. > > can you look at the `org.apache.hadoop.io.retry` package and some uses and think "would it be possible to use -and if not, why not?". If you can use it, then the new policy is simply a RetryUpToMaximumCountWithFixedSleep. > > If not, you have to use a different name for the baseclass than RetryPolicy. [HADOOP-997](https://issues.apache.org/jira/browse/HADOOP-997) was there first. > > TestStaticRetryPolicy is an integration test with the wrong name. ideally, it should be converted to a unit test, so it doesn't skip in yetus. otherwise, make ITestStaticRetryPolicy. Same for TestExponentialRetryPolicy, even though that is an inherited issue. Hi Steve, Thank you so much for reviewing this PR. Regarding your suggestion on using the io.retry.RetryPolicy, we are totally onboad to that and we can see the value add if we inclide to the hadoop-common implementation of retry policies. We would like to take this up as a separate work item as it does not make much sense to change only the new static retry policy and leave the already existing exponential retry policy as it is. We want to restructure our whole implementation of retry policies by inheriting from the hadoop-common code and it requires a larger investement. I have created a Jira specially for this purpose and we will use it to track this refactoring separately. **https://issues.apache.org/jira/browse/HADOOP-18841** Meanwhile for unblocking this Work Item here, I have renamed our retry policy to AbfsRetryPolicy so that it stands independent of io.retry code and does not interfare with that. Hope that sounds good. I have addressed other comments in the PR and comments related to retry policy, I will add in the above Jira so that we do not miss them later when we start working on that. I would request you to review the latest changes. Thanks a lot. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
