qijiale76 opened a new pull request, #8870: URL: https://github.com/apache/hadoop/pull/8870
### Description of PR JIRA: https://issues.apache.org/jira/browse/YARN-12042 The RM removes a completed application once it keeps more than `yarn.resourcemanager.max-completed-applications`, and its home SubCluster is deleted from the Federation StateStore with it. From then on, the Router answered `getApplicationReport` and `forceKillApplication` with a YarnException "does not exist in FederationStateStore", so YarnClient did not fall back to its history server as it does with an RM. The REST `getApp`, `getAppState`, `updateAppState` and the web proxy only looked at the StateStore, and `getApplicationHomeSubCluster` asked the RMs one after the other through the regular proxies, so an RM that is down but still active in the StateStore held the request for `yarn.resourcemanager.connect.max-wait.ms`. This PR adds `FederationApplicationRMProbe`, which asks the RMs of all active SubClusters at once through proxies that try each RM once with short timeouts, and waits at most `yarn.router.application-probe.timeout` (default 10s). The RPC and REST interceptors and the web proxy use it when the StateStore has no home SubCluster. - `ApplicationNotFoundException` is returned, so that clients fall back to their history server, only when every RM answered that it does not know the application; the one of the home SubCluster's RM is passed on as well. - If some RM could not be asked, failed or did not answer in time, a YarnException is returned so that the client retries. - The RPC and REST interceptors ask the RMs as the client, like their other calls; the web proxy as its login user. - The new key is in `yarn-default.xml` and `Federation.md`. ### How was this patch tested? New unit tests: - `TestFederationApplicationRMProbe`: found, not found, unreachable RM, saturated pool, hanging RM bounded by the timeout, probe configuration. - `TestFederationClientInterceptor`: report found without home SubCluster (and recorded again), not found returns `ApplicationNotFoundException`, unreachable SubCluster returns a retriable YarnException. - `TestFederationInterceptorREST`: `getApp` without home SubCluster. - `TestFedAppReportFetcher`: without home SubCluster, in no SubCluster (with and without AHS), unreachable SubCluster. Ran `TestFederationApplicationRMProbe`, `TestFederationClientInterceptor*`, `TestFederationInterceptorREST*`, `TestRouterWebAppProxy`, `TestFedAppReportFetcher` and `TestYarnConfigurationFields` with JDK 17. ### For code changes: - [x] Does the title of this PR start with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: Have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? ### AI Tooling - [x] The PR includes the phrase "Contains content generated by <tool>" where <tool> is the name of the AI tool used. - [x] My use of AI contributions follows the ASF legal policy https://www.apache.org/legal/generative-tooling.html Contains content generated by Claude. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
