qijiale76 opened a new pull request, #8871: URL: https://github.com/apache/hadoop/pull/8871
### Description of PR JIRA: https://issues.apache.org/jira/browse/YARN-12043 Depends on #8870 (YARN-12042); until that one is merged, this PR also contains its commit. Only the last commit belongs to this PR. Once the RM removed a completed application, no RM knows it any more. With SubClusters in different places each one usually has its own Application History Server, while a client of the Router is configured with one at most. This PR lets the Router ask the history servers of the SubClusters for the applications that no RM knows, the one of the home SubCluster first if the StateStore still has it. - New per-SubCluster keys `yarn.router.application-history.address.<subClusterId>` and `yarn.router.application-history.webapp.address.<subClusterId>`. Without them, the RPC and REST interfaces use the Router's own `yarn.timeline-service.address` if the generic application history is enabled, and the web proxy keeps using its own history server as before. - `getApplicationReport`, REST `getApp` and `getAppState` return the report of the history server; `AppInfo` gets a constructor from an `ApplicationReport` for this. - The web proxy redirects to the application page of the history server the report came from, and answers 404 if its web address is not configured, instead of redirecting to another history server. - ACLs: the RPC and REST interceptors ask the history servers as the client, as they ask the RMs, so that the history servers apply their application ACLs. As for the RMs, the history servers must allow the Router's user to impersonate the clients (`hadoop.proxyuser.<router user>.*`). The web proxy asks them as its login user. - An application is reported missing only when every RM and every history server answered that it does not know it; otherwise a retriable error is returned. - Behavior change: the REST `getApp` and `getAppState` now answer 404 for an application that does not exist, as the RM does, and 503 if some RM or history server could not tell. They used to return an empty response (204) in both cases. ### How was this patch tested? New unit tests: - `TestFederationApplicationHistory`: per-SubCluster lookup, home SubCluster first, asked as the given user, unreachable history server, missing web address, Router's own history server. - `TestFederationClientInterceptor`: report from history, unreachable history server. - `TestFederationInterceptorREST`: `getApp`/`getAppState` from history, 404 and 503 answers. - `TestFedAppReportFetcher`: report from the history server of a SubCluster with its page, unreachable history server. - `TestWebAppProxyServlet`: redirect to the page of the history server the report came from, 404 if unknown. Ran the tests above plus `TestFederationInterceptorRESTRetry`, `TestRouterWebServices*`, `TestRouterWebAppProxy` and `TestYarnConfigurationFields` with JDK 17. The full Router and web proxy suites pass except `TestAmFilter#testProxyUpdate`, which fails the same way on trunk, and known flaky `TestYarnFederationWithFairScheduler`. ### For code changes: - [x] Does the title of this PR start with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: Have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? ### AI Tooling - [x] The PR includes the phrase "Contains content generated by <tool>" where <tool> is the name of the AI tool used. - [x] My use of AI contributions follows the ASF legal policy https://www.apache.org/legal/generative-tooling.html Contains content generated by Claude. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
