SEPURI-SAI-KRISHNA commented on PR #13329: URL: https://github.com/apache/seatunnel/pull/13329#issuecomment-6999469466
Thanks @nzw921rx. I read #13328, STIP-41 (#12457) and the full diff at `69f562a5ca`. The last commit already fixed the status cleanup case I had noted. +1 to Issue 4 from @SEZ9 (Hazelcast trust boundary), which I had noted too. One detail if a per-application secret is added: workers get the cluster name as a command-line argument (`WorkerContainerFactory`, `List.of(clusterName, masterAddress, ...)`), which other users on the NodeManager host can usually read with `ps`, so the secret should travel in the container credentials or a localized file instead. Points not covered there: **1. Scope vs. STIP-41.** The STIP lists Kerberos and pre-staged YARN distributions as non-goals for the initial YARN implementation; reviewers on #12457 asked for one self-contained archive per submission, with shared HDFS artifacts in a separate STIP. #13328 notes the Kerberos change. Named snapshots registered in place (`YarnDistributionSource.register`) are the pre-staged case. Could the #12457 reviewers confirm this scope? If not, the simple-auth path could land first, with Kerberos and snapshots as follow-ups. **2. Kerberos limits.** `YarnSecurityContext.masterTokens` gets tokens for the staging filesystem only, and the keytab stays on the submitter. The Kerberos section says the RM renews tokens up to the HDFS token lifetime, but not the effect: by default after 7 days (`dfs.namenode.delegation.token.max-lifetime`) the AM and workers lose HDFS access, so long streaming jobs stop writing HDFS checkpoints. Checkpoint storage on another nameservice gets no token. Please state both. Also, `YarnKerberosIT` checks launch-context tokens; no application runs on the secure cluster. **3. Smaller items** - The README says both suites read jobs from `src/test/resources/common`, which does not exist; YARN scenarios have their own `job.conf`. - No production callers: `YarnDistribution.stage(...)` (both overloads) and the static six-argument `YarnApplicationFileUploader.upload(...)` are unused (`submit` calls the instance `upload()`). `YarnContainerLaunchContextFactory`, `YarnDistribution.inspect(File)`, `archive(Path)` and the "older manifests" branch of `fromManifest` serve only tests. - The AM heartbeats every 500 ms for the whole run (`YarnResourceManagerDriver.java:65`). The #12460 review suggested 1 s. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
