nzw921rx opened a new pull request, #12460: URL: https://github.com/apache/seatunnel/pull/12460
## What does this PR do? This PR implements the MVP proposed in #12457 for running one native Zeta job as one isolated YARN application or Kubernetes Job. Before this change, users who wanted a job-scoped Zeta cluster had to maintain platform-specific scripts for Master and Worker startup, Hazelcast discovery, job submission, failure propagation, and resource cleanup. With this change, `seatunnel-application.sh` submits the job and deployment options to the selected provider. The provider starts one Zeta Master and the configured fixed number of Workers. The Master submits the job after all Workers have joined the cluster and registered their slots. Job completion, failure, and cancellation are propagated to YARN or Kubernetes, and the application-owned compute resources are released. ## Design - `seatunnel-resource-manager-core` defines the platform-neutral application specification, client and deployer contracts, shared configuration options, and localized JAR path resolution. It is built by default. - `seatunnel-resource-manager-yarn` and `seatunnel-resource-manager-kubernetes` are optional provider modules enabled by the `yarn` and `kubernetes` Maven profiles. - The Zeta server `ResourceManagerDriver` SPI allocates external Worker processes. The existing Zeta `ResourceManager` continues to own slot registration and task scheduling. - The standard SeaTunnel distribution packages enabled providers under `resource-managers/<provider>`. This PR does not introduce a separate Application Mode distribution. - The Engine client remains responsible for Hazelcast job communication and does not depend on YARN or Kubernetes SDKs. On YARN, the Zeta Master runs in the ApplicationMaster container and Workers are launched through `AMRMClient` and `NMClient`. On Kubernetes, the Master runs in a Kubernetes Job Pod and creates the fixed Worker Pods through the Kubernetes API. ## Checkpoint recovery and MVP boundary The MVP supports one Master and a configurable fixed number of Workers. Hazelcast backup count is `0`, so losing the Master or a Worker fails the current application. A later submission can set `application.restore-job-id` to restore the job from its latest durable checkpoint. The YARN E2E suite validates this flow with HDFS, and the Kubernetes E2E suite validates it with persistent-volume storage. Multiple Masters, live Master failover, automatic Worker replacement, autoscaling, and Kerberos are outside the scope of this PR. ## Distribution, CI, and documentation - Add `yarn` and `kubernetes` build profiles and package enabled providers into the standard binary distribution. - Add one `seatunnel-resource-managers-e2e` module with package-separated YARN and Kubernetes suites and shared FakeSource, Console, and Assert job configurations. - Add focused GitHub Actions jobs for both providers. Diagnostics are uploaded only when a job fails. - Add English and Chinese overview, architecture, quick-start, configuration, checkpoint-recovery, and FAQ pages. ## Test - 48-module dependency build and packaging: passed. - Engine Core tests: 28 passed. - Resource Manager Core tests: 13 passed. - Starter tests: 15 passed. - YARN provider tests: 16 passed. - Kubernetes provider tests: 11 passed. - Kubernetes Application Mode E2E: 6 passed. - YARN Application Mode E2E: 6 passed, including HDFS checkpoint recovery in a new application. - `git diff --check`, workflow YAML parsing, and changed-document link checks: passed. Maven `verify` was not run because the repository task instructions prohibit that goal. The relevant package, test, and integration-test phases were run directly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
