DanielLeens opened a new issue, #12119: URL: https://github.com/apache/seatunnel/issues/12119
## Description This is a design task for the job admission path. It is not a regression report. Verified at `dev` commit `97d461bc0773399d632fd078735736ecd44f5f0b`: - `CoordinatorService.submitJob()` (`CoordinatorService.java:1374`) constructs `new JobMaster(...)` (`:1419`), calls `init()`, and only then enqueues (`pendingJobQueue.put`, `:1450`). The master-switch restore path does the same (`:1106` -> `:1137` -> `:1158`). - `JobMaster.init()` (`JobMaster.java:230-322`) acquires classloaders (`:257`), restores the DAG, may run cluster-side sink save-mode handling when `SAVEMODE_EXECUTE_LOCATION == CLUSTER` (`:274-285`), builds the physical plan (`:293`), creates the checkpoint manager (`:319`), and releases the acquired classloader references in `finally` (`:313-314`). What a queued job retains today: the initialized `LogicalDag`/`PhysicalPlan`, `CheckpointCoordinator` objects, and futures. It does not retain `DefaultClassLoaderService` reference counts, and it holds no checkpoint threads (schedulers create workers on demand). Retained plan objects can still keep shared cached loaders alive in `cacheMode`. Constraint that rules out the simple reorder: `pendingJobSchedule()` calls `jobMaster.preApplyResources()` (`CoordinatorService.java:422`; `JobMaster.java:505/514`), and `preApplyResourcesForAll` (`:646-662`) sizes the slot request by enumerating `physicalPlan.getPipelineList()`. The plan must exist before the resource check, so "build the `JobMaster` after the check" is circular. Semantics that would move if initialization is deferred: configuration and plugin-load errors and cluster-side save-mode effects currently surface before the submit acknowledgement; a deferred `init()` must surface them through job status rather than lose them. ## Expected outcome - A two-phase admission contract: (1) a durable acceptance record (ids, config/artifact digests, enqueue sequence, priority, version, master epoch); (2) plan construction, either built once with the derived slot request persisted and the heavy objects dropped while queued (accepting a re-plan on admission), or a lightweight resource description that does not need the full plan. - Explicit rules for error timing, pending-job query and cancel, restore, and acceptance semantics. - Legacy `JobStatus` unchanged: queued jobs project to `PENDING`, admitted jobs to `INITIALIZING`. - Heap per queued job measured before and after under `WAIT` with a backlog of 10x capacity. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
