jerryshao opened a new pull request, #13425: URL: https://github.com/apache/gravitino/pull/13425
### What changes were proposed in this pull request? A job's staging directory is now `<gravitino.job.stagingDir>/job-runs/job-<id>`, derived from the job id alone, instead of `<stagingDir>/<metalake>/<template>/job-<id>`. Both `cleanUpStagingDirs()` and `deleteJobTemplate()` delete a job's directory through one helper. It first tries the new location. If that doesn't exist, the job was submitted by an earlier version, and its directory is looked up by id under `<stagingDir>/*/*/job-<id>` instead of being rebuilt from the current names. This also removes directories that were already leaked by an earlier rename. Symbolic links and hidden entries such as `.job-output-index` are skipped, and an unreadable directory (e.g. `lost+found`) doesn't stop the lookup. Also updates the staging paths in the table maintenance docs, and fixes the `JobIT` check for a rejected job's staging directory, which passed vacuously with the new layout. ### Why are the changes needed? The cleanup rebuilt the staging path from the current template and metalake names. After a template or metalake rename, the path didn't exist, nothing was deleted, and the job entity was removed anyway, so the directory (executables, scripts, jars, output logs) leaked forever. The metalake rename case is verified by a new test. Fix: #13372 ### Does this PR introduce _any_ user-facing change? The layout of the job staging directory changes as described above. Directories of jobs submitted before the upgrade are still cleaned up. No API or configuration changes. During a rolling upgrade with a shared staging directory, a server running the old version can't find the directories of jobs submitted by upgraded servers, so the jobs it cleans up leave their directories behind. ### How was this patch tested? - New tests in `TestJobManagerMultiNode` (real JDBC backend and `LocalJobExecutor`) for template rename + expiry, repeated template renames + `deleteJobTemplate`, and metalake rename + expiry. All three fail on main. - New unit tests in `TestJobManager` for the legacy directory lookup, including symlinks, hidden directories, same-name files and unreadable directories. - `./gradlew :core:test -PskipITs --tests 'org.apache.gravitino.job.*'`, and `JobIT` in embedded mode. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
