jerryshao opened a new issue, #13365: URL: https://github.com/apache/gravitino/issues/13365
### What would you like to be improved? `JobManager` works out a job's `startedAt` and `finishedAt` from its periodic status poll (`gravitino.job.statusPullIntervalInMs`, default 5 minutes) rather than from the executor. For short jobs this makes the timestamps missing or misleading: - **`startedAt` is often null for finished jobs.** It is only set when a poll sees the job in `STARTED` (`JobManager#toUpdatedStatusJobEntity`). A job that starts and finishes between two polls goes from `QUEUED` straight to a terminal state and never gets a `startedAt`. Most short jobs end up like this. - **`finishedAt` can be up to one poll interval late.** It records when the poll first saw the terminal state, not when the job actually ended. - **Durations reflect the polling schedule, not the job.** Even when both timestamps are set, they fall on poll boundaries. Users can't tell how long a job waited in the queue versus how long it ran. - **A null `startedAt` is ambiguous.** The `JobHandle#startedAt()` Javadoc says null means "the job has not started execution yet", but finished jobs also return null here. Shortening the poll interval reduces the error but doesn't remove it, and the minimum recommended interval is 1 minute. ### How should we improve? The root cause is that `JobExecutor#getJobStatus` only returns a status. The executor usually knows the real times (for example, `LocalJobExecutor` knows exactly when it launches and when it reaps the process), but they never reach `JobManager`. Proposal: 1. Extend the `JobExecutor` SPI so an executor can report a job's status together with its actual start and finish timestamps. Add it as a `default` method built on `getJobStatus` with no timestamps, so existing executors keep working. 2. Have `LocalJobExecutor` record the real process start and exit times, and let other executors fill these in from their own sources (e.g. application or pod start/end times). 3. Have `JobManager` prefer executor-reported timestamps, and fall back to the poll time only when the executor can't provide one. 4. Define and document what a missing `startedAt` means on a finished job (e.g. "start not observed"), so consumers don't read it as zero or as "never ran". Since this changes a public extension point (`JobExecutor`), it's targeted at the 2.0 release. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
