jerryshao opened a new issue, #13146: URL: https://github.com/apache/gravitino/issues/13146
### Version main branch ### Describe what's wrong When multiple Gravitino servers share the same metadata store and use the local job executor, jobs that exit successfully are intermittently recorded as `FAILED`. `LocalJobExecutor` keeps job status only in the memory of the server that submitted the job. But every server runs `JobManager.pullAndUpdateJobStatus`, which lists all active jobs in the shared store and asks its own executor for their status. When server B polls a job submitted by server A, B's executor throws `NoSuchJobException`, and B marks the job as `FAILED`. The recorded status depends on which server polls first. Related problems in the same setup: 1. Cancelling a job on a server that didn't submit it fails with a 500 error, because that server's executor can't find the job. 2. `LocalJobExecutor` runs only one job at a time regardless of `gravitino.jobExecutor.local.maxRunningJobs`. Its thread pool has a core size of 0 and an unbounded queue, so it never grows beyond one thread. ### Error message and/or stacktrace The server that ran the job logs that it completed successfully. The incorrect `FAILED` status is written by another server, which logs: ``` Job job-xxx with execution id local-job-xxx under metalake xxx is not found in the external job executor, marking it as FAILED. ``` ### How to reproduce 1. Deploy two Gravitino servers sharing the same relational metadata store, both using the local job executor. 2. Register a shell job template, e.g. `/bin/bash -c "echo ok"`, and run it several times through the load balancer. 3. Some successful jobs are recorded as `FAILED` after the status pull, while the server that ran them logs success. 4. Cancel a running job through a server that didn't submit it: the request fails with a 500 error. ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
