mingmwang commented on PR #188:
URL: https://github.com/apache/arrow-ballista/pull/188#issuecomment-1257648819
@andygrove @yahoNanJing @Ted-Jiang @yahoNanJing
BTW, in this PR, the job data in the state store will also be deleted after
300s.
I think we need a following PR to move the completed(Success or Failed) job
data from state store to ObjectStore
for long time storing purpose, and Scheduler UI can read from the
ObjectStore.
Please share your thoughts.
````
const CLEANUP_FINISHED_JOB_DELAY_SECS: u64 = 300;
async fn clean_up_job_data(
state: Arc<dyn StateBackendClient>,
active_job_cache: ExecutionGraphCache,
failed: bool,
job_id: String,
executor_manager: Option<ExecutorManager>,
) -> Result<()> {
let mut active_graph_cache = active_job_cache.write().await;
active_graph_cache.remove(&job_id);
let keyspace = if failed {
Keyspace::FailedJobs
} else {
Keyspace::CompletedJobs
};
let lock = state.lock(keyspace.clone(), "").await?;
with_lock(lock, state.delete(keyspace, &job_id)).await?;
executor_manager
.map(|em| async { Self::clean_up_executors_data(job_id.clone(),
em).await });
Ok(())
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
