milenkovicm commented on PR #2405: URL: https://github.com/apache/datafusion-ballista/pull/2405#issuecomment-5500950370
> One question, and I do not think it blocks merging. After `finish_job` removes the handle, a straggler event for that job would hit `open_for` and recreate an empty `<job_id>.eventlog.running` that nothing will ever rename. The served `.eventlog` is untouched, which is better than the old behaviour of appending records after `JobEnd` into the finished file, so the worst case is a stray file that reads as "this job crashed" for a job that actually succeeded. Do you think late appends after a terminal event are possible in practice, say a task status update arriving after `JobCancel` or after a job has already failed? If they are, it might be worth having `open_for` refuse to reopen a job that has already finished, so the crash signal stays trustworthy. > Will have to investigate. Can't come up with corner case for suggested "check before open" but let me sleep on it. I just wanted to land protocol change before we release. Just a heads up, will change history server as a follow up - Read directory on startup once - Use inotify to listens for new event logs and populate state Will try to make both functions pluggable, -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
