andygrove opened a new issue, #2717:
URL: https://github.com/apache/datafusion-ballista/issues/2717
## Is your feature request related to a problem or challenge?
If a client process is killed outright, instead of dropping its result
stream, its job keeps running on the scheduler until it finishes or is
cancelled through `PATCH /api/job/{job_id}`. Cancel-on-drop only works while
the client process is alive, and the scheduler doesn't notice a client that has
gone away.
Blocking jobs behave the same way, but it matters more for pipelined jobs
(#2652, #2657). A pipelined job holds a reservation of vcores for its whole
life, and its final stage waits for a reader that will never come (until
`ballista.pipelined.registration_timeout_seconds`, or longer while a stalled
stream is open). Meanwhile its reservation can block every other pipelined job
from being placed.
This was found while benchmarking #2657
([comment](https://github.com/apache/datafusion-ballista/pull/2657#issuecomment-6443057416)).
## Describe the solution you'd like
The scheduler should cancel a job whose client has gone away. Options:
- a lease or heartbeat from the client while it waits for or reads results,
or
- cancelling a job once all of its result fetch connections have closed
before the results were fully read.
## Describe alternatives you've considered
Relying on the operator to cancel orphaned jobs through the REST API, which
is the only option today.
## Additional context
Raised in review of #2657.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
