villebro commented on PR #2212: URL: https://github.com/apache/datafusion-ballista/pull/2212#issuecomment-5171553368
> > Is there a strong reason to distinguish between a cluster that has not registered its first executor and one that has lost all registered executors? > > This is a good callout. From a running job's view they are equivalent. I would keep them separate for now because "lost all executors" has an ExecutorLost event to trigger on, while "no first executor yet" has no event and would need a different mechanism (like a startup timeout). This is also already mitigated on k8s by the existing /readyz probe, which keeps an executor-less scheduler out of the k8s service and not accept jobs from clients Thanks for the comment, and makes total sense. The reason I'm raising this is I think it'd be great to aim for simplicity wherever possible. Having fewer knobs and a simpler state machine to reason about can be very helpful, both for admins and end users. But this higher level design can probably be revisited once all these critical issues have first been cleaned up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
