edrevo opened a new pull request #586: URL: https://github.com/apache/arrow-datafusion/pull/586
# Which issue does this PR close? Closes #585. # What changes are included in this PR? This PR provides very primitive support for auto-scaling: it works like an on-off switch. If there are no active tasks, the number of executors drops to 0. If there are active tasks, the number of executors will increase to the maximum number of allowed executors configured in Keda. Further work could refine this so that the number of executors is proportional to the amount of outstanding tasks or something like that, but it is not trivial since we may end up killing executors that hold valuable shuffle data even if the aren't processing anything. Changes in this PR: - The scheduler now implements the necessary gRPC interface for Keda. - k8s user guide has been updated I have tested this manually and seen it scale to 0 and back up when there is a query. # Are there any user-facing changes? No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
