thinkharderdev commented on issue #1848: URL: https://github.com/apache/arrow-datafusion/issues/1848#issuecomment-1042855277
Will do. I think there are a couple of different ways we can approach this: 1. Have the client specify a namespace in the request and use a `ExecutionContext`-per-namespace on the scheduler. We could then dynamically create new contexts whenever a new namespace comes in. 2. Have the scheduler dynamically set target partitions based on executor statistics (e.g. number of available task slots). This would I think require a way to set the target partitions explicitly when creating a sql plan. So maybe add a new method to `ExecutionContext` like `pub async fn sql(&mut self, sql: &str, target_partitions: usize) -> Result<Arc<dyn DataFrame>>` Or both. 1 may be necessary anyway to support multi-tenancy but we may still, within a single namespace, want to allow specifying shuffle settings on a per-query basis. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
