morningman commented on issue #66645: URL: https://github.com/apache/doris/issues/66645#issuecomment-5254513013
Thanks for the detailed use case. Let me make sure I understand the requirement correctly. One possible approach would be to implement this as a variant of federated query. We could potentially build it on top of the [[ADBC Catalog](https://doris.apache.org/docs/dev/lakehouse/catalogs/adbc-catalog)](https://doris.apache.org/docs/dev/lakehouse/catalogs/adbc-catalog), which is planned to replace the current Doris Catalog for this type of cross-Doris query. The basic idea would be: * The optimizer pushes as much computation as possible, especially aggregation and joins, to the remote Doris cluster, so that only the processed/aggregated results are returned to the local cluster. * For this use case, pushdown would need to be mandatory rather than just an optimization. If the optimizer cannot generate a remote plan that satisfies the data residency requirement—for example, if row-level/raw data would need to be returned from the remote cluster—the query should fail instead of falling back to fetching the raw data. At the moment, the optimizer does not yet support aggregation pushdown or join pushdown for this federated query path, but these are capabilities we can consider adding. Would this model satisfy your requirements? In particular, is it acceptable for a query to fail when Doris cannot generate a plan that keeps all raw data processing inside the remote region? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
