Kontinuation commented on PR #20393: URL: https://github.com/apache/datafusion/pull/20393#issuecomment-4003234913
I had a hard time figuring out how to construct an execution plan producing RecordBatches with zero columns while matching the total number of records in the foreign table. This can be done but in a cumbersome way. I don't think we can simply return an empty batch executor for such cases, since we still need to produce batches with no columns. Another approach without breaking the ABI is to expand `None` projection into `Some([0, 1, .. n - 1])`. The relationship between local projection and FFI projection is as follows: ``` Semantics Local FFI Local (scan_fn_wrapper) Project all None --> [0,1,..n-1] --> Some([0,1,..n-1]) Project none Some([]) --> [] --> Some([]) Project Some([a,b,..]) --> [a,b,..] --> Some([a,b,..]) ``` The "select all" projection represented by `None` will become `Some([0,1,...n-1])` after the roundtrip, but the semantics should be the same. I think we should either stick to the current approach to make the FFI APIs look the same as local APIs for better consistency and less mental burden, or expand `None` to `Some([0, 1, .. n-1])` when transferring the projection from local side to FFI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
