alamb commented on issue #24438: URL: https://github.com/apache/datafusion/issues/24438#issuecomment-5409361631
Copy some comments from https://github.com/apache/datafusion/pull/24497 I see now that the idea here is that somehow the external system knows the data is not sorted but is non overlapping I think this is going to be really hard to manage / ensure through the plan -- specifically I think the challenge will be to ensure that every operator properly reports if the property is preserved or not I am not sure this is something we want to complicate datafusion with at this time as it 1. Is complicated (see above) 2. I am not sure how many other systems have data organized in this way / could take advantage of it We are already working on Range Partitioning (which is another form of distribution aware planning) which is also complicated, but I think has a clearly demonstrated wide demand. Let's see if we can find others who need this before we try and code it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
