Re: [I] Physical plan pushdown for volatile predicates [datafusion]
theirix commented on issue #16545: URL: https://github.com/apache/datafusion/issues/16545#issuecomment-3137490258 Thank you for the improvement, @adriangb ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] Physical plan pushdown for volatile predicates [datafusion]
alamb closed issue #16545: Physical plan pushdown for volatile predicates URL: https://github.com/apache/datafusion/issues/16545 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] Physical plan pushdown for volatile predicates [datafusion]
adriangb commented on issue #16545: URL: https://github.com/apache/datafusion/issues/16545#issuecomment-3104993647 It seems reasonable to me to blacklist some filters. But given that users can create arbitrary trait implementations for PhysicalExpr we obviously can't black list everything. Unless there a volatility method on the trait? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] Physical plan pushdown for volatile predicates [datafusion]
theirix commented on issue #16545: URL: https://github.com/apache/datafusion/issues/16545#issuecomment-3104952628 > > I expect the physical plan optimiser doesn't perform pushdown of volatile predicates. > > I am not sure -- does this result in wrong results? We don't observe incorrect results (as was the case with the logical optimiser bug when evaluating a predicate twice). It can produce skewed or non-reproducible results if a volatile predicate evaluates to different values for different row groups (I can create a test to reproduce it). It could probably affect the table sampling implementation #16325 , which relies on inserting a volatile filter. > It does make sense in general to have the physical pushdown optimizer follow the same behavior as the logical optimizer Yes, it is a reasonable assumption about an optimiser. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] Physical plan pushdown for volatile predicates [datafusion]
alamb commented on issue #16545: URL: https://github.com/apache/datafusion/issues/16545#issuecomment-3097890993 > I expect the physical plan optimiser doesn't perform pushdown of volatile predicates. I am not sure -- does this result in wrong results? It does make sense in general to have the physical pushdown optimizer follow the same behavior as the logical optimizer -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] Physical plan pushdown for volatile predicates [datafusion]
theirix commented on issue #16545: URL: https://github.com/apache/datafusion/issues/16545#issuecomment-3092538385 @alamb @adriangb, given your most recent work on pushdowns, does it appear to be a bug? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] Physical plan pushdown for volatile predicates [datafusion]
theirix commented on issue #16545: URL: https://github.com/apache/datafusion/issues/16545#issuecomment-3029427291 @findepi could you please tell if this behaviour makes sense? If not, I could try fixing the physical plan as in #13268, where you have a review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
