szehon-ho commented on PR #58145: URL: https://github.com/apache/spark/pull/58145#issuecomment-6643902551
Thanks, I had a chat with @aokolnychyi and the concern it is getting deeper in a rabbit hole if we try to fix this undercounting etc.. as we discuss on point (2) of https://github.com/apache/spark/pull/58145#issuecomment-6542082777 . Actually originally the idea for this is Iceberg/ Delta derived partition filters for partition transform and generated columns. In these cases, the DSV2 data source connector does _not_ have NDV's for the data columns and so the original filter does _not_ contribute to stats adjusmtent , hence the use is to put 'partition filters' in the Filter node that do allow it to influence stats accurately. So my vote is the simple solution, and not complexity of handling over-counting for now, if there's no easy correct way. There's actually not that much use case, I feel. I should have explained the motivation in the pr earlier, sorry! So with simplified solution, removing these from Filter node is the next step , thanks @yyanyy . its also a bit tricky :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
