fhan688 commented on PR #20392: URL: https://github.com/apache/hudi/pull/20392#issuecomment-6309452209
> #19169 (closes #18791) already fixes this `StartsWith` translation with the same `max >= prefix AND (min < prefix OR startsWith(min, prefix))` bound, so this PR duplicates it, as did my #20226, which I have closed in its favor. Could we consolidate there and land your data-skipping on/off query test in `TestDataSkippingQuery` as a follow-up once #19169 merges? Thanks for pointing this out. I compared the two implementations, and I agree that #19169 should remain the canonical production fix. For non-null binary strings, the bounds are equivalent: this PR uses `min <= prefix`, while #19169 uses `min < prefix`; the equality case is already covered by `startsWith(min, prefix)`. #19169 also preserves support for allowed order-preserving transformations. I’ll close this PR in favor of #19169 and submit the COW/MOR data-skipping on/off query regression test as a focused follow-up after #19169 merges. I also noticed two defensive differences worth discussing separately on #19169: handling missing min/max statistics conservatively, and falling back for Spark 4 non-binary collations. I’ll raise those there rather than maintain a competing implementation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
