voonhous commented on issue #20296: URL: https://github.com/apache/hudi/issues/20296#issuecomment-6367089200
I think this is the same class of problem as: https://github.com/apache/hudi/issues/20227 https://github.com/apache/hudi/issues/20229 https://github.com/apache/hudi/issues/19646 So this is the fourth occurrence on Spark and another on Flink: https://github.com/apache/hudi/issues/20289 Cases i can think of at the top of my head that we should also check are involving transforms in `OrderPreservingTransformation`: - `date_format`, `from_unixtime` with a format: formatted strings don't sort like dates (shown above). - `Multiply` / `Divide`: dividing or multiplying by a negative constant reverses the order. C * -1 = -2 against a file with min 1, max 3 gives -1 <= -2, which is false, so the file is pruned. - `BitwiseOr`: doesn't preserve order. 1 | 2 = 3 but 2 | 2 = 2. - `Add`, `ShiftLeft`: overflow wraps around in non-ANSI mode. - `to_unix_timestamp(stringCol, fmt)`: the string order of stringCol isn't the order of the parsed times. We need to add a test that will exhaustively cover such bugs: 1. Add a property test at `TestDataSkippingUtils` level. No hudi tables so it runs fast 2. Generate random values (use fixed seed) for each file (Must include NULLS) 3. Compute each file's min, max and NULL count from those values. In some files, randomnly null out the stats for a column to simulate missing states 4. For each predicate shape (=, <, IN, IS NULL, AND, OR, NOT, ...) and each whitelisted transform with random literals, check if every file where the orignal predicate is true for some raw row must survive the translated filter. With this the raw data is the reference answer, so the test doesn't depend on anyone's belief about a transform, so there's no ambiguity. With this taest, it would have caught all four bugs above, and it should fail for some of the ones that we are not catching today above. LMK if you guys want me to pick it up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
