vinooganesh commented on PR #119: URL: https://github.com/apache/parquet-testing/pull/119#issuecomment-5186909258
I just ran this against the Java implementation from apache/parquet-java#3397. It reads the file fine, no changes needed on our end. I decoded all six ALP columns and compared them against the matching PLAIN columns as raw bits (`doubleToRawLongBits` / `floatToRawIntBits`, so NaN and -0.0 compare properly). 53,952 values, everything matched. The special values at 1024-2047 all survive: both NaNs, +/-Inf, -0.0 at 2002, and the subnormal at 2003. The constant vector in row group 2 and the 8 nulls in the trailing partial vector are fine too. It's worth calling out that the `_32` columns caught something my own fixtures didn't. Java accepts log_vector_size 3-15, but everything I'd generated was 1024 or 4096, so vector size 32 had never actually been decoded from another writer's file until now. It works, but that path was untested. There are a couple of things I noticed while poking at it, and all of them are fixable with a few extra columns rather than more data: - Every data page in the file is V1. Nothing exercises DataPageV2. - All the NaNs are canonical (0x7FF8000000000000 / 0x7FC00000), so nothing checks that a reader preserves NaN payloads. - The widest FOR bit width in the file is 18. @emkornfield asked us to cover values needing 63/64 bits after FOR on the Java PR, and that case isn't here. I have fixtures for all three if they'd be helpful, but they'd be more columns on this file rather than separate ones. And I'm happy to contribute the Java-side conformance test to parquet-java once this lands on master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
