mbutrovich commented on PR #12289: URL: https://github.com/apache/arrow-rs/pull/12289#issuecomment-6951808658
I ran the new `row_encode` property from mbutrovich/arrow-pbt on this PR at 24c1ede738e7f52f3f90acaa2924d659a5373557 (arrow-pbt at 14985c04ccb9d09c752b3243b433c6ad746923ca, mbutrovich/arrow-pbt#3). ```sh ARROW_PBT_REASONS=1 cargo run -q -- run row_encode 1-3 1000 ``` Seeds 1 to 3 at 1,000 cases each passed with no discards. The same run on `main`'s `arrow-row` at d65329d3e7 passed too. The property checks the bytes of each row, which the order and round-trip checks in `row_format` do not always catch: - Row `i` of one conversion equals the row from converting that row alone, and rows of a slice at a nonzero offset and of `append` after existing rows equal the rows of one conversion. - For keys of only primitive and boolean columns, the bytes equal the same columns' bytes when a `Utf8` column follows them, which takes the general encoder. - Each row equals the bytes the format's documentation gives, for booleans, integers, decimals, dates, times, timestamps, durations, strings and binary of every layout. The model is in `model_rows` in `src/order.rs`. - `LargeUtf8`, `Utf8View`, `LargeBinary` and `BinaryView` encode the same bytes as the same values in `Utf8` or `Binary`. Cases have 400 to 6,000 rows, around the 16 KiB block of `STRIDED_BLOCK_BYTES`, and half of them have only fixed-width columns. Two thirds of the string and binary columns have lengths at the 8-byte and 32-byte block boundaries and up to 101 bytes, as slices of one buffer. To check that the cases reach the changed code, I broke this PR's code five ways. `row_encode` failed on every seed (1 to 3, 300 cases) for each: | Change | `row_encode` | `row_format` alone | |---|---|---| | `encode_strided` writes `1` instead of `0` after a null's sentinel | fails | passes | | The strided write starts at offset 0 instead of `rows.offsets[write_offset]` | fails | passes | | The view loop drops the last byte of a 12-byte inline value | fails | fails | | `copy_short` skips the tail copy for 8 to 15 bytes | fails | fails | | `copy_short` writes a `1` after a 1-byte last block | fails | fails | A sixth change, which shrinks the block by one row when there are more than two blocks, passed. Blocks of any size give the same bytes, so it is not a bug. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
