mbutrovich commented on PR #12289:
URL: https://github.com/apache/arrow-rs/pull/12289#issuecomment-6951808658

   I ran the new `row_encode` property from mbutrovich/arrow-pbt on this PR at 
24c1ede738e7f52f3f90acaa2924d659a5373557 (arrow-pbt at 
14985c04ccb9d09c752b3243b433c6ad746923ca, mbutrovich/arrow-pbt#3).
   
   ```sh
   ARROW_PBT_REASONS=1 cargo run -q -- run row_encode 1-3 1000
   ```
   
   Seeds 1 to 3 at 1,000 cases each passed with no discards. The same run on 
`main`'s `arrow-row` at d65329d3e7 passed too.
   
   The property checks the bytes of each row, which the order and round-trip 
checks in `row_format` do not always catch:
   
   - Row `i` of one conversion equals the row from converting that row alone, 
and rows of a slice at a nonzero offset and of `append` after existing rows 
equal the rows of one conversion.
   - For keys of only primitive and boolean columns, the bytes equal the same 
columns' bytes when a `Utf8` column follows them, which takes the general 
encoder.
   - Each row equals the bytes the format's documentation gives, for booleans, 
integers, decimals, dates, times, timestamps, durations, strings and binary of 
every layout. The model is in `model_rows` in `src/order.rs`.
   - `LargeUtf8`, `Utf8View`, `LargeBinary` and `BinaryView` encode the same 
bytes as the same values in `Utf8` or `Binary`.
   
   Cases have 400 to 6,000 rows, around the 16 KiB block of 
`STRIDED_BLOCK_BYTES`, and half of them have only fixed-width columns. Two 
thirds of the string and binary columns have lengths at the 8-byte and 32-byte 
block boundaries and up to 101 bytes, as slices of one buffer.
   
   To check that the cases reach the changed code, I broke this PR's code five 
ways. `row_encode` failed on every seed (1 to 3, 300 cases) for each:
   
   | Change | `row_encode` | `row_format` alone |
   |---|---|---|
   | `encode_strided` writes `1` instead of `0` after a null's sentinel | fails 
| passes |
   | The strided write starts at offset 0 instead of 
`rows.offsets[write_offset]` | fails | passes |
   | The view loop drops the last byte of a 12-byte inline value | fails | 
fails |
   | `copy_short` skips the tail copy for 8 to 15 bytes | fails | fails |
   | `copy_short` writes a `1` after a 1-byte last block | fails | fails |
   
   A sixth change, which shrinks the block by one row when there are more than 
two blocks, passed. Blocks of any size give the same bytes, so it is not a bug.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to