jhorstmann commented on issue #11985:
URL: https://github.com/apache/arrow-rs/issues/11985#issuecomment-6500117055
Funny, I just noticed the same thing while analyzing how well ALP would work
for our datasets. Some vectors had a bitwidth of 63.
The encoder or writer should be able to write values with bitwidth 0 (taking
up no space), then write all values as exceptions:
> There's no such decision in encode_vector. It unconditionally computes the
FOR range over whatever non-exceptions exist and writes the packed section at
that width. So yes — if two non-exceptions have encoded integers far apart
(e.g. one near
i64::MIN and one near i64::MAX), the encoder writes 1024 × 63-bit packed
values plus 1022 × (8+2) byte exceptions, when it would be cheaper to just
write 1024 × (8+2) byte exceptions and BW=0.
> The fix would be in encode_vector: after computing bit_width and before
writing anything, check whether the packed section cost exceeds the cost of
demoting the remaining non-exceptions to exceptions:
```
packed_cost = n_values * bit_width
demotion_cost = n_non_exceptions * exception_bits (extra exceptions)
if packed_cost > demotion_cost → set bit_width = 0, add all values to
exceptions
```
> This is exactly the same tradeoff estimate_size_bits evaluates during
parameter selection, but it's never applied at the write step. The
parquet-format spec doesn't prohibit encoding non-roundtripping-able values as
exceptions, so this would be a
valid optimization — essentially a per-vector "is the packed section worth
it?" check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
