zeroshade commented on code in PR #943:
URL: https://github.com/apache/arrow-go/pull/943#discussion_r3589006160
##########
parquet/internal/utils/bit_reader.go:
##########
@@ -287,9 +287,12 @@ func (b *BitReader) GetBatchIndex(bits uint, out
[]IndexType) (i int, err error)
b.reader.Seek(b.byteoffset, io.SeekStart)
// grab as many 32 byte chunks as possible in one shot
if i < length { // IndexType should be a 32 bit value so we can do
quick unpacking right into the output
- numUnpacked := unpack32(b.reader,
(*(*[]uint32)(unsafe.Pointer(&out)))[i:], int(bits))
+ numUnpacked, unpackErr := unpack32(b.reader,
(*(*[]uint32)(unsafe.Pointer(&out)))[i:], int(bits))
i += numUnpacked
b.byteoffset += int64(numUnpacked * int(bits) / 8)
+ if unpackErr != nil {
Review Comment:
Good — `unpack32` now surfaces truncation via `unpackErr`, and
`GetBatchIndex`/`GetBatch` return it. But the RLE/dict callers drop it, so the
hardening never reaches the column decoders:
- `parquet/internal/utils/typed_rle_dict.go:166` — `n, _ :=
r.r.GetBatchIndex(...)` then `if n != litbatch { return read, nil }`, so a
truncated literal batch returns a short read with a **nil** error.
- Same pattern at `parquet/internal/utils/rle.go:223`,
`parquet/internal/encoding/levels.go:271` (Encoding_BIT_PACKED levels), and
`typed_rle_dict.go:94` (spaced literals).
Concrete case: an RLE literal run claiming 64 one-bit indices but only 7
payload bytes now yields `(32, io.ErrUnexpectedEOF)` from `GetBatchIndex`, but
`GetBatchWithDict` discards it and returns `(read, nil)`. Please propagate the
error through these callers so truncated input is actually rejected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
