dhruvxvaishnav commented on code in PR #10725:
URL: https://github.com/apache/arrow-rs/pull/10725#discussion_r3810148977
##########
parquet/src/arrow/decoder/dictionary_index.rs:
##########
@@ -43,7 +43,14 @@ impl DictIndexDecoder {
/// Create a new [`DictIndexDecoder`] with the provided data page, the
number of levels
/// associated with this data page, and the number of non-null values (if
known)
pub fn new(data: Bytes, num_levels: usize, num_values: Option<usize>) ->
Result<Self> {
- let bit_width = data[0];
+ let bit_width = *data
+ .first()
+ .ok_or_else(|| general_err!("dictionary index page is empty"))?;
+ if bit_width > 32 {
Review Comment:
Added MAX_RLE_DICTIONARY_BIT_WIDTH = 32 in parquet::encodings::rle and
updated DictIndexDecoder, ByteArrayDictionaryReader (DictionaryDecoder), and
DictDecoder to use it.
##########
parquet/src/encodings/rle.rs:
##########
@@ -361,6 +361,7 @@ pub struct RleDecoder {
impl RleDecoder {
pub fn new(bit_width: u8) -> Self {
+ debug_assert!(bit_width <= 64, "Bit width must be <= 64");
Review Comment:
`RleDecoder` is a generic Parquet RLE / bit-packing hybrid decoder that
supports unpacking values up to 64 bits wide (using a `u64` internal
accumulator). However, Parquet dictionary indices are 32-bit signed integers
(`i32`), so dictionary index bit widths are bounded by
`MAX_RLE_DICTIONARY_BIT_WIDTH` (32). I have added doc comments on
`RleDecoder::new` and `MAX_RLE_DICTIONARY_BIT_WIDTH` explaining this
distinction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
