HippoBaro commented on PR #11262:
URL: https://github.com/apache/arrow-rs/pull/11262#issuecomment-6005241551
> > ```
> > called `Result::unwrap()` on an `Err` value: General("Invalid
FIXED_LEN_BYTE_ARRAY PLAIN data page payload > length: expected 20000 bytes
(10000 values of 2 bytes), got 9978")
> > note: run with `RUST_BACKTRACE=1` environment variable to display a
backtrace
> > error: bench failed, to rerun pass `-p parquet --bench arrow_reader`
> > ```
>
> This is the first time I'm seeing this. I'll investigate...
I have found and pushed a fix in e3e6ece676d4e39fb329b9794513358ec45eb39c.
Apologies for letting that get pushed!
> Not a huge difference, but it seems
DefinitionLevelDecoderImpl::skip_def_levels can be improved some. It resizes
its temp buffer, and then passes that to self.read_def_levels, which resizes it
again. Initializing it's capacity at SKIP_BUFFER_SIZE, and then inlining the
two lines of read_def_levels that we need, I was able to speed up the
problematic round trip benches by around 10%.
@etseidl Thanks for spotting! I added
645d14750c771b30d6840482cec45e6f140267f0 which implements the suggested fix,
but happy to drop it if you'd rather that gets merged separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
