pmoust opened a new issue, #11418: URL: https://github.com/apache/arrow-rs/issues/11418
### What happens now Decoding a Parquet footer reserves a `Vec` sized from the declared row-group count before reading any row group: ```rust // parquet/src/file/metadata/thrift/mod.rs, read on main and in 60.0.0 let mut rg_vec = Vec::with_capacity(list_ident.size as usize); ``` `list_ident.size` comes straight off the wire. `RowGroupMetaData` is a large struct, so a footer that declares a huge row-group count asks for `count * size_of::<RowGroupMetaData>()` up front, before a single row group is read. A footer of a few bytes can therefore drive a multi-gigabyte allocation, which on a host under memory pressure aborts the process. ### Why `read_thrift_vec` does not cover it PR #10979 bounded `read_thrift_vec` by the remaining input, so a declared list size larger than the bytes left is rejected before the reserve. The row-group list is decoded inline in `parquet_metadata_from_bytes` and does not go through `read_thrift_vec`, so that bound does not reach it. The other lists in the footer (schema elements, column chunks, sorting columns, key/values, column orders) do route through `read_thrift_vec` and are bounded. The row-group count is the one that is not. ### Impact Malformed input drives an allocation far larger than the input. Per `SECURITY.md` this is a bug rather than a vulnerability: there is no RCE and no information disclosure, only a denial of service through memory exhaustion. That is why this is a public issue and not a mail to [email protected]. ### Reproducing Build a footer whose `FileMetaData` field 4 (`row_groups`) carries a list header declaring a size far larger than the remaining footer bytes, then decode it. The reserve runs before the decoder reads the first row group, so the allocation request is driven by the header alone. ### Suggested fix Bound `list_ident.size` by the remaining input before the reserve, the same way `read_thrift_vec` now does, or decode the row-group list through that helper so it inherits the bound. ### Context Found while hardening the Parquet external-table reader in [Ravel](https://github.com/NOFireAI/ravel). Ravel pins parquet 58.x/59.x and bounds the footer decode in its own code, so this is not a problem for Ravel in production. Reporting it upstream because the row-group site is still unbounded on the latest release. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
