hanke580 opened a new issue, #54533:
URL: https://github.com/apache/arrow/issues/54533
### Describe the bug, including details regarding any error messages,
version, and platform.
When the first write batch of a list/map column chunk contains a single
record, the Parquet writer can begin the column chunk with a data page that
holds no values (`num_values = 0`, `num_rows = 0`). This happens when
`data_page_size` is small. The Arrow reader takes that empty page as the end of
the column chunk, so every row of that row group is silently missing on read,
while the file metadata still counts them. DuckDB and Polars skip the empty
page and read the rows.
This happens with a one-row file, and with any file whose last row group (or
any row group, e.g. `row_group_size=1`) holds exactly one row.
**Reproducer**
```python
import pyarrow as pa
import pyarrow.parquet as pq
list_bool = pa.list_(pa.bool_())
# 1) one-row table
t = pa.table({"c0": pa.array([[True]], list_bool)})
pq.write_table(t, "one_row.parquet", data_page_version="2.0",
data_page_size=64)
print(pq.ParquetFile("one_row.parquet").metadata.num_rows) # 1
print(pq.read_table("one_row.parquet").num_rows) # expected 1
# 2) the last row group holds a single row
t = pa.table({"c0": pa.array([[True], [False], [True]], list_bool)})
pq.write_table(t, "last_rg.parquet", data_page_version="2.0",
data_page_size=64,
row_group_size=2)
print(pq.read_table("last_rg.parquet")["c0"].to_pylist()) # expected
[[True], [False], [True]]
# 3) default encoding (dictionary), any dictionary-encodable leaf type, V1
pages, very small page size
t = pa.table({"c0": pa.array([[1], [2], [3]], pa.list_(pa.int32()))})
pq.write_table(t, "dict.parquet", data_page_size=4, row_group_size=2)
print(pq.read_table("dict.parquet")["c0"].to_pylist()) # expected
[[1], [2], [3]]
# 4) with a second column the read fails instead
t = pa.table({"id": [1], "c0": pa.array([[True]], list_bool)})
pq.write_table(t, "two_cols.parquet", data_page_version="2.0",
data_page_size=64)
pq.read_table("two_cols.parquet")
```
Output on main:
```
1
0
[[True], [False]]
[[1], [2]]
Traceback (most recent call last):
...
pyarrow.lib.ArrowInvalid: Column 1 named c0 expected length 1 but got length 0
```
DuckDB 1.5.x and Polars 2.0 read `one_row.parquet` as `[[True]]` and
`last_rg.parquet` as all three rows. With `write_page_index=True`, ClickHouse
refuses the file because the offset index lists two pages with the same
first_row_index 0 (the empty page at offset 4, size 26, then the real page at
offset 30, size 32): `Invalid offset index: PageLocation(offset=30,
compressed_page_size=32, first_row_index=0), prev offset: 30, prev row: 0`.
I decoded the page headers of `one_row.parquet`. The column chunk holds two
`DATA_PAGE_V2` pages: first `num_values=0, num_nulls=0, num_rows=0,
encoding=RLE` with a 4-byte body (only the RLE length prefix), then the real
page with `num_values=1, num_rows=1`.
**Scope** (main, a column chunk whose first write batch holds one record):
| leaf / encoding | `data_page_size` values that lose the row |
|---|---|
| BOOLEAN leaf with V2 data pages (RLE encoding) | 1 to 69 |
| any dictionary-encodable leaf with dictionary encoding (the default), V1
or V2 pages | 1 to 6 |
| non-boolean leaf with `use_dictionary=False` | none |
| non-repeated columns | none |
The writer can also emit an empty page in the middle of a chunk. For
example, 1025 one-element `list<bool>` rows with `data_page_size=64` produce
pages of 1024, 0 and 1 values. Arrow happens to read that file correctly.
**Versions:** pyarrow 27.0.0.dev13 (nightly built from main at 9f03413),
Python 3.12, Linux x86_64 (Ubuntu 22.04). With the released wheels, cases 1, 2
and 4 reproduce from 22.0.0 on, and case 3 from 23.0.0 on (on 22.0.0,
dictionary encoding is affected only with V2 pages); 22.0.0 through 26.0.0 were
tested. pyarrow 21.0.0 writes a single page and reads the row back correctly.
**Where it comes from**
- Writer: `DoInBatchesRepeated` calls `action(offset,
check_page_limit_end_offset - offset, /*check_page_limit=*/true)` with zero
levels when the last record of the batch starts at `offset`, which is always
the case for a batch holding a single record
([column_writer.cc#L1219-L1224](https://github.com/apache/arrow/blob/9f0341314ca96eb58081d4132c77b460d880b2b0/cpp/src/parquet/column_writer.cc#L1219-L1224)).
`CommitWriteAndCheckPageLimit` then calls `AddDataPage()` when
`EstimatedDataEncodedSize() >= data_pagesize()`, without checking that anything
is buffered
([column_writer.cc#L1835-L1846](https://github.com/apache/arrow/blob/9f0341314ca96eb58081d4132c77b460d880b2b0/cpp/src/parquet/column_writer.cc#L1835-L1846)).
An empty encoder's estimate is not 0:
- `RleBooleanEncoder` returns 4 + `RlePreserveBufferSize(0, 1)` = 4 + 65 =
69 bytes
([encoder.cc#L1754-L1756](https://github.com/apache/arrow/blob/9f0341314ca96eb58081d4132c77b460d880b2b0/cpp/src/parquet/encoder.cc#L1754-L1756),
[encoder.cc#L444-L451](https://github.com/apache/arrow/blob/9f0341314ca96eb58081d4132c77b460d880b2b0/cpp/src/parquet/encoder.cc#L444-L451)).
- The dictionary encoder returns 1 + 5 = 6 bytes
([encoder.cc#L499-L503](https://github.com/apache/arrow/blob/9f0341314ca96eb58081d4132c77b460d880b2b0/cpp/src/parquet/encoder.cc#L499-L503)).
These numbers match the thresholds in the table above exactly.
- The zero-length call became possible with GH-47027 (#47032, first released
in 22.0.0), which changed `if (offset < last_record_begin_offset)` to `if
(offset <= last_record_begin_offset)`. The later rewrite in #47090 kept this
behaviour.
- Reader: `HasNextInternal()` returns false when the next page has
`num_buffered_values_ == 0`
([column_reader.cc#L776-L785](https://github.com/apache/arrow/blob/9f0341314ca96eb58081d4132c77b460d880b2b0/cpp/src/parquet/column_reader.cc#L776-L785)).
So `ReadRecords` returns 0 on a leading empty page, and
`LeafReader::LoadBatch` moves on to the next row group
([arrow/reader.cc#L510-L514](https://github.com/apache/arrow/blob/9f0341314ca96eb58081d4132c77b460d880b2b0/cpp/src/parquet/arrow/reader.cc#L510-L514)).
Possible fixes:
- Writer: do not call `AddDataPage()` when nothing is buffered
(`num_buffered_values_ == 0`), or skip the zero-length `action` call.
- Reader: skip a data page with zero values instead of treating it as the
end of the column chunk. This would make files already written by 22.0.0–26.0.0
readable by future readers only: pyarrow 21.0.0 also reads `one_row.parquet`
written by main as 0 rows, so a reader-only fix does not help older readers.
**Impact:** `data_page_size` is a public writer option. With small values,
rows of list/map columns written by `pq.write_table` silently disappear when
the file is read back with `pq.read_table` or pandas `read_parquet`, while the
metadata still reports them. Multi-column files fail to load. Found by
differential testing of Parquet writers and readers.
### Component(s)
C++, Parquet, Python
### Component(s)
C++, Parquet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
