fresh-borzoni opened a new issue, #7417:
URL: https://github.com/apache/datafusion-comet/issues/7417
### Describe the bug
When `CometIcebergNativeScan` skips rows in an INT32 or INT64 column encoded
as `DELTA_BINARY_PACKED` whose miniblocks hold more than 64 values, the read
fails with `Parquet error: cannot skip miniblock of size N`. arrow-rs's
`DeltaBitPackDecoder::skip` sized its scratch buffer for the 32- and 64-value
miniblocks that arrow-rs itself writes, but the Parquet spec only requires a
multiple of 32. apache/arrow-rs#9794 turned the original panic
(apache/arrow-rs#9793) into this error. Snowflake writes 512-value miniblocks,
and Databricks Photon writes 256 (apache/arrow-rs#11018).
The scan reads through iceberg-rust, whose `build_deletes_row_selection`
(`crates/iceberg/src/arrow/reader/positional_deletes.rs`) turns position
deletes into a Parquet `RowSelection`. A long run of deleted rows becomes a
`RowSelector::skip`, which reaches the decoder's `skip`. Under
`RowSelectionPolicy::Auto`, short, fragmented selections such as scattered
deletes are evaluated as a mask instead, so they read fine. Spark's reader
reads the same files.
The Delta deletion-vector path builds `RowSelection`s the same way
(`native/core/src/execution/delta_dv.rs`), so Delta tables written by Photon
can likely hit it too. That's from reading the code; I haven't run it.
Found running `compute_table_stats` with Comet 1.1.0 on Iceberg tables
written by Snowflake, after merge-on-read `MERGE`s had added position deletes.
Affected: `main` (parquet 59.3.1), `branch-1.2` (1.2.0-rc1) and `branch-1.1`
(1.1.0), both on 59.3.0. apache/arrow-rs#11021 fixes it in 60.0.0, and
apache/arrow-rs#11754 backports the fix to 59.3.2.
### Steps to reproduce
Spark 3.5.5 and Iceberg 1.12.0 with a Hadoop catalog, with arrow-rs's
`parquet/tests/arrow_reader/bigdelta.parquet` (INT32 and INT64 columns,
128-value miniblocks, 7,300 rows) in `/data/in`:
```sql
CREATE TABLE local.db.t (int_col INT, bigint_col BIGINT) USING iceberg
TBLPROPERTIES ('format-version'='2', 'write.delete.mode'='merge-on-read');
CALL local.system.add_files(table => 'db.t', source_table =>
'`parquet`.`/data/in`');
DELETE FROM local.db.t WHERE _pos + 0 BETWEEN 1000 AND 2999;
SELECT count(*), sum(int_col), sum(bigint_col) FROM local.db.t;
```
(`_pos + 0` stops Iceberg from pushing the `_pos` predicate down, which it
rejects.)
Spark: 5300 rows. Comet 1.1.0: `CometNativeException: Iceberg scan error:
Unexpected => Failed to read a Parquet file, source: Arrow: Parquet argument
error: Parquet error: cannot skip miniblock of size 128`. `CALL
local.system.compute_table_stats(table => 'db.t')` fails the same way.
Deleting every 10th row instead (`WHERE int_col = 3`, 730 rows) reads fine
on 1.1.0.
### Expected behavior
The same results as Spark.
### Additional context
`native/Cargo.toml` already accepts 59.3.2 (`59.2.0`), so a lockfile update
picks up the fix. With parquet 59.3.2 locked, 1.2.0-rc1 returns Spark's results
for the query above, and `compute_table_stats` writes the same NDVs as Spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
