malinjawi opened a new pull request, #13391:
URL: https://github.com/apache/gluten/pull/13391
## What changes are proposed in this pull request?
Fixes two row tracking read bugs in the native Delta scan, seen in the
nightly Delta Lake job (Spark 4.1, Delta 4.4.1).
**Filters on constant `_metadata` fields.** A filter such as
`_metadata.base_row_id = 10` was pushed into the native scan. Velox reads
constant `_metadata` fields as synthesized split columns. Since
facebookincubator/velox#16116 it fails the scan when a pushed filter on one of
them rejects a split, because it expects the planner to have pruned that split.
Delta's file index does not prune by these fields, so any such filter that
selects a subset of files failed with `Synthesized column 'base_row_id' failed
filter validation`. `DeltaScanTransformer` now leaves filters on constant
metadata fields out of `scanFilters` and the pushed-down filters. The Filter
above the scan evaluates them, still in Velox. The `_metadata` file fields,
such as `file_name`, take the same path.
**Missing base row IDs.** Delta fails a read of a non-nullable
`_metadata.base_row_id` or `_metadata.default_row_commit_version` from a file
that has no value for it. The native scan returned NULL instead, so
`_metadata.row_id` returned rows where Delta raises an error. Such a scan now
fails validation and runs on Spark, so it fails as in Delta. The check looks at
the scan's files only when the scan reads one of those two fields. Delta
assigns both values to every file once row tracking is on, so only a table
changed by hand gets here. Delta Lake's test builds one with a manual metadata
commit. The check is per Delta version (`DeltaRowTrackingScanInfo`):
- On Delta 4.x, scans with `nullableRowTrackingConstantFields` stay native
and read NULL, as Delta does.
- Delta 3.3 always fails such reads.
- Delta 2.4 has no row tracking fields.
Removes three `RowIdSuite` entries from the Delta Lake test baseline:
`Filter by base Row IDs`, `Filter by base Row IDs in subquery` and `missing
base row ids and default row commit versions`. The docs listed both bugs under
read problems. They now describe the new behavior.
The other three row tracking entries are not scan bugs:
- `row ids can be read back` and `No dictionary filtering on
_metadata.row_id` fail on the row-group count and encoding names of a plain
Parquet file that the Velox writer writes for the test. Both fail before
reading any row ID.
- `write and read table with all-null materialized columns` fails on a
zero-byte file left by an empty native write, fixed in #13390.
## How was this patch tested?
- `DeltaRowTrackingScanInfoSuite` plans the queries on vanilla Spark. It
checks which filters reach the native scan and which scans the new check flags.
It passes locally on Spark 3.5 (Delta 3.3) and Spark 4.1 (Delta 4.4.1).
- `VeloxDeltaRowTrackingSuite` runs the queries with Gluten:
- Filters on `base_row_id` and `file_name`, also in a subquery, return the
right rows from a native scan.
- Reading row IDs or row commit versions from a table with a file that
lacks them throws.
- A read that data skipping limits to files with the values stays native.
It loads the native library, so it runs in CI.
- Both suites are in the Delta 3.3 tree and linked into the Delta 4 tree.
- Compiled main and test code on Spark 3.5 and 4.1 with
`-Pbackends-velox,spark-<v>,delta,spark-ut`, and `gluten-delta` on Spark 3.4.
cc @rui-mo @felipepessoto
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: IBM Bob
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
