beliefer opened a new pull request, #12453:
URL: https://github.com/apache/gluten/pull/12453
## What changes are proposed in this pull request?
Gluten currently chooses ORC/DWRF column mapping (by name vs. by position)
**globally** for a whole session, driven by
`spark.gluten.sql.columnar.backend.velox.orcUseColumnNames` (default `true`,
map by name) combined with an override for `orc.force.positional.evolution`.
Vanilla Spark does not work that way. `OrcUtils.requestedColumnIds` decides
the mapping mode **per file**:
```scala
if (forcePositionalEvolution || orcFieldNames.forall(_.startsWith("_col"))) {
// map physical schema to data schema by index (position)
} else {
// map by name
}
```
Because Gluten only had a single global switch, it could not match Spark
when a single query touches ORC tables that require **opposite** mapping modes
— for example a join between:
- a table with real physical column names (must be read **by name**), and
- a table written by old Hive whose physical schema is all placeholder names
`_col0, _col1, ...` (must be read **by position**).
With the global switch, one value is always wrong for one of the tables:
`orcUseColumnNames=true` reads the `_col*` table back as NULL (name lookup
fails), while setting `orc.force.positional.evolution=true` globally forces the
real-named table to be read by position and mis-binds its columns.
This PR aligns Gluten with Spark's per-file behavior (backed by a companion
Velox change) and, as requested, **removes the now-redundant
`orcUseColumnNames` config and its accessor**.
### How the per-file decision works now
The native (Velox) reader makes the decision per ORC/DWRF file:
- a file whose physical schema is all `_col*` placeholder names is mapped by
**position**, regardless of any flag;
- a file is also mapped by position when
`orc.force.positional.evolution=true` is forwarded to native;
- otherwise the file is mapped **by name** (the default).
For this to work, Gluten must always hand the table (data) schema to the
native reader for ORC/DWRF so it has a target to remap file columns to.
### Notes
This is the Gluten side of a two-part change. The companion Velox change adds
per-file positional mapping in the DWRF reader: it maps a file by position
when
the physical schema is all `_col*` placeholder names or when the
`orc.force-positional-evolution` session property is set. This Gluten PR
depends
on that Velox change being present in the Velox build.
## How was this patch tested?
- **`gluten-ut` (spark33/34/35/40/41) `GlutenHiveSQLQuerySuite`** — new
regression
test: two ORC tables over the same `_col*` files (placeholder names) plus a
real-named table, all read in one session **without** setting the
positional
flag. Asserts the `_col*` table reads correctly (positional fallback), the
real-named table reads correctly by name (opposite mode), and a join of the
two returns a non-empty result (the original failure folded the join to an
empty `LocalTableScan`).
- **`VeloxScanSuite`** — the former "ORC index based schema evolution" test
(which relied on `orcUseColumnNames=false` to read real-named files by
index,
a non-Spark behavior) is rewritten as "ORC positional schema evolution"
using
`orc.force.positional.evolution=true`, which produces the same positional
mapping via the Spark-compatible path. Expected results are unchanged.
- **`FallbackSuite`** — the "fallback with index based schema evolution" test
drops the `orcUseColumnNames` dimension (the parquet dimension is
retained).
## Was this patch authored or co-authored using generative AI tooling?
co-authored Claude code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
