beliefer opened a new issue, #12708:
URL: https://github.com/apache/gluten/issues/12708
### Backend
VL (Velox)
### Bug description
For an **OUTER** generator, Velox's `Unnest` operator appends a trailing
`BOOLEAN` marker column to its output (output column order: replicated columns
→ unnest value columns → `[ordinality]` → `[marker]`). Gluten's
`GenerateExecTransformer` must consume that marker in `pullOutPostProject` so
the native output aligns positionally with the declared Spark schema.
`pullOutPostProject` handles the OUTER marker for `PosExplode`,
`Inline`/`JsonTupleExplode`, and `Explode` — but **not for `Stack`**. An OUTER
`Stack` falls through to `case _ => generate`, so the marker column is never
consumed. Because Velox binds columns **by position**, the unconsumed marker
makes the native output one column wider than Gluten's declared schema,
shifting every upstream column by one index.
The shift is only observable when a **columnar hash exchange** consumes the
Generate output. Under columnar shuffle, `VeloxShuffleWriter::getFirstColumn`
requires field 0 of the shuffle `RowVector` to be the int32
`hash_partition_key`. With the columns shifted, the boolean marker lands at
field 0 and every task of the stage aborts deterministically:
```
Partition id (field 0) should be integer, but got BOOLEAN
(cpp/velox/shuffle/VeloxShuffleWriter.h — getFirstColumn)
```
The same unconsumed-marker root cause can surface under different messages
depending on the operator downstream of the shift:
- **hash-partition shuffle → SortMergeJoin**: `Partition id (field 0) should
be integer, but got BOOLEAN` (the case above).
- **partial HashAggregate** (e.g. `GROUP BY` on the exploded columns):
crashes earlier in the native input stream with `values_->capacity() >=
byteSize` (`FlatVector.h`).
- **BroadcastHashJoin** downstream: `Serialized encoding is not compatible
... Expected INT_ARRAY. Got BYTE_ARRAY` (`checkTypeEncoding`).
Inner (non-OUTER) `stack` emits no marker column and is unaffected. The
existing `test stack function` only exercises inner `stack`, which is why the
gap was not caught.
Self-contained repro (crashes before the fix, returns correct rows after):
```sql
CREATE OR REPLACE TEMPORARY VIEW t1_stack AS
SELECT * FROM VALUES (1,'james',10,'lucy'),(2,'bond',20,'lily') AS
tbl(id,name,id1,name1);
CREATE OR REPLACE TEMPORARY VIEW t2_dim AS
SELECT * FROM VALUES (1,'a'),(2,'b'),(10,'c'),(20,'d') AS tbl(k,tag);
SET spark.sql.autoBroadcastJoinThreshold=-1; -- force SortMergeJoin
(hash-partition columnar exchange)
SELECT j.eq_pos, t2.tag
FROM (
SELECT eq_pos, val
FROM t1_stack
LATERAL VIEW OUTER stack(2, id, name, id1, name1) v AS eq_pos, val
) j
JOIN t2_dim t2 ON j.eq_pos = t2.k
ORDER BY j.eq_pos, t2.tag;
```
Expected (after fix): `(1,a) (2,b) (10,c) (20,d)`.
Root cause is in `PullOutGenerateProjectHelper.pullOutPostProject`
(`backends-velox/src/main/scala/org/apache/gluten/execution/GenerateExecTransformer.scala`):
`Stack` has no `if generate.outer` branch, unlike
`Explode`/`PosExplode`/`Inline`.
### Gluten version
main branch
### Spark version
Spark-3.5.x
### Spark configurations
Reproduces on the columnar shuffle path. Force a SortMergeJoin so the
exploded key drives a hash-partition columnar exchange:
```
spark.sql.autoBroadcastJoinThreshold = -1
```
### System information
Velox backend. Not environment-specific — the crash is deterministic and
data-independent (every task of the stage fails).
### Relevant logs
```bash
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
