felipepessoto opened a new issue, #13288:
URL: https://github.com/apache/gluten/issues/13288
### Backend
VL (Velox)
### Bug description
Delta's column-mapping concurrency test reaches a Parquet footer assertion
after the observer fix in apache/gluten#13134:
```text
org.apache.spark.sql.delta.FeatureEnablementConcurrencySuite
Enable column mapping feature - txnInterleaved: true
footerMetadata.containsKey(
org.apache.spark.sql.execution.datasources.parquet.ParquetReadSupport.SPARK_METADATA_KEY
) was false
```
The test previously timed out because its transaction observer was replaced.
Preserving that observer lets it execute the UPDATE/column-mapping interleaving
and reach footer validation. The remaining failure is separate from observer
lifecycle handling.
**Observed:** at least one resulting Parquet file lacks Spark's schema
metadata entry. The failure occurs at the presence check, before the test
inspects the embedded column-mapping metadata. This is a native-writer metadata
compatibility gap to investigate, not evidence here of incorrect row values.
**Expected:** native Delta writes should preserve the Spark schema metadata
required by this path, or the test's expectation should be reviewed if that
metadata is not required by the native writer's compatibility contract. Do not
suppress the assertion or remove the known-failure entry without establishing
the intended behavior.
### Reproduction and evidence
Run the existing Delta-on-Gluten harness against Delta v4.2.0 / Spark 4.1
with native Delta writes enabled, selecting:
```text
org.apache.spark.sql.delta.FeatureEnablementConcurrencySuite
test: Enable column mapping feature - txnInterleaved: true
```
The test prepares an UPDATE while column mapping is disabled, enables column
mapping concurrently, completes the transaction, checks the resulting rows, and
then inspects the active files' Parquet footers.
- [Observed failing
job](https://github.com/apache/gluten/actions/runs/36155269247/job/108177714662),
using PR head `b9bb710055aabe37f37a554fb016029d283ec98e`.
- Artifact: `delta-spark-ut-reports-shard-0`, report
`TEST-org.apache.spark.sql.delta.FeatureEnablementConcurrencySuite.xml`.
- [Delta's footer
assertion](https://github.com/delta-io/delta/blob/v4.2.0/spark/src/test/scala/org/apache/spark/sql/delta/FeatureEnablementConcurrencySuite.scala#L101-L111).
- [The interleaved
test](https://github.com/delta-io/delta/blob/v4.2.0/spark/src/test/scala/org/apache/spark/sql/delta/FeatureEnablementConcurrencySuite.scala#L378-L414).
### Gluten version
Observed on the observer-fix branch in apache/gluten#13134; the test remains
in the known-failure baseline. This issue has not yet been reproduced
independently against current main.
### Spark version
Spark 4.1; Delta v4.2.0.
### Spark configurations
```text
spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true
```
Other settings are those supplied by the Delta Spark UT workflow and its
`DeltaSQLCommandTest` harness.
### Related
- Follow-up to apache/gluten#12743.
- Exposed by apache/gluten#13134; intentionally outside that observer fix.
- Separate from apache/gluten#12241, which concerns native reader field-ID
matching.
This issue was written with the assistance of GitHub Copilot CLI 1.0.88.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
