zhouyuan opened a new issue, #13463:
URL: https://github.com/apache/gluten/issues/13463
### Backend
VL (Velox)
### Bug description
`GlutenTransformWithStateUnsafeRowSuite` (Spark 4.2) is flaky in the
`spark-test-spark42 (1)` CI job. The failing test is:
- `test that introducing TTL after restart fails query (encoding =
UnsafeRow) (with changelog checkpointing)`
- `test that introducing TTL after restart fails query (encoding =
UnsafeRow) (without changelog checkpointing)`
The two variants fail intermittently; which one fails changes from run to
run. The test reports a failure with the exception it expects:
```
org.apache.spark.sql.execution.streaming.state.StateStoreValueSchemaNotCompatible:
[STATE_STORE_VALUE_SCHEMA_NOT_COMPATIBLE] Provided value schema does not match
existing state value schema.
Please check number and type of fields.
Existing value_schema=StructType(StructField(value,LongType,false)) and new
value_schema=StructType(StructField(value,StructType(StructField(value,LongType,false)),true),StructField(ttlExpirationMs,LongType,true)).
at
org.apache.spark.sql.execution.streaming.state.StateStoreErrors$.stateStoreValueSchemaNotCompatible(StateStoreErrors.scala:170)
at
org.apache.spark.sql.execution.streaming.state.StateSchemaCompatibilityChecker.check(StateSchemaCompatibilityChecker.scala:247)
...
```
Gluten doesn't execute any part of this query: every operator
(MicroBatchScan, TransformWithState, Exchange, Sort, WriteToDataSourceV2, ...)
falls back to vanilla Spark. The test expects this exception inside
`ExpectFailure[StateStoreValueSchemaNotCompatible]`, but sometimes the error
appears to escape before `ExpectFailure` can catch it. This looks like a timing
race in the Spark test itself, not a Gluten correctness problem.
Recent failures in unrelated PRs and in the scheduled `main` run:
- https://github.com/apache/gluten/actions/runs/40300608111 (PR #13453)
- https://github.com/apache/gluten/actions/runs/40339712752
- https://github.com/apache/gluten/actions/runs/40328474600
- https://github.com/apache/gluten/actions/runs/40286339268
- https://github.com/apache/gluten/actions/runs/40153084868 (scheduled
`main`)
### Proposed action
Short term: exclude the test in
`gluten-ut/spark42/.../VeloxTestSettings.scala` so it stops blocking CI:
```scala
enableSuite[GlutenTransformWithStateUnsafeRowSuite]
.excludeByPrefix("test that introducing TTL after restart fails query")
```
Follow-up: find out why the error escapes `ExpectFailure` (possibly made
more likely by running suites in parallel JVMs), then either fix it or report
it upstream to Spark, and re-enable the test.
### Spark version
Spark 4.2.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
