rdtr opened a new pull request, #13625:
URL: https://github.com/apache/gluten/pull/13625

   ## What changes are proposed in this pull request?
   
   `GlutenTransformWithStateAvroSuite` "test that introducing TTL after restart 
fails query (encoding = Avro)" (both the "with changelog checkpointing" and 
"without changelog checkpointing" variants) fails at random on the Spark 4.2 
shards. It failed on #13574 and #13575 (`spark-test-spark42 (1)`), with the 
same shard green on other PRs; in the last 100 completed runs of 
velox_backend_x86.yml it failed in 8 `spark-test-spark42 (1)` jobs (e.g. job 
128543956370). The failure is the exception the test expects, reported as the 
test failure instead of being caught by `ExpectFailure`: 
`StateStoreInvalidValueSchemaEvolution: 
[STATE_STORE_INVALID_VALUE_SCHEMA_EVOLUTION] Schema evolution is not possible 
new value_schema=... and old value_schema=...`.
   
   This is not Gluten. Vanilla Spark 4.2.0 (no plugin, 
`TransformWithStateAvroSuite` run with the ScalaTest runner against the v4.2.0 
build) fails the same way in 7 of 50 runs, with the same exception. In the 
Gluten suite every operator of the plan falls back (`MicroBatchScan`, 
`Project`, `AppendColumns`, `Exchange`, `Sort`, `TransformWithState`, 
`SerializeFromObject`, `WriteToDataSourceV2`: "Fallback policy is taking 
effect"), so nothing runs in Velox. The test body is identical in v4.0.2, 
v4.1.1 and v4.2.1, but no `spark-test-spark40`/`41` job failed on it in the 
last 100 runs, so this excludes only Spark 4.2. The UnsafeRow variant is 
already excluded for 4.2 (#13453). I did not pin down why the exception is 
raised outside `ExpectFailure`; I filed it as SPARK-61501 
(https://issues.apache.org/jira/browse/SPARK-61501, mirrored in 
apache/spark#60369), and the exclude comment links it.
   
   The Spark 4.3 shard fails the same way (#13590, `spark-test-spark43 (1)`), 
but there is no spark43 module on main yet; the same `excludeByPrefix` needs to 
go into #13520.
   
   ## How was this patch tested?
   
   Spark 4.2.0, same test, 50 iterations each, one JVM per iteration on a 
loaded machine:
   - vanilla Spark (no Gluten): 7 failures (3 with changelog checkpointing, 4 
without)
   - Gluten (spark-4.2 profile, `SPARK_ANSI_SQL_MODE=false 
SPARK_TESTING=true`): 9 failures (7 with, 2 without)
   
   With the exclude, `mvn test -Pbackends-velox -Pspark-4.2 -Pspark-ut -Pdelta 
-pl gluten-ut/spark42` on `GlutenTransformWithStateAvroSuite` passes (86 
succeeded, 0 failed, the two TTL tests ignored); scalastyle and spotless checks 
pass.
   
   ## Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Claude Code (Claude Sonnet 5.5)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

  • [PR] [VL][TEST] Exclude the T... rdtr (via GitHub) via commits

Reply via email to