rdtr opened a new pull request, #13625: URL: https://github.com/apache/gluten/pull/13625
## What changes are proposed in this pull request? `GlutenTransformWithStateAvroSuite` "test that introducing TTL after restart fails query (encoding = Avro)" (both the "with changelog checkpointing" and "without changelog checkpointing" variants) fails at random on the Spark 4.2 shards. It failed on #13574 and #13575 (`spark-test-spark42 (1)`), with the same shard green on other PRs; in the last 100 completed runs of velox_backend_x86.yml it failed in 8 `spark-test-spark42 (1)` jobs (e.g. job 128543956370). The failure is the exception the test expects, reported as the test failure instead of being caught by `ExpectFailure`: `StateStoreInvalidValueSchemaEvolution: [STATE_STORE_INVALID_VALUE_SCHEMA_EVOLUTION] Schema evolution is not possible new value_schema=... and old value_schema=...`. This is not Gluten. Vanilla Spark 4.2.0 (no plugin, `TransformWithStateAvroSuite` run with the ScalaTest runner against the v4.2.0 build) fails the same way in 7 of 50 runs, with the same exception. In the Gluten suite every operator of the plan falls back (`MicroBatchScan`, `Project`, `AppendColumns`, `Exchange`, `Sort`, `TransformWithState`, `SerializeFromObject`, `WriteToDataSourceV2`: "Fallback policy is taking effect"), so nothing runs in Velox. The test body is identical in v4.0.2, v4.1.1 and v4.2.1, but no `spark-test-spark40`/`41` job failed on it in the last 100 runs, so this excludes only Spark 4.2. The UnsafeRow variant is already excluded for 4.2 (#13453). I did not pin down why the exception is raised outside `ExpectFailure`; I filed it as SPARK-61501 (https://issues.apache.org/jira/browse/SPARK-61501, mirrored in apache/spark#60369), and the exclude comment links it. The Spark 4.3 shard fails the same way (#13590, `spark-test-spark43 (1)`), but there is no spark43 module on main yet; the same `excludeByPrefix` needs to go into #13520. ## How was this patch tested? Spark 4.2.0, same test, 50 iterations each, one JVM per iteration on a loaded machine: - vanilla Spark (no Gluten): 7 failures (3 with changelog checkpointing, 4 without) - Gluten (spark-4.2 profile, `SPARK_ANSI_SQL_MODE=false SPARK_TESTING=true`): 9 failures (7 with, 2 without) With the exclude, `mvn test -Pbackends-velox -Pspark-4.2 -Pspark-ut -Pdelta -pl gluten-ut/spark42` on `GlutenTransformWithStateAvroSuite` passes (86 succeeded, 0 failed, the two TTL tests ignored); scalastyle and spotless checks pass. ## Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Sonnet 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
