rdtr opened a new pull request, #13574:
URL: https://github.com/apache/gluten/pull/13574

   ## What changes are proposed in this pull request?
   
   Part of #13573, its first checkbox. Like #12840 did for Spark 3.3, this 
moves the test coverage that exists only in `gluten-ut/spark34` before that 
module is removed.
   
   One case needs to move. `GlutenInsertSuite`'s `offload empty2null when 
v1writes fallback` was added by #6859 to `gluten-ut/spark34` only. It sets 
`spark.sql.maxRecordsPerFile=1000`, which `VeloxBackend` rejects for native 
write, so the dynamic-partition `INSERT OVERWRITE` stays on Spark's 
`WriteFiles`. The case checks that the `empty2null` projection under it still 
runs as a `ProjectExecTransformer`, and that the empty-string partition values 
come back as NULL. This adds it to `GlutenInsertSuite` in `gluten-ut/spark35`, 
`spark40`, `spark41` and `spark42`. One assertion is tightened in the copy. The 
3.4 version called `projectList.find(_.toString().contains("empty2null"))` and 
dropped the result, so it never checked that `empty2null` was there. The copy 
asserts it.
   
   ClickHouse excluded every Gluten-authored `GlutenInsertSuite` case on 3.4 
(`excludeAllGlutenTests()`), and on 3.5+ it excludes the sibling v1writes cases 
one by one. The new case gets the same `excludeCH` entry in the four 
`ClickHouseTestSettings`. Bolt ran the case on 3.4 with the same 
`GlutenInsertSuite` settings it uses on 3.5, so nothing changes there.
   
   The rest of `gluten-ut/spark34` needs no migration:
   
   - `GlutenInnerJoinSuite` / `GlutenOuterJoinSuite` and 
`GlutenFileMetadataStructRowIndexSuite` are enabled only on 3.4, but 3.5+ run 
the same Spark suites as `GlutenInnerJoinSuiteForceShjOn/Off`, 
`GlutenOuterJoinSuiteForceShjOn/Off` and 
`GlutenParquetFileMetadataStructRowIndexSuite`.
   - `GlutenHiveUDFSuite` on 3.4 replaces Spark's `UDTF` and `permanent UDTF` 
with copies that load `TestUDTF.jar` through `hiveResourcePath`. 3.5+ run 
Spark's own two cases.
   - `VeloxSQLQueryTestSettings` on 3.4 enables 39 entries that 3.5 does not. 
22 of them match no file in Spark 3.4.4's `sql-tests/inputs` (for example 
`concat.sql`, whose real path `typeCoercion/native/concat.sql` 3.5 lists too). 
Of the 17 real files, the nine `ansi/*.sql` files turn ANSI on and fall back as 
a whole under the default `spark.gluten.sql.ansiFallback.enabled=true`. 3.5's 
settings say `date.sql` and `timestamp.sql` are covered by 
`datetime-legacy.sql`. #12229 took `extract.sql`, `timestamp-ltz.sql` and 
`timestamp-ntz.sql` off the 3.5 list when it added TIMESTAMP_NTZ support. 
`timestampNTZ/timestamp.sql`, `timestampNTZ/timestamp-ansi.sql` and 
`datetime-formatting-invalid.sql` are not enabled on 3.5 or 4.0. Enabling any 
of them on 3.5 needs a UT run against Spark's golden files, so they are not 
part of this change.
   - In the Velox settings, 18 test cases run on 3.4 but are excluded on 3.5. 
Nine of them do not exist in Spark 3.4.4 (SPARK-48241 in three CSV suites, 
SPARK-43876, SPARK-45171, SPARK-42782, SPARK-48863 and the two 
`INVALID_BITMAP_POSITION` cases). The other nine (five in 
`GlutenBucketedWriteWithoutHiveSupportSuite`, one each in 
`GlutenCachedTableSuite`, `GlutenHeaderCSVReadSchemaSuite`, `GlutenCSVv1Suite` 
and `GlutenCSVv2Suite`) are excluded on 3.5 for 3.5 behavior. That is the same 
situation #12863 handled for 3.3, and they are left for a follow-up.
   
   ## How was this patch tested?
   
   `test-compile` passes with `-Pbackends-velox -Pspark-ut -Pjava-17` on 
`-Pspark-3.5`, `-Pspark-4.0 -Pscala-2.13`, `-Pspark-4.1 -Pscala-2.13` and 
`-Pspark-4.2 -Pscala-2.13`, with scalastyle and `spotless:check` on. The header 
check (`.github/workflows/util/check.py header branch`) passes.
   
   `GlutenInsertSuite` passes on Spark 3.5 with the Velox backend (73 
succeeded, 8 ignored), run with `SPARK_ANSI_SQL_MODE=false SPARK_TESTING=true` 
as CI does. Two mutations make the new case fail. Setting `maxRecordsPerFile` 
to 0 lets the write go native, and the case fails on `VeloxColumnarWriteFiles`. 
Looking for a different expression name makes the tightened `empty2null` 
assertion fail. The 4.x copies are the same code. CI runs them.
   
   ## Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Claude Code (Claude Opus 5.5)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

  • [PR] [GLUTEN-<ISSUE>][... rdtr (via GitHub) via commits

Reply via email to