LuciferYang opened a new pull request, #12888:
URL: https://github.com/apache/gluten/pull/12888
Two `.exclude(...)` entries in `VeloxTestSettings` carried no comment saying
what fails. This writes down the reason for one and completes the rewrite pair
for the other. No production code.
### `Stop task set if FileAlreadyExistsException was thrown`
Excluded in all four modules with nothing said about why. The comment on the
line above it (`the native write staing dir is differnt with vanilla Spark for
coustom partition paths`) belongs to the `SPARK-35106` entry between them.
I removed the exclusion locally and ran `GlutenInsertSuite` against Spark
3.4.4 with Velox. 70 of 71 cases pass; this one fails:
```
- Stop task set if FileAlreadyExistsException was thrown *** FAILED ***
Expected exception org.apache.spark.SparkException to be thrown,
but org.apache.hadoop.fs.FileAlreadyExistsException was thrown
(InsertSuite.scala:2035)
```
The case installs a `FileSystem` that throws `FileAlreadyExistsException` on
create and expects Spark's write path to wrap it, either as `TASK_WRITE_FAILED`
or as the `can not write to output file` message. Gluten's native write
surfaces the raw Hadoop exception instead. So the exclusion is right and only
the reason was missing; that is what this adds.
### `cast from timestamp II`
Spark 3.4.4 and 3.5.5 both assert six values in this case, the four
NaN/Infinity ones plus `Long.MaxValue` and `Long.MinValue`. Spark 3.4.0 only
had the four, which is why the exclusion predates the extra assertions.
4.0 and 4.1 already carry a `testGluten("cast from timestamp II")` rewrite
that keeps five of the six and says why the sixth is dropped:
`glutenCheckExpression` uses `collect()`, which goes through `toJavaTimestamp`
and `rebaseGregorianToJulianMicros`, and `Long.MinValue` microseconds overflows
there. Velox computes it correctly; only the collect path fails. 3.4 and 3.5
have the bare exclusion and no rewrite, so on those two versions the other five
assertions run nowhere.
This ports the 4.0 rewrite to `gluten-ut/spark34` and `gluten-ut/spark35`
verbatim, and adds a one-line comment above the vanilla exclusion in all four
modules so the pair is visible from the settings file.
### Verification
`test-compile` passes on `-Pspark-3.4`, `-Pspark-3.5 -Pscala-2.13`,
`-Pspark-4.0 -Pscala-2.13` and `-Pspark-4.1 -Pscala-2.13`. The
`GlutenInsertSuite` run above is on this machine, which is not authoritative
for a pass but is enough to read a deterministic exception-type mismatch.
Found while auditing the excludes that have no stated reason, in preparation
for removing `gluten-ut/spark33` under #12807. Related: #12863, #12886.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude claude-opus-5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
