wecharyu opened a new issue, #13633:
URL: https://github.com/apache/gluten/issues/13633

   ### Backend
   
   VL (Velox)
   
   ### Bug description
   
   The inherited Spark `SPARK-25158` regression test intermittently fails in 
Gluten's Hive SQL suites because the Python script path points inside a Spark 
test JAR rather than to a filesystem file. The resulting subprocess startup 
failure races with the UDF exception that the test is intended to check.
   
   **Introduced by #12005** (`[GLUTEN-10072][UT] Enable hive execution tests`, 
merge commit 
[356fda012d](https://github.com/apache/gluten/commit/356fda012d859f693fc613e6ac3585b7661bdbc5)).
 That PR added the Hive `GlutenSQLQuerySuite` / `GlutenSQLQuerySuiteAE` 
wrappers and enabled them in `VeloxTestSettings`, bringing the inherited 
`SPARK-25158` test into Gluten's test coverage. It also introduced 
`GlutenHiveResourcePathSupport` and rewrote several tests to use workspace 
resources, but left this inherited test using the upstream resource lookup. 
This attribution concerns its introduction into Gluten CI; the original 
regression test already existed in Spark.
   
   **Expected behavior**
   
   The Python transformation script should resolve to a real file and start 
successfully. When the deliberately failing UDF throws, its exception should 
reach the task and satisfy the expected-message assertion, while the 
uncaught-exception handler remains empty.
   
   **Actual behavior**
   
   Python cannot open the JAR-based script path and exits with code `2`. 
Depending on timing, Spark reports the subprocess error instead of the expected 
UDF error, so the assertion fails and the CI job becomes red.
   
   **Observed CI failures**
   
   | PR | Spark | Failing suite | Job |
   | --- | --- | --- | --- |
   | #13628 | 3.4.4 | `org.apache.spark.sql.hive.execution.GlutenSQLQuerySuite` 
| [spark-test-spark34-slow 
(slow-hive)](https://github.com/apache/gluten/actions/runs/43206708219/job/131146427143)
 |
   | #13613 | 4.0.2 | 
`org.apache.spark.sql.hive.execution.GlutenSQLQuerySuiteAE` | 
[spark-test-spark40-slow 
(slow-hive)](https://github.com/apache/gluten/actions/runs/42909456682/job/130171962940)
 |
   | #13456 | 4.0.2 | 
`org.apache.spark.sql.hive.execution.GlutenSQLQuerySuiteAE` | 
[spark-test-spark40-slow 
(slow-hive)](https://github.com/apache/gluten/actions/runs/43230119992/job/131226342548)
 |
   
   **Root cause and why the failure is flaky**
   
   1. The [upstream 
test](https://github.com/apache/spark/blob/v3.4.4/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala#L2335)
 calls `getTestResourcePath("data")` and appends `/scripts/test_transform.py` 
to construct the Python command. Its UDF deliberately throws `Failed to produce 
data.`.
   2. [SparkFunSuite's resource 
helper](https://github.com/apache/spark/blob/v3.4.4/core/src/test/scala/org/apache/spark/SparkFunSuite.scala#L110)
 converts the ClassLoader resource URL through `new 
File(...getFile).getCanonicalPath`. In these Gluten CI runs, the resource comes 
from `spark-hive_*-tests.jar`. This conversion does not extract the resource; 
it produces a path containing `file:/...tests.jar!/data`, prefixed by the 
current working directory. Python cannot use that as a script file.
   3. 
[HiveScriptTransformationExec](https://github.com/apache/spark/blob/v3.4.4/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationExec.scala#L160)
 starts a writer thread to consume upstream rows, including evaluating the UDF, 
while the task thread reads the subprocess output.
   4. 
[checkFailureAndPropagate](https://github.com/apache/spark/blob/v3.4.4/sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala#L167)
 checks whether the writer has recorded an exception before checking the 
subprocess exit status. If the UDF error is already recorded, the expected 
assertion can pass. If Python exits before that error is recorded, the task can 
propagate the exit-code error and fail the assertion. A passing execution can 
therefore mask the invalid script path.
   
   The Spark 3.4 failure log is consistent with this ordering: the subprocess 
error is logged at `11:31:16.166`, followed by the writer's UDF error and the 
missing-script diagnostic at `11:31:16.171`. The timestamps support the 
analysis but are not a measurement of the precise exception-assignment time. 
The path failure is confirmed by the logs; the timing explanation follows from 
the execution code. No controlled repeated local reproduction or failure-rate 
measurement has been performed.
   
   **Suggested fix**
   
   Rewrite the inherited test in Gluten's Hive suite adapter to use a real 
workspace script path, for example 
`hiveResourcePath("data/scripts/test_transform.py").toFile.getCanonicalPath`, 
following the existing resource adaptations. Preserve both the UDF-message 
assertion and the uncaught-exception assertion. Cover both regular and AQE 
suites and audit the other Spark versions that inherit this test.
   
   Related tracking issue: #10072.
   
   This issue was prepared with assistance from OpenAI Codex (GPT-6).
   
   ### Gluten version
   
   Main branch. Example failing PR head: 
`0bd72a021508a708338bae5cedc501a244e2dafd` (#13628).
   
   ### Spark version
   
   Confirmed in CI with Spark 3.4.4 and Spark 4.0.2.
   
   ### Spark configurations
   
   The `slow-hive` jobs in the Velox Backend (x86) workflow, including the 
regular and AQE Hive SQL suites listed above.
   
   ### System information
   
   GitHub Actions CI. The linked jobs contain the runner, container, and build 
details.
   
   ### Relevant logs
   
   The Spark 3.4 run contains the following filesystem diagnostic:
   
   ```text
   python3: can't open file 
'/__w/gluten/gluten/gluten-ut/spark34/file:/root/.m2/repository/org/apache/spark/spark-hive_2.12/3.4.4/spark-hive_2.12-3.4.4-tests.jar!/data/scripts/test_transform.py'
   ```
   
   The task reports `Subprocess exited with status 2`, and the assertion fails 
because the message does not contain the expected UDF text. The Spark 4.0 runs 
show the same malformed path pattern under `spark40`, using 
`spark-hive_2.13-4.0.2-tests.jar`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to