pedrumj2 opened a new issue, #13647:
URL: https://github.com/apache/gluten/issues/13647

   ### Description
   
   Hey folks
   
   I was hoping to get feedback on a proposal. 
   
   ### Proposal
   Translate Velox generated exception to Spark exceptions to maintain 
consistent exception semantics with Spark. The necessary hooks have already 
been provided in 
[translateException](https://github.com/pedrumj2/gluten/blob/dbec1c00cf1321dcd0e7a2ad13e229f1001a334c/gluten-arrow/src/main/java/org/apache/gluten/vectorized/ColumnarBatchOutIterator.java#L176)
   
   ----
   
   ### Current State
   - Today when we enable new Velox operators in gluten we 
[propagate](https://github.com/apache/gluten/blob/3342737a427bac513dfd5d931ed93619110da215/cpp/core/jni/JniError.h#L45-L53)
 the Velox exception message via a gluten exception
   - The Velox exception can be different than that of Spark. For example:
   
   
   Spark JVM:
   ```
   "Exceeds char/varchar type length limitation: 5"
   ```
   
   Velox:
   ```
   "Exceeds allowed length limitation: 5"
   ```
   ----
   
   ### Why It's an Issue
   Let me explain using an example. In #12976 a new Velox operator was enabled 
in Gluten:
   - Enabling the operator was 1 line of change (remove if from the 
`kBlackList` array:
   
   ```
   const std::unordered_set<std::string> kBlackList = {"split_part", 
"sequence", "approx_percentile", "map_from_arrays"};
   ```
   - However after enabling the operator it starts to raise a different type of 
exception, causing the spark tests that assert the exception to fail. This 
caused 20+ sparks tests that relied on the operator to fail which required 
exclusion such as:
   
   ```
   ...
   .excludeGlutenTest("NOT NULL checks for nullable map with required values 
(byName)")
   .excludeGlutenTest("NOT NULL checks for nullable map with required values 
(byPosition)")
   .excludeGlutenTest("NOT NULL checks for fields inside nullable maps 
(byPosition)")
   ...
   ```
   
   To avoid losing test coverage spark tests were copied over adding 800 lines 
of test code (duplicate code with spark tests with a few lines different during 
exception assertions). 
   
   ----
   ### User Experience
   This would change user experience though which needs a decision on the repo. 
User would no longer be receiving a gluten specific error message and the 
experience would be transparent regardless of which execution engine is running 
spark. A decision that the repo would need to make if its the correct 
experience. 
   
   ----
   ### Alternative Solution, Option 2
   
   Regarding **reason2** we could rely on Apache spark to enable the exception 
assertions in their tests to be extensible such as 
[assertLengthCheckFailure](https://github.com/apache/spark/blob/v4.0.2/sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala#L64-L74).
 
   
   The challenge with this is:
   - The spark team seems reluctant of adding these extensions (please see 
[comment](https://github.com/apache/spark/pull/52996#pullrequestreview-3449022354))
   - The extensions would only help with spark versions going forward and 
previous spark versions would still have the code duplication
   
   ----
   ### Alternative Solution, Option 3
   We rely on Velox to throw exceptions consistent with Spark. My understanding 
is that Velox does strive to be agnostic to the specific data processing system 
that uses it so incorporating these changes there might prove challenging. 
   
   
   
   
   
   ### Gluten version
   
   main branch


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to