pedrumj2 commented on PR #13639:
URL: https://github.com/apache/gluten/pull/13639#issuecomment-6900943914

   > @pedrumj2, thanks for the PR!
   > 
   > Could we adapt the test checker instead? On Spark 4.x, all length checks 
go through `assertLengthCheckFailure`, so the Gluten suite can override it to 
accept any exception whose cause chain contains Velox's `Exceeds allowed length 
limitation: <n>`. Then the copied 4.x tests can be removed. On Spark 3.4 / 3.5, 
the checks are inline in each test, so I think we can keep the existing Gluten 
copies there. Since Spark 3.4 support will be removed soon (#13573), that 
leaves only the Spark 3.5 copies, which should be fine to keep.
   > 
   > The point is that a different exception type or message should be fine for 
users, as long as the query fails for the same reason and the message clearly 
explains it. Does this make sense?
   
   Thanks for the review @philo-he 
   
   > Could we adapt the test checker instead? On Spark 4.x, all length checks 
go through `assertLengthCheckFailure`, so the Gluten suite can override it to 
accept any exception whose cause chain contains Velox's `Exceeds allowed length 
limitation: <n>`
   
   I can definitely follow with using the test checker. However I was thinking 
a bit more broadly how we can solve this test duplication overall. Seems to be 
common pattern. 
   
   My only concern is that with this approach we would rely on the spark repo 
to provide us with these extensions so we can leverage them in our tests. 
Please see this 
[comment](https://github.com/apache/spark/pull/52996#pullrequestreview-3449022354)
 while they do add these helper functions from time to time but it seems they 
are reluctant to provide them consistently, meaning I'm not sure we can solve 
this for all our tests. 
   
   > The point is that a different exception type or message should be fine for 
users, as long as the query fails for the same reason and the message clearly 
explains it. Does this make sense?
   
   I agree, this definitely make sense. There are two options for raising 
exceptions each have their pros and cons:
   - Have a gluten exception type with a different user facing message than 
Spark
   - Maintain same exception with Spark wrapped in a gluten exception
   
   Curious I looked through the repo but wasn't able to find a discussion on 
which direction the repo prefers to take. Do you think it would be worthwhile 
to have a broader discussion on this? I opened an issue here 
https://github.com/apache/gluten/issues/13647
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to