rdtr commented on PR #13389:
URL: https://github.com/apache/gluten/pull/13389#issuecomment-6809582601

   @Mariamalmesfer Thanks for picking up `GetTimestamp`, the tests here cover 
the cases well.
   
   One thing on direction first. In 
https://github.com/facebookincubator/velox/issues/3869 Velox community 
concluded on 11/04 that Spark functions whose behavior depends on ANSI mode 
read `spark.ansi_enabled` from the query config, and that no new `checked_*` 
functions are added (the existing ones are being removed). The model is in 
https://github.com/facebookincubator/velox/issues/3869#issuecomment-6607421212 
and rui-mo's approval in 
https://github.com/facebookincubator/velox/issues/3869#issuecomment-6641765130. 
Picking the function from the expression's own `failOnError` is what Gluten 
does for `checked_add` today, so it was a fair choice before #3869 settled; the 
difference is only where the mode is read.
   
   For `get_timestamp` that change is already open in 
https://github.com/facebookincubator/velox/pull/19333. It reads the config in 
`initialize()` and throws on a parse error when ANSI is on, like 
`unix_timestamp`. With it, Gluten keeps mapping `GetTimestamp` to 
`get_timestamp` and needs neither the overlay nor a new name. When the mode 
captured in the expression differs from the session (a view or a DataFrame 
created in the other mode), https://github.com/apache/gluten/pull/13529 falls 
back to Spark.
   
   This also affects `try_to_timestamp`. Spark builds it as `GetTimestamp` with 
`failOnError = false`, which this PR maps to `get_timestamp`. Once #19333 is in 
the pin, `get_timestamp` throws whenever the session has ANSI on, so that 
mapping would make `try_to_timestamp` throw. #13529 falls back for that case 
for now.
   
   So I'd suggest dropping `CheckedGetTimestamp.h`, its registration and 
`genGetTimestampTransformer`, and keeping `GetTimestampAnsiValidateSuite`. The 
suite is useful either way. It can go in once the Velox pin includes #19333, 
with the `try_to_timestamp` cases under ANSI on expecting a fallback. Would 
that work for you? I'll ping you here when #19333 lands.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to