rdtr commented on PR #13389: URL: https://github.com/apache/gluten/pull/13389#issuecomment-6809582601
@Mariamalmesfer Thanks for picking up `GetTimestamp`, the tests here cover the cases well. One thing on direction first. In https://github.com/facebookincubator/velox/issues/3869 Velox community concluded on 11/04 that Spark functions whose behavior depends on ANSI mode read `spark.ansi_enabled` from the query config, and that no new `checked_*` functions are added (the existing ones are being removed). The model is in https://github.com/facebookincubator/velox/issues/3869#issuecomment-6607421212 and rui-mo's approval in https://github.com/facebookincubator/velox/issues/3869#issuecomment-6641765130. Picking the function from the expression's own `failOnError` is what Gluten does for `checked_add` today, so it was a fair choice before #3869 settled; the difference is only where the mode is read. For `get_timestamp` that change is already open in https://github.com/facebookincubator/velox/pull/19333. It reads the config in `initialize()` and throws on a parse error when ANSI is on, like `unix_timestamp`. With it, Gluten keeps mapping `GetTimestamp` to `get_timestamp` and needs neither the overlay nor a new name. When the mode captured in the expression differs from the session (a view or a DataFrame created in the other mode), https://github.com/apache/gluten/pull/13529 falls back to Spark. This also affects `try_to_timestamp`. Spark builds it as `GetTimestamp` with `failOnError = false`, which this PR maps to `get_timestamp`. Once #19333 is in the pin, `get_timestamp` throws whenever the session has ANSI on, so that mapping would make `try_to_timestamp` throw. #13529 falls back for that case for now. So I'd suggest dropping `CheckedGetTimestamp.h`, its registration and `genGetTimestampTransformer`, and keeping `GetTimestampAnsiValidateSuite`. The suite is useful either way. It can go in once the Velox pin includes #19333, with the `try_to_timestamp` cases under ANSI on expecting a fallback. Would that work for you? I'll ping you here when #19333 lands. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
