rdtr opened a new pull request, #13268: URL: https://github.com/apache/gluten/pull/13268
## What changes are proposed in this pull request? This is a new version of #11674, which was closed as stale without a review. `monotonically_increasing_id` falls back to vanilla Spark in the Velox backend. Velox has a Spark implementation, and Gluten already passes the partition ID it uses (`SparkQueryConfig::kPartitionId`), but the function is not mapped. An earlier attempt (#10097) was closed because of a result mismatch (#7628): with two calls in one query, such as `SELECT monotonically_increasing_id(), monotonically_increasing_id()`, the second column returned 2, 3, ... instead of 0, 1, .... Velox deduplicated the two identical calls into one, so they shared one counter. Velox has since added `expression.dedup_non_deterministic` (facebookincubator/velox#15008) to turn this off. This PR: - Maps `MonotonicallyIncreasingID` in the Velox backend's expression mappings. - Sets `expression.dedup_non_deterministic` to `false`. Spark keeps each call of a non-deterministic function separate, with its own state. Deterministic expressions are still deduplicated. - Re-enables the `monotonically_increasing_id` test in `ScalarFunctionsValidateSuite`, fixes its name, and adds a query with two calls. Question for reviewers: the config is set for all queries, not only the ones that use `monotonically_increasing_id`. I think that matches Spark, which never merges non-deterministic calls, but please let me know if you'd prefer to limit it. Closes #7628. ## How was this patch tested? Re-enabled and extended the `monotonically_increasing_id` test in `ScalarFunctionsValidateSuite`. ## Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
