This is an automated email from the ASF dual-hosted git repository.
FelixYBW pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gluten.git
The following commit(s) were added to refs/heads/main by this push:
new 6b96bc721d [VL] Add function overlay to manage function
implementations in Gluten (#12817)
6b96bc721d is described below
commit 6b96bc721dad682a2092c752c5cef7f4e8c5525c
Author: Yuan <[email protected]>
AuthorDate: Fri Aug 21 08:35:37 2026 +0100
[VL] Add function overlay to manage function implementations in Gluten
(#12817)
* [VL] Add function overlay to manage function implementations in Gluten
Adding or fixing a function currently requires upstreaming it to Velox
first, which can take a long time. This introduces a function overlay
under cpp/velox/operators/functions/overlay to host Velox-compatible
function implementations managed on the Gluten side.
The overlay is registered via gluten::registerFunctionOverlay() at the
end of registerAllFunctions(), after all Velox function registrations,
so overlay functions take precedence over same-name, same-signature
Velox functions. The existing round override is moved into the overlay
as the first example, and the developer guide documents how to add a
function through the overlay.
* Apply suggestions from code review
Co-authored-by: Copilot Autofix powered by AI
<[email protected]>
---------
Co-authored-by: Copilot Autofix powered by AI
<[email protected]>
---
cpp/velox/CMakeLists.txt | 1 +
.../functions/RegistrationAllFunctions.cc | 15 +++-----
cpp/velox/operators/functions/overlay/README.md | 45 ++++++++++++++++++++++
.../functions/overlay/RegisterFunctionOverlay.cc | 45 ++++++++++++++++++++++
.../functions/overlay/RegisterFunctionOverlay.h | 30 +++++++++++++++
.../functions/{Arithmetic.h => overlay/Round.h} | 3 ++
.../developers/velox-function-development-guide.md | 27 +++++++++++++
7 files changed, 156 insertions(+), 10 deletions(-)
diff --git a/cpp/velox/CMakeLists.txt b/cpp/velox/CMakeLists.txt
index e347381887..c881ea8d57 100644
--- a/cpp/velox/CMakeLists.txt
+++ b/cpp/velox/CMakeLists.txt
@@ -176,6 +176,7 @@ set(VELOX_SRCS
memory/VeloxMemoryManager.cc
operators/functions/RegistrationAllFunctions.cc
operators/functions/delta/DeltaBitmapAggregator.cc
+ operators/functions/overlay/RegisterFunctionOverlay.cc
operators/functions/RowConstructorWithNull.cc
operators/functions/SparkExprToSubfieldFilterParser.cc
operators/plannodes/RowVectorStream.cc
diff --git a/cpp/velox/operators/functions/RegistrationAllFunctions.cc
b/cpp/velox/operators/functions/RegistrationAllFunctions.cc
index 752953390d..7ed6308193 100644
--- a/cpp/velox/operators/functions/RegistrationAllFunctions.cc
+++ b/cpp/velox/operators/functions/RegistrationAllFunctions.cc
@@ -16,15 +16,14 @@
*/
#include "operators/functions/RegistrationAllFunctions.h"
-#include "operators/functions/Arithmetic.h"
#include "operators/functions/RowConstructorWithNull.h"
#include "operators/functions/RowFunctionWithNull.h"
#include "operators/functions/delta/DeltaBitmapAggregator.h"
+#include "operators/functions/overlay/RegisterFunctionOverlay.h"
#include "velox/expression/SpecialFormRegistry.h"
#include "velox/expression/VectorFunction.h"
#include "velox/functions/iceberg/Register.h"
#include "velox/functions/lib/CheckedArithmetic.h"
-#include "velox/functions/lib/RegistrationHelpers.h"
#include "velox/functions/prestosql/aggregates/RegisterAggregateFunctions.h"
#include "velox/functions/prestosql/registration/RegistrationFunctions.h"
#include "velox/functions/prestosql/window/WindowFunctionsRegistration.h"
@@ -50,14 +49,6 @@ namespace gluten {
namespace {
void registerFunctionOverwrite() {
- velox::functions::registerUnaryNumeric<RoundFunction>({"round"});
- velox::registerFunction<RoundFunction, int8_t, int8_t, int32_t>({"round"});
- velox::registerFunction<RoundFunction, int16_t, int16_t, int32_t>({"round"});
- velox::registerFunction<RoundFunction, int32_t, int32_t, int32_t>({"round"});
- velox::registerFunction<RoundFunction, int64_t, int64_t, int32_t>({"round"});
- velox::registerFunction<RoundFunction, double, double, int32_t>({"round"});
- velox::registerFunction<RoundFunction, float, float, int32_t>({"round"});
-
auto kRowConstructorWithNull =
RowConstructorWithNullCallToSpecialForm::kRowConstructorWithNull;
velox::exec::registerVectorFunction(
kRowConstructorWithNull,
@@ -96,6 +87,10 @@ void registerAllFunctions() {
velox::functions::iceberg::registerFunctions();
registerDeltaBitmapAggregator();
+
+ // Gluten-managed function implementations. Registered last so they take
+ // precedence over same-name, same-signature Velox functions.
+ registerFunctionOverlay();
}
} // namespace gluten
diff --git a/cpp/velox/operators/functions/overlay/README.md
b/cpp/velox/operators/functions/overlay/README.md
new file mode 100644
index 0000000000..c265c22158
--- /dev/null
+++ b/cpp/velox/operators/functions/overlay/README.md
@@ -0,0 +1,45 @@
+# Gluten Function Overlay
+
+This directory is Gluten's function overlay: a place to host Velox-compatible
+function implementations that are managed on the Gluten side, so a new or fixed
+function does not have to be upstreamed to Velox before it can ship in Gluten.
+
+Typical use cases:
+
+* A Spark function that is missing in Velox. Implement it here first, so Gluten
+ can offload it right away, and upstream it to Velox later.
+* A Velox `sparksql` function whose semantics diverge from Spark. Implement the
+ corrected version here to override the Velox one while the fix is pending
+ upstream.
+
+## How it works
+
+`registerFunctionOverlay()` (see `RegisterFunctionOverlay.h`) is called at the
+end of `gluten::registerAllFunctions()`, after all Velox presto/spark function
+registrations. Velox's function registries let a later registration with the
+same name and signature replace an earlier one, so any function registered in
+the overlay takes precedence over the Velox implementation.
+
+## Adding a function
+
+1. Implement the function in a header/source file in this directory, following
+ Velox's function authoring APIs (simple function, vector function,
aggregate,
+ or window function). See `Round.h` for a simple-function example and
+ [Velox scalar functions
guide](https://github.com/facebookincubator/velox/blob/main/velox/docs/develop/scalar-functions.rst).
+2. Register it in `RegisterFunctionOverlay.cc` inside
+ `registerFunctionOverlay()`. Use the same name Gluten's Substrait plan
+ conversion emits (Spark's function name in most cases).
+3. If you added a new `.cc` file, add it to the source list in
+ `cpp/velox/CMakeLists.txt`.
+4. Add unit tests in `cpp/velox/tests/SparkFunctionTest.cc` (function overlay
is
+ registered there through `registerAllFunctions()`), and Scala side tests
+ where applicable.
+5. If the function is brand new (not just an override), make sure it is mapped
+ on the Scala side (e.g. expression mappings in
+ `ExpressionMappings.scala` / `ExpressionNames`) so the planner offloads it.
+
+## Lifecycle
+
+The overlay is a staging area, not a permanent fork. When an overlay function
+is accepted into upstream Velox, remove it from the overlay in the same PR that
+bumps the Velox version, so the Velox implementation takes effect.
diff --git a/cpp/velox/operators/functions/overlay/RegisterFunctionOverlay.cc
b/cpp/velox/operators/functions/overlay/RegisterFunctionOverlay.cc
new file mode 100644
index 0000000000..d16e7636bd
--- /dev/null
+++ b/cpp/velox/operators/functions/overlay/RegisterFunctionOverlay.cc
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include "operators/functions/overlay/RegisterFunctionOverlay.h"
+
+#include "operators/functions/overlay/Round.h"
+#include "velox/functions/lib/RegistrationHelpers.h"
+
+using namespace facebook;
+
+namespace gluten {
+namespace {
+
+// Spark's round differs from Velox's in the handling of negative decimals and
+// floating point rounding semantics.
+void registerRoundFunction() {
+ velox::functions::registerUnaryNumeric<RoundFunction>({"round"});
+ velox::registerFunction<RoundFunction, int8_t, int8_t, int32_t>({"round"});
+ velox::registerFunction<RoundFunction, int16_t, int16_t, int32_t>({"round"});
+ velox::registerFunction<RoundFunction, int32_t, int32_t, int32_t>({"round"});
+ velox::registerFunction<RoundFunction, int64_t, int64_t, int32_t>({"round"});
+ velox::registerFunction<RoundFunction, double, double, int32_t>({"round"});
+ velox::registerFunction<RoundFunction, float, float, int32_t>({"round"});
+}
+
+} // namespace
+
+void registerFunctionOverlay() {
+ registerRoundFunction();
+}
+
+} // namespace gluten
diff --git a/cpp/velox/operators/functions/overlay/RegisterFunctionOverlay.h
b/cpp/velox/operators/functions/overlay/RegisterFunctionOverlay.h
new file mode 100644
index 0000000000..e5627897e0
--- /dev/null
+++ b/cpp/velox/operators/functions/overlay/RegisterFunctionOverlay.h
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+namespace gluten {
+
+/// Registers all functions implemented in Gluten's function overlay
+/// (operators/functions/overlay). The overlay hosts function implementations
+/// managed on the Gluten side, either functions not yet available in Velox or
+/// Gluten-specific overrides of Velox functions. It is registered after all
+/// Velox functions, so a function registered here with the same name and
+/// signature takes precedence over the Velox implementation.
+void registerFunctionOverlay();
+
+} // namespace gluten
diff --git a/cpp/velox/operators/functions/Arithmetic.h
b/cpp/velox/operators/functions/overlay/Round.h
similarity index 99%
rename from cpp/velox/operators/functions/Arithmetic.h
rename to cpp/velox/operators/functions/overlay/Round.h
index 9b2e22ac41..c9487c109b 100644
--- a/cpp/velox/operators/functions/Arithmetic.h
+++ b/cpp/velox/operators/functions/overlay/Round.h
@@ -14,6 +14,9 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
+
+#pragma once
+
#include <folly/CPortability.h>
#include <stdint.h>
#include <cmath>
diff --git a/docs/developers/velox-function-development-guide.md
b/docs/developers/velox-function-development-guide.md
index 6b928bc8af..a73aefe64d 100644
--- a/docs/developers/velox-function-development-guide.md
+++ b/docs/developers/velox-function-development-guide.md
@@ -39,6 +39,33 @@ registerBinaryIntegral<BitwiseAndFunction>({prefix +
"bitwise_and"});
Functions for complex types have similar implementations.
See `ArrayAverageFunction` in
[velox/functions/prestosql/ArrayFunctions.h](https://github.com/facebookincubator/velox/blob/main/velox/functions/prestosql/ArrayFunctions.h).
+## Gluten Function Overlay
+
+Upstreaming a function to Velox can take a long time. To avoid being blocked
on that, Gluten provides a function overlay in
+[cpp/velox/operators/functions/overlay](https://github.com/apache/gluten/tree/main/cpp/velox/operators/functions/overlay),
+where Velox-compatible function implementations can be hosted and managed on
the Gluten side. The overlay is registered by
+`gluten::registerFunctionOverlay()` at the end of
`gluten::registerAllFunctions()`, after all Velox presto/spark functions.
+Since a later registration with the same name and signature replaces the
earlier one in Velox's registries, an overlay function
+takes precedence over the Velox implementation.
+
+Use the overlay when:
+* A Spark function is missing in Velox. Implement it in the overlay first so
Gluten can offload it immediately, then upstream it
+ to Velox at your own pace.
+* A Velox `sparksql` function has a semantic gap with Spark. Put the corrected
implementation in the overlay to override it while
+ the fix is pending upstream. The `round` function (`overlay/Round.h`) is an
example of such an override.
+
+To add a function:
+1. Implement it in a file under `cpp/velox/operators/functions/overlay/`,
using Velox's function authoring APIs (simple function,
+ vector function, aggregate, or window function).
+2. Register it in `overlay/RegisterFunctionOverlay.cc`, using the function
name Gluten's Substrait plan conversion emits.
+3. If a new `.cc` file is added, list it in `cpp/velox/CMakeLists.txt`.
+4. Add C++ unit tests in `cpp/velox/tests/SparkFunctionTest.cc` and Scala side
query tests where applicable.
+5. For a brand-new function, also add the Scala side expression mapping
(`ExpressionNames.scala` / `ExpressionMappings.scala`)
+ so the planner offloads it.
+
+The overlay is a staging area, not a permanent fork: once a function is
accepted into upstream Velox, remove it from the overlay
+in the same PR that bumps the Velox version.
+
### Reference:
Velox’s official developer guide:
*
[velox/docs/develop/scalar-functions.rst](https://github.com/facebookincubator/velox/blob/main/velox/docs/develop/scalar-functions.rst)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
