pedrumj2 opened a new issue, #13273: URL: https://github.com/apache/gluten/issues/13273
### Description ### Backend VL (Velox) ### Description A UDF loaded through `udfLibraryPaths` and called by its own name (apache/gluten#13014, apache/gluten#13016) fails as soon as it takes or returns a named struct: - Passing a named struct to a struct argument is rejected as `not registered`. - Selecting a field of a returned struct fails analysis with `FIELD_NOT_FOUND`. The `FIELD_NOT_FOUND` error lists the struct's fields with empty names: the struct types in the UDF's signature reach Spark without their field names. apache/gluten#1878 added field names to the struct types Gluten sends to Velox, but a UDF signature travels the other way, from Velox back to Spark. **Expected:** the first query in the reproduction returns `3` and `7`, and the second returns `2` and `3`. **Actual:** both fail with the errors under Relevant logs. This issue was written with the assistance of AI: co-authored with Claude Code (claude-opus-5-5). ### Gluten version main branch, reproduced at 0259f981258c3ce906c4fab643a7dfdb03d96514 ### Spark version Spark-3.5.x (3.5.5) ### Spark configurations ``` spark.plugins=org.apache.gluten.GlutenPlugin spark.memory.offHeap.enabled=true spark.memory.offHeap.size=2g spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager spark.gluten.sql.columnar.backend.velox.udfLibraryPaths=file:///work/gluten/cpp/build/velox/udf/examples/libmyudf.so spark.gluten.sql.columnar.backend.velox.driver.udfLibraryPaths=file:///work/gluten/cpp/build/velox/udf/examples/libmyudf.so spark.gluten.sql.columnar.backend.velox.nativeUDF.bypassRegistration=true ``` ### Reproduction Everything runs in the `apache/gluten:centos-9-jdk8` image. Run each block from the same empty directory. **1. Get the source and Spark** ```bash git clone https://github.com/apache/gluten.git git -C gluten checkout 0259f981258c3ce906c4fab643a7dfdb03d96514 curl -LO https://archive.apache.org/dist/spark/spark-3.5.5/spark-3.5.5-bin-hadoop3.tgz tar -xzf spark-3.5.5-bin-hadoop3.tgz ``` **2. Add two example UDFs.** `myudf_sum_row` takes a `struct<a:bigint,b:bigint>`, and `myudf_make_row` returns one. <details> <summary>Apply the patch to <code>cpp/velox/udf/examples/MyUDF.cc</code></summary> ```bash cat > myudf.patch <<'EOF' diff --git a/cpp/velox/udf/examples/MyUDF.cc b/cpp/velox/udf/examples/MyUDF.cc index 687cdd2d3..b070a4156 100644 --- a/cpp/velox/udf/examples/MyUDF.cc +++ b/cpp/velox/udf/examples/MyUDF.cc @@ -31,6 +31,7 @@ static const char* kInteger = "int"; static const char* kBigInt = "bigint"; static const char* kDate = "date"; static const char* kVarChar = "varchar"; +static const char* kStructAB = "struct<a:bigint,b:bigint>"; namespace hivestringstring { @@ -108,6 +109,84 @@ class MyUdfPlusOneRegisterer final : public gluten::UdfRegisterer { } // namespace myudfplusone +namespace myudfsumrow { + +template <typename T> +struct MyUdfSumRowFunction { + VELOX_DEFINE_FUNCTION_TYPES(T); + + FOLLY_ALWAYS_INLINE bool call(int64_t& result, const arg_type<Row<int64_t, int64_t>>& row) { + const auto a = row.template at<0>(); + const auto b = row.template at<1>(); + if (!a.has_value() || !b.has_value()) { + return false; + } + result = a.value() + b.value(); + return true; + } +}; + +// name: myudf_sum_row +// signatures: +// struct<a:bigint,b:bigint> -> bigint +// type: SimpleFunction +class MyUdfSumRowRegisterer final : public gluten::UdfRegisterer { + public: + int getNumUdf() override { + return 1; + } + + void populateUdfEntries(int& index, gluten::UdfEntry* udfEntries) override { + udfEntries[index++] = {name_.c_str(), kBigInt, 1, arg_, false, false}; + } + + void registerSignatures() override { + facebook::velox::registerFunction<MyUdfSumRowFunction, int64_t, Row<int64_t, int64_t>>({name_}); + } + + private: + const std::string name_ = "myudf_sum_row"; + const char* arg_[1] = {kStructAB}; +}; + +} // namespace myudfsumrow + +namespace myudfmakerow { + +template <typename T> +struct MyUdfMakeRowFunction { + VELOX_DEFINE_FUNCTION_TYPES(T); + + FOLLY_ALWAYS_INLINE void call(out_type<Row<int64_t, int64_t>>& result, const int64_t& a) { + result = std::make_tuple(a, a + 1); + } +}; + +// name: myudf_make_row +// signatures: +// bigint -> struct<a:bigint,b:bigint> +// type: SimpleFunction +class MyUdfMakeRowRegisterer final : public gluten::UdfRegisterer { + public: + int getNumUdf() override { + return 1; + } + + void populateUdfEntries(int& index, gluten::UdfEntry* udfEntries) override { + udfEntries[index++] = {name_.c_str(), kStructAB, 1, arg_, false, false}; + } + + void registerSignatures() override { + facebook::velox::registerFunction<MyUdfMakeRowFunction, Row<int64_t, int64_t>, int64_t>({name_}); + } + + private: + const std::string name_ = "myudf_make_row"; + const char* arg_[1] = {kBigInt}; +}; + +} // namespace myudfmakerow + std::vector<std::shared_ptr<gluten::UdfRegisterer>>& globalRegisters() { static std::vector<std::shared_ptr<gluten::UdfRegisterer>> registerers; return registerers; @@ -121,6 +200,8 @@ void setupRegisterers() { auto& registerers = globalRegisters(); registerers.push_back(std::make_shared<hivestringstring::HiveStringStringRegisterer>()); registerers.push_back(std::make_shared<myudfplusone::MyUdfPlusOneRegisterer>()); + registerers.push_back(std::make_shared<myudfsumrow::MyUdfSumRowRegisterer>()); + registerers.push_back(std::make_shared<myudfmakerow::MyUdfMakeRowRegisterer>()); inited = true; } } // namespace EOF git -C gluten apply ../myudf.patch ``` </details> **3. Write the queries** ```bash cat > repro.sql <<'EOF' SELECT myudf_sum_row(named_struct('a', a, 'b', b)) AS sum_ab FROM VALUES (1L, 2L), (3L, 4L) AS t(a, b); SELECT myudf_make_row(id).b AS b FROM VALUES (1L), (2L) AS t(id); EXPLAIN SELECT myudf_make_row(id).b AS b FROM VALUES (1L) AS t(id); EOF ``` **4. Build Gluten with the UDF examples** ```bash docker run --rm -v "$PWD":/work:z -w /work/gluten apache/gluten:centos-9-jdk8 bash -c ' source /opt/rh/gcc-toolset-12/enable export VELOX_BUILD_SHARED=ON ./dev/builddeps-veloxbe.sh --build_examples=ON --build_arrow=OFF --run_setup_script=OFF ./build/mvn clean install -Pbackends-velox -Pspark-3.5 -DskipTests' ``` **5. Run the queries** ```bash docker run --rm -v "$PWD":/work:z -w /work apache/gluten:centos-9-jdk8 \ spark-3.5.5-bin-hadoop3/bin/spark-sql --master 'local[2]' \ --jars gluten/package/target/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.8.0-SNAPSHOT.jar \ --driver-class-path gluten/package/target/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.8.0-SNAPSHOT.jar \ --conf spark.plugins=org.apache.gluten.GlutenPlugin \ --conf spark.memory.offHeap.enabled=true \ --conf spark.memory.offHeap.size=2g \ --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \ --conf spark.gluten.sql.columnar.backend.velox.udfLibraryPaths=file:///work/gluten/cpp/build/velox/udf/examples/libmyudf.so \ --conf spark.gluten.sql.columnar.backend.velox.driver.udfLibraryPaths=file:///work/gluten/cpp/build/velox/udf/examples/libmyudf.so \ --conf spark.gluten.sql.columnar.backend.velox.nativeUDF.bypassRegistration=true \ --hiveconf hive.cli.errors.ignore=true \ -f repro.sql ``` ### Relevant logs ``` 26/10/13 19:44:41 ERROR SparkSQLDriver: Failed in [SELECT myudf_sum_row(named_struct('a', a, 'b', b)) AS sum_ab FROM VALUES (1L, 2L), (3L, 4L) AS t(a, b)] org.apache.gluten.exception.GlutenNotSupportException: UDF myudf_sum_row -> struct<a:bigint,b:bigint> is not registered. at org.apache.spark.sql.expression.UDFResolver$.getUdfExpression(UDFResolver.scala:407) at org.apache.spark.sql.expression.UDFResolver$.$anonfun$getFunctionDescriptions$10(UDFResolver.scala:385) ... [FIELD_NOT_FOUND] No such struct field `b` in ``, ``. ... == Physical Plan == org.apache.spark.sql.AnalysisException: [FIELD_NOT_FOUND] No such struct field `b` in ``, ``. ``` ### Gluten version main branch -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
