This is an automated email from the ASF dual-hosted git repository. philo-he pushed a commit to branch branch-1.7 in repository https://gitbox.apache.org/repos/asf/gluten.git
commit 18cec7066b05e4dc228b63880c69fb948a95c149 Author: Philo He <[email protected]> AuthorDate: Thu Aug 6 02:19:33 2026 +0800 [DOC] Polish wording and fix inaccuracies in docs (#12674) --- README.md | 16 +++++----- docs/developers/ProfileMemoryOfGlutenWithVelox.md | 6 ++-- docs/developers/velox-backend-CI.md | 38 +++++++++++------------ docs/get-started/Velox.md | 38 +++++++++++------------ docs/get-started/VeloxLocalCache.md | 25 +++++++++------ docs/velox-backend-limitations.md | 19 ++++-------- docs/velox-backend-scalar-function-support.md | 7 +++-- 7 files changed, 76 insertions(+), 73 deletions(-) diff --git a/README.md b/README.md index ea54568d33..b20be1ce37 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,7 @@ -<img src="docs/image/gluten-logo.svg" alt="Gluten" width="200"> +<img src="docs/image/gluten-logo.svg" alt="Gluten" width="260"> -# Apache Gluten -**A Middle Layer for Offloading JVM-based SQL Engines' Execution to Native Engines** +**A Middle Layer for Offloading JVM-Based SQL Execution to Native Engines** [](https://www.bestpractices.dev/projects/8452) @@ -27,7 +26,7 @@ data processing, these engines can outperform Spark’s JVM-based SQL engine. Ho The basic design principle is to reuse Spark’s control flow, while offloading compute-intensive data processing to the native side. More specifically: -* Transform Spark’s physical plan to Substrait plan, then transform it to native engine's plan. +* Transform Spark’s physical plan into a Substrait plan, then transform it into the native engine's plan. * Offload performance-critical data processing to native engine. * Define clear JNI interfaces for native SQL engines. * Allow easy switching between available native backends. @@ -47,7 +46,7 @@ and Spark’s Columnar API (introduced in Spark 3.0) is used during execution. G <p align="center"> <img src="https://user-images.githubusercontent.com/47296334/199617207-1140698a-4d53-462d-9bc7-303d14be060b.png"; width="700"> </p> -Currently, Gluten supports only ClickHouse and Velox backends. Velox is a C++ database acceleration library which provides reusable, extensible and high-performance data processing components. In addition, Gluten is designed to be extensible, +Currently, Gluten supports only the ClickHouse and Velox backends. Velox is a C++ database acceleration library that provides reusable, extensible, and high-performance data processing components. In addition, Gluten is designed to be extensible, allowing support for additional backends in the future. Gluten's key components: @@ -79,7 +78,7 @@ There are two ways to acquire Gluten jar for the above configuration. ### Use Released JAR -Please download the tar package [here](https://gluten.apache.org/downloads/), then extract Gluten JAR from it. +Please download the tar package [here](https://gluten.apache.org/downloads/), then extract the Gluten JAR from it. Additionally, Gluten provides nightly builds based on the main branch for early testing. The nightly build JARs are available at [Apache Gluten Nightlies](https://nightlies.apache.org/gluten/). They have been verified on Centos 7/8/9, Ubuntu 20.04/22.04. @@ -113,7 +112,7 @@ Welcome to contribute to the Gluten project! See [CONTRIBUTING.md](CONTRIBUTING. ## 6. Community -Gluten successfully became an Apache Incubator project in March 2024 and graduated as an Apache Top-Level Project in March 2026. Here are several ways to connect with the community. +Here are several ways to connect with the community. ### GitHub @@ -164,7 +163,8 @@ Gluten is licensed under [Apache License Version 2.0](https://www.apache.org/lic ## 10. Acknowledgements -Gluten was initiated by Intel and Kyligence in 2022. Several other companies are also actively contributing to its development, including BIGO, Meituan, Alibaba Cloud, NetEase, Baidu, Microsoft, IBM, Google, etc. +Gluten was initiated by Intel and Kyligence in 2022 and became an Apache Top-Level Project in March 2026. +Several other companies are also actively contributing to its development, including BIGO, Meituan, Alibaba Cloud, NetEase, Baidu, Microsoft, IBM, Google, and others. <a href="https://github.com/apache/gluten/graphs/contributors";> <img src="https://contrib.rocks/image?repo=apache/gluten&columns=25"; /> diff --git a/docs/developers/ProfileMemoryOfGlutenWithVelox.md b/docs/developers/ProfileMemoryOfGlutenWithVelox.md index 4705e4f592..82eda3fa49 100644 --- a/docs/developers/ProfileMemoryOfGlutenWithVelox.md +++ b/docs/developers/ProfileMemoryOfGlutenWithVelox.md @@ -113,10 +113,12 @@ spark.files /path/to/jemalloc/libjemalloc.so Example of enabling heap profile in spark executor: ``` -spark.executorEnv.LD_PRELOAD ./libjemalloc.so -spark.executorEnv.MALLOC_CONF prof:true,lg_prof_interval:30,prof_prefix:/tmp/gluten_heap_perf +spark.executorEnv.LD_PRELOAD /path/to/jemalloc/libjemalloc.so +spark.executorEnv.MALLOC_CONF prof:true,prof_final:true,lg_prof_interval:30,prof_prefix:/tmp/gluten_heap_perf ``` +Note: `lg_prof_interval:30` only dumps a profile after every 2^30 bytes (1GB) of cumulative allocation activity, so a short-running or low-allocation job may never trigger it. `prof_final:true` ensures a profile is dumped when the process exits, regardless of the interval. + Finally, profiling files prefixed with `/tmp/gluten_heap_perf.${PID}` will be generated for each spark executor. ## Memory dump on spark executor exit diff --git a/docs/developers/velox-backend-CI.md b/docs/developers/velox-backend-CI.md index a0af0b0f39..d35e5d9b31 100644 --- a/docs/developers/velox-backend-CI.md +++ b/docs/developers/velox-backend-CI.md @@ -6,36 +6,36 @@ parent: Developer Overview --- # Velox Backend CI -GHA workflows are defined under `.github/workflows/`. +GitHub Actions (GHA) workflows are defined under `.github/workflows/`. ## Docker Build -We have a weekly job defined in `docker_image.yml` to build docker images for CI verification. The docker files and images are listed below: +A weekly job defined in `docker_image.yml` builds the Docker images used for CI verification. The Dockerfiles (under `dev/docker/`) and their corresponding images are listed below: file | images | comments -- | -- | -- -dev/docker/Dockerfile.centos7-static-build | apache/gluten:vcpkg-centos-7 | centos 7, static link, jdk8 -dev/docker/Dockerfile.centos8-static-build | apache/gluten:vcpkg-centos-8 | centos 8, static link, jdk8 -dev/docker/Dockerfile.centos8-dynamic-build | apache/gluten:centos-8-jdk8 | centos 8, dynamic link, jdk8 -dev/docker/Dockerfile.centos8-dynamic-build | apache/gluten:centos-8-jdk11 | centos 8, dynamic link, jdk11 -dev/docker/Dockerfile.centos8-dynamic-build | apache/gluten:centos-8-jdk17 | centos 8, dynamic link, jdk17 -dev/docker/cudf/Dockerfile | apache/gluten:centos-9-jdk8-cudf | centos 9, dynamic link, jdk8 +Dockerfile.centos7-gcc13-static-build | apache/gluten:vcpkg-centos-7-gcc13 | centos 7, static link, jdk8 +Dockerfile.centos8-gcc13-static-build | apache/gluten:vcpkg-centos-8-gcc13 | centos 8, static link, jdk8 +Dockerfile.centos8-dynamic-build | apache/gluten:centos-8-jdk8 | centos 8, dynamic link, jdk8 +Dockerfile.centos8-dynamic-build | apache/gluten:centos-8-jdk11 | centos 8, dynamic link, jdk11 +Dockerfile.centos8-dynamic-build | apache/gluten:centos-8-jdk17 | centos 8, dynamic link, jdk17 +cudf/Dockerfile | apache/gluten:centos-9-jdk8-cudf | centos 9, dynamic link, jdk8 -Docker images can be found from https://hub.docker.com/r/apache/gluten/tags +The Docker images can be found at [https://hub.docker.com/r/apache/gluten/tags](https://hub.docker.com/r/apache/gluten/tags). ## Vcpkg Caching -Gluten main branch is pulled down during static build in docker. And vcpkg will cache binary data of all dependencies defined under dev/vcpkg. -These binary data is cached into `/var/cache/vcpkg` and CI job can re-use them in new build. By setting `VCPKG_BINARY_SOURCES=clear` in env., -reusing vcpkg cache can be disabled. +The Gluten main branch is pulled during the static build in Docker, and vcpkg caches binary data for all dependencies defined under `dev/vcpkg`. +This binary data is cached into `/var/cache/vcpkg`, and CI jobs can reuse it in later builds. Setting `VCPKG_BINARY_SOURCES=clear` in the +environment disables reuse of the vcpkg cache. ## Arrow Libs Pre-installation -Arrow libs are pre-installed in docker, assuming they are not actively changed, then not necessarily to be re-built every time. +Arrow libs are pre-installed in the Docker image, since they don't change often and don't need to be rebuilt on every run. -## .M2 cache -The dependency libraries are pre installed in to /root/.m2 by `mvn dependency:go-offline` Spark is set to 3.5 by default. +## .M2 Cache +Dependency libraries are pre-installed into `/root/.m2` via `mvn dependency:go-offline`. Spark is set to 3.5 by default. ## Ccache -Since the docker image is rebuilt weekly, the ccache mostly are outdated. So the cache is removed from the image. +Since the Docker image is rebuilt weekly, the ccache is mostly outdated, so it is removed from the image. -## Updating Docker Image -Two GitHub secrets `DOCKERHUB_USER` & `DOCKERHUB_TOKEN` can be used to push docker image to docker hub: https://hub.docker.com/r/apache/gluten/tags. -Note GitHub secrets are not retrievable in PR from forked repo. \ No newline at end of file +## Updating the Docker Image +The GitHub secrets `DOCKERHUB_USER` and `DOCKERHUB_TOKEN` are used to push Docker images to [Docker Hub](https://hub.docker.com/r/apache/gluten/tags). +Note that GitHub secrets are not accessible in PRs from forked repos. \ No newline at end of file diff --git a/docs/get-started/Velox.md b/docs/get-started/Velox.md index 427a774958..4e6060d6f7 100644 --- a/docs/get-started/Velox.md +++ b/docs/get-started/Velox.md @@ -16,7 +16,7 @@ parent: Getting-Started # Prerequisite -Currently, with static build Gluten+Velox backend supports all the Linux OSes, but is only tested on **Ubuntu20.04/Ubuntu22.04/Centos7/Centos8**. With dynamic build, Gluten+Velox backend support **Ubuntu20.04/Ubuntu22.04/Centos7/Centos8** and their variants. +Currently, the statically built Gluten+Velox backend supports all Linux OSes but is only tested on **Ubuntu 20.04/22.04/CentOS 7/8**. The dynamically built backend supports **Ubuntu 20.04/22.04/CentOS 7/8** and their variants. Currently, the officially supported Spark versions are 3.2.2, 3.3.1, 3.4.4 and 3.5.5. @@ -77,8 +77,8 @@ export CPU_TARGET="aarch64" **Step-by-step build** -Alternative to the above one-step build, you can follow the below guide for step-by-step build. -Currently, Gluten is using a [IBM Velox](https://github.com/IBM/velox) which is daily updated based on [upstream Velox](https://github.com/facebookincubator/velox). +As an alternative to the above one-step build, you can follow the step-by-step guide below. +Currently, Gluten uses an [IBM Velox fork](https://github.com/IBM/velox), which is updated daily based on [upstream Velox](https://github.com/facebookincubator/velox). ```bash @@ -106,7 +106,7 @@ mvn clean package -Pbackends-velox -Pspark-4.1 -Pjava-17 -Pscala-2.13 -DskipTest Notes： Building Velox may fail caused by OOM. You can prevent this failure by adjusting `NUM_THREADS` (e.g., `export NUM_THREADS=4`) before building Gluten/Velox. The recommended minimal memory size is 64G. -After the above build process, the Jar file will be generated under `package/target/`. +After the above build process, the jar file will be generated under `package/target/`. Alternatively you may refer to [build in docker](../developers/velox-backend-build-in-docker.md) to build the gluten jar in docker. @@ -149,8 +149,9 @@ This enables OpenSSL with FIPS features and requires OpenSSL shared libraries to ## HDFS support -Gluten supports dynamically loading both libhdfs.so and libhdfs3.so at runtime by using dlopen, allowing the JVM to load the appropriate shared library file as needed. This means you do not need to set the library path during the compilation phase. -To enable this functionality, you must set the JAVA_HOME and HADOOP_HOME environment variables. Gluten will then locate and load the ${HADOOP_HOME}/lib/native/libhdfs.so file at runtime. If you prefer to use libhdfs3.so instead, simply replace the ${HADOOP_HOME}/lib/native/libhdfs.so file with libhdfs3.so. +Gluten supports dynamically loading libhdfs.so or libhdfs3.so at runtime via dlopen, so the library path does not need to be set at compile time. +To enable this, set the JAVA_HOME and HADOOP_HOME environment variables. Gluten will then locate and load `${HADOOP_HOME}/lib/native/libhdfs.so` at runtime. +To use libhdfs3.so instead, replace that path with libhdfs3.so. ### Build libhdfs3 @@ -380,13 +381,13 @@ Gluten with velox backend supports [Iceberg](https://iceberg.apache.org/) table. ### How to use -First of all, compile gluten-iceberg module by a `iceberg` profile, as follows: +First, compile the gluten-iceberg module with the `iceberg` profile, as follows: ``` mvn clean package -Pbackends-velox -Pspark-3.3 -Piceberg -DskipTests ``` -Once built successfully, iceberg features will be included in gluten-velox-bundle-X jar. Then you can query iceberg table by gluten/velox without scan's fallback. +Once built successfully, iceberg features will be included in the gluten-velox-bundle-X jar. You can then query iceberg tables via Gluten/Velox without falling back on scan. ## Hudi Support @@ -404,17 +405,17 @@ Compile gluten-paimon module by a `paimon` profile, as follows: mvn clean package -Pbackends-velox -Pspark-3.5 -Ppaimon -DskipTests ``` -Once built successfully, paimon features will be included in gluten-velox-bundle-X jar. Then you can query paimon non-pk table by gluten/velox without scan's fallback. +Once built successfully, paimon features will be included in the gluten-velox-bundle-X jar. You can then query paimon non-pk tables via Gluten/Velox without falling back on scan. ### How to use -First of all, compile gluten-hudi module by a `hudi` profile, as follows: +First, compile the gluten-hudi module with the `hudi` profile, as follows: ``` mvn clean package -Pbackends-velox -Pspark-3.3 -Phudi -DskipTests ``` -Once built successfully, hudi features will be included in gluten-velox-bundle-X jar. Then you can query hudi **COW** table by gluten/velox without scan's fallback. +Once built successfully, hudi features will be included in the gluten-velox-bundle-X jar. You can then query hudi **COW** tables via Gluten/Velox without falling back on scan. # Coverage @@ -422,7 +423,7 @@ Spark3.5 has 400+ functions in total. ~240 are commonly used. To get the support > Velox doesn't support [ANSI > mode](https://spark.apache.org/docs/latest/sql-ref-ansi-compliance.html)), > so as Gluten. Once ANSI mode is enabled in Spark config, Gluten will > fallback to Vanilla Spark. -To identify what can be offloaded in a query and detailed fallback reasons, user can follow below steps to retrieve corresponding logs. +To identify what can be offloaded in a query and see the detailed fallback reasons, follow the steps below to retrieve the corresponding logs. ``` 1) Enable Gluten by proper [configuration](https://github.com/apache/gluten/blob/main/docs/Configuration.md). @@ -453,8 +454,8 @@ With above steps, you will get a physical plan output like: native validation failed due to: in ProjectRel, Scalar function name not registered: get_struct_field, called with arguments: (ROW<col_0:INTEGER,col_1:BIGINT,col_2:BIGINT>, INTEGER). ``` -In the above, the symbol `^` indicates a plan is offloaded to Velox in a stage. In Spark DAG, all such pipelined plans (consecutive plans marked with `^`) are plotted -inside an umbrella node named `WholeStageCodegenTransformer` (It's not codegen node. The naming is just for making it well plotted like Spark Whole Stage Codegen). +In the above, the symbol `^` indicates a plan is offloaded to Velox in a stage. In the Spark DAG, all such pipelined plans (consecutive plans marked with `^`) are plotted +inside an umbrella node named `WholeStageCodegenTransformer`. Despite the name, this is not a codegen node; it's named this way simply to display similarly to Spark's Whole Stage Codegen. # Spill @@ -533,8 +534,7 @@ Refer to [Gluten configuration](../Configuration.md) for more details. ## Performance -Below table shows the TPC-H Q1 and Q6 Performance in a multiple-thread test (--num-executors 6 --executor-cores 6) for Velox and vanilla Spark. -Both Parquet and ORC datasets are sf1024. +The table below shows TPC-H Q1 and Q6 performance for Velox and vanilla Spark in a multi-thread test (`--num-executors 6 --executor-cores 6`), using scale factor 1024 datasets in both Parquet and ORC formats. | Query Performance (s) | Velox (ORC) | Vanilla Spark (Parquet) | Vanilla Spark (ORC) | |---------------- | ----------- | ------------- | ------------- | @@ -562,8 +562,8 @@ cause the inconsistent results with a materialized query. However, we have no ch ## Native Plan in Spark's Explain Output -Gluten supports inject native plan string into Spark explain with formatted mode by setting `--conf spark.gluten.sql.injectNativePlanStringToExplain=true`. -Here is an example, how Gluten shows the native plan string. +Gluten supports injecting the native plan string into Spark's explain output in formatted mode by setting `--conf spark.gluten.sql.injectNativePlanStringToExplain=true`. +Here is an example of how Gluten shows the native plan string. ``` (9) WholeStageCodegenTransformer (2) @@ -610,7 +610,7 @@ To enable this feature, you can set the following Spark configuration: | Property | Default | Description | |-------------------------------------------------------------|---------|-------------------------------------------------------------------| -| `spark.gluten.velox.offHeapBroadcastBuildRelation.enabled` | `false` | Enable/disable off-heap storage for broadcast build relations. | +| `spark.gluten.velox.offHeapBroadcastBuildRelation.enabled` | `false` | Whether to store broadcast build relations off-heap. | This feature has been tested through a series of tests, and we are collecting more feedback from users. If you have memory problem on broadcast build relations, please try this feature and give more feedbacks. diff --git a/docs/get-started/VeloxLocalCache.md b/docs/get-started/VeloxLocalCache.md index 1c7c40ced0..93a4d8d0f5 100644 --- a/docs/get-started/VeloxLocalCache.md +++ b/docs/get-started/VeloxLocalCache.md @@ -5,16 +5,23 @@ nav_order: 7 parent: Getting-Started --- -Velox supports a local cache when reading data from HDFS/S3/ABFS. With this feature, Velox can asynchronously cache the data on local disk when reading from remote storage and future read requests on previously cached blocks will be serviced from local cache files. To enable the local caching feature, the following configurations are required: +Velox supports local caching when reading data from HDFS/S3/ABFS. With this feature, Velox asynchronously caches data on +local disk when reading from remote storage, and future read requests for previously cached blocks are served from the +local cache files. To enable local caching, the following configurations are required: ``` -spark.gluten.sql.columnar.backend.velox.cacheEnabled // enable or disable velox cache, default false. -spark.gluten.sql.columnar.backend.velox.memCacheSize // the total size of in-mem cache, default is 128MB. -spark.gluten.sql.columnar.backend.velox.ssdCachePath // the folder to store the cache files, default is "/tmp". -spark.gluten.sql.columnar.backend.velox.ssdCacheSize // the total size of the SSD cache, default is 128MB. Velox will do in-mem cache only if this value is 0. -spark.gluten.sql.columnar.backend.velox.ssdCacheShards // the shards of the SSD cache, default is 1. -spark.gluten.sql.columnar.backend.velox.ssdCacheIOThreads // the IO threads for cache promoting, default is 1. Velox will try to do "read-ahead" if this value is bigger than 1 -spark.gluten.sql.columnar.backend.velox.ssdODirect // enable or disable O_DIRECT on cache write, default false. +spark.gluten.sql.columnar.backend.velox.cacheEnabled // Enable Velox cache. Default: false. +spark.gluten.sql.columnar.backend.velox.memCacheSize // In-memory cache size. Default: 128MB. +spark.gluten.sql.columnar.backend.velox.ssdCachePath // Folder to store cache files, preferably on SSD. Default: "/tmp". +spark.gluten.sql.columnar.backend.velox.ssdCacheSize // SSD cache size. Memory-only caching is used when set to 0. Default: 128MB. +spark.gluten.sql.columnar.backend.velox.ssdCacheShards // Number of SSD cache shards. Default: 1. +spark.gluten.sql.columnar.backend.velox.ssdCacheIOThreads // Number of IO threads for SSD cache read/write. Enables read-ahead when > 1. Default: 4. +spark.gluten.sql.columnar.backend.velox.loadQuantum // Load quantum size. Must be at most 8MB when Velox cache is enabled, otherwise Velox fails. Default: 256MB. +spark.gluten.sql.columnar.backend.velox.ssdODirect // Enable O_DIRECT on cache write. Currently causes SSD cache writes to fail if enabled; see Velox issue #10597. Default: false. +spark.gluten.soft-affinity.enabled // Enable Soft Affinity scheduling. Should be enabled together with Velox cache. Default: false. ``` -It's recommended to mount SSDs to the cache path to get the best performance of local caching. Cache files will be written to "spark.gluten.sql.columnar.backend.velox.cachePath", with UUID based suffix, e.g. "/tmp/cache.13e8ab65-3af4-46ac-8d28-ff99b2a9ec9b0". Gluten cannot reuse older caches for now, and the old cache files are left after Spark context shutdown. +It's recommended to mount SSDs to the cache path to get the best performance of local caching. Cache files will be written +to "spark.gluten.sql.columnar.backend.velox.ssdCachePath", with UUID based suffix, +e.g. "/tmp/cache.13e8ab65-3af4-46ac-8d28-ff99b2a9ec9b0". Gluten cannot reuse older caches for now, and the old cache files +are left after Spark context shutdown. diff --git a/docs/velox-backend-limitations.md b/docs/velox-backend-limitations.md index 9dc830fb33..01130bde3d 100644 --- a/docs/velox-backend-limitations.md +++ b/docs/velox-backend-limitations.md @@ -146,17 +146,10 @@ Gluten's. Only reading with INT96 and dictionary encoding is supported. When reading INT64 represented millisecond/microsecond timestamps, or INT96 represented timestamps of other encodings, exceptions can occur. -- Complex types - - Parquet scan of nested array with struct or array as element type is not supported in Velox (fallback behavior). - - Parquet scan of nested map with struct as key type, or array type as value type is not supported in Velox (fallback behavior). - -### CSV Read -The header option should be true. And now we only support DatasourceV1, i.e., user should set `spark.sql.sources.useV1SourceList=csv`. User defined read option is not supported, which will make CSV read fall back to vanilla Spark in most case. -CSV read will also fall back to vanilla Spark and log warning when user specifies schema is different with file schema. - ### Utilizing Map Type as Hash Keys in ColumnarShuffleExchange -Spark uses the `spark.sql.legacy.allowHashOnMapType` configuration to support hash map key functions. -Gluten enables this configuration during the creation of ColumnarShuffleExchange, as shown in the code [link](https://github.com/apache/gluten/blob/0dacac84d3bf3d2759a5dd7e0735147852d2845d/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxSparkPlanExecApi.scala#L355-L363). -This method bypasses Spark's unresolved checks and creates projects with the hash(mapType) operator before ColumnarShuffleExchange. -However, if `spark.sql.legacy.allowHashOnMapType` is disabled in a test environment, projects using the hash(mapType) expression may throw an `Invalid call to dataType on unresolved object` exception during validation, causing them to fallback to vanilla Spark, as referenced in the code [link](https://github.com/apache/spark/blob/de5fa426e23b84fc3c2bddeabcd2e1eda515abd5/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala#L291-L296). - Enabling this configuration allows the project to be offloaded to Velox. +Spark's `spark.sql.legacy.allowHashOnMapType` configuration controls whether hashing is allowed on map-type keys. +Gluten enables this configuration when creating `ColumnarShuffleExchange`, as shown [here](https://github.com/apache/gluten/blob/0dacac84d3bf3d2759a5dd7e0735147852d2845d/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxSparkPlanExecApi.scala#L355-L363). +This bypasses Spark's unresolved-expression checks and lets projects using the `hash(mapType)` operator be created before `ColumnarShuffleExchange`. +However, if `spark.sql.legacy.allowHashOnMapType` is disabled in a test environment, projects using the `hash(mapType)` expression may throw an +`Invalid call to dataType on unresolved object` exception during validation, causing them to fall back to vanilla Spark, as referenced [here](https://github.com/apache/spark/blob/de5fa426e23b84fc3c2bddeabcd2e1eda515abd5/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala#L291-L296). +Enabling this configuration allows the project to be offloaded to Velox. diff --git a/docs/velox-backend-scalar-function-support.md b/docs/velox-backend-scalar-function-support.md index e16baf3701..f86af55484 100644 --- a/docs/velox-backend-scalar-function-support.md +++ b/docs/velox-backend-scalar-function-support.md @@ -1,6 +1,6 @@ # Scalar Functions Support Status -**Out of 357 scalar functions in Spark 3.5, Gluten currently fully supports 242 functions and partially supports 25 functions.** +**Out of 357 scalar functions in Spark 3.5, Gluten currently fully supports 245 functions and partially supports 27 functions.** ## Array Functions @@ -220,6 +220,7 @@ ## Mathematical Functions + | Spark Functions | Spark Expressions | Status | Restrictions | |-------------------|------------------------|----------|----------------| | % | Remainder | S | | @@ -281,8 +282,8 @@ | sin | Sin | | | | sinh | Sinh | S | | | sqrt | Sqrt | S | | -| tan | Tan | | | -| tanh | Tanh | | | +| tan | Tan | S | | +| tanh | Tanh | S | | | try_add | TryAdd | PS | | | try_divide | TryDivide | | | | try_multiply | TryMultiply | | | --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
![https://user-images.githubusercontent.com/47296334/199617207-1140698a-4d53-462d-9bc7-303d14be060b.png"](https://user-images.githubusercontent.com/47296334/199617207-1140698a-4d53-462d-9bc7-303d14be060b.png"){kind=link}