This is an automated email from the ASF dual-hosted git repository.
philo-he pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gluten.git
The following commit(s) were added to refs/heads/main by this push:
new cfbc08e48e [DOC] Polish wording and fix inaccuracies in docs (#12675)
cfbc08e48e is described below
commit cfbc08e48ee40ae3175c66ff653869c09190520b
Author: Philo He <[email protected]>
AuthorDate: Thu Aug 6 02:19:33 2026 +0800
[DOC] Polish wording and fix inaccuracies in docs (#12675)
---
README.md | 16 +++++-----
docs/developers/ProfileMemoryOfGlutenWithVelox.md | 6 ++--
docs/developers/velox-backend-CI.md | 38 +++++++++++------------
docs/get-started/Velox.md | 38 +++++++++++------------
docs/get-started/VeloxLocalCache.md | 25 +++++++++------
docs/velox-backend-limitations.md | 19 ++++--------
docs/velox-backend-scalar-function-support.md | 7 +++--
7 files changed, 76 insertions(+), 73 deletions(-)
diff --git a/README.md b/README.md
index ea54568d33..b20be1ce37 100644
--- a/README.md
+++ b/README.md
@@ -1,8 +1,7 @@
-<img src="docs/image/gluten-logo.svg" alt="Gluten" width="200">
+<img src="docs/image/gluten-logo.svg" alt="Gluten" width="260">
-# Apache Gluten
-**A Middle Layer for Offloading JVM-based SQL Engines' Execution to Native
Engines**
+**A Middle Layer for Offloading JVM-Based SQL Execution to Native Engines**
[](https://www.bestpractices.dev/projects/8452)
@@ -27,7 +26,7 @@ data processing, these engines can outperform Spark’s
JVM-based SQL engine. Ho
The basic design principle is to reuse Spark’s control flow, while offloading
compute-intensive data processing to the native side. More specifically:
-* Transform Spark’s physical plan to Substrait plan, then transform it to
native engine's plan.
+* Transform Spark’s physical plan into a Substrait plan, then transform it
into the native engine's plan.
* Offload performance-critical data processing to native engine.
* Define clear JNI interfaces for native SQL engines.
* Allow easy switching between available native backends.
@@ -47,7 +46,7 @@ and Spark’s Columnar API (introduced in Spark 3.0) is used
during execution. G
<p align="center">
<img
src="https://user-images.githubusercontent.com/47296334/199617207-1140698a-4d53-462d-9bc7-303d14be060b.png";
width="700">
</p>
-Currently, Gluten supports only ClickHouse and Velox backends. Velox is a C++
database acceleration library which provides reusable, extensible and
high-performance data processing components. In addition, Gluten is designed to
be extensible,
+Currently, Gluten supports only the ClickHouse and Velox backends. Velox is a
C++ database acceleration library that provides reusable, extensible, and
high-performance data processing components. In addition, Gluten is designed to
be extensible,
allowing support for additional backends in the future.
Gluten's key components:
@@ -79,7 +78,7 @@ There are two ways to acquire Gluten jar for the above
configuration.
### Use Released JAR
-Please download the tar package [here](https://gluten.apache.org/downloads/),
then extract Gluten JAR from it.
+Please download the tar package [here](https://gluten.apache.org/downloads/),
then extract the Gluten JAR from it.
Additionally, Gluten provides nightly builds based on the main branch for
early testing. The nightly build JARs are available at [Apache Gluten
Nightlies](https://nightlies.apache.org/gluten/).
They have been verified on Centos 7/8/9, Ubuntu 20.04/22.04.
@@ -113,7 +112,7 @@ Welcome to contribute to the Gluten project! See
[CONTRIBUTING.md](CONTRIBUTING.
## 6. Community
-Gluten successfully became an Apache Incubator project in March 2024 and
graduated as an Apache Top-Level Project in March 2026. Here are several ways
to connect with the community.
+Here are several ways to connect with the community.
### GitHub
@@ -164,7 +163,8 @@ Gluten is licensed under [Apache License Version
2.0](https://www.apache.org/lic
## 10. Acknowledgements
-Gluten was initiated by Intel and Kyligence in 2022. Several other companies
are also actively contributing to its development, including BIGO, Meituan,
Alibaba Cloud, NetEase, Baidu, Microsoft, IBM, Google, etc.
+Gluten was initiated by Intel and Kyligence in 2022 and became an Apache
Top-Level Project in March 2026.
+Several other companies are also actively contributing to its development,
including BIGO, Meituan, Alibaba Cloud, NetEase, Baidu, Microsoft, IBM, Google,
and others.
<a href="https://github.com/apache/gluten/graphs/contributors";>
<img src="https://contrib.rocks/image?repo=apache/gluten&columns=25"; />
diff --git a/docs/developers/ProfileMemoryOfGlutenWithVelox.md
b/docs/developers/ProfileMemoryOfGlutenWithVelox.md
index 4705e4f592..82eda3fa49 100644
--- a/docs/developers/ProfileMemoryOfGlutenWithVelox.md
+++ b/docs/developers/ProfileMemoryOfGlutenWithVelox.md
@@ -113,10 +113,12 @@ spark.files /path/to/jemalloc/libjemalloc.so
Example of enabling heap profile in spark executor:
```
-spark.executorEnv.LD_PRELOAD ./libjemalloc.so
-spark.executorEnv.MALLOC_CONF
prof:true,lg_prof_interval:30,prof_prefix:/tmp/gluten_heap_perf
+spark.executorEnv.LD_PRELOAD /path/to/jemalloc/libjemalloc.so
+spark.executorEnv.MALLOC_CONF
prof:true,prof_final:true,lg_prof_interval:30,prof_prefix:/tmp/gluten_heap_perf
```
+Note: `lg_prof_interval:30` only dumps a profile after every 2^30 bytes (1GB)
of cumulative allocation activity, so a short-running or low-allocation job may
never trigger it. `prof_final:true` ensures a profile is dumped when the
process exits, regardless of the interval.
+
Finally, profiling files prefixed with `/tmp/gluten_heap_perf.${PID}` will be
generated for each spark executor.
## Memory dump on spark executor exit
diff --git a/docs/developers/velox-backend-CI.md
b/docs/developers/velox-backend-CI.md
index a0af0b0f39..d35e5d9b31 100644
--- a/docs/developers/velox-backend-CI.md
+++ b/docs/developers/velox-backend-CI.md
@@ -6,36 +6,36 @@ parent: Developer Overview
---
# Velox Backend CI
-GHA workflows are defined under `.github/workflows/`.
+GitHub Actions (GHA) workflows are defined under `.github/workflows/`.
## Docker Build
-We have a weekly job defined in `docker_image.yml` to build docker images for
CI verification. The docker files and images are listed below:
+A weekly job defined in `docker_image.yml` builds the Docker images used for
CI verification. The Dockerfiles (under `dev/docker/`) and their corresponding
images are listed below:
file | images | comments
-- | -- | --
-dev/docker/Dockerfile.centos7-static-build | apache/gluten:vcpkg-centos-7 |
centos 7, static link, jdk8
-dev/docker/Dockerfile.centos8-static-build | apache/gluten:vcpkg-centos-8 |
centos 8, static link, jdk8
-dev/docker/Dockerfile.centos8-dynamic-build | apache/gluten:centos-8-jdk8 |
centos 8, dynamic link, jdk8
-dev/docker/Dockerfile.centos8-dynamic-build | apache/gluten:centos-8-jdk11 |
centos 8, dynamic link, jdk11
-dev/docker/Dockerfile.centos8-dynamic-build | apache/gluten:centos-8-jdk17 |
centos 8, dynamic link, jdk17
-dev/docker/cudf/Dockerfile | apache/gluten:centos-9-jdk8-cudf | centos 9,
dynamic link, jdk8
+Dockerfile.centos7-gcc13-static-build | apache/gluten:vcpkg-centos-7-gcc13 |
centos 7, static link, jdk8
+Dockerfile.centos8-gcc13-static-build | apache/gluten:vcpkg-centos-8-gcc13 |
centos 8, static link, jdk8
+Dockerfile.centos8-dynamic-build | apache/gluten:centos-8-jdk8 | centos 8,
dynamic link, jdk8
+Dockerfile.centos8-dynamic-build | apache/gluten:centos-8-jdk11 | centos 8,
dynamic link, jdk11
+Dockerfile.centos8-dynamic-build | apache/gluten:centos-8-jdk17 | centos 8,
dynamic link, jdk17
+cudf/Dockerfile | apache/gluten:centos-9-jdk8-cudf | centos 9, dynamic link,
jdk8
-Docker images can be found from https://hub.docker.com/r/apache/gluten/tags
+The Docker images can be found at
[https://hub.docker.com/r/apache/gluten/tags](https://hub.docker.com/r/apache/gluten/tags).
## Vcpkg Caching
-Gluten main branch is pulled down during static build in docker. And vcpkg
will cache binary data of all dependencies defined under dev/vcpkg.
-These binary data is cached into `/var/cache/vcpkg` and CI job can re-use them
in new build. By setting `VCPKG_BINARY_SOURCES=clear` in env.,
-reusing vcpkg cache can be disabled.
+The Gluten main branch is pulled during the static build in Docker, and vcpkg
caches binary data for all dependencies defined under `dev/vcpkg`.
+This binary data is cached into `/var/cache/vcpkg`, and CI jobs can reuse it
in later builds. Setting `VCPKG_BINARY_SOURCES=clear` in the
+environment disables reuse of the vcpkg cache.
## Arrow Libs Pre-installation
-Arrow libs are pre-installed in docker, assuming they are not actively
changed, then not necessarily to be re-built every time.
+Arrow libs are pre-installed in the Docker image, since they don't change
often and don't need to be rebuilt on every run.
-## .M2 cache
-The dependency libraries are pre installed in to /root/.m2 by `mvn
dependency:go-offline` Spark is set to 3.5 by default.
+## .M2 Cache
+Dependency libraries are pre-installed into `/root/.m2` via `mvn
dependency:go-offline`. Spark is set to 3.5 by default.
## Ccache
-Since the docker image is rebuilt weekly, the ccache mostly are outdated. So
the cache is removed from the image.
+Since the Docker image is rebuilt weekly, the ccache is mostly outdated, so it
is removed from the image.
-## Updating Docker Image
-Two GitHub secrets `DOCKERHUB_USER` & `DOCKERHUB_TOKEN` can be used to push
docker image to docker hub: https://hub.docker.com/r/apache/gluten/tags.
-Note GitHub secrets are not retrievable in PR from forked repo.
\ No newline at end of file
+## Updating the Docker Image
+The GitHub secrets `DOCKERHUB_USER` and `DOCKERHUB_TOKEN` are used to push
Docker images to [Docker Hub](https://hub.docker.com/r/apache/gluten/tags).
+Note that GitHub secrets are not accessible in PRs from forked repos.
\ No newline at end of file
diff --git a/docs/get-started/Velox.md b/docs/get-started/Velox.md
index 5c504607fb..9eead57d0d 100644
--- a/docs/get-started/Velox.md
+++ b/docs/get-started/Velox.md
@@ -18,7 +18,7 @@ Note: Spark 4.0 and 4.1 require JDK 17+ and Scala 2.13 (build
with `-Pspark-4.0`
# Prerequisite
-Currently, with static build Gluten+Velox backend supports all the Linux OSes,
but is only tested on **Ubuntu20.04/Ubuntu22.04/Centos7/Centos8**. With dynamic
build, Gluten+Velox backend support **Ubuntu20.04/Ubuntu22.04/Centos7/Centos8**
and their variants.
+Currently, the statically built Gluten+Velox backend supports all Linux OSes
but is only tested on **Ubuntu 20.04/22.04/CentOS 7/8**. The dynamically built
backend supports **Ubuntu 20.04/22.04/CentOS 7/8** and their variants.
Currently, the officially supported Spark versions are 3.3.1, 3.4.4, 3.5.5,
4.0.2 and 4.1.1.
@@ -79,8 +79,8 @@ export CPU_TARGET="aarch64"
**Step-by-step build**
-Alternative to the above one-step build, you can follow the below guide for
step-by-step build.
-Currently, Gluten is using a [IBM Velox](https://github.com/IBM/velox) which
is daily updated based on [upstream
Velox](https://github.com/facebookincubator/velox).
+As an alternative to the above one-step build, you can follow the step-by-step
guide below.
+Currently, Gluten uses an [IBM Velox fork](https://github.com/IBM/velox),
which is updated daily based on [upstream
Velox](https://github.com/facebookincubator/velox).
```bash
@@ -108,7 +108,7 @@ mvn clean package -Pbackends-velox -Pspark-4.1 -Pjava-17
-Pscala-2.13 -DskipTest
Notes： Building Velox may fail caused by OOM. You can prevent this failure by
adjusting `NUM_THREADS` (e.g., `export NUM_THREADS=4`) before building
Gluten/Velox. The recommended minimal memory size is 64G.
-After the above build process, the Jar file will be generated under
`package/target/`.
+After the above build process, the jar file will be generated under
`package/target/`.
Alternatively you may refer to [build in
docker](../developers/velox-backend-build-in-docker.md) to build the gluten jar
in docker.
@@ -151,8 +151,9 @@ This enables OpenSSL with FIPS features and requires
OpenSSL shared libraries to
## HDFS support
-Gluten supports dynamically loading both libhdfs.so and libhdfs3.so at runtime
by using dlopen, allowing the JVM to load the appropriate shared library file
as needed. This means you do not need to set the library path during the
compilation phase.
-To enable this functionality, you must set the JAVA_HOME and HADOOP_HOME
environment variables. Gluten will then locate and load the
${HADOOP_HOME}/lib/native/libhdfs.so file at runtime. If you prefer to use
libhdfs3.so instead, simply replace the ${HADOOP_HOME}/lib/native/libhdfs.so
file with libhdfs3.so.
+Gluten supports dynamically loading libhdfs.so or libhdfs3.so at runtime via
dlopen, so the library path does not need to be set at compile time.
+To enable this, set the JAVA_HOME and HADOOP_HOME environment variables.
Gluten will then locate and load `${HADOOP_HOME}/lib/native/libhdfs.so` at
runtime.
+To use libhdfs3.so instead, replace that path with libhdfs3.so.
### Build libhdfs3
@@ -382,13 +383,13 @@ Gluten with velox backend supports
[Iceberg](https://iceberg.apache.org/) table.
### How to use
-First of all, compile gluten-iceberg module by a `iceberg` profile, as follows:
+First, compile the gluten-iceberg module with the `iceberg` profile, as
follows:
```
mvn clean package -Pbackends-velox -Pspark-3.3 -Piceberg -DskipTests
```
-Once built successfully, iceberg features will be included in
gluten-velox-bundle-X jar. Then you can query iceberg table by gluten/velox
without scan's fallback.
+Once built successfully, iceberg features will be included in the
gluten-velox-bundle-X jar. You can then query iceberg tables via Gluten/Velox
without falling back on scan.
## Hudi Support
@@ -406,17 +407,17 @@ Compile gluten-paimon module by a `paimon` profile, as
follows:
mvn clean package -Pbackends-velox -Pspark-3.5 -Ppaimon -DskipTests
```
-Once built successfully, paimon features will be included in
gluten-velox-bundle-X jar. Then you can query paimon non-pk table by
gluten/velox without scan's fallback.
+Once built successfully, paimon features will be included in the
gluten-velox-bundle-X jar. You can then query paimon non-pk tables via
Gluten/Velox without falling back on scan.
### How to use
-First of all, compile gluten-hudi module by a `hudi` profile, as follows:
+First, compile the gluten-hudi module with the `hudi` profile, as follows:
```
mvn clean package -Pbackends-velox -Pspark-3.3 -Phudi -DskipTests
```
-Once built successfully, hudi features will be included in
gluten-velox-bundle-X jar. Then you can query hudi **COW** table by
gluten/velox without scan's fallback.
+Once built successfully, hudi features will be included in the
gluten-velox-bundle-X jar. You can then query hudi **COW** tables via
Gluten/Velox without falling back on scan.
# Coverage
@@ -424,7 +425,7 @@ Spark3.5 has 400+ functions in total. ~240 are commonly
used. To get the support
> Velox doesn't support [ANSI
> mode](https://spark.apache.org/docs/latest/sql-ref-ansi-compliance.html)),
> so as Gluten. Once ANSI mode is enabled in Spark config, Gluten will
> fallback to Vanilla Spark.
-To identify what can be offloaded in a query and detailed fallback reasons,
user can follow below steps to retrieve corresponding logs.
+To identify what can be offloaded in a query and see the detailed fallback
reasons, follow the steps below to retrieve the corresponding logs.
```
1) Enable Gluten by proper
[configuration](https://github.com/apache/gluten/blob/main/docs/Configuration.md).
@@ -455,8 +456,8 @@ With above steps, you will get a physical plan output like:
native validation failed due to: in ProjectRel, Scalar function name not
registered: get_struct_field, called with arguments:
(ROW<col_0:INTEGER,col_1:BIGINT,col_2:BIGINT>, INTEGER).
```
-In the above, the symbol `^` indicates a plan is offloaded to Velox in a
stage. In Spark DAG, all such pipelined plans (consecutive plans marked with
`^`) are plotted
-inside an umbrella node named `WholeStageCodegenTransformer` (It's not codegen
node. The naming is just for making it well plotted like Spark Whole Stage
Codegen).
+In the above, the symbol `^` indicates a plan is offloaded to Velox in a
stage. In the Spark DAG, all such pipelined plans (consecutive plans marked
with `^`) are plotted
+inside an umbrella node named `WholeStageCodegenTransformer`. Despite the
name, this is not a codegen node; it's named this way simply to display
similarly to Spark's Whole Stage Codegen.
# Spill
@@ -535,8 +536,7 @@ Refer to [Gluten configuration](../Configuration.md) for
more details.
## Performance
-Below table shows the TPC-H Q1 and Q6 Performance in a multiple-thread test
(--num-executors 6 --executor-cores 6) for Velox and vanilla Spark.
-Both Parquet and ORC datasets are sf1024.
+The table below shows TPC-H Q1 and Q6 performance for Velox and vanilla Spark
in a multi-thread test (`--num-executors 6 --executor-cores 6`), using scale
factor 1024 datasets in both Parquet and ORC formats.
| Query Performance (s) | Velox (ORC) | Vanilla Spark (Parquet) | Vanilla
Spark (ORC) |
|---------------- | ----------- | ------------- | ------------- |
@@ -564,8 +564,8 @@ cause the inconsistent results with a materialized query.
However, we have no ch
## Native Plan in Spark's Explain Output
-Gluten supports inject native plan string into Spark explain with formatted
mode by setting `--conf spark.gluten.sql.injectNativePlanStringToExplain=true`.
-Here is an example, how Gluten shows the native plan string.
+Gluten supports injecting the native plan string into Spark's explain output
in formatted mode by setting `--conf
spark.gluten.sql.injectNativePlanStringToExplain=true`.
+Here is an example of how Gluten shows the native plan string.
```
(9) WholeStageCodegenTransformer (2)
@@ -612,7 +612,7 @@ To enable this feature, you can set the following Spark
configuration:
| Property | Default |
Description |
|-------------------------------------------------------------|---------|-------------------------------------------------------------------|
-| `spark.gluten.velox.offHeapBroadcastBuildRelation.enabled` | `false` |
Enable/disable off-heap storage for broadcast build relations. |
+| `spark.gluten.velox.offHeapBroadcastBuildRelation.enabled` | `false` |
Whether to store broadcast build relations off-heap. |
This feature has been tested through a series of tests, and we are collecting
more feedback from users. If you have memory problem on broadcast build
relations, please try this feature and give more feedbacks.
diff --git a/docs/get-started/VeloxLocalCache.md
b/docs/get-started/VeloxLocalCache.md
index 432bfc38d2..93a4d8d0f5 100644
--- a/docs/get-started/VeloxLocalCache.md
+++ b/docs/get-started/VeloxLocalCache.md
@@ -5,16 +5,23 @@ nav_order: 7
parent: Getting-Started
---
-Velox supports a local cache when reading data from HDFS/S3/ABFS. With this
feature, Velox can asynchronously cache the data on local disk when reading
from remote storage and future read requests on previously cached blocks will
be serviced from local cache files. To enable the local caching feature, the
following configurations are required:
+Velox supports local caching when reading data from HDFS/S3/ABFS. With this
feature, Velox asynchronously caches data on
+local disk when reading from remote storage, and future read requests for
previously cached blocks are served from the
+local cache files. To enable local caching, the following configurations are
required:
```
-spark.gluten.sql.columnar.backend.velox.cacheEnabled // enable or disable
velox cache, default false.
-spark.gluten.sql.columnar.backend.velox.memCacheSize // the total size of
in-mem cache, default is 128MB.
-spark.gluten.sql.columnar.backend.velox.ssdCachePath // the folder to
store the cache files, default is "/tmp".
-spark.gluten.sql.columnar.backend.velox.ssdCacheSize // the total size of
the SSD cache, default is 128MB. Velox will do in-mem cache only if this value
is 0.
-spark.gluten.sql.columnar.backend.velox.ssdCacheShards // the shards of the
SSD cache, default is 1.
-spark.gluten.sql.columnar.backend.velox.ssdCacheIOThreads // the number of IO
threads for SSD cache read/write operations, default is 4. Velox will try to do
"read-ahead" if this value is bigger than 1
-spark.gluten.sql.columnar.backend.velox.ssdODirect // enable or disable
O_DIRECT on cache write, default false.
+spark.gluten.sql.columnar.backend.velox.cacheEnabled // Enable Velox
cache. Default: false.
+spark.gluten.sql.columnar.backend.velox.memCacheSize // In-memory cache
size. Default: 128MB.
+spark.gluten.sql.columnar.backend.velox.ssdCachePath // Folder to store
cache files, preferably on SSD. Default: "/tmp".
+spark.gluten.sql.columnar.backend.velox.ssdCacheSize // SSD cache size.
Memory-only caching is used when set to 0. Default: 128MB.
+spark.gluten.sql.columnar.backend.velox.ssdCacheShards // Number of SSD
cache shards. Default: 1.
+spark.gluten.sql.columnar.backend.velox.ssdCacheIOThreads // Number of IO
threads for SSD cache read/write. Enables read-ahead when > 1. Default: 4.
+spark.gluten.sql.columnar.backend.velox.loadQuantum // Load quantum
size. Must be at most 8MB when Velox cache is enabled, otherwise Velox fails.
Default: 256MB.
+spark.gluten.sql.columnar.backend.velox.ssdODirect // Enable O_DIRECT
on cache write. Currently causes SSD cache writes to fail if enabled; see Velox
issue #10597. Default: false.
+spark.gluten.soft-affinity.enabled // Enable Soft
Affinity scheduling. Should be enabled together with Velox cache. Default:
false.
```
-It's recommended to mount SSDs to the cache path to get the best performance
of local caching. Cache files will be written to
"spark.gluten.sql.columnar.backend.velox.cachePath", with UUID based suffix,
e.g. "/tmp/cache.13e8ab65-3af4-46ac-8d28-ff99b2a9ec9b0". Gluten cannot reuse
older caches for now, and the old cache files are left after Spark context
shutdown.
+It's recommended to mount SSDs to the cache path to get the best performance
of local caching. Cache files will be written
+to "spark.gluten.sql.columnar.backend.velox.ssdCachePath", with UUID based
suffix,
+e.g. "/tmp/cache.13e8ab65-3af4-46ac-8d28-ff99b2a9ec9b0". Gluten cannot reuse
older caches for now, and the old cache files
+are left after Spark context shutdown.
diff --git a/docs/velox-backend-limitations.md
b/docs/velox-backend-limitations.md
index 22e7b3de59..a39f2a40fb 100644
--- a/docs/velox-backend-limitations.md
+++ b/docs/velox-backend-limitations.md
@@ -144,17 +144,10 @@ Gluten's.
Only reading with INT96 and dictionary encoding is supported. When reading
INT64 represented millisecond/microsecond timestamps, or INT96 represented
timestamps of other encodings, exceptions can occur.
-- Complex types
- - Parquet scan of nested array with struct or array as element type is not
supported in Velox (fallback behavior).
- - Parquet scan of nested map with struct as key type, or array type as value
type is not supported in Velox (fallback behavior).
-
-### CSV Read
-The header option should be true. And now we only support DatasourceV1, i.e.,
user should set `spark.sql.sources.useV1SourceList=csv`. User defined read
option is not supported, which will make CSV read fall back to vanilla Spark in
most case.
-CSV read will also fall back to vanilla Spark and log warning when user
specifies schema is different with file schema.
-
### Utilizing Map Type as Hash Keys in ColumnarShuffleExchange
-Spark uses the `spark.sql.legacy.allowHashOnMapType` configuration to support
hash map key functions.
-Gluten enables this configuration during the creation of
ColumnarShuffleExchange, as shown in the code
[link](https://github.com/apache/gluten/blob/0dacac84d3bf3d2759a5dd7e0735147852d2845d/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxSparkPlanExecApi.scala#L355-L363).
-This method bypasses Spark's unresolved checks and creates projects with the
hash(mapType) operator before ColumnarShuffleExchange.
-However, if `spark.sql.legacy.allowHashOnMapType` is disabled in a test
environment, projects using the hash(mapType) expression may throw an `Invalid
call to dataType on unresolved object` exception during validation, causing
them to fallback to vanilla Spark, as referenced in the code
[link](https://github.com/apache/spark/blob/de5fa426e23b84fc3c2bddeabcd2e1eda515abd5/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala#L291-L296).
- Enabling this configuration allows the project to be offloaded to Velox.
+Spark's `spark.sql.legacy.allowHashOnMapType` configuration controls whether
hashing is allowed on map-type keys.
+Gluten enables this configuration when creating `ColumnarShuffleExchange`, as
shown
[here](https://github.com/apache/gluten/blob/0dacac84d3bf3d2759a5dd7e0735147852d2845d/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxSparkPlanExecApi.scala#L355-L363).
+This bypasses Spark's unresolved-expression checks and lets projects using the
`hash(mapType)` operator be created before `ColumnarShuffleExchange`.
+However, if `spark.sql.legacy.allowHashOnMapType` is disabled in a test
environment, projects using the `hash(mapType)` expression may throw an
+`Invalid call to dataType on unresolved object` exception during validation,
causing them to fall back to vanilla Spark, as referenced
[here](https://github.com/apache/spark/blob/de5fa426e23b84fc3c2bddeabcd2e1eda515abd5/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala#L291-L296).
+Enabling this configuration allows the project to be offloaded to Velox.
diff --git a/docs/velox-backend-scalar-function-support.md
b/docs/velox-backend-scalar-function-support.md
index e16baf3701..f86af55484 100644
--- a/docs/velox-backend-scalar-function-support.md
+++ b/docs/velox-backend-scalar-function-support.md
@@ -1,6 +1,6 @@
# Scalar Functions Support Status
-**Out of 357 scalar functions in Spark 3.5, Gluten currently fully supports
242 functions and partially supports 25 functions.**
+**Out of 357 scalar functions in Spark 3.5, Gluten currently fully supports
245 functions and partially supports 27 functions.**
## Array Functions
@@ -220,6 +220,7 @@
## Mathematical Functions
+
| Spark Functions | Spark Expressions | Status | Restrictions |
|-------------------|------------------------|----------|----------------|
| % | Remainder | S | |
@@ -281,8 +282,8 @@
| sin | Sin | | |
| sinh | Sinh | S | |
| sqrt | Sqrt | S | |
-| tan | Tan | | |
-| tanh | Tanh | | |
+| tan | Tan | S | |
+| tanh | Tanh | S | |
| try_add | TryAdd | PS | |
| try_divide | TryDivide | | |
| try_multiply | TryMultiply | | |
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
![https://user-images.githubusercontent.com/47296334/199617207-1140698a-4d53-462d-9bc7-303d14be060b.png"](https://user-images.githubusercontent.com/47296334/199617207-1140698a-4d53-462d-9bc7-303d14be060b.png"){kind=link}