philo-he commented on code in PR #12675:
URL: https://github.com/apache/gluten/pull/12675#discussion_r3703266285
##########
dev/release/build-release.sh:
##########
@@ -62,7 +62,7 @@ do
-Piceberg,delta,hudi,paimon -DskipTests
done
-for spark_version in 4.0
+for spark_version in 4.0 4.1
do
${GLUTEN_HOME}/build/mvn clean install -Pjava-17 -Pscala-2.13
-Pbackends-velox -Pspark-${spark_version} -Piceberg,paimon -DskipTests
done
Review Comment:
Updated PR description.
##########
docs/get-started/VeloxLocalCache.md:
##########
@@ -5,16 +5,23 @@ nav_order: 7
parent: Getting-Started
---
-Velox supports a local cache when reading data from HDFS/S3/ABFS. With this
feature, Velox can asynchronously cache the data on local disk when reading
from remote storage and future read requests on previously cached blocks will
be serviced from local cache files. To enable the local caching feature, the
following configurations are required:
+Velox supports local caching when reading data from HDFS/S3/ABFS. With this
feature, Velox asynchronously caches data on
+local disk when reading from remote storage, and future read requests for
previously cached blocks are served from the
+local cache files. To enable local caching, the following configurations are
required:
```
-spark.gluten.sql.columnar.backend.velox.cacheEnabled // enable or disable
velox cache, default false.
-spark.gluten.sql.columnar.backend.velox.memCacheSize // the total size of
in-mem cache, default is 128MB.
-spark.gluten.sql.columnar.backend.velox.ssdCachePath // the folder to
store the cache files, default is "/tmp".
-spark.gluten.sql.columnar.backend.velox.ssdCacheSize // the total size of
the SSD cache, default is 128MB. Velox will do in-mem cache only if this value
is 0.
-spark.gluten.sql.columnar.backend.velox.ssdCacheShards // the shards of the
SSD cache, default is 1.
-spark.gluten.sql.columnar.backend.velox.ssdCacheIOThreads // the number of IO
threads for SSD cache read/write operations, default is 4. Velox will try to do
"read-ahead" if this value is bigger than 1
-spark.gluten.sql.columnar.backend.velox.ssdODirect // enable or disable
O_DIRECT on cache write, default false.
+spark.gluten.sql.columnar.backend.velox.cacheEnabled // Enable Velox
cache. Default: false.
+spark.gluten.sql.columnar.backend.velox.memCacheSize // In-memory cache
size. Default: 128MB.
+spark.gluten.sql.columnar.backend.velox.ssdCachePath // Folder to store
cache files, preferably on SSD. Default: "/tmp".
+spark.gluten.sql.columnar.backend.velox.ssdCacheSize // SSD cache size.
Memory-only caching is used when set to 0. Default: 128MB.
+spark.gluten.sql.columnar.backend.velox.ssdCacheShards // Number of SSD
cache shards. Default: 1.
+spark.gluten.sql.columnar.backend.velox.ssdCacheIOThreads // Number of IO
threads for SSD cache read/write. Enables read-ahead when > 1. Default: 4.
+spark.gluten.sql.columnar.backend.velox.loadQuantum // Load quantum
size. Must be set to 8MB when Velox cache is enabled, otherwise Velox fails.
Default: 256MB.
Review Comment:
Just updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
