malinjawi commented on code in PR #13338: URL: https://github.com/apache/gluten/pull/13338#discussion_r4312797887
########## docs/get-started/VeloxDelta.md: ########## @@ -1,62 +1,572 @@ -# Delta Lake Feature Support Status in Apache Gluten (Velox Backend) +--- +layout: page +title: Delta Lake Support in Velox Backend +nav_order: 13 +parent: Getting-Started +--- -This document summarizes the support status of **Delta Lake table features** when used with **Apache Gluten (Velox backend)**. +# Delta Lake Support in Velox Backend -## Supported Spark / Delta combinations +## Supported Spark version -| Spark profile | Spark version | Scala version | Delta Lake version | Status | +| Spark | Delta Lake | Delta Lake jar | Read | Native write | Tested by | +|---|---|---|---|---|---| +| 3.4.4 | 2.4.0 | `io.delta:delta-core_2.12:2.4.0` | Offload. Deletion vector scans fall back. | None | Gluten suites | +| 3.5.5 | 3.3.2 | `io.delta:delta-spark_2.12:3.3.2` (`_2.13` for Scala 2.13) | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites (Scala 2.12 only) | +| 4.0.2 | 4.0.1 | `io.delta:delta-spark_2.13:4.0.1` | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites tagged `ExtendedSQLTest` | +| 4.1.1 | 4.1.0 | `io.delta:delta-spark_4.1_2.13:4.1.0` | Offload | Experimental, off by default | Gluten suites, nightly Delta Lake 4.2.0 job | +| 4.2 | None on main | | None | None | None | + +Each Spark profile from 3.4 to 4.1 builds against the one Delta Lake release in the table. The copied Delta Lake suites run with native write on. + +Spark 4.1 is the most tested. A nightly job builds Delta Lake 4.2.0 from source and runs its Spark test suites on Spark 4.1. It puts a Gluten bundle compiled against Delta Lake 4.1.0 on the test classpath. It patches Delta Lake's `DeltaSQLCommandTest` to load Gluten with native write on. "Seen on 4.1" on this page refers to that job. Its known failures are listed in `.github/workflows/util/delta-spark-ut/known-failures.txt`. Four entries are two 2-billion-row deletion vector tests, run in two suites, that the job force-fails. Some entries check a query plan that changes once Gluten offloads it. Read the README in that directory before treating an entry as a Gluten bug. + +On Spark 4.1, Gluten's own suites check deletion vector reads and a partitioned native `save(path)`. Other native writes on Spark 4.1 are checked only by the nightly job. + +Delta Lake support for Spark 4.2 is in progress in [#13271](https://github.com/apache/gluten/pull/13271). + +### Build and run + +Delta Lake support is compiled in only with the `-Pdelta` Maven profile. Build the native library first, as described in [Gluten with Velox Backend](Velox.md). + +```` +./build/mvn clean install -Pbackends-velox -Pspark-3.5 -Pdelta -DskipTests +./build/mvn clean install -Pbackends-velox -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pdelta \ + -DskipTests -Dmaven.compiler.release=17 +```` + +`dev/buildbundle-veloxbe.sh` does not pass `-Pdelta`, so its bundle has no Delta Lake support. No Gluten bundle contains the Delta Lake jar, including one built with `-Pdelta`. Add the jar from the table above. When `io.delta.sql.DeltaSparkSessionExtension` is not on the classpath, Gluten logs `Excluding runtime-incompatible component` and leaves its Delta support off. + +Native writes to HDFS, S3, GCS or ABFS paths need the native library built with `--enable_hdfs=ON`, `--enable_s3=ON`, `--enable_gcs=ON` or `--enable_abfs=ON`. Without that connector, scans of those paths fall back and native writes fail (see [Write problems](#write-problems)). From code. + +For example, on Spark 3.5: + +```` +spark-shell \ + --packages io.delta:delta-spark_2.12:3.3.2 \ + --jars gluten-velox-bundle-spark3.5_2.12-<os>_<arch>-<version>.jar \ + --conf spark.plugins=org.apache.gluten.GlutenPlugin \ + --conf spark.memory.offHeap.enabled=true \ + --conf spark.memory.offHeap.size=4g \ + --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \ + --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \ + --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog +```` + +On Spark 4.0 or 4.1, use the Delta Lake jar for that version and add `--conf spark.sql.ansi.enabled=false`. + +### ANSI mode on Spark 4.x + +Spark 4.0 and 4.1 turn ANSI mode on by default. With Gluten's default `spark.gluten.sql.ansiFallback.enabled=true`, every query then falls back, Delta scans and native writes included. From code. + +```` +spark.sql.ansi.enabled=false +```` + +Gluten's Delta tests run with ANSI mode off in CI. The nightly Delta Lake job turns ANSI mode off in the test setup that loads Gluten. That setup and some Gluten Delta suites also set `spark.gluten.sql.ansiFallback.enabled=false`. Spark 3.x behaves the same when ANSI mode is turned on. From code. + +Keeping ANSI on and setting `spark.gluten.sql.ansiFallback.enabled=false` removes that fallback. Velox then runs the offloaded operators with ANSI mode on. For Delta Lake, only the nightly Delta Lake job tests that combination, in the Delta Lake 4.2.0 suites that turn ANSI on. It has a known MERGE problem (see [MERGE, UPDATE and DELETE problems](#merge-update-and-delete-problems)). + +## Support Status +The following values show the Delta Lake support status: + +| Value | Description | +|----------------|----------------------------------------------------------------------------| +| Offload | Runs in the Velox backend. For writes, Velox writes the data files. | +| PartialOffload | Some operators run in Velox and the rest fall back | +| Fallback | Runs on Spark or Delta Lake | +| Exception | Fails instead of falling back | +| ResultMismatch | Can return wrong results or write wrong files | + +Not tested: no test covers the case.<br> +From code: the status comes from reading the source. No test that runs in CI checks it.<br> +Seen on 4.1: the nightly Delta Lake job on Spark 4.1 showed the result.<br> + +## Adding catalogs +Fallback + +Catalog operations run on the driver. + +## Creating a table +Fallback + +```` +CREATE TABLE t (id INT, name STRING) USING delta; +```` + +A table created without `AS SELECT` is a metadata-only commit. There is nothing to offload. + +Offload + +```` +SET spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true; +CREATE TABLE t2 USING delta AS SELECT * FROM t; +```` + +With `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true` on Spark 3.5 to 4.1, Velox writes the data files of `CREATE TABLE ... AS SELECT` and `REPLACE TABLE ... AS SELECT` when the target catalog is a DeltaCatalog. Seen on 4.1. On 3.5 and 4.0 this is from code. + +## Writing +PartialOffload + +```` +INSERT INTO t SELECT id, name FROM source WHERE length(name) = 1; +```` + +Delta's writer writes the files of `INSERT`, with or without native write. The `SELECT` can run in Velox. From code. + +Fallback + +```` +INSERT INTO t VALUES (1, 'a'), (2, 'b'), (3, 'c'); +```` + +With the default `spark.gluten.sql.columnar.localTableScan=false`, the `VALUES` scan runs on Spark. Delta's writer writes the files. From code. + +Offload + +```` +SET spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true; +UPDATE t SET name = 'b' WHERE id = 2; +```` + Review Comment: Fixed. The legend rows and the other table descriptions now end with a period consistently. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
