malinjawi opened a new pull request, #13595:
URL: https://github.com/apache/gluten/pull/13595

   ## What changes are proposed in this pull request?
   
   This is the write half of #12263, using the lance-spark path agreed there. 
Gluten passes Velox's batches to Lance as Arrow, with no row conversion, and 
lance-spark still does the commit. It replaces #13515, which only moved 
`lance.version`; that bump is part of the first commit here.
   
   The feature is experimental and off by default. With 
`spark.gluten.sql.columnar.backend.velox.lance.enableNativeWrite=true`, writes 
to Lance tables run through Velox:
   - appends and overwrites (`INSERT INTO`, `INSERT OVERWRITE`, 
`mode("append")` and `mode("overwrite")`)
   - the nested writes of `CREATE TABLE AS SELECT` and `REPLACE TABLE AS 
SELECT` (Spark 3.5 and later)
   
   Two commits, reviewable one at a time:
   
   1. **Offload Lance table writes to Velox.**
      - Moves `lance.version` from 0.8.0 to 0.9.0-beta.2, the first lance-spark 
release with `LanceBatchWrite.taskCommit` (lance-format/lance-spark#715) and 
`getFragmentWriteSpec` (lance-format/lance-spark#892). #13527 tracks moving to 
a stable lance-spark release before the next Gluten release.
      - Each task exports its Velox batches as an Arrow C stream and calls 
`Fragment.create` with the write parameters lance-spark resolves for the table: 
write mode, file format version, file size limits and storage options. It 
returns `LanceBatchWrite.taskCommit(fragments)`, and lance-spark commits the 
fragments on the driver as it does for its own writer.
      - Velox may leave any value under a null slot, which Arrow allows, but 
Lance counts those values when it picks bit widths and dictionaries. The writer 
zeroes them, in a copy, so the files are as compact as lance-spark's.
      - If a task fails, its stream fails, and Lance deletes the files it wrote 
for that task. lance-spark's writer ends its stream instead, which leaves those 
files behind.
      - Tasks report the same `recordsWritten` and `bytesWritten` metrics as 
lance-spark's writer, plus Gluten's native write metrics.
   2. **Write Lance fixed-size list columns natively.** Velox has no fixed-size 
list type and exports vector columns as plain lists. The writer moves each 
top-level fixed-size list of signed integers or floats into the commit schema's 
layout: without a copy when no row is null, with one memory copy per row 
otherwise, and element by element only when elements are null. A list with the 
wrong number of elements fails the task.
   
   Some writes stay on lance-spark's writer, decided at plan time:
   - sharded tables
   - blob columns (a follow-up writes them natively)
   - large strings and binaries, half floats and unsigned integers
   - fixed-size lists inside other columns
   - `TimestampNTZ` columns, which Gluten's validation keeps on Spark
   - timestamps inside a struct that may be null. Velox's Arrow export converts 
the timestamps of null struct rows too, where an expression or a join may leave 
any value, and a value out of range fails the export. Velox's own Parquet 
writer fails the same way.
   
   The first batch each task exports is also checked against the table's Arrow 
schema, so a layout mismatch fails the task instead of writing bad data.
   
   ## How was this patch tested?
   
   All Lance suites pass on Spark 3.5, 4.0, 4.1 and 4.2 with both commits, and 
on Spark 3.5 and 4.0 with the first commit alone, with no Arrow or Velox memory 
leaks. On Spark 3.4 every writer test passes; two scan tests fail there as they 
do on main (reading large strings, and the filter/shuffle/sort test from 
#13488).
   - `VeloxLanceWriteSuite` (25 tests in the first commit, 27 with the second):
     - append and overwrite, including `write_mode=overwrite`; several tasks 
and batches per task; an empty append; a table with dropped columns
     - struct, array, map and vector columns
     - every offloaded type with nulls, written from a Parquet scan: values 
match, and no non-zero value is left under a null slot on disk
     - a session time zone other than UTC, `decimal(38,10)`, and `TimestampNTZ`
     - a table pinned to file format 2.1; metrics
     - SQL `INSERT INTO`, `INSERT OVERWRITE`, CTAS and RTAS through a Lance 
catalog
     - a task that fails mid-write leaves the table unchanged and no files 
behind; a failure inside Lance fails the task; no Arrow memory leaks
     - every fallback listed above
   - `LanceNullSlotsSuite` (4) and `LanceFixedSizeListsSuite` (5) unit-test the 
null-slot zeroing and the vector conversion.
   - `VeloxLanceSuite` and the scan's `LanceFixedSizeListsSuite` still pass.
   
   Exact comparison (laptop, Spark 3.5): native output equals both the Parquet 
input and lance-spark's output with `EXCEPT ALL` in both directions, on a 
`store_sales`-like table (12M rows, identical file sizes), a nullable 
`customer`-like table (20M rows) and 1M 768-float vectors with 1% null vectors.
   
   Cluster benchmark (8 x r6id.4xlarge, Spark 3.5, 5 rotating rounds per setup 
and dataset, medians; measured before the null-slot zeroing was added):
   
   | Data | Native | Gluten, lance-spark writer | Vanilla Spark | Native vs 
lance-spark writer | Native vs vanilla |
   |---|---|---|---|---|---|
   | TPC-DS `store_sales`, 1.44B rows | 39.2 s | 72.4 s | 89.4 s | 1.86x, 44% 
less CPU | 2.26x, 50% less CPU |
   | 60M x 768-float vectors | 26.9 s | 35.6 s | 43.7 s | 1.28x, 31% less CPU | 
1.62x |
   | 30M rows, vectors and nine strings | 67.2 s | 83.6 s | 125.3 s | 1.22x, 
30% less CPU | 1.89x |
   
   ## Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Claude Code (claude-opus-5-5)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to