malinjawi commented on PR #11924: URL: https://github.com/apache/gluten/pull/11924#issuecomment-4305273896
> @malinjawi > > 1. PR title is still within `[WIP]`, is there anything remaining for this PR? > 2. Have you checked locally, ensuring the `GlutenDeltaFileFormatWriter` code path is in use? > 3. Non-blocking - can you also share some perf numbers? Thanks @zhztheplayer 1. PR is now ready removing WIP. 2. Yes, I validated this locally. With native write enabled, the executed plan goes through the Gluten Delta write command path (`GlutenDeltaFileFormatWriter`); with native write disabled, it does not. 3. I did a local Perf run (same workload, Delta 4.0.1, 20M rows, partitioned write, 1 warmup + 3 measured runs): - Vanilla Spark: `24.502s` median (`24.502, 41.538, 23.823`) - Gluten (native write OFF): `24.504s` median (`24.783, 24.405, 24.504`) - Gluten (native write ON): `6.972s` median (`6.972, 7.052, 6.816`) So native Delta 4 write is about `3.51x` faster than vanilla in this run (~`71.5%` lower write time). I also have the full logs/scripts bundle if you want it attached. [delta4-native-write-perf-2026-04-23.zip](https://github.com/user-attachments/files/27015192/delta4-native-write-perf-2026-04-23.zip) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
