FelixYBW commented on PR #59313: URL: https://github.com/apache/spark/pull/59313#issuecomment-6210894578
Glad to see Spark community started to adding native support. We are thrilled to contribute and improve. I quickly implemented a velox datasource PoC with int column only: https://github.com/FelixYBW/velox/tree/spark-native-datasource-poc/velox/experimental/spark_datasource Here are some benchmark results. Single Parquet file with zstd compression, 553 MiB file (73,420,230 rows, 5 row groups), 11 columns all are INT Benchmark is quite simple, scan all 11 columns then sum. parquet is on local disk, single core on a m7i EC2. Most of the time is actually spent on zstd decompression. Here is a quick benchmark: | Configuration | Time (s) | Speedup vs. Spark | |---|---|---| | Spark | 6.72 | 1.00x | | Spark + Velox datasource | 5.83 | **1.15x** | | Gluten scan only | 10.03 | 0.67x | | Gluten full | 5.44 | **1.24x** | @HyukjinKwon We need to disable arrow check (`-Darrow.enable_unsafe_memory_access=true -Darrow.enable_null_check_for_get=false`) in spark+velox datasource, otherwise perf is too bad. See if we can disable it by default. Then the PR can get expected improvement. Gluten scan only performance is too bad due to the VeloxC2R operator. Looks like we need to use Spark's default one instead. All done with Claude -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
