andygrove opened a new issue, #5647: URL: https://github.com/apache/datafusion-comet/issues/5647
### What is the problem the feature request solves? The only performance numbers for native Iceberg writes are the ad hoc measurements in the #5361 description (2.9x unpartitioned, 5.5x partitioned fanout, from a harness in a personal fork). There is no benchmark in the repository, so a regression in the writer, in the shuffle feeding it, or in the JVM metrics rebuild would not be noticed, and the partitioned number was measured with `write.distribution-mode=none` rather than the default plan shape. ### Describe the potential solution Add a Spark benchmark (the `org.apache.spark.sql.benchmark` framework used by `CometReadBenchmark` and the shuffle benchmark) that measures `INSERT INTO ... SELECT` into an Iceberg table for: - unpartitioned - partitioned with the default hash distribution (clustered writer, native shuffle and sort on the partition transforms after #5635) - partitioned with `write.spark.fanout.enabled=true` - a copy-on-write `DELETE` (rewrite of affected files) Each case runs Spark, Comet scan only, and Comet scan plus native write, so the writer's contribution is isolated the way the #5361 table did. Wire it into the benchmark targets in the `Makefile` and document how to run it. Follow-on: run it on the EC2 benchmark runner and keep results with the other benchmark files. Keep the Spark benchmark framework's warmup and minimum-time floors in mind when sizing the data (see the shuffle benchmark work in #5388 for how that dominated wall clock). ### Additional context Part of the native Iceberg writes epic. Related: #5361, #3595 (fused sort plus write, which this benchmark would measure). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
