andygrove commented on code in PR #5064: URL: https://github.com/apache/datafusion-comet/pull/5064#discussion_r3666576048
########## docs/source/contributor-guide/roadmap.md: ########## @@ -41,25 +41,28 @@ supported functions, frames, and fallback cases. [#4836]: https://github.com/apache/datafusion-comet/issues/4836 [#4837]: https://github.com/apache/datafusion-comet/issues/4837 -## Lambda Expressions +## Native Lambda Evaluation Spark supports higher-order functions on arrays and maps that take a lambda, including `transform`, `exists`, -`forall`, `aggregate`, `zip_with`, `map_filter`, and `map_zip_with`. Comet currently lacks a general mechanism -for serializing lambda expressions and evaluating them in DataFusion. Adding this capability will unlock a -significant family of Spark expressions in one effort. - -## Dynamic Partition Pruning - -Native Parquet scans (`CometNativeScanExec`) support Dynamic Partition Pruning (DPP) both with and without -Adaptive Query Execution. Non-AQE DPP landed in [#4011] and AQE DPP with broadcast reuse landed in [#4112]. -Iceberg native scans support non-AQE DPP ([#3349], [#3511]) and, on Spark 3.5+, AQE DPP with broadcast reuse -([#4215]); on Spark 3.4 Iceberg AQE DPP falls back to Spark without reuse. - -[#3349]: https://github.com/apache/datafusion-comet/pull/3349 -[#3511]: https://github.com/apache/datafusion-comet/pull/3511 -[#4011]: https://github.com/apache/datafusion-comet/pull/4011 -[#4112]: https://github.com/apache/datafusion-comet/pull/4112 -[#4215]: https://github.com/apache/datafusion-comet/pull/4215 +`forall`, `aggregate`, `zip_with`, `map_filter`, and `map_zip_with`. Comet evaluates these today through a JVM +codegen-dispatch bridge (`CometScalaUDF`, `CometBatchKernelCodegen`) instead of falling back to Spark, but the +lambda body is still interpreted row-at-a-time on the JVM rather than natively in DataFusion. Moving evaluation +into native Rust code would remove that JVM round-trip and let these expressions benefit from vectorized native +execution. + +## Iceberg Table Format V3 Support + +Comet landed its first Iceberg table format V3 feature (native data file decryption, [#4991]), and we want to +add more V3 features to Comet's native Iceberg scans so they don't fall back to Spark. The work is tracked +phase-by-phase in [#3376]: detecting the V3 format and supporting new V3 data types, deletion vector reads, row +lineage, and table encryption. Native deletion vector reads are prototyped in a draft PR ([#4887]), but are +blocked on upstream `iceberg-rust` support, tracked in [iceberg-rust #2792] and [iceberg-rust #2411]. + +[#3376]: https://github.com/apache/datafusion-comet/issues/3376 +[#4887]: https://github.com/apache/datafusion-comet/pull/4887 +[#4991]: https://github.com/apache/datafusion-comet/pull/4991 +[iceberg-rust #2411]: https://github.com/apache/iceberg-rust/issues/2411 +[iceberg-rust #2792]: https://github.com/apache/iceberg-rust/issues/2792 ## TPC-H and TPC-DS Performance Review Comment: I think we could rewrite the TPC performance section to make it more positive. We can point out that current performance is already quite good, and share results from AWS Labs and then talk about ongoing optimization work? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
