mbutrovich commented on issue #2921: URL: https://github.com/apache/datafusion-comet/issues/2921#issuecomment-3662247190
I made my initial argument in #2528, particularly in "Benefits over iceberg_compat". While there are execution/runtime/behavior differences between the two code paths, I think our challenges with shading and circular dependencies forced our hand trying the Iceberg Rust approach. It's been tough to make significant progress enhancing the Iceberg Java + Comet code path with new features and bug fixes, not for lack of trying from @parthchandra and @huaxingao. I believe our ability to rapidly iterate in both the Comet and [Iceberg Rust repos](https://github.com/apache/iceberg-rust/pulls?q=is%3Apr+author%3Ambutrovich+is%3Aclosed) (shouts out to @Xuanwo, @kevinjqliu, and others) has been promising so far. Iceberg Rust is about to cut their 0.8 release with significant enhancements to the reader from the Comet team. There are definitely implementation and feature gaps to work on, but I feel like we can close them much more quickly with the Iceberg Rust approach. My opinion is that we should deprecate the Iceberg Java + Comet hybrid scan approach, and focus on hardening and enhancing the Iceberg Rust code path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
