Re: [DISCUSS] Pre-Proposal: Improving Merge-On-Read Query Performance With Indexing

2025-06-10 Thread Xiaoxuan Li
Thank you for the thoughtful feedback, Yan, and for bringing up these important questions. > How realistic is the scenario I've described, and what's the likelihood of encountering it in production environments? I don’t have direct visibility into that either, but I’ve seen some vendors claim the

Re: [DISCUSS] Standardize cross client integration tests

2025-06-10 Thread Jayce Slesar
Hi all, also happy to support this! I think one thing I'm looking forward out of this work in addition to the general improvements is being able to use this as a building block for instrumenting different Iceberg implementations. From what I currently understand maintainers need to know what diffe

[DISCUSS] Remove usage of allowEmptyValue

2025-06-10 Thread Elphas Toringepi
Hi Team, I’d like to start a discussion about removing the use of allowEmptyValue in the REST Catalog API, in light of OAI/OpenAPI-Specification#1632. The use of allowEmptyValue is not recommended, and it is likely to be removed in a future revision of the OpenAPI specification. This option has

Re: [DISCUSS] Kafka Connect delta writer support

2025-06-10 Thread Ryan Blue
I'm strongly against writing equality deletes from the KC writer because it can't sort to make the deletes more efficient to apply. I don't think that equality deletes should be used in situations like this and that it is only going to cause pain for users that don't understand that they need offli

Re: [DISCUSS] Standardize cross client integration tests

2025-06-10 Thread Fokko Driesprong
Thanks Leon for bringing this up. The main reason that all the implementations test against Spark is that it is well supported and has a nice SQL API to easily set up test cases. But most importantly, it uses the Iceberg Java SDK underneath, which we consider the reference implementation of Iceber

Re: [DISCUSS] June board report

2025-06-10 Thread Fokko Driesprong
Thanks Ryan, looks good. Thanks for mentioning the PyIceberg milestone 🎉 Kind regards, Fokko Op zo 8 jun 2025 om 22:08 schreef Matt Topol : > Thanks Ryan! The Go notes look good to me. > > On Fri, Jun 6, 2025, 11:15 PM Manu Zhang wrote: > >> Hi Ryan, >> >> Thanks for drafting the report. We als

Re: [DISCUSS] Standardize cross client integration tests

2025-06-10 Thread Renjie Liu
Hi, Leon: > How complex would it be to integrate sqllogictest into non-Rust clients? This seems non-trivial to me. Note that it's not only about parsing/executing sqllogictest, the underlying sql engine needs to integrate with iceberg's language client. > Should we centralize the shared Docker i

[DISCUSS] Kafka Connect delta writer support

2025-06-10 Thread ismail simsek
Hi All, I have ported the Kafka Connect delta writer feature, however there are some questions around should we support it or wait for the new solution for Equality deletes. And there is some interest from the community to have it, so opening this threat to discuss it. PR https://github.com/apac