lliangyu-lin opened a new issue, #13229: URL: https://github.com/apache/iceberg/issues/13229
### Feature Request / Improvement ### Motivation As Iceberg adopts across platforms, languages, and engines, it's increasingly important to ensure that all client implementations behave consistently when interacting with the same table. Clients in Java, Python, Go, Rust, and C++ should be able to read and write tables created by any other client without compatibility issues. A shared, extensible testing mechanism that supports all common catalog types with any engine and storages, enabling full read and write interoperability testing running on pipeline would help the projects finding incompatibilities early. ### Proposed Changes **Iceberg Java Implementation Repository** Add an Iceberg Spark Environment Docker Image that will be built and published nightly, similar to the [iceberg rest catalog fixture image](https://github.com/apache/iceberg/blob/main/docker/iceberg-rest-fixture/Dockerfile). **Other Client Implementation Repositories** * Define **Docker compose files** that create containers that host iceberg spark runtime, catalog services, and storage services. * **Provision PySpark Test File**: Defines various test table setup through Spark SQL or DataFrame APIs * **Read Test Suites**: Reads tables that are pre-provisioned in the catalog using the above PySpark provisioning script. * **Write Test Suites**: Writes various table states with the client implementation. * **Validation PySpark Test File**: Defines Spark SQL statements or DataFrame operations on the tables created in the write test suites. The results are verified through assertions within the python script. * Add **GitHub Workflow** that sets up the infrastructure and runs two steps verification: submit provision test script and verify with read integration tests, submit write integration tests and verify with validation test script. Document: https://docs.google.com/document/d/1vZfVzGZucsDc35uoRrbn5CKHGd0muzcY7FAwmKt-KNI/edit?usp=sharing ### Query engine None ### Willingness to contribute - [x] I can contribute this improvement/feature independently - [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community - [ ] I cannot contribute this improvement/feature at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
