Fokko commented on PR #11283:
URL: https://github.com/apache/iceberg/pull/11283#issuecomment-2470436838
Hey @ajantha-bhat Thanks for taking a stab at this again. My main goal is to
use this docker image to replace PyIceberg, Iceberg-Rust, and Iceberg-Go. These
repositories still rely on a third-pary container that we want to get rid of (I
believe you also raised this earlier). I tried this, but failed because it
didn't come with the AWS Runtime:
```
pyiceberg-rest | [main] INFO org.apache.iceberg.rest.RESTCatalogServer -
Creating catalog with properties: {jdbc.password=password,
s3.endpoint=http://minio:9000, jdbc.user=user,
io-impl=org.apache.iceberg.aws.s3.S3FileIO,
catalog-impl=org.apache.iceberg.jdbc.JdbcCatalog, jdbc.schema-version=V1,
warehouse=s3://warehouse/, uri=jdbc:sqlite::memory:}
pyiceberg-rest | [main] INFO org.apache.iceberg.CatalogUtil - Loading
custom FileIO implementation: org.apache.iceberg.aws.s3.S3FileIO
pyiceberg-rest | Exception in thread "main"
java.lang.IllegalArgumentException: Cannot initialize FileIO implementation
org.apache.iceberg.aws.s3.S3FileIO: Cannot find constructor for interface
org.apache.iceberg.io.FileIO
pyiceberg-rest | Missing org.apache.iceberg.aws.s3.S3FileIO
[java.lang.NoClassDefFoundError:
software/amazon/awssdk/core/exception/SdkException]
pyiceberg-rest | at
org.apache.iceberg.CatalogUtil.loadFileIO(CatalogUtil.java:356)
pyiceberg-rest | at
org.apache.iceberg.jdbc.JdbcCatalog.initialize(JdbcCatalog.java:132)
pyiceberg-rest | at
org.apache.iceberg.CatalogUtil.loadCatalog(CatalogUtil.java:274)
pyiceberg-rest | at
org.apache.iceberg.CatalogUtil.buildIcebergCatalog(CatalogUtil.java:328)
pyiceberg-rest | at
org.apache.iceberg.rest.RESTCatalogServer.initializeBackendCatalog(RESTCatalogServer.java:88)
```
This is because we store the metadata in Minio, so the metadata is easily
accessible outside of the container as well. How do you feel about adding the
S3 runtime?
Steps to run the tests:
```sh
git clone [email protected]:apache/iceberg-python.git
cd iceberg-python
```
Apply patch as described in
https://github.com/apache/iceberg/pull/11283#pullrequestreview-2419506245:
```
➜ iceberg-python git:(main) ✗ git diff
diff --git a/dev/docker-compose-integration.yml
b/dev/docker-compose-integration.yml
index fccdcdc75..9a807fca3 100644
--- a/dev/docker-compose-integration.yml
+++ b/dev/docker-compose-integration.yml
@@ -41,7 +41,7 @@ services:
- hive:hive
- minio:minio
rest:
- image: tabulario/iceberg-rest
+ image: apache/iceberg-rest-adapter
container_name: pyiceberg-rest
networks:
iceberg_net:
```
And run the tests:
```sh
make install
make test-integration
```
Tail the logs using:
```sh
docker compose -f dev/docker-compose-integration.yml logs -f
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
