HonahX commented on issue #938:
URL: https://github.com/apache/iceberg-python/issues/938#issuecomment-2235570532
> But maybe there is value allowing load_catalog to parse properties for
custom catalogs for ease of use.
+1. I think it is beneficial for `load_catalog` to support additional
methods for fetching catalog implementations and initializing them with
properties parsed from ~/.pyiceberg.yaml, PYICEBERG__* environment variables,
and parameters would be highly beneficial.
For example, it would allow users to define custom catalogs in the following
way:
```yaml
custom:
py-catalog-impl: example.catalog.ExampleCatalog
custom.property-a: a
custom.property-b: b
```
Then, they can load the catalog in their code with:
```python
catalog = load_catalog("custom")
```
IMO, `load_catalog` should serve as the primary entry point for users to
initialize catalog instances. Extending its capabilities to support loading
custom implementations could make non-rest private catalog implementations also
benefit from the util.
Inspired by the discussions of custom implementation of `Catalog` interface,
I think we can also consider to make `_commit_table` public
https://github.com/apache/iceberg-python/blob/301e336926d950f1d3424b8f87423504150368e0/pyiceberg/catalog/__init__.py#L459-L473
`_commit_table` is crucial if user wants to use the custom catalog
implementation to update table. Although in python there is no real "private"
method, making this one public could make it clear that custom catalog need to
implement this one to enable the usage of transaction and other places where
table updates take place. On the doc side, it will be good if `_commit_table`
could show up in the [Catalog Code
Reference](https://py.iceberg.apache.org/reference/pyiceberg/catalog/#pyiceberg.catalog.Catalog).
We may also add some guide like [Java Custom
Catalog](https://iceberg.apache.org/docs/nightly/custom-catalog/), especially
for `_commit_table` which is very different from [Java Catalog's
commit](https://github.com/apache/iceberg/blob/36ecab4601b34114224e10b579eaee9018e57344/core/src/main/java/org/apache/iceberg/TableOperations.java#L64)
Would love to hear more thoughts on these! Thanks for all the valuable
discussions above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
