kevinjqliu commented on code in PR #1238:
URL: https://github.com/apache/iceberg-python/pull/1238#discussion_r1807441718
##########
mkdocs/docs/configuration.md:
##########
@@ -47,6 +55,8 @@ Iceberg tables support table properties to configure table
behavior.
| `commit.manifest.target-size-bytes` | Size in bytes | 8388608 (8MB) |
Target size when merging manifest files |
| `commit.manifest.min-count-to-merge` | Number of manifests | 100 |
Target size when merging manifest files |
| `commit.manifest-merge.enabled` | Boolean | False |
Controls whether to automatically merge manifests on writes |
+| `schema.name-mapping.default` | Name mapping strategy
| N/A | Default name mapping for schema evolution.
|
+| `format-version` | `{1, 2}`
| 2 | The version of the Iceberg table format to use.
|
Review Comment:
these changes are from the other PR.
Can you rebase this PR with main so that only changes for this PR will show?
##########
mkdocs/docs/configuration.md:
##########
@@ -151,6 +161,62 @@ For the FileIO there are several configuration options
available:
<!-- markdown-link-check-enable-->
+### Custom FileIO Implementations
+
+<!-- markdown-link-check-disable -->
+
+The `pyIceberg` library allows you to use custom FileIO implementations,
enabling flexible file handling tailored to your specific needs. This feature
is particularly useful when working with different storage backends or file
formats.
+
+#### Bringing Your Own FileIO with `PY_IO_IMPL`
Review Comment:
nit: merge with above section
##########
mkdocs/docs/configuration.md:
##########
@@ -151,6 +161,62 @@ For the FileIO there are several configuration options
available:
<!-- markdown-link-check-enable-->
+### Custom FileIO Implementations
+
+<!-- markdown-link-check-disable -->
+
+The `pyIceberg` library allows you to use custom FileIO implementations,
enabling flexible file handling tailored to your specific needs. This feature
is particularly useful when working with different storage backends or file
formats.
Review Comment:
nit: `PyIceberg`
##########
mkdocs/docs/configuration.md:
##########
@@ -151,6 +161,62 @@ For the FileIO there are several configuration options
available:
<!-- markdown-link-check-enable-->
+### Custom FileIO Implementations
+
+<!-- markdown-link-check-disable -->
Review Comment:
nit: this is not necessary unless there are links here that would not work
##########
mkdocs/docs/configuration.md:
##########
@@ -151,6 +161,62 @@ For the FileIO there are several configuration options
available:
<!-- markdown-link-check-enable-->
+### Custom FileIO Implementations
+
+<!-- markdown-link-check-disable -->
+
+The `pyIceberg` library allows you to use custom FileIO implementations,
enabling flexible file handling tailored to your specific needs. This feature
is particularly useful when working with different storage backends or file
formats.
+
+#### Bringing Your Own FileIO with `PY_IO_IMPL`
+
+To implement a custom FileIO, you can specify the `PY_IO_IMPL` property in
your configuration. The property should point to the custom FileIO class you
wish to use. Below is a brief guide on how to use this feature.
+
+```python
+PY_IO_IMPL = "py-io-impl"
+```
+
+#### Implementation Details
+
+The following functions are key to inferring and loading custom FileIO
implementations:
+
+##### `_infer_file_io_from_scheme`
+
+```python
+def _infer_file_io_from_scheme(path: str, properties: Properties) ->
Optional[FileIO]:
+```
+
+- **Purpose**: Infers the appropriate FileIO implementation based on the
scheme of the provided file path.
+- **Parameters**:
+ - `path` (str): The file path from which to infer the scheme.
+ - `properties` (Properties): Configuration properties to assist with
loading.
+- **Returns**: An instance of `FileIO` if a suitable implementation is found;
otherwise, `None`.
+
+##### Usage Example
+
+```python
+file_io = _infer_file_io_from_scheme("s3://my-bucket/my-file.txt", properties)
+```
+
+##### `load_file_io`
+
+```python
+def load_file_io(properties: Properties = EMPTY_DICT, location: Optional[str]
= None) -> FileIO:
+```
+
+- **Purpose**: Loads the custom FileIO implementation specified in the
`properties`.
+- **Parameters**:
+ - `properties` (Properties): A dictionary of configuration properties,
which may include `PY_IO_IMPL`.
+ - `location` (Optional[str]): An optional location to specify the file
path.
+- **Returns**: An instance of `FileIO`.
Review Comment:
these are not necessary as part of docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
