This is an automated email from the ASF dual-hosted git repository.
yuqi1129 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new bda25bd5c1 [#12211] feat(client-python): split optional storage
dependencies (#13750)
bda25bd5c1 is described below
commit bda25bd5c16a7f3020ae848caff91b0179409507
Author: Thunderkeg <[email protected]>
AuthorDate: Fri Oct 23 08:34:48 2026 -0700
[#12211] feat(client-python): split optional storage dependencies (#13750)
### What changes were proposed in this pull request?
- Restrict the base `apache-gravitino` installation to metadata-client
runtime dependencies.
- Add `[gvfs]` for local GVFS use, backend extras for HDFS, S3, GCS,
OSS, and Azure, and `[storage]` for all storage backends; share GVFS
requirements while keeping provider dependencies isolated.
- Load `gravitino.gvfs` only when requested and provide installation
guidance when GVFS or a storage-provider dependency is missing.
- Move development and test dependencies out of the base install,
include requirement files in the source distribution, and document
installation and migration options.
### Why are the changes needed?
Ray Data applications need the official Gravitino Python client to
resolve catalog metadata. A metadata-only base install lets applications
co-install the client with `ray[data]` without installing unused GVFS
backends or building a separate HTTP client. General dependency-range
changes remain tracked by #13556.
Fix: #12211
### Does this PR introduce any user-facing change?
Yes. `pip install apache-gravitino` no longer installs GVFS or
storage-provider dependencies. Install `apache-gravitino[gvfs]` for
local paths, a backend extra such as `apache-gravitino[s3]` or
`apache-gravitino[hdfs]` for remote storage, or
`apache-gravitino[storage]` for all backends. The existing `lance` extra
remains available.
### How was this patch tested?
Focused Python unit suites were run from `clients/client-python` with:
```shell
python -m unittest discover -s tests/unittests -p
'test_client_import_without_storage_dependencies.py' -v
python -m unittest discover -s tests/unittests -p 'test_storage_handler.py'
-v
python -m unittest discover -s tests/unittests -p
'test_package_metadata.py' -v
python -m unittest discover -s tests/unittests -p
'test_gravitino_client.py' -v
```
All four focused suites passed (2 import-boundary tests, 4
storage-handler tests, 1 package-metadata test, and 3 Gravitino client
tests).
- Black checks and `flake8 --select=F` passed for changed Python files;
the commit's Ruff and Ruff-format hooks also passed.
- The source distribution installed alongside `ray[data]==2.59.0` on
Python 3.13.7; `uv pip check` passed, `ray.data` and `GravitinoClient`
imported, and `GravitinoClient.list_catalogs()` returned the expected
catalog from a loopback Gravitino-compatible endpoint.
- Isolated Python 3.13 environments installed `[gvfs]`, `[hdfs]`,
`[s3]`, `[gcs]`, `[oss]`, `[azure]`, and `[storage]` from the source
distribution; `uv pip check`, provider imports, and safe handler checks
passed without contacting storage services.
- The full client test suite, Gradle tasks, documentation build, and
live Gravitino or storage-service compatibility were not run.
---------
Signed-off-by: jiangxt2 <[email protected]>
Co-authored-by: Qi Yu <[email protected]>
---
clients/client-python/MANIFEST.in | 6 +
clients/client-python/gravitino/__init__.py | 29 +++-
.../gravitino/filesystem/gvfs_storage_handler.py | 36 ++++-
.../{MANIFEST.in => requirements-azure.txt} | 10 +-
clients/client-python/requirements-dev.txt | 8 +
.../{MANIFEST.in => requirements-gcs.txt} | 10 +-
.../{MANIFEST.in => requirements-gvfs.txt} | 13 +-
.../{MANIFEST.in => requirements-hdfs.txt} | 10 +-
.../{MANIFEST.in => requirements-oss.txt} | 10 +-
.../{MANIFEST.in => requirements-s3.txt} | 10 +-
clients/client-python/requirements.txt | 14 +-
clients/client-python/setup.py | 58 ++++++-
...t_client_import_without_storage_dependencies.py | 103 ++++++++++++
.../tests/unittests/test_package_metadata.py | 173 +++++++++++++++++++++
.../tests/unittests/test_storage_handler.py | 97 +++++++++++-
docs/fileset-catalog-with-adls.md | 2 +-
docs/fileset-catalog-with-gcs.md | 2 +-
docs/fileset-catalog-with-oss.md | 2 +-
docs/fileset-catalog-with-s3.md | 2 +-
docs/how-to-use-gvfs.md | 13 +-
docs/how-to-use-python-client.md | 24 ++-
21 files changed, 544 insertions(+), 88 deletions(-)
diff --git a/clients/client-python/MANIFEST.in
b/clients/client-python/MANIFEST.in
index f75d13e184..22762bc8e1 100644
--- a/clients/client-python/MANIFEST.in
+++ b/clients/client-python/MANIFEST.in
@@ -18,7 +18,13 @@
include requirements.txt
include requirements-dev.txt
+include requirements-gvfs.txt
+include requirements-hdfs.txt
include requirements-lance.txt
+include requirements-s3.txt
+include requirements-gcs.txt
+include requirements-oss.txt
+include requirements-azure.txt
include README.md
include LICENSE
include NOTICE
diff --git a/clients/client-python/gravitino/__init__.py
b/clients/client-python/gravitino/__init__.py
index 63ccaacf73..55970fb9a7 100644
--- a/clients/client-python/gravitino/__init__.py
+++ b/clients/client-python/gravitino/__init__.py
@@ -15,6 +15,9 @@
# specific language governing permissions and limitations
# under the License.
+import importlib
+from typing import TYPE_CHECKING
+
from gravitino.api.catalog import Catalog
from gravitino.api.schema import Schema
from gravitino.api.file.fileset import Fileset
@@ -25,7 +28,31 @@ from gravitino.client.gravitino_client import GravitinoClient
from gravitino.client.gravitino_admin_client import GravitinoAdminClient
from gravitino.client.gravitino_metalake import GravitinoMetalake
from gravitino.name_identifier import NameIdentifier
-from gravitino.filesystem import gvfs
+
+if TYPE_CHECKING:
+ from gravitino.filesystem import gvfs
+
+
+def __getattr__(name):
+ """Load the optional GVFS module only when a caller requests it."""
+ if name != "gvfs":
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+
+ try:
+ gvfs = importlib.import_module("gravitino.filesystem.gvfs")
+ except ModuleNotFoundError as error:
+ if error.name not in {"fsspec", "cachetools", "readerwriterlock"}:
+ raise
+ raise ImportError(
+ "GVFS support requires optional filesystem dependencies. "
+ "Install `apache-gravitino[gvfs]` for local files, a backend extra
"
+ "such as `apache-gravitino[hdfs]`, or `apache-gravitino[storage]` "
+ "for all backends."
+ ) from error
+
+ globals()[name] = gvfs
+ return gvfs
+
__all__ = [
"Catalog",
diff --git a/clients/client-python/gravitino/filesystem/gvfs_storage_handler.py
b/clients/client-python/gravitino/filesystem/gvfs_storage_handler.py
index 22abc27aa9..1fdbe317f0 100644
--- a/clients/client-python/gravitino/filesystem/gvfs_storage_handler.py
+++ b/clients/client-python/gravitino/filesystem/gvfs_storage_handler.py
@@ -55,6 +55,21 @@ SLASH = "/"
logger = logging.getLogger(__name__)
+def _import_optional_storage_module(module_name: str, extra_name: str):
+ """Import a provider dependency with an actionable optional-extra error."""
+ try:
+ return importlib.import_module(module_name)
+ except ModuleNotFoundError as error:
+ provider_module = module_name.split(".", maxsplit=1)[0]
+ if error.name != provider_module:
+ raise
+ raise GravitinoRuntimeException(
+ f"Storage provider dependency '{provider_module}' "
+ "is not installed. "
+ f'Install it with `pip install "apache-gravitino[{extra_name}]"`.'
+ ) from error
+
+
class StorageType(Enum):
HDFS = "hdfs"
LOCAL = "file"
@@ -256,7 +271,9 @@ class HDFSStorageHandler(StorageHandler):
def get_filesystem(
self, actual_path: Optional[str] = None, **kwargs
) -> AbstractFileSystem:
- fs_class = importlib.import_module("pyarrow.fs").HadoopFileSystem
+ fs_class = _import_optional_storage_module(
+ "pyarrow.fs", "hdfs"
+ ).HadoopFileSystem
return ArrowFSWrapper(fs_class.from_uri(actual_path))
def get_filesystem_with_expiration(
@@ -362,7 +379,8 @@ class S3StorageHandler(StorageHandler):
def get_filesystem(
self, actual_path: Optional[str] = None, **kwargs
) -> AbstractFileSystem:
- return importlib.import_module("s3fs").S3FileSystem(**kwargs)
+ s3fs = _import_optional_storage_module("s3fs", "s3")
+ return s3fs.S3FileSystem(**kwargs)
def _get_actual_prefix(
self,
@@ -395,7 +413,8 @@ class GCSStorageHandler(StorageHandler):
def get_filesystem(
self, actual_path: Optional[str] = None, **kwargs
) -> AbstractFileSystem:
- return importlib.import_module("gcsfs").GCSFileSystem(**kwargs)
+ gcsfs = _import_optional_storage_module("gcsfs", "gcs")
+ return gcsfs.GCSFileSystem(**kwargs)
def get_filesystem_with_expiration(
self,
@@ -415,7 +434,8 @@ class GCSStorageHandler(StorageHandler):
if not GOOGLE_AUTH_AVAILABLE:
raise GravitinoRuntimeException(
"Failed to import google.auth. "
- "Please ensure google-auth is installed."
+ "Install the GCS extra with "
+ '`pip install "apache-gravitino[gcs]"`.'
)
# gcsfs expects a google.auth.credentials.Credentials
object
@@ -509,7 +529,8 @@ class OSSStorageHandler(StorageHandler):
def get_filesystem(
self, actual_path: Optional[str] = None, **kwargs
) -> AbstractFileSystem:
- return importlib.import_module("ossfs").OSSFileSystem(**kwargs)
+ ossfs = _import_optional_storage_module("ossfs", "oss")
+ return ossfs.OSSFileSystem(**kwargs)
def get_filesystem_with_expiration(
self,
@@ -568,7 +589,7 @@ class OSSStorageHandler(StorageHandler):
return (
TIME_WITHOUT_EXPIRATION,
- importlib.import_module("ossfs").OSSFileSystem(
+ self.get_filesystem(
key=oss_access_key_id,
secret=oss_secret_access_key,
endpoint=oss_endpoint,
@@ -613,7 +634,8 @@ class ABSStorageHandler(StorageHandler):
def get_filesystem(
self, actual_path: Optional[str] = None, **kwargs
) -> AbstractFileSystem:
- return importlib.import_module("adlfs").AzureBlobFileSystem(**kwargs)
+ adlfs = _import_optional_storage_module("adlfs", "azure")
+ return adlfs.AzureBlobFileSystem(**kwargs)
def get_filesystem_with_expiration(
self,
diff --git a/clients/client-python/MANIFEST.in
b/clients/client-python/requirements-azure.txt
similarity index 81%
copy from clients/client-python/MANIFEST.in
copy to clients/client-python/requirements-azure.txt
index f75d13e184..fc1fb1561c 100644
--- a/clients/client-python/MANIFEST.in
+++ b/clients/client-python/requirements-azure.txt
@@ -15,12 +15,4 @@
# specific language governing permissions and limitations
# under the License.
-
-include requirements.txt
-include requirements-dev.txt
-include requirements-lance.txt
-include README.md
-include LICENSE
-include NOTICE
-include gravitino/version.ini
-graft licenses
+adlfs==2026.2.0
diff --git a/clients/client-python/requirements-dev.txt
b/clients/client-python/requirements-dev.txt
index 8d13479d0a..852d7d301c 100644
--- a/clients/client-python/requirements-dev.txt
+++ b/clients/client-python/requirements-dev.txt
@@ -17,12 +17,20 @@
requests==2.32.5
dataclasses-json==0.6.7
+simplejson==3.20.1
pylint==4.0.5
black==26.3.1
+flake8==7.0.0
+pre-commit==3.5.0
twine==6.2.0
coverage==7.14.3
pandas==2.3.3
pyarrow==23.0.1
+fsspec==2026.2.0
+gcsfs==2026.2.0
+s3fs==2026.2.0
+ossfs==2025.5.0
+adlfs==2026.2.0
llama-index==0.13.0
tenacity==9.1.4
cachetools==7.1.4
diff --git a/clients/client-python/MANIFEST.in
b/clients/client-python/requirements-gcs.txt
similarity index 81%
copy from clients/client-python/MANIFEST.in
copy to clients/client-python/requirements-gcs.txt
index f75d13e184..d5ad973db7 100644
--- a/clients/client-python/MANIFEST.in
+++ b/clients/client-python/requirements-gcs.txt
@@ -15,12 +15,4 @@
# specific language governing permissions and limitations
# under the License.
-
-include requirements.txt
-include requirements-dev.txt
-include requirements-lance.txt
-include README.md
-include LICENSE
-include NOTICE
-include gravitino/version.ini
-graft licenses
+gcsfs==2026.2.0
diff --git a/clients/client-python/MANIFEST.in
b/clients/client-python/requirements-gvfs.txt
similarity index 81%
copy from clients/client-python/MANIFEST.in
copy to clients/client-python/requirements-gvfs.txt
index f75d13e184..d6cac76aa4 100644
--- a/clients/client-python/MANIFEST.in
+++ b/clients/client-python/requirements-gvfs.txt
@@ -15,12 +15,7 @@
# specific language governing permissions and limitations
# under the License.
-
-include requirements.txt
-include requirements-dev.txt
-include requirements-lance.txt
-include README.md
-include LICENSE
-include NOTICE
-include gravitino/version.ini
-graft licenses
+# Shared runtime dependencies for the Gravitino virtual filesystem.
+cachetools==7.1.4
+fsspec==2026.2.0
+readerwriterlock==1.0.9
diff --git a/clients/client-python/MANIFEST.in
b/clients/client-python/requirements-hdfs.txt
similarity index 81%
copy from clients/client-python/MANIFEST.in
copy to clients/client-python/requirements-hdfs.txt
index f75d13e184..ac734a8cca 100644
--- a/clients/client-python/MANIFEST.in
+++ b/clients/client-python/requirements-hdfs.txt
@@ -15,12 +15,4 @@
# specific language governing permissions and limitations
# under the License.
-
-include requirements.txt
-include requirements-dev.txt
-include requirements-lance.txt
-include README.md
-include LICENSE
-include NOTICE
-include gravitino/version.ini
-graft licenses
+pyarrow==23.0.1
diff --git a/clients/client-python/MANIFEST.in
b/clients/client-python/requirements-oss.txt
similarity index 81%
copy from clients/client-python/MANIFEST.in
copy to clients/client-python/requirements-oss.txt
index f75d13e184..08824ddf18 100644
--- a/clients/client-python/MANIFEST.in
+++ b/clients/client-python/requirements-oss.txt
@@ -15,12 +15,4 @@
# specific language governing permissions and limitations
# under the License.
-
-include requirements.txt
-include requirements-dev.txt
-include requirements-lance.txt
-include README.md
-include LICENSE
-include NOTICE
-include gravitino/version.ini
-graft licenses
+ossfs==2025.5.0
diff --git a/clients/client-python/MANIFEST.in
b/clients/client-python/requirements-s3.txt
similarity index 81%
copy from clients/client-python/MANIFEST.in
copy to clients/client-python/requirements-s3.txt
index f75d13e184..ba3c9c935e 100644
--- a/clients/client-python/MANIFEST.in
+++ b/clients/client-python/requirements-s3.txt
@@ -15,12 +15,4 @@
# specific language governing permissions and limitations
# under the License.
-
-include requirements.txt
-include requirements-dev.txt
-include requirements-lance.txt
-include README.md
-include LICENSE
-include NOTICE
-include gravitino/version.ini
-graft licenses
+s3fs==2026.2.0
diff --git a/clients/client-python/requirements.txt
b/clients/client-python/requirements.txt
index 8f586463d4..bc339201da 100644
--- a/clients/client-python/requirements.txt
+++ b/clients/client-python/requirements.txt
@@ -16,18 +16,6 @@
# under the License.
-# the tools to publish the python client to Pypi
-requests==2.32.5
+# Runtime dependencies for the core metadata client.
dataclasses-json==0.6.7
simplejson==3.20.1
-readerwriterlock==1.0.9
-fsspec==2026.2.0
-pyarrow==23.0.1
-cachetools==7.1.4
-gcsfs==2026.2.0
-s3fs==2026.2.0
-ossfs==2025.5.0
-adlfs==2026.2.0
-black==26.3.1
-flake8==7.0.0
-pre-commit==3.5.0
diff --git a/clients/client-python/setup.py b/clients/client-python/setup.py
index 56eade07c9..7c9df24b5c 100644
--- a/clients/client-python/setup.py
+++ b/clients/client-python/setup.py
@@ -15,8 +15,57 @@
# specific language governing permissions and limitations
# under the License.
+from pathlib import Path
+
from setuptools import find_packages, setup
+CLIENT_PYTHON_ROOT = Path(__file__).resolve().parent
+
+
+def read_requirements(file_name):
+ """Read active requirement lines from a project file."""
+ requirements_path = CLIENT_PYTHON_ROOT / file_name
+ with requirements_path.open(encoding="utf-8") as requirements_file:
+ return [
+ line.strip()
+ for line in requirements_file
+ if line.strip() and not line.lstrip().startswith("#")
+ ]
+
+
+def combine_requirements(*requirement_groups):
+ """Combine requirement groups and remove duplicates without reordering."""
+ return list(
+ dict.fromkeys(
+ requirement
+ for requirement_group in requirement_groups
+ for requirement in requirement_group
+ )
+ )
+
+
+gvfs_requirements = read_requirements("requirements-gvfs.txt")
+provider_requirement_files = {
+ "hdfs": "requirements-hdfs.txt",
+ "s3": "requirements-s3.txt",
+ "gcs": "requirements-gcs.txt",
+ "oss": "requirements-oss.txt",
+ "azure": "requirements-azure.txt",
+}
+provider_requirements = {
+ extra_name: combine_requirements(
+ gvfs_requirements, read_requirements(requirements_file)
+ )
+ for extra_name, requirements_file in provider_requirement_files.items()
+}
+storage_requirements = combine_requirements(
+ gvfs_requirements,
+ *(
+ read_requirements(file_name)
+ for file_name in provider_requirement_files.values()
+ ),
+)
+
try:
with open("README.md") as f:
@@ -53,10 +102,13 @@ setup(
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
],
- install_requires=open("requirements.txt").read(),
+ install_requires=read_requirements("requirements.txt"),
extras_require={
- "dev": open("requirements-dev.txt").read(),
- "lance": open("requirements-lance.txt").read(),
+ "dev": read_requirements("requirements-dev.txt"),
+ "lance": read_requirements("requirements-lance.txt"),
+ "gvfs": gvfs_requirements,
+ **provider_requirements,
+ "storage": storage_requirements,
},
include_package_data=True,
)
diff --git
a/clients/client-python/tests/unittests/test_client_import_without_storage_dependencies.py
b/clients/client-python/tests/unittests/test_client_import_without_storage_dependencies.py
new file mode 100644
index 0000000000..e7d431df22
--- /dev/null
+++
b/clients/client-python/tests/unittests/test_client_import_without_storage_dependencies.py
@@ -0,0 +1,103 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+import subprocess
+import sys
+import unittest
+from pathlib import Path
+
+
+class TestClientImportWithoutStorageDependencies(unittest.TestCase):
+ """Verify core imports do not eagerly load optional GVFS packages."""
+
+ def setUp(self):
+ self.client_root = Path(__file__).resolve().parents[2]
+
+ def test_metadata_client_import_does_not_load_storage_packages(self):
+ script = r"""
+import builtins
+
+blocked = {
+ "cachetools",
+ "fsspec",
+ "gcsfs",
+ "adlfs",
+ "ossfs",
+ "pyarrow",
+ "readerwriterlock",
+ "s3fs",
+}
+original_import = builtins.__import__
+
+def import_without_storage(
+ name, globals=None, locals=None, fromlist=(), level=0
+):
+ if name.split(".", maxsplit=1)[0] in blocked:
+ missing_module = name.split(".", maxsplit=1)[0]
+ raise ModuleNotFoundError(
+ f"No module named '{name}'", name=missing_module
+ )
+ return original_import(name, globals, locals, fromlist, level)
+
+builtins.__import__ = import_without_storage
+from gravitino import GravitinoClient
+assert GravitinoClient.__name__ == "GravitinoClient"
+"""
+ result = subprocess.run(
+ [sys.executable, "-c", script],
+ cwd=self.client_root,
+ env=os.environ.copy(),
+ capture_output=True,
+ check=False,
+ text=True,
+ )
+
+ self.assertEqual(result.returncode, 0, result.stderr)
+
+ def test_gvfs_access_without_common_extra_has_install_guidance(self):
+ script = r"""
+import builtins
+
+original_import = builtins.__import__
+
+def import_without_fsspec(
+ name, globals=None, locals=None, fromlist=(), level=0
+):
+ if name.split(".", maxsplit=1)[0] == "fsspec":
+ raise ModuleNotFoundError("No module named 'fsspec'", name="fsspec")
+ return original_import(name, globals, locals, fromlist, level)
+
+builtins.__import__ = import_without_fsspec
+try:
+ from gravitino import gvfs
+except ImportError as error:
+ assert "apache-gravitino[gvfs]" in str(error)
+ assert "apache-gravitino[storage]" in str(error)
+else:
+ raise AssertionError("Accessing GVFS without its extra should fail")
+"""
+ result = subprocess.run(
+ [sys.executable, "-c", script],
+ cwd=self.client_root,
+ env=os.environ.copy(),
+ capture_output=True,
+ check=False,
+ text=True,
+ )
+
+ self.assertEqual(result.returncode, 0, result.stderr)
diff --git a/clients/client-python/tests/unittests/test_package_metadata.py
b/clients/client-python/tests/unittests/test_package_metadata.py
new file mode 100644
index 0000000000..2fe22668b4
--- /dev/null
+++ b/clients/client-python/tests/unittests/test_package_metadata.py
@@ -0,0 +1,173 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import re
+import subprocess
+import sys
+import tempfile
+import unittest
+from email.parser import Parser
+from pathlib import Path
+
+
+class TestPackageMetadata(unittest.TestCase):
+ """Verify the built package separates core and optional dependencies."""
+
+ def test_core_requirements_and_storage_extras(self):
+ metadata, source_files = self._read_package_metadata()
+
+ self._assert_core_requirements(metadata)
+ self._assert_source_manifest(source_files)
+ self._assert_storage_extras(metadata)
+
+ def _read_package_metadata(self):
+ client_root = Path(__file__).resolve().parents[2]
+ with tempfile.TemporaryDirectory() as egg_base:
+ result = subprocess.run(
+ [
+ sys.executable,
+ "setup.py",
+ "egg_info",
+ "--egg-base",
+ egg_base,
+ ],
+ cwd=client_root,
+ capture_output=True,
+ check=False,
+ text=True,
+ )
+ self.assertEqual(
+ result.returncode,
+ 0,
+ result.stdout + result.stderr,
+ )
+
+ metadata_files = list(Path(egg_base).glob("*.egg-info/PKG-INFO"))
+ self.assertEqual(len(metadata_files), 1)
+ metadata_content = metadata_files[0].read_text(encoding="utf-8")
+ metadata = Parser().parsestr(metadata_content)
+ source_files = set(
+ (metadata_files[0].parent / "SOURCES.txt")
+ .read_text(encoding="utf-8")
+ .splitlines()
+ )
+ return metadata, source_files
+
+ def _assert_core_requirements(self, metadata):
+ requirements = metadata.get_all("Requires-Dist", [])
+ base_requirements = [
+ requirement for requirement in requirements if "extra ==" not in
requirement
+ ]
+ base_names = {
+ re.split(r"[<>=!~; ]", requirement, maxsplit=1)[0]
+ .strip()
+ .lower()
+ .replace("_", "-")
+ for requirement in base_requirements
+ }
+ self.assertIn("dataclasses-json", base_names)
+ self.assertIn("simplejson", base_names)
+ for optional_name in (
+ "black",
+ "cachetools",
+ "fsspec",
+ "gcsfs",
+ "adlfs",
+ "ossfs",
+ "pyarrow",
+ "pre-commit",
+ "readerwriterlock",
+ "requests",
+ "s3fs",
+ "flake8",
+ ):
+ with self.subTest(requirement=optional_name):
+ self.assertNotIn(optional_name, base_names)
+
+ extras = set(metadata.get_all("Provides-Extra", []))
+ self.assertTrue(
+ {
+ "dev",
+ "lance",
+ "gvfs",
+ "hdfs",
+ "s3",
+ "gcs",
+ "oss",
+ "azure",
+ "storage",
+ }.issubset(extras)
+ )
+
+ def _assert_source_manifest(self, source_files):
+ for requirements_file in (
+ "requirements.txt",
+ "requirements-dev.txt",
+ "requirements-gvfs.txt",
+ "requirements-hdfs.txt",
+ "requirements-s3.txt",
+ "requirements-gcs.txt",
+ "requirements-oss.txt",
+ "requirements-azure.txt",
+ "requirements-lance.txt",
+ ):
+ with self.subTest(source=requirements_file):
+ self.assertIn(requirements_file, source_files)
+
+ def _assert_storage_extras(self, metadata):
+ shared_gvfs_requirements = {"cachetools", "fsspec", "readerwriterlock"}
+ gvfs_requirements = self._requirement_names_for_extra(metadata, "gvfs")
+ self.assertTrue(shared_gvfs_requirements.issubset(gvfs_requirements))
+ self.assertTrue(
+ gvfs_requirements.isdisjoint({"pyarrow", "s3fs", "gcsfs", "ossfs",
"adlfs"})
+ )
+
+ for extra_name, provider in (
+ ("hdfs", "pyarrow"),
+ ("s3", "s3fs"),
+ ("gcs", "gcsfs"),
+ ("oss", "ossfs"),
+ ("azure", "adlfs"),
+ ):
+ with self.subTest(extra=extra_name):
+ extra_requirements = self._requirement_names_for_extra(
+ metadata, extra_name
+ )
+
self.assertTrue(shared_gvfs_requirements.issubset(extra_requirements))
+ self.assertIn(provider, extra_requirements)
+
+ storage_requirements = self._requirement_names_for_extra(metadata,
"storage")
+ self.assertTrue(
+ shared_gvfs_requirements.union(
+ {"pyarrow", "s3fs", "gcsfs", "ossfs", "adlfs"}
+ ).issubset(storage_requirements)
+ )
+
+ @staticmethod
+ def _requirement_names_for_extra(metadata, extra_name):
+ extra_requirements = [
+ requirement
+ for requirement in metadata.get_all("Requires-Dist", [])
+ if f'extra == "{extra_name}"' in requirement
+ ]
+ return {
+ re.split(r"[<>=!~; ]", requirement, maxsplit=1)[0]
+ .strip()
+ .lower()
+ .replace("_", "-")
+ for requirement in extra_requirements
+ }
diff --git a/clients/client-python/tests/unittests/test_storage_handler.py
b/clients/client-python/tests/unittests/test_storage_handler.py
index 53871eb33a..3d5e04d755 100644
--- a/clients/client-python/tests/unittests/test_storage_handler.py
+++ b/clients/client-python/tests/unittests/test_storage_handler.py
@@ -15,12 +15,24 @@
# specific language governing permissions and limitations
# under the License.
+import re
import unittest
-from unittest.mock import patch
+from unittest.mock import MagicMock, patch
from fsspec.implementations.memory import MemoryFileSystem
-from gravitino.filesystem.gvfs_storage_handler import S3StorageHandler
+from gravitino.exceptions.base import GravitinoRuntimeException
+from gravitino.filesystem.gvfs_storage_handler import (
+ ABSStorageHandler,
+ GCSStorageHandler,
+ HDFSStorageHandler,
+ OSSStorageHandler,
+ S3StorageHandler,
+)
+
+STORAGE_IMPORT_MODULE = (
+ "gravitino.filesystem.gvfs_storage_handler.importlib.import_module"
+)
class TestStorageHandler(unittest.TestCase):
@@ -84,3 +96,84 @@ class TestStorageHandler(unittest.TestCase):
self.assertEqual(captured_args["key"], "access_key_from_client")
self.assertEqual(captured_args["secret"], "secret_key_from_client")
self.assertEqual(captured_args["endpoint_url"],
"endpoint_from_catalog")
+
+ def test_missing_provider_dependency_has_install_guidance(self):
+ handlers = [
+ (HDFSStorageHandler(), "pyarrow", "hdfs"),
+ (S3StorageHandler(), "s3fs", "s3"),
+ (GCSStorageHandler(), "gcsfs", "gcs"),
+ (OSSStorageHandler(), "ossfs", "oss"),
+ (ABSStorageHandler(), "adlfs", "azure"),
+ ]
+
+ for handler, missing_module, extra_name in handlers:
+ with self.subTest(extra=extra_name):
+ with patch(
+ STORAGE_IMPORT_MODULE,
+ side_effect=ModuleNotFoundError(
+ f"No module named '{missing_module}'",
+ name=missing_module,
+ ),
+ ):
+ with self.assertRaisesRegex(
+ GravitinoRuntimeException,
+ re.escape(f"apache-gravitino[{extra_name}]"),
+ ):
+ handler.get_filesystem("unused://path")
+
+ def test_provider_handlers_dispatch_to_their_filesystem_class(self):
+ handlers = [
+ (HDFSStorageHandler(), "pyarrow.fs", "HadoopFileSystem"),
+ (S3StorageHandler(), "s3fs", "S3FileSystem"),
+ (GCSStorageHandler(), "gcsfs", "GCSFileSystem"),
+ (OSSStorageHandler(), "ossfs", "OSSFileSystem"),
+ (ABSStorageHandler(), "adlfs", "AzureBlobFileSystem"),
+ ]
+
+ for handler, module_name, class_name in handlers:
+ with self.subTest(provider=module_name):
+ provider_module = MagicMock()
+ filesystem_class = getattr(provider_module, class_name)
+ filesystem = MagicMock()
+
+ if module_name == "pyarrow.fs":
+ filesystem_class.from_uri.return_value = filesystem
+ with (
+ patch(
+
"gravitino.filesystem.gvfs_storage_handler.ArrowFSWrapper",
+ return_value=filesystem,
+ ) as arrow_wrapper,
+ patch(
+ STORAGE_IMPORT_MODULE, return_value=provider_module
+ ) as import_module,
+ ):
+ result =
handler.get_filesystem("hdfs://namenode:8020/path")
+
+ filesystem_class.from_uri.assert_called_once_with(
+ "hdfs://namenode:8020/path"
+ )
+ arrow_wrapper.assert_called_once_with(filesystem)
+ else:
+ filesystem_class.return_value = filesystem
+ with patch(
+ STORAGE_IMPORT_MODULE, return_value=provider_module
+ ) as import_module:
+ result = handler.get_filesystem(
+ "unused://path", test_option="value"
+ )
+
+
filesystem_class.assert_called_once_with(test_option="value")
+
+ import_module.assert_called_once_with(module_name)
+ self.assertIs(result, filesystem)
+
+ def test_provider_transitive_import_error_is_preserved(self):
+ handler = S3StorageHandler()
+ with patch(
+ STORAGE_IMPORT_MODULE,
+ side_effect=ModuleNotFoundError(
+ "No module named 'botocore'", name="botocore"
+ ),
+ ):
+ with self.assertRaisesRegex(ModuleNotFoundError, "botocore"):
+ handler.get_filesystem("s3a://bucket/path")
diff --git a/docs/fileset-catalog-with-adls.md
b/docs/fileset-catalog-with-adls.md
index 9c0a076300..b54189037d 100644
--- a/docs/fileset-catalog-with-adls.md
+++ b/docs/fileset-catalog-with-adls.md
@@ -419,7 +419,7 @@ ${HADOOP_HOME}/bin/hadoop fs -put /path/to/local/file
gvfs://fileset/adls_catalo
### GVFS Python client
```bash
-pip install apache-gravitino==${GRAVITINO_VERSION}
+pip install "apache-gravitino[azure]==${GRAVITINO_VERSION}"
```
On top of the [base GVFS configuration](./how-to-use-gvfs.md#configuration-1),
pass the Azure Data Lake Storage
diff --git a/docs/fileset-catalog-with-gcs.md b/docs/fileset-catalog-with-gcs.md
index 68a8b41eb4..303c63372d 100644
--- a/docs/fileset-catalog-with-gcs.md
+++ b/docs/fileset-catalog-with-gcs.md
@@ -403,7 +403,7 @@ ${HADOOP_HOME}/bin/hadoop fs -put /path/to/local/file
gvfs://fileset/gcs_catalog
### GVFS Python client
```bash
-pip install apache-gravitino==${GRAVITINO_VERSION}
+pip install "apache-gravitino[gcs]==${GRAVITINO_VERSION}"
```
On top of the [base GVFS configuration](./how-to-use-gvfs.md#configuration-1),
pass the Google Cloud Storage
diff --git a/docs/fileset-catalog-with-oss.md b/docs/fileset-catalog-with-oss.md
index c2fe45ef41..633a1f5eb3 100644
--- a/docs/fileset-catalog-with-oss.md
+++ b/docs/fileset-catalog-with-oss.md
@@ -427,7 +427,7 @@ ${HADOOP_HOME}/bin/hadoop fs -put /path/to/local/file
gvfs://fileset/oss_catalog
### GVFS Python client
```bash
-pip install apache-gravitino==${GRAVITINO_VERSION}
+pip install "apache-gravitino[oss]==${GRAVITINO_VERSION}"
```
On top of the [base GVFS configuration](./how-to-use-gvfs.md#configuration-1),
pass the Alibaba Cloud OSS
diff --git a/docs/fileset-catalog-with-s3.md b/docs/fileset-catalog-with-s3.md
index b09fbdf2b2..21ef6f6d65 100644
--- a/docs/fileset-catalog-with-s3.md
+++ b/docs/fileset-catalog-with-s3.md
@@ -440,7 +440,7 @@ ${HADOOP_HOME}/bin/hadoop fs -put /path/to/local/file
gvfs://fileset/s3_catalog/
### GVFS Python client
```bash
-pip install apache-gravitino==${GRAVITINO_VERSION}
+pip install "apache-gravitino[s3]==${GRAVITINO_VERSION}"
```
On top of the [base GVFS configuration](./how-to-use-gvfs.md#configuration-1),
pass the Amazon S3
diff --git a/docs/how-to-use-gvfs.md b/docs/how-to-use-gvfs.md
index 17cb874591..1b080cad1f 100644
--- a/docs/how-to-use-gvfs.md
+++ b/docs/how-to-use-gvfs.md
@@ -519,13 +519,20 @@ For fileset with multiple locations, you can specify
which location to access us
3. If neither is specified, the system will use the value of
`default-location-name` from the fileset properties
:::
-1. Make sure to obtain the Gravitino library.
- You can get it by [pip](https://pip.pypa.io/en/stable/installation/):
+1. Install the Gravitino Python client with the extra for your storage backend
using
+ [pip](https://pip.pypa.io/en/stable/installation/). The base installation
provides only the
+ metadata client and does not install GVFS dependencies. The following HDFS
example requires
+ the `hdfs` extra:
```shell
- pip install apache-gravitino
+ pip install "apache-gravitino[hdfs]"
```
+ For local `file://` paths, use `apache-gravitino[gvfs]`. For cloud storage,
choose
+ `apache-gravitino[s3]`, `apache-gravitino[gcs]`, `apache-gravitino[oss]`, or
+ `apache-gravitino[azure]` for the corresponding backend. Each backend extra
includes the shared
+ GVFS dependencies. Use `apache-gravitino[storage]` to install all supported
GVFS backends.
+
2. Configuring the Hadoop environment.
You should ensure that the Python client has Kerberos authentication
information and
configure Hadoop environments in the system environment:
diff --git a/docs/how-to-use-python-client.md b/docs/how-to-use-python-client.md
index ef84c623c8..c62e9a1071 100644
--- a/docs/how-to-use-python-client.md
+++ b/docs/how-to-use-python-client.md
@@ -29,6 +29,28 @@ install it in your local.
pip install apache-gravitino
```
+The base installation provides the Gravitino metadata client. For GVFS with
local `file://` paths
+and no remote backend, install only the shared GVFS dependencies:
+
+```shell
+pip install "apache-gravitino[gvfs]"
+```
+
+For a remote storage backend, install its extra, for example:
+
+```shell
+pip install "apache-gravitino[hdfs]"
+pip install "apache-gravitino[s3]"
+pip install "apache-gravitino[gcs]"
+pip install "apache-gravitino[oss]"
+pip install "apache-gravitino[azure]"
+```
+
+To preserve the previous installation profile that included all GVFS backends,
install
+`apache-gravitino[storage]`. Ray Data applications that only use Gravitino for
metadata can install
+the base Gravitino client alongside their Ray Data dependencies without
installing these storage
+extras.
+
1. [Manage metalake using Gravitino Python
API](./manage-metalake-using-gravitino.md?language=python)
2. [Manage fileset metadata using Gravitino Python
API](./manage-fileset-metadata-using-gravitino.md?language=python)
3. [Manage Semantic Model metadata using Gravitino Python
API](./manage-semantic-model-metadata-using-gravitino.md?language=python)
@@ -58,7 +80,7 @@ contains the following code snippets:
1. Install HDFS Python client.
2. Create a HDFS client to connect HDFS and to do some test operations.
-3. Install Gravitino Python client.
+3. Install the Gravitino Python client with HDFS support: `pip install
"apache-gravitino[hdfs]"`.
4. Initialize Gravitino admin client and create a Gravitino metalake.
5. Initialize Gravitino client and list metalakes.
6. Create a Gravitino `Catalog` and special `type` is `Catalog.Type.FILESET`
and `provider` is
