yuqi1129 commented on code in PR #12399: URL: https://github.com/apache/gravitino/pull/12399#discussion_r3757920845
########## docs/fileset-catalog.md: ########## @@ -1,214 +1,266 @@ --- title: "Fileset Catalog" slug: "/fileset-catalog" -date: 2024-4-2 -keyword: "fileset catalog" +keywords: + - fileset + - catalog + - storage + - s3 + - gcs + - adls + - oss + - cos license: "This software is licensed under the Apache License version 2." --- -## Introduction +## Overview -Fileset catalog is a fileset catalog that using Hadoop Compatible File System (HCFS) to manage -the storage location of the fileset. It supports the local filesystem and HDFS. -Gravitino supports [S3](fileset-catalog-with-s3.md), [GCS](fileset-catalog-with-gcs.md), -[OSS](fileset-catalog-with-oss.md) and [Azure Blob Storage](fileset-catalog-with-adls.md) through Fileset catalog. -Gravitino also supports [Tencent Cloud COS](fileset-catalog-with-cos.md). +A fileset catalog manages filesets over a Hadoop Compatible File System. Gravitino owns the catalog rather than federating an external one, so no provider is needed when creating it, and the same catalog, schema, and fileset model works over HDFS, a local filesystem, or object storage. -The rest of this document will use HDFS or local file as an example to illustrate how to use the Fileset catalog. -For S3, GCS, OSS, Azure Blob Storage and COS, the configuration is similar to HDFS, -refer to the corresponding document for more details. +What changes per storage system is small: a bundle jar on the classpath, the URI scheme in the location, and a few credential properties. Creating and managing the objects is covered in [Manage Fileset Metadata](./manage-fileset-metadata-using-gravitino.md), and reading and writing the files in [How to Use GVFS](./how-to-use-gvfs.md). Neither changes because the data sits in S3 rather than HDFS, which is the point of the indirection described in [Filesets](./filesets.md). -Note that Gravitino uses Hadoop 3 dependencies to build Fileset catalog. Theoretically, it should be -compatible with both Hadoop 2.x and 3.x, since Gravitino doesn't leverage any new features in -Hadoop 3. If there's any compatibility issue, create an [issue](https://github.com/apache/gravitino/issues). +The catalog is built against Hadoop 3 but uses no Hadoop 3 features, so Hadoop 2.x should also work. Report any incompatibility as an [issue](https://github.com/apache/gravitino/issues). -## Catalog +## Quick Start -### Catalog Properties +**1. Create the catalog.** No provider is needed. Give it a base `location` and, for object +storage, the credential properties for that backend. -Besides the [common catalog properties](./gravitino-server-config.md#catalog-properties-configuration), -the Fileset catalog has the following properties: +```shell +curl -X POST -H "Content-Type: application/json" \ + -d '{ + "name": "{catalog_name}", + "type": "FILESET", + "comment": "", + "properties": { + "location": "hdfs://{cluster}/{path}" + } + }' \ + http://localhost:8090/api/metalakes/{metalake}/catalogs +``` -| Property Name | Description | Default Value | Required | -|--------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------|----------| -| `location` | The storage location managed by Fileset catalog. Its location name is `unknown`. The value should always a directory(HDFS) or path prefix(cloud storage like S3, GCS.) and does not support a single file. | (none) | No | -| `location-` | The property prefix. User can use `location-{name}={path}` to set multiple locations with different names for the catalog. | (none) | No | -| `default-filesystem-provider` | (deprecated) The default filesystem provider of this Fileset catalog if users do not specify the scheme in the URI. Candidate values are 'builtin-local', 'builtin-hdfs', 's3', 'gcs', 'abs' and 'oss'. Default value is `builtin-local`. For S3, if we set this value to 's3', we can omit the prefix 's3a://' in the location. | `builtin-local` | No | -| `filesystem-providers` | (deprecated) The file system providers to add. Users need to set this configuration to support cloud storage or custom HCFS. For instance, set it to `s3` or a comma separated string that contains `s3` like `gs,s3` to support multiple kinds of fileset including `s3`. | (none) | NO | -| `credential-providers` | The credential provider types, separated by comma. | (none) | No | -| `filesystem-conn-timeout-secs` | The timeout of getting the file system using Hadoop FileSystem client instance. Time unit: seconds. | 6 | No | -| `disable-filesystem-ops` | The configuration to disable file system operations in the server side. If set to true, the Fileset catalog in the server side will not create, drop files or folder when the schema, fileset is created, dropped. | false | No | -| `fileset-cache-eviction-interval-ms` | The interval in milliseconds to evict the fileset cache, -1 means never evict. | 3600000 | No | -| `fileset-cache-max-size` | The maximum number of the filesets the cache may contain, -1 means no limit. | 200000 | No | -| `config.resources` | The configuration resources, separated by comma. For example, `hdfs-site.xml,core-site.xml`. | (none) | No | -| `fs.path.config.<name>` | Defines a logical location entry. Set `fs.path.config.<name>` to the real base URI (for example, `hdfs://cluster1/`). Any key that starts with the same prefix (such as `fs.path.config.<name>.config.resource`) is treated as a location-scoped property and will be forwarded to the underlying filesystem client. | (none) | No | +**2. Create a schema.** The directory is created under the catalog location unless +`disable-filesystem-ops` is set. -:::note -`default-filesystem-provider` and `filesystem-providers` are deprecated. The fileset catalog automatically loads filesystem providers on the classpath, including the built-in filesystem provider and cloud providers when the corresponding bundle jar is present (for example, `gravitino-aws-bundle`, `gravitino-azure-bundle`, `gravitino-aliyun-bundle`, `gravitino-gcp-bundle`, or `gravitino-tencent-bundle`). -::: +```shell +curl -X POST -H "Content-Type: application/json" \ + -d '{"name": "{schema_name}", "comment": "", "properties": {}}' \ + http://localhost:8090/api/metalakes/{metalake}/catalogs/{catalog_name}/schemas +``` -Refer to [Credential vending](./security/credential-vending.md) for more details about credential vending. +**3. Create a fileset.** + +```shell +curl -X POST -H "Content-Type: application/json" \ + -d '{ + "name": "{fileset_name}", + "type": "MANAGED", + "comment": "", + "properties": {} + }' \ + http://localhost:8090/api/metalakes/{metalake}/catalogs/{catalog_name}/schemas/{schema_name}/filesets +``` -### HDFS Fileset +**4. Read and write the files.** The fileset is addressable as +`gvfs://fileset/{catalog_name}/{schema_name}/{fileset_name}` from the Java client, the Python +client, Spark, and the Hadoop shell. See [How to Use GVFS](./how-to-use-gvfs.md). + +Only step 1 changes per storage backend, and only in the `location` scheme and the credential +properties. Steps 2 through 4 are the same over HDFS, S3, GCS, ADLS, OSS, and COS. + +## Catalog Properties + +These apply in addition to the [common catalog properties](./gravitino-server-config.md#catalog-properties-configuration). + +| Property Name | Description | Default Value | +|--------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------|---------------| +| `location` | Base storage location, named `unknown`. Always a directory or path prefix, never a single file | (none) | +| `location-` | Prefix for named locations, as `location-{name}={path}` | (none) | +| `credential-providers` | Credential provider types, separated by commas | (none) | +| `config.resources` | Configuration files to load, separated by commas, such as `hdfs-site.xml,core-site.xml` | (none) | +| `filesystem-conn-timeout-secs` | Timeout when obtaining a filesystem client, in seconds | `6` | +| `disable-filesystem-ops` | Stops the server creating and removing directories when schemas and filesets are created and dropped | `false` | +| `fileset-cache-eviction-interval-ms` | Fileset cache eviction interval, where `-1` never evicts | `3600000` | +| `fileset-cache-max-size` | Maximum filesets held in the cache, where `-1` is unlimited | `200000` | +| `fs.path.config.<n>` | A logical location entry set to a base URI such as `hdfs://cluster1/`. Keys sharing the prefix are forwarded to that filesystem client | (none) | Review Comment: fs.path.config.<n>, It should be `name` NOT `n` here. ########## docs/fileset-catalog-with-s3.md: ########## @@ -1,568 +1,200 @@ --- -title: "Fileset Catalog with S3" +title: "Fileset Catalog with Amazon S3" slug: "/fileset-catalog-with-s3" -date: 2025-01-03 keyword: "Fileset catalog S3" license: "This software is licensed under the Apache License version 2." --- -## Introduction +## Overview -This document explains how to configure a Fileset catalog with S3 in Gravitino. +A fileset catalog backed by Amazon S3 stores fileset data in S3 while Gravitino manages the metadata. Clients reach the data through the Gravitino Virtual File System (GVFS) using a `gvfs://` path, so the storage backend stays behind the catalog. -## Prerequisites +Everything here is specific to S3. For the fileset model, the shared catalog, schema, and fileset properties, and property inheritance, see [Fileset Catalog](./fileset-catalog.md). -To create a Fileset catalog with S3, follow these steps: +## Quick Start -1. Download the [`gravitino-aws-bundle-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-aws-bundle) file. -2. Place this file in the Gravitino Fileset catalog classpath at `${GRAVITINO_HOME}/catalogs/fileset/libs/`. -3. Start the Gravitino server using the following command: - -```bash -$ ${GRAVITINO_HOME}/bin/gravitino-server.sh start -``` - -Once the server is up and running, you can proceed to configure the Fileset catalog with S3. In the rest of this document we will use `http://localhost:8090` as the Gravitino server URL, replace with your actual server URL. - -## S3 Catalog Configuration - -### S3 Fileset Catalog Configuration - -In addition to the basic configurations mentioned in [Fileset-catalog-catalog-configuration](./fileset-catalog.md#catalog-properties), the following properties are necessary to configure a Fileset catalog with S3: - -| Configuration item | Description | Default value | Required | -|-------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------|----------| -| `filesystem-providers` | (deprecated) The file system providers to add. Set it to `s3` if it's a S3 fileset, or a comma separated string that contains `s3` like `gs,s3` to support multiple kinds of fileset including `s3`. | (none) | Yes | -| `default-filesystem-provider` | (deprecated) The name default filesystem providers of this Fileset catalog if users do not specify the scheme in the URI. Default value is `builtin-local`, for S3, if we set this value, we can omit the prefix 's3a://' in the location. | `builtin-local` | No | -| `s3-endpoint` | The endpoint of the AWS S3. | (none) | Yes | -| `s3-access-key-id` | The access key of the AWS S3. | (none) | Yes | -| `s3-secret-access-key` | The secret key of the AWS S3. | (none) | Yes | -| `credential-providers` | The credential provider types, separated by comma, possible value can be `s3-token`, `s3-secret-key`. As the default authentication type is using AKSK as the above, this configuration can enable credential vending provided by Gravitino server and client will no longer need to provide authentication information like AKSK to access S3 by GVFS. Once it's set, more configuration items are needed to make it works, see [s3-credential-vending](security/credential-vending.md#s3-credentials) | (none) | No | - -:::note -`default-filesystem-provider` and `filesystem-providers` are deprecated. The fileset catalog automatically loads filesystem providers on the classpath, including buildin filesystem provider and cloud providers when the corresponding bundle jar is present (for example, `gravitino-aws-bundle`). -::: - -### Schema Configuration - -To learn how to create a schema, refer to [Schema configurations](./fileset-catalog.md#schema-properties). - -### Fileset Configuration - -For more details on creating a fileset, Refer to [Fileset configurations](./fileset-catalog.md#fileset-properties). - -## Create the Catalog, Schema, and Fileset - -This section demonstrates how to use the Fileset catalog with S3 in Gravitino, with a complete example. - -### Step 1: Create a Fileset Catalog with S3 - -First of all, you need to create a Fileset catalog with S3. The following example shows how to create a Fileset catalog with S3: - -<Tabs groupId="language" queryString> -<TabItem value="shell" label="Shell"> +**1. Install the bundle.** Download [`gravitino-aws-bundle`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-aws-bundle), place it in `${GRAVITINO_HOME}/catalogs/fileset/libs/`, and start the server. ```shell -curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ --H "Content-Type: application/json" -d '{ - "name": "test_catalog", - "type": "FILESET", - "comment": "This is a S3 fileset catalog", - "properties": { - "location": "s3a://bucket/root", - "s3-access-key-id": "access_key", - "s3-secret-access-key": "secret_key", - "s3-endpoint": "http://s3.ap-northeast-1.amazonaws.com"; - } -}' http://localhost:8090/api/metalakes/metalake/catalogs -``` - -</TabItem> -<TabItem value="java" label="Java"> - -```java -GravitinoClient gravitinoClient = GravitinoClient - .builder("http://localhost:8090";) - .withMetalake("metalake") - .build(); - -Map<String, String> s3Properties = ImmutableMap.<String, String>builder() - .put("location", "s3a://bucket/root") - .put("s3-access-key-id", "access_key") - .put("s3-secret-access-key", "secret_key") - .put("s3-endpoint", "http://s3.ap-northeast-1.amazonaws.com";) - .build(); - -Catalog s3Catalog = gravitinoClient.createCatalog("test_catalog", - Type.FILESET, - "This is a S3 fileset catalog", - s3Properties); -// ... - -``` - -</TabItem> -<TabItem value="python" label="Python"> - -```python -gravitino_client: GravitinoClient = GravitinoClient(uri="http://localhost:8090";, metalake_name="metalake") -s3_properties = { - "location": "s3a://bucket/root", - "s3-access-key-id": "access_key" - "s3-secret-access-key": "secret_key", - "s3-endpoint": "http://s3.ap-northeast-1.amazonaws.com"; -} - -s3_catalog = gravitino_client.create_catalog(name="test_catalog", - catalog_type=Catalog.Type.FILESET, - provider=None, - comment="This is a S3 fileset catalog", - properties=s3_properties) -``` - -</TabItem> -</Tabs> - -:::note -- When using S3, ensure that the location value starts with s3a:// (not s3://) for AWS S3. For example, use s3a://bucket/root, as the s3:// format is not supported by the hadoop-aws library. -- When using MinIO or other S3-compatible storage services, make sure to set the `s3-endpoint` property to the appropriate endpoint URL. -- When using MinIO or other S3-compatible storage services, you may need to set additional properties such as `s3-path-style-access` to `true` depending on the storage service requirements. You can do this in Gravitino's `fileset.conf` file with the "gravitino.bypass." prefix: -```bash -$ cat $GRAVITINO_HOME/catalogs/fileset/conf/fileset.conf -gravitino.bypass.fs.s3a.path.style.access=true +${GRAVITINO_HOME}/bin/gravitino-server.sh start ``` -::: - -### Step 2: Create a Schema - -Once your Fileset catalog with S3 is created, you can create a schema under the catalog. Here are examples of how to do that: -<Tabs groupId="language" queryString> -<TabItem value="shell" label="Shell"> +**2. Create the catalog.** ```shell -curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ --H "Content-Type: application/json" -d '{ - "name": "test_schema", - "comment": "This is a S3 schema", - "properties": { - "location": "s3a://bucket/root/schema" - } -}' http://localhost:8090/api/metalakes/metalake/catalogs/test_catalog/schemas -``` - -</TabItem> -<TabItem value="java" label="Java"> - -```java -Catalog catalog = gravitinoClient.loadCatalog("hive_catalog"); - -SupportsSchemas supportsSchemas = catalog.asSchemas(); - -Map<String, String> schemaProperties = ImmutableMap.<String, String>builder() - .put("location", "s3a://bucket/root/schema") - .build(); -Schema schema = supportsSchemas.createSchema("test_schema", - "This is a S3 schema", - schemaProperties -); -// ... -``` - -</TabItem> -<TabItem value="python" label="Python"> - -```python -gravitino_client: GravitinoClient = GravitinoClient(uri="http://localhost:8090";, metalake_name="metalake") -catalog: Catalog = gravitino_client.load_catalog(name="test_catalog") -catalog.as_schemas().create_schema(name="test_schema", - comment="This is a S3 schema", - properties={"location": "s3a://bucket/root/schema"}) -``` - -</TabItem> -</Tabs> - -### Step 3: Create a Fileset - -After creating the schema, you can create a fileset. Here are examples for creating a fileset: - -<Tabs groupId="language" queryString> -<TabItem value="shell" label="Shell"> +curl -X POST -H "Content-Type: application/json" \ + -d '{ + "name": "{catalog_name}", + "type": "FILESET", + "comment": "", + "properties": { + "location": "s3a://{bucket}/{prefix}", + "s3-endpoint": "{s3_endpoint}", + "s3-access-key-id": "{s3_access_key_id}", + "s3-secret-access-key": "{s3_secret_access_key}", Review Comment: Redundant ',' at the end. The other five pages also have the same problem. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
