This is an automated email from the ASF dual-hosted git repository.
mchades pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new 8289f20805 [#11521] docs(glue): Add AWS Glue catalog and connector
documentation (#11523)
8289f20805 is described below
commit 8289f20805b626dd30b5bc733c53ed0e0ee27f3a
Author: Yuhui <[email protected]>
AuthorDate: Tue Jun 16 10:05:24 2026 +0800
[#11521] docs(glue): Add AWS Glue catalog and connector documentation
(#11523)
### What changes were proposed in this pull request?
This PR adds documentation for AWS Glue catalog:
- AWS Glue catalog core documentation
- Trino connector Glue catalog guide
- Spark connector Glue catalog guide
### Why are the changes needed?
Users need complete documentation to use AWS Glue catalog with Gravitino
and its connectors.
Fix: #11521
### Does this PR introduce _any_ user-facing change?
- New docs: `docs/aws-glue-catalog.md`
- New docs: `docs/trino-connector/catalog-glue.md`
- New docs: `docs/spark-connector/spark-catalog-glue.md`
- Updated: `docs/trino-connector/index.md`
- Updated: `docs/trino-connector/supported-catalog.md`
- Updated: `docs/spark-connector/spark-connector.md`
### How was this patch tested?
Documentation changes validated by manual review.
---
docs/aws-glue-catalog.md | 311 +++++++++++++++++++++++++++
docs/security/how-to-use-built-in-idp.md | 4 +-
docs/spark-connector/spark-catalog-glue.md | 325 +++++++++++++++++++++++++++++
docs/spark-connector/spark-connector.md | 2 +-
docs/trino-connector/catalog-glue.md | 303 +++++++++++++++++++++++++++
docs/trino-connector/index.md | 1 +
docs/trino-connector/supported-catalog.md | 4 +-
7 files changed, 946 insertions(+), 4 deletions(-)
diff --git a/docs/aws-glue-catalog.md b/docs/aws-glue-catalog.md
new file mode 100644
index 0000000000..ccdff0fb99
--- /dev/null
+++ b/docs/aws-glue-catalog.md
@@ -0,0 +1,311 @@
+---
+title: "AWS Glue catalog"
+slug: /aws-glue-catalog
+keywords:
+ - glue
+ - aws
+ - metadata
+license: "This software is licensed under the Apache License version 2."
+---
+
+## Introduction
+
+Apache Gravitino uses [AWS Glue Data Catalog](https://aws.amazon.com/glue/) as
a metadata catalog.
+
+### Requirements
+
+* The Glue catalog requires network access to the AWS Glue API.
+* Gravitino uses the AWS SDK v2 to communicate with Glue.
+
+:::note
+The Glue catalog is case-insensitive for schema and table names. AWS Glue
folds database and table names to lowercase on storage.
+:::
+
+## Catalog
+
+### Catalog Capabilities
+
+The Glue catalog supports creating, updating, and deleting databases and
tables in the AWS Glue Data Catalog.
+
+- Supports all table types stored in Glue (Hive, Iceberg, Delta, Parquet, and
others) by default.
+- Supports Hive-format table partitioning, bucketing, and sort orders.
+- Does not support views. Glue views (tables with `TableType=VIRTUAL_VIEW`)
are filtered out.
+
+### Catalog Properties
+
+Besides the [common catalog
properties](./gravitino-server-config.md#catalog-properties-configuration), the
Glue catalog has the following properties:
+
+| Property Name | Description
|
Default Value | Required | Immutable | Since Version |
+|--------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------|----------|-----------|---------------|
+| `aws-region` | AWS region for the Glue Data Catalog (e.g.
`us-east-1`).
| (none) | Yes | Yes | 1.3.0 |
+| `aws-glue-catalog-id` | The 12-digit AWS account ID that owns the Glue
catalog. When omitted, defaults to the caller's AWS account ID.
|
(none) | No | Yes | 1.3.0 |
+| `aws-access-key-id` | AWS access key ID for static credential
authentication. When omitted, the default credential chain is used.
| (none) | No | No | 1.3.0 |
+| `aws-secret-access-key` | AWS secret access key paired with
`aws-access-key-id`. When omitted, the default credential chain is used.
| (none) | No | No | 1.3.0 |
+| `aws-glue-endpoint` | Custom Glue endpoint URL for VPC endpoints or
LocalStack testing (e.g. `http://localhost:4566`).
|
(none) | No | No | 1.3.0 |
+| `warehouse` | Base storage path used as the warehouse when no
explicit `location` is specified at table creation time (e.g.
`s3://my-bucket/warehouse`). Table location is derived as
`warehouse/database/table`. | (none) | Yes | No | 1.3.0 |
+| `default-table-format` | Default format for tables created via Gravitino's
`createTable()` API. Accepted values: `iceberg`, `hive`.
|
`hive` | No | No | 1.3.0 |
+| `table-format-filter` | Comma-separated list of table formats exposed by
`listTables()` and `loadTable()`. Accepted values: `all`, `hive`, `iceberg`,
`delta`, `parquet`. Use to restrict visible table types. |
`all` | No | No | 1.3.0 |
+
+:::note
+**Authentication priority**: Static credentials (`aws-access-key-id` +
`aws-secret-access-key`) take precedence over the default credential chain
(environment variables, instance profile, container credentials).
+:::
+
+### Catalog Operations
+
+Refer to [Manage Relational Metadata Using
Gravitino](./manage-relational-metadata-using-gravitino.md#catalog-operations)
for more details.
+
+## Schema
+
+### Schema Capabilities
+
+The Glue catalog supports creating, updating, and deleting databases in the
AWS Glue Data Catalog.
+
+### Schema Properties
+
+The Glue catalog defines no predefined schema properties beyond `comment`.
Additional key-value properties pass through to the underlying Glue database.
+
+### Schema Operations
+
+See [Manage Relational Metadata Using
Gravitino](./manage-relational-metadata-using-gravitino.md#schema-operations).
+
+## Table
+
+### Table Capabilities
+
+- The Glue catalog supports creating, updating, and deleting tables in the AWS
Glue Data Catalog.
+- All entries in the Glue `Table.parameters()` pass through Gravitino intact,
so downstream tools can correctly identify the table format.
+- Does not support column default value.
+- Does not support NOT NULL constraints on columns.
+- Does not support table indexes.
+
+### Table Partitioning
+
+The Glue catalog supports [partitioned
tables](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-PartitionedTables).
Create partitioned tables in the Glue catalog by specifying the partitioning
attribute.
+
+The supported partitioning strategies depend on the table format:
+
+- **Hive-format tables**: Only `Identity` partitioning is supported, because
the native Glue partition model is Hive-style key=value.
+- **Iceberg-format tables**: All Iceberg partition transforms are supported:
`identity`, `year`, `month`, `day`, `hour`, `bucket`, and `truncate`.
+
+:::caution
+The `fieldName` specified in the partitioning attribute must be the name of a
column defined in the table.
+:::
+
+### Table Sort Orders and Distributions
+
+The Glue catalog supports [bucketed sorted
tables](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-BucketedSortedTables).
Create bucketed sorted tables by setting the `distribution` and `sortOrders`
attributes.
+Although Gravitino supports several distribution strategies, AWS Glue
inherently only supports a single distribution strategy (clustered by column).
Therefore, the Glue catalog only supports `Hash` distribution.
+
+:::caution
+The `fieldName` specified in the `distribution` and `sortOrders` attribute
must be the name of a column defined in the table.
+:::
+
+### Table Column Types
+
+The Glue catalog supports all data types defined in the [Hive Language
Manual](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types).
+The following table lists the data types mapped from the Glue catalog to
Gravitino.
+
+| Glue Data Type | Gravitino Data Type | Since Version |
+|---------------------------|---------------------|---------------|
+| `boolean` | `boolean` | 1.3.0 |
+| `tinyint` | `byte` | 1.3.0 |
+| `smallint` | `short` | 1.3.0 |
+| `int` / `integer` | `integer` | 1.3.0 |
+| `bigint` | `long` | 1.3.0 |
+| `float` | `float` | 1.3.0 |
+| `double` | `double` | 1.3.0 |
+| `decimal` | `decimal` | 1.3.0 |
+| `string` | `string` | 1.3.0 |
+| `char` | `char` | 1.3.0 |
+| `varchar` | `varchar` | 1.3.0 |
+| `timestamp` | `timestamp` | 1.3.0 |
+| `date` | `date` | 1.3.0 |
+| `interval_year_month` | `interval_year` | 1.3.0 |
+| `interval_day_time` | `interval_day` | 1.3.0 |
+| `binary` | `binary` | 1.3.0 |
+| `array` | `list` | 1.3.0 |
+| `map` | `map` | 1.3.0 |
+| `struct` | `struct` | 1.3.0 |
+| `uniontype` | `union` | 1.3.0 |
+
+:::info
+Data types not listed above map to Gravitino **[External
Type](./manage-relational-metadata-using-gravitino.md#external-type)**, which
represents an unresolvable data type from the Glue catalog.
+:::
+
+### Table Properties
+
+The following table lists predefined properties for Glue tables. Additional
key-value properties pass through to the underlying Glue database.
+
+:::note
+**Reserved**: Fields that cannot be passed to the Gravitino server.
+
+**Immutable**: Fields that cannot be modified once set.
+:::
+
+| Property Name | Description
| Default Value
| Required | Reserved | Immutable | Since Version |
+|-------------------------|--------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------|----------|----------|-----------|---------------|
+| `location` | The location for table storage, such as
`s3://bucket/prefix/test_table`. Derived from `warehouse/database/table` when
not specified. | (derived from warehouse)
| No | No | No | 1.3.0 |
+| `format` | The table file format (`parquet`, `orc`,
`textfile`, etc.). When set, `input-format`, `output-format`, and `serde-lib`
are derived automatically. Used primarily when creating Hive-format tables via
Trino. | (none) | No | No | Yes | 1.3.0 |
+| `input-format` | The input format class for the table, such as
`org.apache.hadoop.hive.ql.io.orc.OrcInputFormat`.
| `org.apache.hadoop.mapred.TextInputFormat`
| No | No | Yes | 1.3.0 |
+| `output-format` | The output format class for the table, such as
`org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat`.
| `org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat`
| No | No | Yes | 1.3.0 |
+| `serde-lib` | The serde library class for the table, such as
`org.apache.hadoop.hive.ql.io.orc.OrcSerde`.
| `org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe`
| No | No | Yes | 1.3.0 |
+| `serde-name` | The name of the serde.
| (none)
| No | No | No | 1.3.0 |
+| `serde.parameter.` | The prefix of the serde parameter, such as
`"serde.parameter.orc.create.index" = "true"`, indicating ORC serde lib to
create row indexes. | (none)
| No | No | No | 1.3.0 |
+| `table-format` | Table format stored in Glue `Table.parameters()`.
Use `ICEBERG` to create an Iceberg table. Common values: `ICEBERG`, `HIVE`.
| (none)
| No | No | No | 1.3.0 |
+| `metadata_location` | Iceberg table metadata file location stored in
Glue `Table.parameters()`. When set during `createTable()`, registers an
existing Iceberg table rather than creating a new one. | (none)
| No | No | No
| 1.3.0 |
+| `comment` | Used to store a table comment.
| (none)
| No | Yes | No | 1.3.0 |
+
+:::note
+All entries in the Glue `Table.parameters()` pass through Gravitino's API
layer intact. This passthrough ensures `table_type=ICEBERG`,
`metadata_location=s3://...`, `spark.sql.sources.provider=delta`, and any other
format indicators survive Gravitino's metadata proxy layer.
+:::
+
+### Table Operations
+
+Refer to [Manage Relational Metadata Using
Gravitino](./manage-relational-metadata-using-gravitino.md#table-operations)
for more details.
+
+#### Alter Operations
+
+Gravitino defines a unified set of [metadata operation
interfaces](./manage-relational-metadata-using-gravitino.md#alter-a-table). The
following table maps Glue alter operations to Gravitino table update requests.
+
+##### Alter table
+
+| Glue Alter Operation | Gravitino Table Update Request | Since Version |
+|--------------------------|--------------------------------|---------------|
+| `Alter Table Properties` | `Set a table property` | 1.3.0 |
+| `Alter Table Comment` | `Update comment` | 1.3.0 |
+| `Remove Properties` | `Remove a table property` | 1.3.0 |
+
+:::caution
+Hive-format table rename is not supported. AWS Glue does not provide a native
rename API for tables; renaming would require recreating the table.
+Iceberg-format table rename is supported.
+:::
+
+##### Alter column
+
+| Glue Alter Operation | Gravitino Table Update Request | Since Version
|
+|--------------------------|-----------------------------------|---------------|
+| `Change Column Name` | `Rename a column` | 1.3.0
|
+| `Change Column Type` | `Update the type of a column` | 1.3.0
|
+| `Change Column Position` | `Update the position of a column` | 1.3.0
|
+| `Change Column Comment` | `Update the column comment` | 1.3.0
|
+
+##### Alter partition
+
+The Glue catalog supports partition operations via `SupportsPartitions` for
Hive-format identity-partitioned tables:
+
+- `listPartitions()` / `listPartitionNames()`
+- `getPartition(partitionName)`
+- `addPartition(partition)`
+- `dropPartition(partitionName)`
+
+:::caution
+Only `IdentityPartition` is supported because the Glue partition model is
Hive-style key=value.
+:::
+
+## Iceberg Tables
+
+The Glue catalog supports creating and managing Iceberg-format tables through
the Apache Iceberg SDK's `GlueCatalog`. When an Iceberg table is created,
Gravitino writes the `metadata.json` file to S3 and registers the table in Glue
with the correct `metadata_location` parameter, making it usable by Trino
(Lakehouse connector), Spark, and other Iceberg-native query engines.
+
+### Create an Iceberg Table
+
+Set `table-format=ICEBERG` in the table properties, or configure
`default-table-format=iceberg` on the catalog to make all tables Iceberg by
default.
+
+The `warehouse` catalog property must be configured. The table location is
derived as `warehouse/database/table` when no explicit `location` is specified.
+
+### Register an Existing Iceberg Table
+
+To register an Iceberg table that already exists in S3, set
`metadata_location` to the path of the existing `metadata.json` file during
`createTable()`. In this mode, Gravitino registers the table in Glue without
creating new metadata.
+
+### Iceberg Column Types
+
+Iceberg tables use the Iceberg type system, which differs from Hive types. The
following table lists the Gravitino types supported for Iceberg tables and how
they map to Iceberg types:
+
+| Gravitino Data Type | Iceberg Data Type | Notes
|
+|--------------------------|-----------------------|-----------------------------------------------------|
+| `boolean` | `boolean` |
|
+| `byte` | `int` | Widened to 32-bit integer
|
+| `short` | `int` | Widened to 32-bit integer
|
+| `integer` | `int` |
|
+| `long` | `long` |
|
+| `float` | `float` |
|
+| `double` | `double` |
|
+| `decimal(p, s)` | `decimal(p, s)` |
|
+| `string` | `string` |
|
+| `varchar` | `string` | Iceberg has no
variable-length char types |
+| `char` | `string` | Iceberg has no
variable-length char types |
+| `date` | `date` |
|
+| `time(6)` | `time` | Only microsecond
precision (6) is supported |
+| `timestamp(6)` | `timestamp` | Only microsecond
precision (6) is supported |
+| `timestamptz(6)` | `timestamptz` | Only microsecond
precision (6) is supported |
+| `binary` | `binary` |
|
+| `fixed(n)` | `binary` | Mapped to variable-length
binary |
+| `uuid` | `uuid` |
|
+| `list` | `list` |
|
+| `map` | `map` |
|
+| `struct` | `struct` |
|
+
+### Iceberg Table Alter Operations
+
+For Iceberg tables, the following alter operations are supported:
+
+| Operation | Gravitino Table Update Request |
+|---------------------------------|-----------------------------------|
+| Add column | `Add a column` |
+| Delete column | `Delete a column` |
+| Rename column | `Rename a column` |
+| Update column type | `Update the type of a column` |
+| Update column comment | `Update the column comment` |
+| Update column nullability | `Update column nullability` |
+| Set table property | `Set a table property` |
+| Remove table property | `Remove a table property` |
+
+:::caution
+Schema changes and property changes are committed in two separate Iceberg
transactions. If the schema commit succeeds but the property commit fails, the
table is left in a partially altered state.
+:::
+
+:::caution
+Nested column operations (add, delete, rename, type update) are not supported
for Iceberg tables via this catalog.
+:::
+
+## Security
+
+### AWS IAM Permissions
+
+The IAM policy attached to the credential used by the Glue catalog must cover
both Glue metadata access and S3 data access:
+
+```json
+{
+ "Version": "2012-10-17",
+ "Statement": [
+ {
+ "Sid": "GlueMetadataAccess",
+ "Effect": "Allow",
+ "Action": [
+ "glue:GetCatalog",
+ "glue:GetDatabase", "glue:GetDatabases",
+ "glue:CreateDatabase", "glue:UpdateDatabase", "glue:DeleteDatabase",
+ "glue:GetTable", "glue:GetTables",
+ "glue:CreateTable", "glue:UpdateTable", "glue:DeleteTable",
+ "glue:GetPartition", "glue:GetPartitions",
+ "glue:CreatePartition", "glue:DeletePartition"
+ ],
+ "Resource": [
+ "arn:aws:glue:<region>:<account-id>:catalog",
+ "arn:aws:glue:<region>:<account-id>:database/*",
+ "arn:aws:glue:<region>:<account-id>:table/*/*"
+ ]
+ },
+ {
+ "Sid": "S3DataAccess",
+ "Effect": "Allow",
+ "Action": [
+ "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucket"
+ ],
+ "Resource": [
+ "arn:aws:s3:::<warehouse-bucket>",
+ "arn:aws:s3:::<warehouse-bucket>/*"
+ ]
+ }
+ ]
+}
+```
diff --git a/docs/security/how-to-use-built-in-idp.md
b/docs/security/how-to-use-built-in-idp.md
index 0c0a621bc3..09dd0e0ad5 100644
--- a/docs/security/how-to-use-built-in-idp.md
+++ b/docs/security/how-to-use-built-in-idp.md
@@ -21,10 +21,10 @@ request and response schemas, see the [Built-in IDP
OpenAPI](../open-api/idp/ope
## Web UI
-Built-in IdP is **incompatible** with the `simple` authenticator (the
default). The Web UI does not
+Built-in IDP is **incompatible** with the `simple` authenticator (the
default). The Web UI does not
support signing in with built-in IDP Basic credentials. Login pages only
handle `simple`
(username-only when authorization is enabled) and `oauth` (OIDC/OAuth) flows.
If you need the Web UI
-while built-in IdP is enabled, prefer [OAuth
mode](how-to-authenticate.md#oauth-mode). For Basic
+while built-in IDP is enabled, prefer [OAuth
mode](how-to-authenticate.md#oauth-mode). For Basic
authentication, use the REST API, [Java/Python
clients](how-to-authenticate.md#basic-mode), or
engine connectors instead.
diff --git a/docs/spark-connector/spark-catalog-glue.md
b/docs/spark-connector/spark-catalog-glue.md
new file mode 100644
index 0000000000..97b3c0693b
--- /dev/null
+++ b/docs/spark-connector/spark-catalog-glue.md
@@ -0,0 +1,325 @@
+---
+title: "Spark connector Glue catalog"
+slug: /spark-connector/spark-catalog-glue
+keyword: spark connector glue catalog aws
+license: "This software is licensed under the Apache License version 2."
+---
+
+## Overview
+
+With the Apache Gravitino Spark connector, accessing data or managing metadata
in AWS Glue Data Catalog becomes straightforward, enabling seamless federated
queries across Glue catalogs.
+
+## Capabilities
+
+Supports most DDL and DML operations in SparkSQL, except these operations:
+
+- Function operations (Gravitino UDFs are supported, see [Spark connector -
User-defined functions](spark-connector-udf.md))
+- Partition operations
+- View operations
+- `LOAD` clause
+- `CREATE TABLE LIKE` clause
+- `TRUNCATE TABLE` clause
+
+## Table Format Support
+
+The Glue catalog supports mixed table formats within a single database:
+
+- **Hive-format tables** (PARQUET, ORC, TEXTFILE, etc.): routed to
HiveTableCatalog using the AWS Glue Data Catalog Hive client
+- **Iceberg-format tables**: routed to Iceberg's SparkCatalog (GlueCatalog)
for I/O
+
+Table routing is based on the `table-format` property in Glue table
parameters. Tables with `table-format=ICEBERG` are delegated to the Iceberg
backend.
+
+## Requirements
+
+- Network access to the AWS Glue API and Amazon S3
+- AWS IAM credentials with necessary Glue and S3 permissions.
+ See [AWS IAM permissions](../aws-glue-catalog.md#aws-iam-permissions) for
the required policy
+- Apache Spark 3.3, 3.4, or 3.5
+- Patched Hive and AWS Glue client JARs (see [Setup](#setup); pre-installed on
Amazon EMR)
+- `iceberg-spark-runtime` JAR on the Spark classpath for Iceberg table support
+
+## Setup
+
+Spark's bundled Hive 2.3.9 does not include the `HiveMetaStoreClientFactory`
interface
+(added by [HIVE-12679](https://issues.apache.org/jira/browse/HIVE-12679)) that
the AWS Glue client
+requires. Replace the bundled Hive JARs with patched versions bundled together
with the
+Glue client.
+
+:::note
+On AWS managed Spark environments such as Amazon EMR, the Hive libraries are
already patched and
+the AWS Glue Data Catalog client is pre-installed. Skip Steps 1 and 2 below.
+For a complete walkthrough on Amazon EMR, see [Deploy on Amazon
EMR](#deploy-on-amazon-emr).
+:::
+
+[spark-hive-glue-libs](https://github.com/datastrato/spark-hive-glue-libs)
provides pre-built JARs
+that include the patched Hive 2.3.10 and the AWS Glue Data Catalog client for
Spark.
+
+### Step 1: Download the JARs
+
+```bash
+BASE=https://raw.githubusercontent.com/datastrato/spark-hive-glue-libs/main/spark3/glue-3.4.0
+mkdir -p /opt/glue-hive-jars
+for jar in \
+ aws-glue-datacatalog-spark-client-3.4.0.jar \
+ hive-exec-2.3.10.jar \
+ hive-metastore-2.3.10.jar \
+ hive-common-2.3.10.jar \
+ hive-serde-2.3.10.jar \
+ hive-shims-2.3.10.jar \
+ aws-java-sdk-glue-1.12.31.jar \
+ aws-java-sdk-core-1.12.31.jar \
+ jmespath-java-1.12.31.jar; do
+ wget "$BASE/$jar" -P /opt/glue-hive-jars/
+done
+```
+
+### Step 2: Configure Spark
+
+Add the following configurations when starting Spark:
+
+```bash
+spark-submit \
+ --conf spark.sql.hive.metastore.version=2.3.10 \
+ --conf spark.sql.hive.metastore.jars=path \
+ --conf spark.sql.hive.metastore.jars.path=/opt/glue-hive-jars/* \
+ --conf
spark.sql.hive.metastore.sharedPrefixes=com.amazonaws,org.apache.thrift,org.slf4j,com.google.common
\
+ ...
+```
+
+:::note
+The `spark.sql.hive.metastore.jars=path` configuration instructs Spark to load
the Hive metastore
+client from the specified directory instead of its bundled Hive JARs.
+The AWS SDK JARs in the directory are loaded in Spark's `IsolatedClientLoader`
to prevent version
+conflicts with `hadoop-aws`.
+:::
+
+## Deploy on Amazon EMR
+
+Amazon EMR 7.x pre-installs the patched Hive libraries and the AWS Glue Data
Catalog client,
+so the manual JAR setup described in [Setup](#setup) is not required.
+
+### Prerequisites
+
+- AWS CLI configured with an IAM user or role that has
`AmazonEMRFullAccessPolicy_v2` and EC2 permissions
+- EC2 instance profile (`EMR_EC2_DefaultRole`) with Glue read and S3
read/write permissions
+- An S3 bucket for table storage (e.g. `s3://my-bucket/warehouse`)
+- A Gravitino server reachable from the EMR cluster
+
+### Step 1: Create an EMR cluster
+
+```bash
+aws emr create-cluster \
+ --name "gravitino-glue" \
+ --release-label emr-7.2.0 \
+ --applications Name=Spark \
+ --instance-type m5.xlarge \
+ --instance-count 1 \
+ --use-default-roles \
+ --region <your-region> \
+ --tags 'for-use-with-amazon-emr-managed-policies=true' \
+ --configurations
'[{"Classification":"spark-hive-site","Properties":{"hive.metastore.client.factory.class":"com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory"}}]'
\
+ --ec2-attributes KeyName=<your-key-pair>
+```
+
+The `spark-hive-site` configuration routes Spark's Hive metastore client to
the AWS Glue Data
+Catalog. The `for-use-with-amazon-emr-managed-policies=true` tag is required by
+`AmazonEMRFullAccessPolicy_v2`.
+
+### Step 2: Add JARs to the Spark classpath
+
+Follow the [Spark connector setup](spark-connector.md#usage) to obtain the
+`gravitino-spark-connector-runtime-3.5` JAR. On EMR, place it in
`/usr/lib/spark/jars/` instead
+of a custom path — that directory is automatically on Spark's system
classpath. Using `--jars` or
+`--driver-class-path` is not sufficient because the Gravitino plugin
classloader must find the JAR
+at Spark startup.
+
+For Iceberg table support, also add the Iceberg Spark runtime JAR to
`/usr/lib/spark/jars/`:
+
+-
[iceberg-spark-runtime-3.5_2.12](https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-spark-runtime-3.5_2.12)
version **1.10.1**
+
+:::warning
+Use `iceberg-spark-runtime` version **1.10.1**, not 1.11.0. EMR 7.2.0 ships
with AWS SDK v2
+2.23.18, which does not include `RetryMode.ADAPTIVE_V2`. Iceberg 1.11.0
references this field
+at runtime and throws `NoSuchFieldError: ADAPTIVE_V2`.
+:::
+
+### Step 3: Create a Glue catalog
+
+```bash
+curl -X POST -H "Content-Type: application/json" \
+ -d '{
+ "name": "glue_catalog",
+ "type": "RELATIONAL",
+ "provider": "glue",
+ "properties": {
+ "aws-region": "<your-region>",
+ "warehouse": "s3://<your-bucket>/warehouse"
+ }
+ }' http://<gravitino-host>:8090/api/metalakes/<metalake>/catalogs
+```
+
+When running on EMR with an EC2 instance role, AWS credentials are picked up
automatically —
+no static `aws-access-key-id` or `aws-secret-access-key` is needed.
+
+### Step 4: Submit a Spark job
+
+:::warning
+Use `spark-submit`, not `spark-sql`. The `spark-sql` CLI does not initialize
the Gravitino
+plugin classloader in the correct order, so registered catalogs will not
appear in `SHOW CATALOGS`.
+:::
+
+Pass the Gravitino plugin configuration via `spark-submit`:
+
+```bash
+spark-submit --master yarn \
+ --conf
spark.plugins=org.apache.gravitino.spark.connector.plugin.GravitinoSparkPlugin \
+ --conf spark.sql.gravitino.uri=http://<gravitino-host>:8090 \
+ --conf spark.sql.gravitino.metalake=<metalake> \
+ --conf spark.sql.gravitino.enableIcebergSupport=true \
+ --conf spark.sql.catalogImplementation=hive \
+ --conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
+ --conf spark.hadoop.fs.s3.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
+ your_job.py
+```
+
+The following PySpark snippet shows how to query Hive-format and Iceberg
tables in the Glue catalog:
+
+```python
+from pyspark.sql import SparkSession
+
+spark = SparkSession.builder.getOrCreate()
+
+spark.sql("USE glue_catalog.mydb")
+
+# Query a Hive-format table
+spark.sql("SELECT * FROM employees WHERE department = 'Engineering'").show()
+
+# Query an Iceberg table
+spark.sql("SELECT * FROM orders WHERE order_ts >= DATE '2024-01-01'").show()
+
+spark.stop()
+```
+
+## Create a Catalog
+
+Use the Gravitino REST API or the Gravitino CLI to create a Glue catalog:
+
+```bash
+curl -X POST -H "Content-Type: application/json" \
+-d '{
+ "name": "glue_catalog",
+ "type": "RELATIONAL",
+ "provider": "glue",
+ "properties": {
+ "aws-region": "us-east-1",
+ "aws-access-key-id": "<your-access-key-id>",
+ "aws-secret-access-key": "<your-secret-access-key>",
+ "warehouse": "s3://my-bucket/warehouse"
+ }
+}' http://gravitino-host:8090/api/metalakes/{metalake}/catalogs
+```
+
+For more information about the Glue catalog properties, refer to [AWS Glue
catalog](../aws-glue-catalog.md).
+
+## SQL Example
+
+```sql
+-- Suppose glue_catalog is the Glue catalog name managed by Gravitino
+USE glue_catalog;
+
+CREATE DATABASE IF NOT EXISTS mydatabase;
+USE mydatabase;
+
+-- Create a Hive-format Parquet table
+CREATE TABLE IF NOT EXISTS employees (
+ id INT,
+ name STRING,
+ age INT
+)
+PARTITIONED BY (department STRING)
+STORED AS PARQUET;
+
+DESC TABLE EXTENDED employees;
+
+INSERT OVERWRITE TABLE employees PARTITION(department='Engineering')
+VALUES (1, 'John Doe', 30), (2, 'Jane Smith', 28);
+
+SELECT * FROM employees WHERE department = 'Engineering';
+
+-- Create an Iceberg table partitioned by day
+CREATE TABLE IF NOT EXISTS orders (
+ order_id BIGINT,
+ customer STRING,
+ amount DECIMAL(10, 2),
+ order_ts TIMESTAMP
+)
+USING iceberg
+PARTITIONED BY (days(order_ts));
+
+INSERT INTO orders VALUES (1, 'alice', 99.99, TIMESTAMP '2024-01-01 00:00:00');
+
+SELECT * FROM orders WHERE order_ts >= DATE '2024-01-01';
+```
+
+
+## Catalog Properties
+
+The Gravitino Spark connector maps the following catalog property names to
Spark Hive/Iceberg connector configuration.
+
+| Property name in Gravitino catalog properties | Spark Hive connector
configuration | Description | Since Version |
+|-----------------------------------------------|------------------------------------|---------------------------------------------|---------------|
+| `aws-region` | `aws.region`
| AWS region for Glue Data Catalog | 1.3.0 |
+| `aws-glue-catalog-id` | `aws.glue.catalog.id`
| 12-digit AWS account ID owning Glue catalog | 1.3.0 |
+| `aws-glue-endpoint` | `aws.glue.endpoint`
| Custom Glue endpoint URL | 1.3.0 |
+| `warehouse` | (Iceberg) `warehouse`
| Base storage path for Iceberg tables | 1.3.0 |
+
+For Iceberg tables, Gravitino properties are mapped to Iceberg GlueCatalog
configuration:
+
+| Gravitino property | Iceberg GlueCatalog property |
Description |
+|-------------------------|---------------------------------------|-------------------------------------------|
+| `aws-region` | `client.region` | AWS region
|
+| `aws-glue-catalog-id` | `glue.id` | Glue
catalog ID |
+| `aws-glue-endpoint` | `glue.endpoint` | Glue
endpoint URL |
+| `aws-access-key-id` | `client.credentials-provider.*` | AWS access
key (via custom provider) |
+| `aws-secret-access-key` | `client.credentials-provider.*` | AWS secret
key (via custom provider) |
+
+Gravitino catalog property names with the prefix `spark.bypass.` are passed
directly to the Spark Hive connector. For example, using
`spark.bypass.hive.exec.dynamic.partition.mode` to pass
`hive.exec.dynamic.partition.mode` to the Spark Hive connector.
+
+## S3 Storage
+
+When using AWS S3, configure S3 credentials as global Hadoop properties via
Spark configuration:
+
+```bash
+spark-submit \
+ --conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
+ --conf spark.hadoop.fs.s3a.access.key=<your-access-key> \
+ --conf spark.hadoop.fs.s3a.secret.key=<your-secret-key> \
+ --conf spark.hadoop.fs.s3a.endpoint.region=<your-region> \
+ --conf spark.hadoop.fs.s3.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
+ --conf spark.hadoop.fs.s3.access.key=<your-access-key> \
+ --conf spark.hadoop.fs.s3.secret.key=<your-secret-key> \
+ ...
+```
+
+:::note
+Both `fs.s3a.*` and `fs.s3.*` must be configured. The AWS Glue Data Catalog
Hive client stores table
+locations using the `s3://` scheme, while Spark uses `s3a://` for S3 access.
Mapping both schemes to
+`S3AFileSystem` ensures Hive-format tables are readable.
+:::
+
+Alternatively, use `spark.bypass.` prefix in catalog properties to pass S3
configuration:
+
+```json
+{
+ "properties": {
+ "aws-region": "us-east-1",
+ "warehouse": "s3a://my-bucket/warehouse",
+ "spark.bypass.fs.s3a.access.key": "<your-access-key>",
+ "spark.bypass.fs.s3a.secret.key": "<your-secret-key>"
+ }
+}
+```
+
+:::note
+Ensure you have the AWS Java SDK and Hadoop AWS JARs in your Spark classpath
when using S3 storage.
+:::
diff --git a/docs/spark-connector/spark-connector.md
b/docs/spark-connector/spark-connector.md
index b0bbffe247..dc36ab117c 100644
--- a/docs/spark-connector/spark-connector.md
+++ b/docs/spark-connector/spark-connector.md
@@ -11,7 +11,7 @@ The Apache Gravitino Spark connector leverages the Spark
DataSourceV2 interface
## Capabilities
-1. Supports [Hive catalog](spark-catalog-hive.md), [Iceberg
catalog](spark-catalog-iceberg.md), [Paimon catalog](spark-catalog-paimon.md)
and [Jdbc catalog](spark-catalog-jdbc.md).
+1. Supports [Hive catalog](spark-catalog-hive.md), [Iceberg
catalog](spark-catalog-iceberg.md), [Paimon catalog](spark-catalog-paimon.md),
[Jdbc catalog](spark-catalog-jdbc.md), and [AWS Glue
catalog](spark-catalog-glue.md).
2. Supports federation query.
3. Supports most DDL and DML SQLs.
diff --git a/docs/trino-connector/catalog-glue.md
b/docs/trino-connector/catalog-glue.md
new file mode 100644
index 0000000000..94d3dda342
--- /dev/null
+++ b/docs/trino-connector/catalog-glue.md
@@ -0,0 +1,303 @@
+---
+title: "Apache Gravitino Trino connector - Glue catalog"
+slug: /trino-connector/catalog-glue
+keyword: gravitino connector trino glue aws
+license: "This software is licensed under the Apache License version 2."
+---
+
+## Overview
+
+The Glue catalog allows Trino to query data stored in tables registered in the
AWS Glue Data Catalog.
+It supports both Hive-format and Iceberg-format tables on Amazon S3.
+
+## Requirements
+
+- Network access from the Trino coordinator and workers to the AWS Glue API
and Amazon S3.
+- An AWS IAM credential (access key pair or instance profile) with the
necessary Glue and S3 permissions.
+ See [AWS IAM permissions](../aws-glue-catalog.md#aws-iam-permissions) for
the required policy.
+- Data files stored on Amazon S3.
+
+## Schema Operations
+
+### Create a Schema
+
+```sql
+CREATE SCHEMA glue_test.schema_name;
+```
+
+## Table Operations
+
+### Create a Hive-Format Table
+
+The default table format is Hive. The following example creates an ORC-format
partitioned table:
+
+```sql
+CREATE TABLE glue_test.db01.orders
+(
+ order_id bigint,
+ customer varchar,
+ amount decimal(10, 2),
+ order_dt date
+)
+WITH (
+ format = 'ORC',
+ location = 's3://my-bucket/warehouse/db01/orders',
+ partitioned_by = ARRAY['order_dt']
+);
+```
+
+To create a bucketed, sorted table:
+
+```sql
+CREATE TABLE glue_test.db01.events
+(
+ event_id bigint,
+ user_id bigint,
+ ts timestamp
+)
+WITH (
+ format = 'PARQUET',
+ location = 's3://my-bucket/warehouse/db01/events',
+ bucketed_by = ARRAY['user_id'],
+ bucket_count = 8,
+ sorted_by = ARRAY['ts DESC']
+);
+```
+
+### Create an Iceberg-Format Table
+
+Set `type = 'ICEBERG'` to create an Iceberg table. Iceberg tables support
richer partition transforms.
+
+```sql
+CREATE TABLE glue_test.db01.logs
+(
+ log_id bigint,
+ message varchar,
+ event_ts timestamp(6) with time zone
+)
+WITH (
+ type = 'ICEBERG',
+ location = 's3://my-bucket/warehouse/db01/logs',
+ partitioned_by = ARRAY['hour(event_ts)']
+);
+```
+
+Supported Iceberg partition transform expressions in `partitioned_by`:
+
+| Expression | Description |
+|---------------------|--------------------------|
+| `column` | Identity partition |
+| `year(column)` | Partition by year |
+| `month(column)` | Partition by month |
+| `day(column)` | Partition by day |
+| `hour(column)` | Partition by hour |
+| `bucket(column, N)` | Hash into N buckets |
+| `truncate(column, W)` | Truncate to width W |
+
+:::note
+`CREATE OR REPLACE TABLE AS SELECT` is not supported. Use `DROP TABLE`
followed by `CREATE TABLE AS SELECT` as an alternative.
+:::
+
+### Alter Table
+
+The following alter table operations are supported:
+
+- Add a column
+- Drop a column
+- Rename a column
+- Change a column type
+- Set a table property
+
+:::caution
+Hive-format table rename is not supported. AWS Glue does not provide a native
rename API for tables.
+Iceberg-format table rename is supported.
+:::
+
+### Select
+
+```sql
+SELECT * FROM glue_test.db01.orders WHERE order_dt = DATE '2024-01-01';
+```
+
+### Insert
+
+```sql
+INSERT INTO glue_test.db01.orders (order_id, customer, amount, order_dt)
+VALUES (1, 'alice', 99.99, DATE '2024-01-01');
+```
+
+### Update
+
+`UPDATE` is supported only for Iceberg tables (v2 spec or higher).
+
+```sql
+UPDATE glue_test.db01.logs SET message = 'updated' WHERE log_id = 1;
+```
+
+### Delete
+
+For Hive-format tables, `DELETE` is only supported when the `WHERE` clause
matches entire partitions.
+For Iceberg tables, row-level deletion is supported.
+
+```sql
+-- Hive table: delete an entire partition
+DELETE FROM glue_test.db01.orders WHERE order_dt = DATE '2024-01-01';
+
+-- Iceberg table: row-level delete
+DELETE FROM glue_test.db01.logs WHERE log_id = 42;
+```
+
+### Drop
+
+```sql
+DROP TABLE glue_test.db01.orders;
+
+DROP SCHEMA glue_test.db01;
+```
+
+## Schema and Table Properties
+
+### Create a Schema with Properties
+
+```sql
+CREATE SCHEMA glue_test.db01
+WITH (
+ location = 's3://my-bucket/warehouse/db01'
+);
+```
+
+| Property | Description | Default Value | Required | Since
Version |
+|------------|--------------------------|---------------|----------|---------------|
+| `location` | S3 location for the schema | (none) | No | 1.3.0
|
+
+### Create a Table with Properties
+
+```sql
+CREATE TABLE glue_test.db01.table_name
+(
+ name varchar,
+ salary integer
+) WITH (
+ type = 'HIVE',
+ format = 'PARQUET',
+ location = 's3://my-bucket/warehouse/db01/table_name',
+ partitioned_by = ARRAY['salary'],
+ bucketed_by = ARRAY['name'],
+ bucket_count = 4,
+ sorted_by = ARRAY['name']
+);
+```
+
+| Property | Description
| Default Value | Required | Since Version |
+|-----------------|-----------------------------------------------------------------------------|---------------|----------|---------------|
+| `type` | Table format: `HIVE` or `ICEBERG`
| `HIVE` | No | 1.3.0 |
+| `format` | File format for Hive-format tables: `PARQUET`, `ORC`,
`TEXTFILE`, etc. | `TEXTFILE` | No | 1.3.0 |
+| `location` | S3 storage location for the table
| (derived from catalog `warehouse`) | No | 1.3.0 |
+| `partitioned_by`| Partition columns or expressions. For Iceberg, use
transform syntax such as `year(col)`. | (none) | No | 1.3.0 |
+| `bucketed_by` | Bucket columns (Hive-format tables only)
| (none) | No | 1.3.0 |
+| `bucket_count` | Number of buckets (required when `bucketed_by` is set)
| (none) | No | 1.3.0 |
+| `sorted_by` | Sort order columns, e.g. `ARRAY['col ASC NULLS LAST',
'col2 DESC']` | (none) | No | 1.3.0 |
+
+## Examples
+
+Follow these steps to use the Glue catalog in Trino through Gravitino.
+
+### Create a Catalog in Gravitino
+
+Use the Trino CLI to create the catalog. Assuming the metalake is `test` and
the catalog name is `glue_test`:
+
+```sql
+CALL gravitino.system.create_catalog(
+ 'glue_test',
+ 'glue',
+ MAP(
+ ARRAY['aws-region', 'aws-access-key-id', 'aws-secret-access-key',
'warehouse'],
+ ARRAY['us-east-1', '<aws-access-key-id>', '<aws-secret-access-key>',
's3://my-bucket/warehouse']
+ )
+);
+```
+
+For more information about the Glue catalog, refer to [AWS Glue
catalog](../aws-glue-catalog.md).
+
+### Connect and List Catalogs
+
+Set `gravitino.metalake` to `test` and start the Trino container. Then list
catalogs:
+
+```sql
+SHOW CATALOGS;
+```
+
+The results are similar to:
+
+```text
+ Catalog
+----------------
+ gravitino
+ glue_test
+ jmx
+ system
+(4 rows)
+```
+
+The `glue_test` catalog corresponds to the catalog created in Gravitino.
+
+## Data Type Mapping
+
+The Glue connector extends the Hive data type mapping with additional support
for Iceberg types.
+
+| Gravitino Type | Trino Type | Notes
|
+|------------------------------|------------------------------|----------------------------------------------|
+| `boolean` | `BOOLEAN` |
|
+| `byte` | `TINYINT` |
|
+| `short` | `SMALLINT` |
|
+| `integer` | `INTEGER` |
|
+| `long` | `BIGINT` |
|
+| `float` | `REAL` |
|
+| `double` | `DOUBLE` |
|
+| `decimal(p, s)` | `DECIMAL(p, s)` |
|
+| `string` | `VARCHAR` |
|
+| `varchar(n)` | `VARCHAR(n)` |
|
+| `char(n)` | `CHAR(n)` |
|
+| `binary` | `VARBINARY` |
|
+| `date` | `DATE` |
|
+| `time(6)` | `TIME(6)` | Iceberg tables
only; microsecond precision |
+| `timestamp` | `TIMESTAMP(3)` | Hive-format
tables; millisecond precision |
+| `timestamp(6)` | `TIMESTAMP(6)` | Iceberg
tables; microsecond precision |
+| `timestamptz(6)` | `TIMESTAMP(6) WITH TIME ZONE`| Iceberg tables
only |
+| `list` | `ARRAY` |
|
+| `map` | `MAP` |
|
+| `struct` | `ROW` |
|
+
+:::note
+`TIME` and `TIMESTAMP WITH TIME ZONE` are available for Iceberg-format tables
only. Hive-format tables use millisecond-precision `TIMESTAMP` without time
zone.
+:::
+
+## Trino Connector Configuration
+
+Gravitino passes catalog properties to the underlying Trino Hive connector.
Supply additional Trino connector properties using the `trino.bypass.` prefix:
+
+```sql
+CALL gravitino.system.create_catalog(
+ 'glue_test',
+ 'glue',
+ MAP(
+ ARRAY['aws-region', 'aws-access-key-id', 'aws-secret-access-key',
'warehouse',
+ 'trino.bypass.hive.metastore.glue.max-connections'],
+ ARRAY['us-east-1', '<aws-access-key-id>', '<aws-secret-access-key>',
's3://my-bucket/warehouse',
+ '50']
+ )
+);
+```
+
+The following Gravitino catalog properties are automatically forwarded to the
Trino connector and cannot be overridden via `trino.bypass.*`:
+
+| Gravitino property | Trino connector property |
+|-------------------------|---------------------------------------|
+| `aws-region` | `hive.metastore.glue.region` |
+| `aws-glue-catalog-id` | `hive.metastore.glue.catalogid` |
+| `aws-access-key-id` | `hive.metastore.glue.aws-access-key`,
`hive.s3.aws-access-key` |
+| `aws-secret-access-key` | `hive.metastore.glue.aws-secret-key`,
`hive.s3.aws-secret-key` |
+| `aws-glue-endpoint` | `hive.metastore.glue.endpoint-url` |
+
+For additional Trino Hive connector configuration options, refer to the
+[Trino Hive connector
documentation](https://trino.io/docs/current/connector/hive.html).
diff --git a/docs/trino-connector/index.md b/docs/trino-connector/index.md
index 901d98df32..6c5a9f7b5d 100644
--- a/docs/trino-connector/index.md
+++ b/docs/trino-connector/index.md
@@ -18,6 +18,7 @@ Apache Gravitino Trino connector index:
- [Iceberg](catalog-iceberg.md)
- [MySQL](catalog-mysql.md)
- [PostgreSQL](catalog-postgresql.md)
+ - [AWS Glue](catalog-glue.md)
- [Trino cascading query](trino-cascading-query.md)
- [Supported SQL](sql-support.md)
- [UDF support](udf-support.md)
diff --git a/docs/trino-connector/supported-catalog.md
b/docs/trino-connector/supported-catalog.md
index fd14f9deec..66e39fdc36 100644
--- a/docs/trino-connector/supported-catalog.md
+++ b/docs/trino-connector/supported-catalog.md
@@ -13,6 +13,7 @@ The catalogs currently supported by the Apache Gravitino
Trino connector are as
- [Iceberg](catalog-iceberg.md)
- [MySQL](catalog-mysql.md)
- [PostgreSQL](catalog-postgresql.md)
+- [AWS Glue](catalog-glue.md)
## Create Catalog
@@ -27,7 +28,7 @@ create_catalog(CATALOG varchar, PROVIDER varchar, PROPERTIES
MAP(VARCHAR, VARCHA
```
- CATALOG: The catalog name to be created.
-- PROVIDER: The catalog provider. Supported values: `hive`,
`lakehouse-iceberg`, `jdbc-mysql`, `jdbc-postgresql`.
+- PROVIDER: The catalog provider. Supported values: `hive`,
`lakehouse-iceberg`, `jdbc-mysql`, `jdbc-postgresql`, `glue`.
- PROPERTIES: The properties of the catalog.
- IGNORE_EXIST: The flag to ignore the error if the catalog already exists.
It's optional, the default value is `false`.
@@ -36,6 +37,7 @@ The type of catalog properties reference:
- [Iceberg catalog](../lakehouse-iceberg-catalog.md#catalog-properties)
- [MySQL catalog](../jdbc-mysql-catalog.md#catalog-properties)
- [PostgreSQL catalog](../jdbc-postgresql-catalog.md#catalog-properties)
+- [AWS Glue catalog](../aws-glue-catalog.md#catalog-properties)
Drop catalog:
