fmorillo7694 commented on code in PR #206: URL: https://github.com/apache/flink-connector-aws/pull/206#discussion_r4130486518
########## docs/content/docs/connectors/table/glue.md: ########## @@ -0,0 +1,399 @@ +--- +title: "AWS Glue Catalog" +weight: 11 +type: docs +aliases: + - /dev/table/connectors/glue.html +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# AWS Glue Catalog + +The AWS Glue Catalog provides a way to use [AWS Glue](https://aws.amazon.com/glue) as a catalog for Apache Flink. +This allows users to access Glue's metadata store directly from Flink SQL and Table API. + +## Features + +- Register AWS Glue as a catalog in Flink applications +- Access Glue databases and tables through Flink SQL +- Support for various AWS data sources (S3, Kinesis, MSK) +- Mapping between Flink and AWS Glue data types +- Compatibility with Flink's Table API and SQL interface + +The Glue Catalog is registered through the Table API / SQL. DataStream applications can also +use it by converting between DataStreams and Tables with the +[DataStream API integration]({{< ref "docs/dev/table/data_stream_api" >}}), so tables backed by +Glue metadata are accessible from DataStream programs through a `StreamTableEnvironment`. + +## Dependencies + +{{< sql_connector_download_table "glue" >}} + +## Prerequisites + +Before getting started, ensure you have the following: + +- **AWS account** with appropriate permissions for AWS Glue and other required services +- **AWS credentials** properly configured, see the + [AWS documentation](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials.html) + for the supported configuration options + +## How to create a Glue Catalog + +### SQL + +```sql +CREATE CATALOG glue_catalog WITH ( + 'type' = 'glue', + 'catalog-name' = 'glue_catalog', Review Comment: Good catch, and this one was self-inflicted: round 1 added strict FactoryUtil validation (your ask) and the doc snippet was never re-run against it. `catalog-name` is not a declared option (the catalog name is the `CREATE CATALOG <name>` identifier), so it fails with a ValidationException. Removed in both `content` and `content.zh` in 79cdcff. To stop this class of drift, the same commit adds `GlueCatalogFactoryTest#testDocumentedCreateCatalogExamplesAreAcceptedByTheFactory`: it parses every `CREATE CATALOG ... WITH (...)` block from the two documentation pages and runs the option set through `GlueCatalogFactory`, so a doc example that the factory rejects now fails the build rather than the user. ########## flink-catalog-aws/flink-catalog-aws-glue/src/main/java/org/apache/flink/table/catalog/glue/util/GlueTypeConverter.java: ########## @@ -0,0 +1,352 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.table.catalog.glue.util; + +import org.apache.flink.annotation.Internal; +import org.apache.flink.table.api.DataTypes; +import org.apache.flink.table.catalog.glue.exception.UnsupportedDataTypeMappingException; +import org.apache.flink.table.types.DataType; +import org.apache.flink.table.types.logical.ArrayType; +import org.apache.flink.table.types.logical.DecimalType; +import org.apache.flink.table.types.logical.LogicalType; +import org.apache.flink.table.types.logical.LogicalTypeRoot; +import org.apache.flink.table.types.logical.MapType; +import org.apache.flink.table.types.logical.RowType; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.ArrayList; +import java.util.List; +import java.util.regex.Matcher; +import java.util.regex.Pattern; + +/** + * Utility class for converting Flink types to Glue types and vice versa. Supports the conversion of + * common primitive, array, map, and struct types. + */ +@Internal +public class GlueTypeConverter { + + /** Logger for tracking Glue type conversions. */ + private static final Logger LOG = LoggerFactory.getLogger(GlueTypeConverter.class); + + /** Regular expressions for handling specific Glue types. */ + private static final Pattern DECIMAL_PATTERN = Pattern.compile("decimal\\((\\d+),(\\d+)\\)"); + + /** + * Parameterized Hive/Glue string types, e.g. {@code varchar(255)} or {@code char(10)}, written + * by other engines (Athena, Spark, Hive). Flink's Glue catalog stores strings as + * unparameterized {@code string}, but foreign tables may use the parameterized form, so the + * read path must accept it. The declared length is preserved as a Flink {@code + * VARCHAR(n)}/{@code CHAR(n)}. + */ + private static final Pattern VARCHAR_PATTERN = + Pattern.compile("varchar\\((\\d+)\\)", Pattern.CASE_INSENSITIVE); + + private static final Pattern CHAR_PATTERN = + Pattern.compile("char\\((\\d+)\\)", Pattern.CASE_INSENSITIVE); + + private static final Pattern ARRAY_PATTERN = Pattern.compile("array<(.+)>"); + private static final Pattern MAP_PATTERN = Pattern.compile("map<(.+),(.+)>"); + private static final Pattern STRUCT_PATTERN = Pattern.compile("struct<(.+)>"); Review Comment: Yes. I had fixed the two instances you pointed at rather than the class. Glue type strings written by Athena, Spark, Hive or crawlers are not normalised, so `ARRAY<STRING>`, `MAP<STRING,INT>`, `STRUCT<...>` and `DECIMAL(p,s)` need the same treatment. All patterns are now `CASE_INSENSITIVE` (a18aca5); struct field names keep their case because the content is sliced from the original string, not the lowercased one. Tests: `GlueTypeConverterTest#testEveryGlueTypeIsMatchedCaseInsensitively` feeds every entry of the Glue to Flink mapping table in upper case and capitalised keyword form, plus an explicit `ARRAY<STRUCT<UserId:BIGINT,Tags:MAP<STRING,INT>>>` case checking field names survive. And since Flink's own writer never produces these strings, the e2e now creates an `EXTERNAL_TABLE` and a `VIRTUAL_VIEW` out of band with the raw Glue client, exactly as Athena DDL stores them (uppercase, `varchar(255)`, `CHAR(3)`, `DECIMAL(10,2)`, `ARRAY`, `MAP`, `STRUCT`, no Flink properties), and reads them back through Flink with `DESCRIBE`, `SHOW CREATE TABLE` and `CREATE TABLE ... LIKE`. Green against real Glue in eu-central-1 (6/6 e2e). ########## flink-catalog-aws/flink-catalog-aws-glue/pom.xml: ########## @@ -0,0 +1,83 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> +<project xmlns="http://maven.apache.org/POM/4.0.0"; + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd";> + <modelVersion>4.0.0</modelVersion> + + <parent> + <groupId>org.apache.flink</groupId> + <artifactId>flink-catalog-aws-parent</artifactId> + <version>6.1-SNAPSHOT</version> + <relativePath>..</relativePath> + </parent> + + <artifactId>flink-catalog-aws-glue</artifactId> + <name>Flink : Catalogs : AWS Glue</name> + <packaging>jar</packaging> + + <dependencies> + <!-- Flink dependencies specific to Glue module --> + <dependency> + <groupId>org.apache.flink</groupId> + <artifactId>flink-table-planner-loader</artifactId> + <version>${flink.version}</version> + <scope>provided</scope> + </dependency> + + <!-- AWS dependencies specific to Glue --> + <dependency> + <groupId>software.amazon.awssdk</groupId> + <artifactId>glue</artifactId> + </dependency> + <dependency> + <groupId>software.amazon.awssdk</groupId> + <artifactId>apache-client</artifactId> + <version>${aws.sdkv2.version}</version> + </dependency> + + <!-- Shared AWS client-creation and configuration utilities --> + <dependency> + <groupId>org.apache.flink</groupId> + <artifactId>flink-connector-aws-base</artifactId> + <version>${project.version}</version> + </dependency> Review Comment: Added in 72b0a70, mirroring the other modules: the dependency, `archunit.properties`, `TestCodeArchitectureTest` for `org.apache.flink.table.catalog.glue`, and the violation store. Worth noting the rules paid for themselves immediately: they flagged both moto ITCases for not declaring a `MiniClusterExtension`, which is now fixed in the same commit. Unit suite 254/254 with the architecture tests included. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
