pvary commented on code in PR #15062: URL: https://github.com/apache/iceberg/pull/15062#discussion_r2703865726
########## site/docs/flink-quickstart.md: ########## @@ -0,0 +1,183 @@ +--- +title: "Flink and Iceberg Quickstart" +--- +<!-- + - Licensed to the Apache Software Foundation (ASF) under one or more + - contributor license agreements. See the NOTICE file distributed with + - this work for additional information regarding copyright ownership. + - The ASF licenses this file to You under the Apache License, Version 2.0 + - (the "License"); you may not use this file except in compliance with + - the License. You may obtain a copy of the License at + - + - http://www.apache.org/licenses/LICENSE-2.0 + - + - Unless required by applicable law or agreed to in writing, software + - distributed under the License is distributed on an "AS IS" BASIS, + - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + - See the License for the specific language governing permissions and + - limitations under the License. + --> + +This guide will get you up and running with Apache Iceberg™ using Apache Flink™, including sample code to +highlight some powerful features. You can learn more about Iceberg's Flink runtime by checking out the [Flink](docs/latest/flink.md) section. + +## Docker Compose + +The fastest way to get started is to use a Docker Compose file. +To use this, you'll need to install the [Docker CLI](https://docs.docker.com/get-docker/) as well as the [Docker Compose CLI](https://github.com/docker/compose-cli/blob/main/INSTALL.md). + +Once you have those, save these two files into a new folder: + +* [`docker-compose.yml`](https://raw.githubusercontent.com/apache/iceberg/refs/heads/main/flink/v2.0/quickstart/docker-compose.yml) + + This contains: + + * A local Flink cluster (Job Manager and Task Manager) + * Iceberg REST Catalog + * SeaweedFS (local S3 storage) + * AWS CLI (to create the S3 bucket) + +* [`Dockerfile.flink`](https://raw.githubusercontent.com/apache/iceberg/refs/heads/main/flink/v2.0/quickstart/Dockerfile.flink) - base Flink image, plus some required JARs for S3 and Iceberg. + +Next, start up the docker containers with this command: + +```sh +docker compose up -d +``` + +Launch a Flink SQL client session: + +```sh +docker compose exec -it jobmanager ./bin/sql-client.sh +``` + +## Creating an Iceberg Catalog in Flink + +Iceberg has several catalog back-ends that can be used to track tables, like JDBC, Hive MetaStore and Glue. +In this guide we use a REST catalog, backed by S3. +To learn more, check out the [Catalog](docs/latest/flink-configuration.md#catalog-configuration) page in the Flink section. + +First up, we need to define a Flink catalog. +Tables within this catalog will be stored as Iceberg tables in the defined S3 warehouse: + +```sql +CREATE CATALOG iceberg_catalog WITH ( + 'type' = 'iceberg', + 'catalog-impl' = 'org.apache.iceberg.rest.RESTCatalog', + 'uri' = 'http://iceberg-rest:8181', + 'warehouse' = 's3://warehouse/', + 'io-impl' = 'org.apache.iceberg.aws.s3.S3FileIO', + 's3.endpoint' = 'http://seaweedfs:9000', Review Comment: Maybe highlight the values which should be overwritten by the user even if they use S3 to store the data and a REST catalog? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
