fhueske commented on a change in pull request #9192: [FLINK-12749] [docs] [examples] Initial Version of Flink Cluster Playground URL: https://github.com/apache/flink/pull/9192#discussion_r306898498
########## File path: docs/getting-started/docker-playgrounds/flink_cluster_playground.md ########## @@ -0,0 +1,680 @@ +--- +title: "Flink Cluster Playground" +nav-title: 'Flink Cluster Playground' +nav-parent_id: docker-playgrounds +nav-pos: 1 +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +There are many ways to deploy and operate Apache Flink in various environments. Regardless of this +variety, the fundamental building blocks of a Flink Cluster remain the same and similar +operational principles apply. + +This docker compose-based playground will get you started with Apache Flink operations quickly and +will briefly introduce you to the main components that make up a Flink Cluster. + +* This will be replaced by the TOC +{:toc} + +## Anatomy of this Playground + +This playground consists of a long living +[Flink Session Cluster]({{ site.baseurl }}/concepts/glossary.html#flink-session-cluster) and a Kafka +Cluster. + +A Flink Cluster always consists of a +[Flink Master]({{ site.baseurl }}/concepts/glossary.html#flink-master) and one or more +[Flink TaskManagers]({{ site.baseurl }}/concepts/glossary.html#flink-taskmanager). The Flink Master +is responsible to handle Job submissions, the supervision of Jobs as well as resource +management. The Flink TaskManagers are the worker processes and are responsible for the execution of +the actual [Tasks]({{ site.baseurl }}/concepts/glossary.html#task) which make up a Flink Job. In +this playground you will start with a single TaskManager, but scale out to more TaskManagers later. +Additionally, this playground comes with a dedicated *client* container, which we use to submit the +Flink Job initially and to perform various operational tasks later on. + +The Kafka Cluster consists of a Zookeeper server and a Kafka Broker. + +<img src="{{ site.baseurl }}/fig/flink-docker-playground.svg" alt="Flink Docker Playground" +class="offset" width="80%" /> + +When the playground is started a Flink Job called *Flink Event Count* will be submitted to the +Flink Master. Additionally, two Kafka Topics *input* and *output* are created. + +<img src="{{ site.baseurl }}/fig/click-event-count-example.svg" alt="Click Event Count Example" +class="offset" width="80%" /> + +The Job consumes `ClickEvent`s from the *input* topic, each with a `timestamp` and a `page`. The +events are then keyed by `page` and counted in one minute +[windows]({{ site.baseurl }}/dev/stream/operators/windows.html). The results are written to the +*output* topic. + +There are six different `page`s and the **events are generated so that each window contains exactly Review comment: Extend to: > There are six different `page`s and we generate 1000 click events per page and minute. Hence, the output of the Flink job should show 1000 views per page and window. ? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services