http://git-wip-us.apache.org/repos/asf/incubator-flink/blob/4895131e/docs/internal_add_operator.md ---------------------------------------------------------------------- diff --git a/docs/internal_add_operator.md b/docs/internal_add_operator.md index e370c14..ddcb6b5 100644 --- a/docs/internal_add_operator.md +++ b/docs/internal_add_operator.md @@ -243,4 +243,9 @@ public <R> DataSet<R> mapPartition(MapPartitionFunction<T, R> function) { } ~~~ +--- + +*This documentation is maintained by the contributors of the individual components. +We kindly ask anyone that adds and changes components to eventually provide a patch +or pull request that updates these documents as well.*
http://git-wip-us.apache.org/repos/asf/incubator-flink/blob/4895131e/docs/internal_distributed_akka.md ---------------------------------------------------------------------- diff --git a/docs/internal_distributed_akka.md b/docs/internal_distributed_akka.md index f0bf8e0..49497b2 100644 --- a/docs/internal_distributed_akka.md +++ b/docs/internal_distributed_akka.md @@ -39,3 +39,9 @@ via Akka (http://akka.io) ## Failure Detection + +--- + +*This documentation is maintained by the contributors of the individual components. +We kindly ask anyone that adds and changes components to eventually provide a patch +or pull request that updates these documents as well.* http://git-wip-us.apache.org/repos/asf/incubator-flink/blob/4895131e/docs/internal_howto.md ---------------------------------------------------------------------- diff --git a/docs/internal_howto.md b/docs/internal_howto.md new file mode 100644 index 0000000..415bda7 --- /dev/null +++ b/docs/internal_howto.md @@ -0,0 +1,39 @@ +--- +title: "Overview of Flink System Architecture & Internals" +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +This documentation provides an overview of the "How-To's" for +Flink developers. It is intended as guide to contributors and people +that are interested in the technology behind Flink. + + +## Architectures and Components + +- [How-to: Adding a new Operator](internal_add_operator.html) + +- [How-to: Using logging in Flink](internal_logging.html) + +--- + +*This documentation is maintained by the contributors of the individual components. +We kindly ask anyone that adds and changes components to eventually provide a patch +or pull request that updates these documents as well.* + http://git-wip-us.apache.org/repos/asf/incubator-flink/blob/4895131e/docs/internal_job_scheduling.md ---------------------------------------------------------------------- diff --git a/docs/internal_job_scheduling.md b/docs/internal_job_scheduling.md index 1237c85..7fb4500 100644 --- a/docs/internal_job_scheduling.md +++ b/docs/internal_job_scheduling.md @@ -19,3 +19,64 @@ KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> + +This document briefly describes how Flink schedules jobs and +how it represents and tracks job status on the JobManager. + +* This will be replaced by the TOC +{:toc} + + +## Scheduling + +Execution resources in Flink are defined through _Task Slots_. Each TaskManager will have one or more task slots, +each of which can run one pipeline of parallel tasks. A pipeline consists of multiple successive tasks, such as the +*n-th* parallel instance of a MapFunction together with the *n-th* parallel instance of a ReduceFunction. +Note that Flink often executes successive tasks concurrently: For Streaming programs, that happens in any case, +but also for batch programs, it happens frequently. + +The figure below illustrates that. Consider a program with a data source, a *MapFunction*, and a *ReduceFunctoin*. +The source and MapFunction are executed with a parallelism of 4, while the ReduceFunction is executed with a +parallism of 3. A pipeline consists of the sequence Source - Map - Reduce. On a cluster with 2 TaskManagers with +3 slots each, the program will be executed as described below. + +<div style="text-align: center;"> +<img src="img/slots.svg" alt="Assigning Pipelines of Tasks to Slots" height="250px" style="text-align: center;"/> +</div> + +Internally, Flink defines through [SlotSharingGroup](https://github.com/apache/incubator-flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/jobmanager/scheduler/SlotSharingGroup.java) +and [CoLocationGroup](https://github.com/apache/incubator-flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/jobmanager/scheduler/CoLocationGroup.java) +which tasks may share a slot (permissive), respectively which tasks must be strictly placed into the same slot. + + +## JobManager Data Structures + +During job execution, the JobManager keeps track of distributed tasks, decides when to schedule the next task (or set of tasks), +and reacts to finished tasks or execution failures. + +The JobManager receives the [JobGraph](https://github.com/apache/incubator-flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/), +which is a representation of the data flow consisting of operators ([JobVertex](https://github.com/apache/incubator-flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/AbstractJobVertex.java)) +and intermediate results ([IntermediateDataSet](https://github.com/apache/incubator-flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/IntermediateDataSet.java)). +Each operator has properies, like the degree of parallelism and the code that it executes. +In addition, the JobGraph has a set of attached libraries, that are neccessary to execute the code of the operators. + +The JobManager transforms the JobGraph into an [ExecutionGraph](https://github.com/apache/incubator-flink/tree/master/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph). +The ExecutionGraph is a parallel version of the JobGraph: For each JobVertex, it contains an [ExecutionVertex](https://github.com/apache/incubator-flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ExecutionVertex.java) per parallel subtask. An operator with a parallelism of 100 will have one JobVertex and 100 ExecutionVertices. +The ExecutionVertex tracks the state of execution of a particular subtask. All ExecutionVertices from one JobVertex are held in an +[ExecutionJobVertex](https://github.com/apache/incubator-flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ExecutionJobVertex.java), +which tracks the status of the operator as a whole. +Besides the vertices, the ExecutionGraph also contains the [IntermediateResult](https://github.com/apache/incubator-flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/IntermediateResult.java) and the [IntermediateResultPartition](https://github.com/apache/incubator-flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/IntermediateResultPartition.java). The former tracks the state of the *IntermediateDataSet*, the latter the state of each of its partitions. + +<div style="text-align: center;"> +<img src="img/job_and_execution_graph.svg" alt="JobGraph and ExecutionGraph" height="400px" style="text-align: center;"/> +</div> + +During its execution, each parallel task goes through multiple stages, from *created* to *finished* or *failed*. The diagram below illustrates the +states and possible transitions between them. A task may be executed multiple times (for example in the course of failure recovery). +For that reason, the execution of an ExecutionVertex is tracked in an [Execution](https://github.com/apache/incubator-flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/Execution.java). Each ExecutionVertex has a current Execution, and prior Executions. + +<div style="text-align: center;"> +<img src="img/state_machine.svg" alt="States and Transitions of Task Executions" height="300px" style="text-align: center;"/> +</div> + + http://git-wip-us.apache.org/repos/asf/incubator-flink/blob/4895131e/docs/internal_logging.md ---------------------------------------------------------------------- diff --git a/docs/internal_logging.md b/docs/internal_logging.md index 79e3656..42fd317 100644 --- a/docs/internal_logging.md +++ b/docs/internal_logging.md @@ -86,4 +86,11 @@ Placeholders can also be used in conjunction with exceptions which shall be logg catch(Exception exception){ LOG.error("An {} occurred.", "error", exception); } -~~~ \ No newline at end of file +~~~ + +--- + +*This documentation is maintained by the contributors of the individual components. +We kindly ask anyone that adds and changes components to eventually provide a patch +or pull request that updates these documents as well.* + http://git-wip-us.apache.org/repos/asf/incubator-flink/blob/4895131e/docs/internal_overview.md ---------------------------------------------------------------------- diff --git a/docs/internal_overview.md b/docs/internal_overview.md deleted file mode 100644 index 88fe6f6..0000000 --- a/docs/internal_overview.md +++ /dev/null @@ -1,58 +0,0 @@ ---- -title: "Overview of Flink System Architecture & Internals" ---- -<!-- -Licensed to the Apache Software Foundation (ASF) under one -or more contributor license agreements. See the NOTICE file -distributed with this work for additional information -regarding copyright ownership. The ASF licenses this file -to you under the Apache License, Version 2.0 (the -"License"); you may not use this file except in compliance -with the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, -software distributed under the License is distributed on an -"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -KIND, either express or implied. See the License for the -specific language governing permissions and limitations -under the License. ---> - -This documentation provides an overview of the architecture of the Flink system -and its components. It is intended as guide to contributors, and people -that are interested in the technology behind Flink. - -*This documentation is maintained by the contributors of the individual components. -We kindly ask anyone that adds and changes components to eventually provide a patch -or pull request that updates these documents as well.* - - -## Architectures and Components - -- [General Architecture and Process Model](internal_general_arch.html) - -<!-- -- [Life Cycle of a Program](program_life_cycle.html) ---> - -- [Distributed Communication via Akka](internal_distributed_akka.html) - -<!-- -- [Jobs and Scheduling](job_scheduling.html) - -- [Distributed Runtime](distributed_runtime.html) - -- [Runtime Algorithms and Memory Management](operators_and_memory.html) - -- [Program Optimizer](optimizer.html) ---> - -- [How-to: Adding a new Operator](internal_add_operator.html) - -<!-- -- [Java API, Types, and Type Extraction](types.html) ---> - -- [How-to: Using logging in Flink](internal_logging.html)
