subject:"\[GitHub\] \[flink\] azagrebin commented on a change in pull request #12131\: \[FLINK\-17161\]\[docs\]\[docker\] Document the official docker hub image and examples of how to use it"

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

2020-05-15 Thread GitBox



azagrebin commented on a change in pull request #12131:
URL: https://github.com/apache/flink/pull/12131#discussion_r425772745



##
File path: docs/ops/deployment/kubernetes.md
##
@@ -274,44 +341,121 @@ spec:
 path: flink-conf.yaml
   - key: log4j.properties
 path: log4j.properties
-
 {% endhighlight %}
 
-`jobmanager-service.yaml`
+### Job cluster resource definitions
+
+`jobmanager-job.yaml`
 {% highlight yaml %}
-apiVersion: v1
-kind: Service
+apiVersion: batch/v1

Review comment:
   I am not fully familiar with versioning of Kubernetes components but it 
matters. The versions correspond to the current examples in Kubernetes docs.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

2020-05-15 Thread GitBox



azagrebin commented on a change in pull request #12131:
URL: https://github.com/apache/flink/pull/12131#discussion_r425765241



##
File path: docs/ops/deployment/kubernetes.md
##
@@ -76,57 +74,92 @@ You can then access the Flink UI via different ways:
 {% highlight bash %}./bin/flink run -m localhost:8081 
./examples/streaming/WordCount.jar{% endhighlight %}
 
 *  Create a `NodePort` service on the rest service of jobmanager:
-1. Run `kubectl create -f jobmanager-rest-service.yaml` to create the 
`NodePort` service on jobmanager. The example of `jobmanager-rest-service.yaml` 
can be found in [appendix](#session-cluster-resource-definitions).
+1. Run `kubectl create -f jobmanager-rest-service.yaml` to create the 
`NodePort` service on jobmanager. The example of `jobmanager-rest-service.yaml` 
can be found in [appendix](#common-cluster-resource-definitions).
 2. Run `kubectl get svc flink-jobmanager-rest` to know the `node-port` of 
this service and navigate to 
[http://:;](http://:)
 in your browser.
-3. Similarly to `port-forward` solution, you could also use the following 
command below to submit jobs to the cluster:
+3. If you use minikube, you can get its public ip by running `minikube ip`.
+4. Similarly to `port-forward` solution, you could also use the following 
command below to submit jobs to the cluster:
 
 {% highlight bash %}./bin/flink run -m : 
./examples/streaming/WordCount.jar{% endhighlight %}
 
-In order to terminate the Flink session cluster, use `kubectl`:
+In order to terminate the Flink cluster, terminate the specific 
[Session](#deploy-session-cluster) or [Job](#deploy-job-cluster) cluster 
components
+and use `kubectl` to terminate the common components:
 
-kubectl delete -f jobmanager-deployment.yaml
-kubectl delete -f taskmanager-deployment.yaml
+```sh
 kubectl delete -f jobmanager-service.yaml
 kubectl delete -f flink-configuration-configmap.yaml
+# if created then also the rest service
+kubectl delete -f jobmanager-rest-service.yaml
+```
+
+### Deploy Session Cluster
+
+A Flink Session cluster is executed as a long-running Kubernetes Deployment.
+Note that you can run multiple Flink jobs on a Session cluster.
+Each job needs to be submitted to the cluster after the cluster has been 
deployed.
 
-## Flink job cluster on Kubernetes
+A basic Flink Session cluster deployment in Kubernetes has three components:
 
-A Flink job cluster is a dedicated cluster which runs a single job.
-The job is part of the image and, thus, there is no extra job submission 
needed.
+* a Deployment which runs a Flink Master
+* a Deployment for a pool of TaskManagers
+* a Service exposing the Flink Master's REST and UI ports
 
-### Creating the job-specific image
+After creating [the common cluster 
components](#deploy-flink-cluster-on-kubernetes), use [the Session specific 
resource definitions](#session-cluster-resource-definitions)
+to launch the Session cluster with the `kubectl` command:
 
-The Flink job cluster image needs to contain the user code jars of the job for 
which the cluster is started.
-Therefore, one needs to build a dedicated container image for every job.
-Please follow these [instructions](https://github.com/apache/flink/blob/{{ 
site.github_branch }}/flink-container/docker/README.md) to build the Docker 
image.
+```sh
+kubectl create -f jobmanager-session-deployment.yaml
+kubectl create -f taskmanager-session-deployment.yaml
+```
 
-### Using plugins
+To terminate the Session cluster, these components can be deleted before [the 
common ones](#deploy-flink-cluster-on-kubernetes) with the `kubectl` command:

Review comment:
   Asaik, in general you can create and remove resources in an unordered 
way in k8s.
   I can change this:
   `before` -> `along with`
   Moreover, whatever order you choose k8s does this async so the components 
get online randomly in time but if you do not make any assumption about the 
order in your app then the whole deployment should eventually get up and running





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

2020-05-15 Thread GitBox



azagrebin commented on a change in pull request #12131:
URL: https://github.com/apache/flink/pull/12131#discussion_r425765241



##
File path: docs/ops/deployment/kubernetes.md
##
@@ -76,57 +74,92 @@ You can then access the Flink UI via different ways:
 {% highlight bash %}./bin/flink run -m localhost:8081 
./examples/streaming/WordCount.jar{% endhighlight %}
 
 *  Create a `NodePort` service on the rest service of jobmanager:
-1. Run `kubectl create -f jobmanager-rest-service.yaml` to create the 
`NodePort` service on jobmanager. The example of `jobmanager-rest-service.yaml` 
can be found in [appendix](#session-cluster-resource-definitions).
+1. Run `kubectl create -f jobmanager-rest-service.yaml` to create the 
`NodePort` service on jobmanager. The example of `jobmanager-rest-service.yaml` 
can be found in [appendix](#common-cluster-resource-definitions).
 2. Run `kubectl get svc flink-jobmanager-rest` to know the `node-port` of 
this service and navigate to 
[http://:;](http://:)
 in your browser.
-3. Similarly to `port-forward` solution, you could also use the following 
command below to submit jobs to the cluster:
+3. If you use minikube, you can get its public ip by running `minikube ip`.
+4. Similarly to `port-forward` solution, you could also use the following 
command below to submit jobs to the cluster:
 
 {% highlight bash %}./bin/flink run -m : 
./examples/streaming/WordCount.jar{% endhighlight %}
 
-In order to terminate the Flink session cluster, use `kubectl`:
+In order to terminate the Flink cluster, terminate the specific 
[Session](#deploy-session-cluster) or [Job](#deploy-job-cluster) cluster 
components
+and use `kubectl` to terminate the common components:
 
-kubectl delete -f jobmanager-deployment.yaml
-kubectl delete -f taskmanager-deployment.yaml
+```sh
 kubectl delete -f jobmanager-service.yaml
 kubectl delete -f flink-configuration-configmap.yaml
+# if created then also the rest service
+kubectl delete -f jobmanager-rest-service.yaml
+```
+
+### Deploy Session Cluster
+
+A Flink Session cluster is executed as a long-running Kubernetes Deployment.
+Note that you can run multiple Flink jobs on a Session cluster.
+Each job needs to be submitted to the cluster after the cluster has been 
deployed.
 
-## Flink job cluster on Kubernetes
+A basic Flink Session cluster deployment in Kubernetes has three components:
 
-A Flink job cluster is a dedicated cluster which runs a single job.
-The job is part of the image and, thus, there is no extra job submission 
needed.
+* a Deployment which runs a Flink Master
+* a Deployment for a pool of TaskManagers
+* a Service exposing the Flink Master's REST and UI ports
 
-### Creating the job-specific image
+After creating [the common cluster 
components](#deploy-flink-cluster-on-kubernetes), use [the Session specific 
resource definitions](#session-cluster-resource-definitions)
+to launch the Session cluster with the `kubectl` command:
 
-The Flink job cluster image needs to contain the user code jars of the job for 
which the cluster is started.
-Therefore, one needs to build a dedicated container image for every job.
-Please follow these [instructions](https://github.com/apache/flink/blob/{{ 
site.github_branch }}/flink-container/docker/README.md) to build the Docker 
image.
+```sh
+kubectl create -f jobmanager-session-deployment.yaml
+kubectl create -f taskmanager-session-deployment.yaml
+```
 
-### Using plugins
+To terminate the Session cluster, these components can be deleted before [the 
common ones](#deploy-flink-cluster-on-kubernetes) with the `kubectl` command:

Review comment:
   Asaik, in general you can create and remove resources in an unordered 
way in k8s.
   I can change this:
   `before` -> `along with`
   Moreover, whatever order you choose k8s does this async so the components 
get online randomly in time





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

2020-05-15 Thread GitBox



azagrebin commented on a change in pull request #12131:
URL: https://github.com/apache/flink/pull/12131#discussion_r425765241



##
File path: docs/ops/deployment/kubernetes.md
##
@@ -76,57 +74,92 @@ You can then access the Flink UI via different ways:
 {% highlight bash %}./bin/flink run -m localhost:8081 
./examples/streaming/WordCount.jar{% endhighlight %}
 
 *  Create a `NodePort` service on the rest service of jobmanager:
-1. Run `kubectl create -f jobmanager-rest-service.yaml` to create the 
`NodePort` service on jobmanager. The example of `jobmanager-rest-service.yaml` 
can be found in [appendix](#session-cluster-resource-definitions).
+1. Run `kubectl create -f jobmanager-rest-service.yaml` to create the 
`NodePort` service on jobmanager. The example of `jobmanager-rest-service.yaml` 
can be found in [appendix](#common-cluster-resource-definitions).
 2. Run `kubectl get svc flink-jobmanager-rest` to know the `node-port` of 
this service and navigate to 
[http://:;](http://:)
 in your browser.
-3. Similarly to `port-forward` solution, you could also use the following 
command below to submit jobs to the cluster:
+3. If you use minikube, you can get its public ip by running `minikube ip`.
+4. Similarly to `port-forward` solution, you could also use the following 
command below to submit jobs to the cluster:
 
 {% highlight bash %}./bin/flink run -m : 
./examples/streaming/WordCount.jar{% endhighlight %}
 
-In order to terminate the Flink session cluster, use `kubectl`:
+In order to terminate the Flink cluster, terminate the specific 
[Session](#deploy-session-cluster) or [Job](#deploy-job-cluster) cluster 
components
+and use `kubectl` to terminate the common components:
 
-kubectl delete -f jobmanager-deployment.yaml
-kubectl delete -f taskmanager-deployment.yaml
+```sh
 kubectl delete -f jobmanager-service.yaml
 kubectl delete -f flink-configuration-configmap.yaml
+# if created then also the rest service
+kubectl delete -f jobmanager-rest-service.yaml
+```
+
+### Deploy Session Cluster
+
+A Flink Session cluster is executed as a long-running Kubernetes Deployment.
+Note that you can run multiple Flink jobs on a Session cluster.
+Each job needs to be submitted to the cluster after the cluster has been 
deployed.
 
-## Flink job cluster on Kubernetes
+A basic Flink Session cluster deployment in Kubernetes has three components:
 
-A Flink job cluster is a dedicated cluster which runs a single job.
-The job is part of the image and, thus, there is no extra job submission 
needed.
+* a Deployment which runs a Flink Master
+* a Deployment for a pool of TaskManagers
+* a Service exposing the Flink Master's REST and UI ports
 
-### Creating the job-specific image
+After creating [the common cluster 
components](#deploy-flink-cluster-on-kubernetes), use [the Session specific 
resource definitions](#session-cluster-resource-definitions)
+to launch the Session cluster with the `kubectl` command:
 
-The Flink job cluster image needs to contain the user code jars of the job for 
which the cluster is started.
-Therefore, one needs to build a dedicated container image for every job.
-Please follow these [instructions](https://github.com/apache/flink/blob/{{ 
site.github_branch }}/flink-container/docker/README.md) to build the Docker 
image.
+```sh
+kubectl create -f jobmanager-session-deployment.yaml
+kubectl create -f taskmanager-session-deployment.yaml
+```
 
-### Using plugins
+To terminate the Session cluster, these components can be deleted before [the 
common ones](#deploy-flink-cluster-on-kubernetes) with the `kubectl` command:

Review comment:
   Asaik, in general you can create and remove resources in an unordered 
way in k8s.
   I can change this:
   `before` -> `along with`





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

2020-05-15 Thread GitBox



azagrebin commented on a change in pull request #12131:
URL: https://github.com/apache/flink/pull/12131#discussion_r425777675



##
File path: docs/ops/deployment/docker.md
##
@@ -24,119 +24,490 @@ under the License.
 -->
 
 [Docker](https://www.docker.com) is a popular container runtime. 
-There are Docker images for Apache Flink available on Docker Hub which can be 
used to deploy a session cluster.
-The Flink repository also contains tooling to create container images to 
deploy a job cluster.
+There are Docker images for Apache Flink available [on Docker 
Hub](https://hub.docker.com/_/flink).
+You can use the docker images to deploy a *Session* or *Job cluster* in a 
containerized environment, e.g.,
+[standalone Kubernetes](kubernetes.html) or [native 
Kubernetes](native_kubernetes.html).
 
 * This will be replaced by the TOC
 {:toc}
 
-## Flink session cluster
-
-A Flink session cluster can be used to run multiple jobs. 
-Each job needs to be submitted to the cluster after it has been deployed. 
-
-### Docker images
+## Docker Hub Flink images
 
 The [Flink Docker repository](https://hub.docker.com/_/flink/) is hosted on
 Docker Hub and serves images of Flink version 1.2.1 and later.
 
-Images for each supported combination of Hadoop and Scala are available, and 
tag aliases are provided for convenience.
+### Image tags
 
-Beginning with Flink 1.5, image tags that omit a Hadoop version (e.g.
-`-hadoop28`) correspond to Hadoop-free releases of Flink that do not include a
-bundled Hadoop distribution.
+Images for each supported combination of Flink and Scala versions are 
available, and
+[tag aliases](https://hub.docker.com/_/flink?tab=tags) are provided for 
convenience.
 
-For example, the following aliases can be used: *(`1.5.y` indicates the latest
-release of Flink 1.5)*
+For example, you can use the following aliases: *(`1.11.y` indicates the 
latest release of Flink 1.11)*
 
 * `flink:latest` → `flink:-scala_`
-* `flink:1.5` → `flink:1.5.y-scala_2.11`
-* `flink:1.5-hadoop27` → `flink:1.5.y-hadoop27-scala_2.11`
+* `flink:1.11` → `flink:1.11.y-scala_2.11`
+
+Note Prio to Flink 1.5 version, Hadoop 
dependencies were always bundled with Flink.
+You can see that certain tags include the version of Hadoop, e.g. (e.g. 
`-hadoop28`).
+Beginning with Flink 1.5, image tags that omit the Hadoop version correspond 
to Hadoop-free releases of Flink
+that do not include a bundled Hadoop distribution.
+
+## How to run Flink image
+
+The Flink image contains a regular Flink distribution with its default 
configuration and a standard entry point script.
+You can run its entry point in the following modes:
+* *Flink Master* for [a Session cluster](#start-a-session-cluster)
+* *Flink Master* for [a Single Job cluster](#start-a-single-job-cluster)
+* *TaskManager* for any cluster
+
+This allows you to deploy a standalone cluster (Session or Single Job) in any 
containerised environment, for example:
+* manually in a local docker setup,
+* [in a Kubernetes cluster](kubernetes.html),
+* [with Docker Compose](#flink-with-docker-compose),
+* [with Docker swarm](#flink-with-docker-swarm).
+
+Note [The native 
Kubernetes](native_kubernetes.html) also runs the same image by default
+and deploys *TaskManagers* on demand so that you do not have to do it manually.
+
+The next chapters describe how to start a single Flink docker container for 
various purposes.
+
+### Start a Session Cluster
+
+A *Flink Session cluster* can be used to run multiple jobs. Each job needs to 
be submitted to the cluster after it has been deployed.
+To deploy a *Flink Session cluster* with docker, you need to start a *Flink 
Master* container:
+
+```sh
+docker run flink:{% if site.is_stable 
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif 
%} jobmanager
+```
+
+and one or more *TaskManager* containers:
+
+```sh
+docker run flink:{% if site.is_stable 
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif 
%} taskmanager
+```
+
+### Start a Job Cluster
+
+A *Flink Job cluster* is a dedicated cluster which runs a single job.
+In this case, you deploy the cluster with the job as one step, thus, there is 
no extra job submission needed.
+Therefore, the *job artifacts* should be already available locally in the 
container.
+
+The *job artifacts* are included into the class path of Flink's JVM process 
within the container and consist of:
+* your job jar, which you would normally submit to a *Session cluster* and
+* all other necessary dependencies or resources, not included into Flink.
+
+To deploy a cluster for a single job with docker, you need to
+* make *job artifacts* available locally *in all containers* under 
`/opt/flink/usrlib`,
+* start a *Flink Master* container in the *Job Cluster* mode
+* start the required number of *TaskManager* containers.
+
+To make the **job artifacts available** locally in the container, you can
+
+* **either mount a volume** (or multiple volumes)

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

2020-05-15 Thread GitBox



azagrebin commented on a change in pull request #12131:
URL: https://github.com/apache/flink/pull/12131#discussion_r425777456



##
File path: docs/ops/deployment/docker.md
##
@@ -24,119 +24,490 @@ under the License.
 -->
 
 [Docker](https://www.docker.com) is a popular container runtime. 
-There are Docker images for Apache Flink available on Docker Hub which can be 
used to deploy a session cluster.
-The Flink repository also contains tooling to create container images to 
deploy a job cluster.
+There are Docker images for Apache Flink available [on Docker 
Hub](https://hub.docker.com/_/flink).
+You can use the docker images to deploy a *Session* or *Job cluster* in a 
containerized environment, e.g.,
+[standalone Kubernetes](kubernetes.html) or [native 
Kubernetes](native_kubernetes.html).
 
 * This will be replaced by the TOC
 {:toc}
 
-## Flink session cluster
-
-A Flink session cluster can be used to run multiple jobs. 
-Each job needs to be submitted to the cluster after it has been deployed. 
-
-### Docker images
+## Docker Hub Flink images
 
 The [Flink Docker repository](https://hub.docker.com/_/flink/) is hosted on
 Docker Hub and serves images of Flink version 1.2.1 and later.
 
-Images for each supported combination of Hadoop and Scala are available, and 
tag aliases are provided for convenience.
+### Image tags
 
-Beginning with Flink 1.5, image tags that omit a Hadoop version (e.g.
-`-hadoop28`) correspond to Hadoop-free releases of Flink that do not include a
-bundled Hadoop distribution.
+Images for each supported combination of Flink and Scala versions are 
available, and
+[tag aliases](https://hub.docker.com/_/flink?tab=tags) are provided for 
convenience.
 
-For example, the following aliases can be used: *(`1.5.y` indicates the latest
-release of Flink 1.5)*
+For example, you can use the following aliases: *(`1.11.y` indicates the 
latest release of Flink 1.11)*
 
 * `flink:latest` → `flink:-scala_`
-* `flink:1.5` → `flink:1.5.y-scala_2.11`
-* `flink:1.5-hadoop27` → `flink:1.5.y-hadoop27-scala_2.11`
+* `flink:1.11` → `flink:1.11.y-scala_2.11`
+
+Note Prio to Flink 1.5 version, Hadoop 
dependencies were always bundled with Flink.
+You can see that certain tags include the version of Hadoop, e.g. (e.g. 
`-hadoop28`).
+Beginning with Flink 1.5, image tags that omit the Hadoop version correspond 
to Hadoop-free releases of Flink
+that do not include a bundled Hadoop distribution.
+
+## How to run Flink image
+
+The Flink image contains a regular Flink distribution with its default 
configuration and a standard entry point script.
+You can run its entry point in the following modes:
+* *Flink Master* for [a Session cluster](#start-a-session-cluster)
+* *Flink Master* for [a Single Job cluster](#start-a-single-job-cluster)
+* *TaskManager* for any cluster
+
+This allows you to deploy a standalone cluster (Session or Single Job) in any 
containerised environment, for example:
+* manually in a local docker setup,
+* [in a Kubernetes cluster](kubernetes.html),
+* [with Docker Compose](#flink-with-docker-compose),
+* [with Docker swarm](#flink-with-docker-swarm).
+
+Note [The native 
Kubernetes](native_kubernetes.html) also runs the same image by default
+and deploys *TaskManagers* on demand so that you do not have to do it manually.
+
+The next chapters describe how to start a single Flink docker container for 
various purposes.
+
+### Start a Session Cluster
+
+A *Flink Session cluster* can be used to run multiple jobs. Each job needs to 
be submitted to the cluster after it has been deployed.
+To deploy a *Flink Session cluster* with docker, you need to start a *Flink 
Master* container:
+
+```sh
+docker run flink:{% if site.is_stable 
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif 
%} jobmanager
+```
+
+and one or more *TaskManager* containers:
+
+```sh
+docker run flink:{% if site.is_stable 
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif 
%} taskmanager
+```
+
+### Start a Job Cluster
+
+A *Flink Job cluster* is a dedicated cluster which runs a single job.
+In this case, you deploy the cluster with the job as one step, thus, there is 
no extra job submission needed.
+Therefore, the *job artifacts* should be already available locally in the 
container.
+
+The *job artifacts* are included into the class path of Flink's JVM process 
within the container and consist of:
+* your job jar, which you would normally submit to a *Session cluster* and
+* all other necessary dependencies or resources, not included into Flink.
+
+To deploy a cluster for a single job with docker, you need to
+* make *job artifacts* available locally *in all containers* under 
`/opt/flink/usrlib`,
+* start a *Flink Master* container in the *Job Cluster* mode
+* start the required number of *TaskManager* containers.
+
+To make the **job artifacts available** locally in the container, you can
+
+* **either mount a volume** (or multiple volumes)

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

2020-05-15 Thread GitBox



azagrebin commented on a change in pull request #12131:
URL: https://github.com/apache/flink/pull/12131#discussion_r425776072



##
File path: docs/ops/deployment/docker.md
##
@@ -246,10 +246,15 @@ At the end you can call the standard 
`/docker-entrypoint.sh` script of the Flink
 # create custom_plugin.jar
 
 echo "
-ln -fs /opt/flink/opt/flink-metrics-graphite-*.jar /opt/flink/lib/.  # 
enable an optional library
+ln -fs /opt/flink/opt/flink-queryable-state-runtime-*.jar /opt/flink/lib/. 
 # enable an optional library
 ln -fs /mnt/custom_lib.jar /opt/flink/lib/.  # enable a custom library
-ln -fs /opt/flink/opt/flink-s3-fs-hadoop-*.jar /opt/flink/lib/.  # enable 
an optional plugin
-ln -fs /mnt/custom_plugin.jar /opt/flink/plugins/.  # enable a custom 
plugin
+
+mkdir -p /opt/flink/plugins/flink-s3-fs-hadoop

Review comment:
   not atm as I understand, it is basically copied from the `plugins` 
how-to.
   we should have packed it respectively in `/opt/flink/opt` which is another 
issue.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

2020-05-15 Thread GitBox



azagrebin commented on a change in pull request #12131:
URL: https://github.com/apache/flink/pull/12131#discussion_r425773202



##
File path: docs/ops/deployment/kubernetes.md
##
@@ -274,44 +341,121 @@ spec:
 path: flink-conf.yaml
   - key: log4j.properties
 path: log4j.properties
-
 {% endhighlight %}
 
-`jobmanager-service.yaml`
+### Job cluster resource definitions
+
+`jobmanager-job.yaml`
 {% highlight yaml %}
-apiVersion: v1
-kind: Service
+apiVersion: batch/v1
+kind: Job
 metadata:
   name: flink-jobmanager
 spec:
-  type: ClusterIP
-  ports:
-  - name: rpc
-port: 6123
-  - name: blob
-port: 6124
-  - name: ui
-port: 8081
+  replicas: 1
   selector:
-app: flink
-component: jobmanager
+matchLabels:
+  app: flink
+  component: jobmanager
+  template:
+metadata:
+  labels:
+app: flink
+component: jobmanager
+spec:
+  restartPolicy: OnFailure
+  containers:
+- name: jobmanager
+  image: flink:{% if site.is_stable 
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %}
+  env:
+  workingDir: /opt/flink

Review comment:
   true, this leftover looks redundant





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

2020-05-15 Thread GitBox



azagrebin commented on a change in pull request #12131:
URL: https://github.com/apache/flink/pull/12131#discussion_r425772745



##
File path: docs/ops/deployment/kubernetes.md
##
@@ -274,44 +341,121 @@ spec:
 path: flink-conf.yaml
   - key: log4j.properties
 path: log4j.properties
-
 {% endhighlight %}
 
-`jobmanager-service.yaml`
+### Job cluster resource definitions
+
+`jobmanager-job.yaml`
 {% highlight yaml %}
-apiVersion: v1
-kind: Service
+apiVersion: batch/v1

Review comment:
   I am fully familiar with versioning of Kubernetes components but it 
matters. The versions correspond to the current examples on Kubernetes docs.

##
File path: docs/ops/deployment/kubernetes.md
##
@@ -274,44 +341,121 @@ spec:
 path: flink-conf.yaml
   - key: log4j.properties
 path: log4j.properties
-
 {% endhighlight %}
 
-`jobmanager-service.yaml`
+### Job cluster resource definitions
+
+`jobmanager-job.yaml`
 {% highlight yaml %}
-apiVersion: v1
-kind: Service
+apiVersion: batch/v1

Review comment:
   I am fully familiar with versioning of Kubernetes components but it 
matters. The versions correspond to the current examples in Kubernetes docs.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

2020-05-15 Thread GitBox



azagrebin commented on a change in pull request #12131:
URL: https://github.com/apache/flink/pull/12131#discussion_r425770459



##
File path: docs/ops/deployment/kubernetes.md
##
@@ -143,8 +176,10 @@ data:
 blob.server.port: 6124
 jobmanager.rpc.port: 6123
 taskmanager.rpc.port: 6122
-jobmanager.memory.process.size: 1472m
-taskmanager.memory.process.size: 1024m
+queryable-state.server.ports: 6125
+jobmanager.memory.process.size: 1600m

Review comment:
   I think only `jobmanager.rpc.address` is required





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

2020-05-15 Thread GitBox



azagrebin commented on a change in pull request #12131:
URL: https://github.com/apache/flink/pull/12131#discussion_r425765241



##
File path: docs/ops/deployment/kubernetes.md
##
@@ -76,57 +74,92 @@ You can then access the Flink UI via different ways:
 {% highlight bash %}./bin/flink run -m localhost:8081 
./examples/streaming/WordCount.jar{% endhighlight %}
 
 *  Create a `NodePort` service on the rest service of jobmanager:
-1. Run `kubectl create -f jobmanager-rest-service.yaml` to create the 
`NodePort` service on jobmanager. The example of `jobmanager-rest-service.yaml` 
can be found in [appendix](#session-cluster-resource-definitions).
+1. Run `kubectl create -f jobmanager-rest-service.yaml` to create the 
`NodePort` service on jobmanager. The example of `jobmanager-rest-service.yaml` 
can be found in [appendix](#common-cluster-resource-definitions).
 2. Run `kubectl get svc flink-jobmanager-rest` to know the `node-port` of 
this service and navigate to 
[http://:;](http://:)
 in your browser.
-3. Similarly to `port-forward` solution, you could also use the following 
command below to submit jobs to the cluster:
+3. If you use minikube, you can get its public ip by running `minikube ip`.
+4. Similarly to `port-forward` solution, you could also use the following 
command below to submit jobs to the cluster:
 
 {% highlight bash %}./bin/flink run -m : 
./examples/streaming/WordCount.jar{% endhighlight %}
 
-In order to terminate the Flink session cluster, use `kubectl`:
+In order to terminate the Flink cluster, terminate the specific 
[Session](#deploy-session-cluster) or [Job](#deploy-job-cluster) cluster 
components
+and use `kubectl` to terminate the common components:
 
-kubectl delete -f jobmanager-deployment.yaml
-kubectl delete -f taskmanager-deployment.yaml
+```sh
 kubectl delete -f jobmanager-service.yaml
 kubectl delete -f flink-configuration-configmap.yaml
+# if created then also the rest service
+kubectl delete -f jobmanager-rest-service.yaml
+```
+
+### Deploy Session Cluster
+
+A Flink Session cluster is executed as a long-running Kubernetes Deployment.
+Note that you can run multiple Flink jobs on a Session cluster.
+Each job needs to be submitted to the cluster after the cluster has been 
deployed.
 
-## Flink job cluster on Kubernetes
+A basic Flink Session cluster deployment in Kubernetes has three components:
 
-A Flink job cluster is a dedicated cluster which runs a single job.
-The job is part of the image and, thus, there is no extra job submission 
needed.
+* a Deployment which runs a Flink Master
+* a Deployment for a pool of TaskManagers
+* a Service exposing the Flink Master's REST and UI ports
 
-### Creating the job-specific image
+After creating [the common cluster 
components](#deploy-flink-cluster-on-kubernetes), use [the Session specific 
resource definitions](#session-cluster-resource-definitions)
+to launch the Session cluster with the `kubectl` command:
 
-The Flink job cluster image needs to contain the user code jars of the job for 
which the cluster is started.
-Therefore, one needs to build a dedicated container image for every job.
-Please follow these [instructions](https://github.com/apache/flink/blob/{{ 
site.github_branch }}/flink-container/docker/README.md) to build the Docker 
image.
+```sh
+kubectl create -f jobmanager-session-deployment.yaml
+kubectl create -f taskmanager-session-deployment.yaml
+```
 
-### Using plugins
+To terminate the Session cluster, these components can be deleted before [the 
common ones](#deploy-flink-cluster-on-kubernetes) with the `kubectl` command:

Review comment:
   Asaik, in general you create and remove resources unordered in k8s.
   I can change this:
   `before` -> `along with`





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

2020-05-15 Thread GitBox



azagrebin commented on a change in pull request #12131:
URL: https://github.com/apache/flink/pull/12131#discussion_r425763837



##
File path: docs/ops/deployment/kubernetes.md
##
@@ -76,57 +74,92 @@ You can then access the Flink UI via different ways:
 {% highlight bash %}./bin/flink run -m localhost:8081 
./examples/streaming/WordCount.jar{% endhighlight %}
 
 *  Create a `NodePort` service on the rest service of jobmanager:
-1. Run `kubectl create -f jobmanager-rest-service.yaml` to create the 
`NodePort` service on jobmanager. The example of `jobmanager-rest-service.yaml` 
can be found in [appendix](#session-cluster-resource-definitions).
+1. Run `kubectl create -f jobmanager-rest-service.yaml` to create the 
`NodePort` service on jobmanager. The example of `jobmanager-rest-service.yaml` 
can be found in [appendix](#common-cluster-resource-definitions).
 2. Run `kubectl get svc flink-jobmanager-rest` to know the `node-port` of 
this service and navigate to 
[http://:;](http://:)
 in your browser.
-3. Similarly to `port-forward` solution, you could also use the following 
command below to submit jobs to the cluster:
+3. If you use minikube, you can get its public ip by running `minikube ip`.
+4. Similarly to `port-forward` solution, you could also use the following 
command below to submit jobs to the cluster:
 
 {% highlight bash %}./bin/flink run -m : 
./examples/streaming/WordCount.jar{% endhighlight %}
 
-In order to terminate the Flink session cluster, use `kubectl`:
+In order to terminate the Flink cluster, terminate the specific 
[Session](#deploy-session-cluster) or [Job](#deploy-job-cluster) cluster 
components
+and use `kubectl` to terminate the common components:
 
-kubectl delete -f jobmanager-deployment.yaml
-kubectl delete -f taskmanager-deployment.yaml
+```sh
 kubectl delete -f jobmanager-service.yaml
 kubectl delete -f flink-configuration-configmap.yaml
+# if created then also the rest service
+kubectl delete -f jobmanager-rest-service.yaml
+```
+
+### Deploy Session Cluster
+
+A Flink Session cluster is executed as a long-running Kubernetes Deployment.
+Note that you can run multiple Flink jobs on a Session cluster.
+Each job needs to be submitted to the cluster after the cluster has been 
deployed.
 
-## Flink job cluster on Kubernetes
+A basic Flink Session cluster deployment in Kubernetes has three components:
 
-A Flink job cluster is a dedicated cluster which runs a single job.
-The job is part of the image and, thus, there is no extra job submission 
needed.
+* a Deployment which runs a Flink Master
+* a Deployment for a pool of TaskManagers
+* a Service exposing the Flink Master's REST and UI ports
 
-### Creating the job-specific image
+After creating [the common cluster 
components](#deploy-flink-cluster-on-kubernetes), use [the Session specific 
resource definitions](#session-cluster-resource-definitions)
+to launch the Session cluster with the `kubectl` command:
 
-The Flink job cluster image needs to contain the user code jars of the job for 
which the cluster is started.
-Therefore, one needs to build a dedicated container image for every job.
-Please follow these [instructions](https://github.com/apache/flink/blob/{{ 
site.github_branch }}/flink-container/docker/README.md) to build the Docker 
image.
+```sh
+kubectl create -f jobmanager-session-deployment.yaml
+kubectl create -f taskmanager-session-deployment.yaml
+```
 
-### Using plugins
+To terminate the Session cluster, these components can be deleted before [the 
common ones](#deploy-flink-cluster-on-kubernetes) with the `kubectl` command:
 
-As described in the [plugins]({{ site.baseurl }}/ops/plugins.html) 
documentation page: in order to use plugins they must be
-copied to the correct location in the flink installation for them to work.
+```sh
+kubectl delete -f taskmanager-session-deployment.yaml

Review comment:
   I do not think so





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

2020-05-15 Thread GitBox



azagrebin commented on a change in pull request #12131:
URL: https://github.com/apache/flink/pull/12131#discussion_r425742232



##
File path: docs/ops/deployment/kubernetes.md
##
@@ -76,57 +74,92 @@ You can then access the Flink UI via different ways:
 {% highlight bash %}./bin/flink run -m localhost:8081 
./examples/streaming/WordCount.jar{% endhighlight %}
 
 *  Create a `NodePort` service on the rest service of jobmanager:
-1. Run `kubectl create -f jobmanager-rest-service.yaml` to create the 
`NodePort` service on jobmanager. The example of `jobmanager-rest-service.yaml` 
can be found in [appendix](#session-cluster-resource-definitions).
+1. Run `kubectl create -f jobmanager-rest-service.yaml` to create the 
`NodePort` service on jobmanager. The example of `jobmanager-rest-service.yaml` 
can be found in [appendix](#common-cluster-resource-definitions).
 2. Run `kubectl get svc flink-jobmanager-rest` to know the `node-port` of 
this service and navigate to 
[http://:;](http://:)
 in your browser.
-3. Similarly to `port-forward` solution, you could also use the following 
command below to submit jobs to the cluster:
+3. If you use minikube, you can get its public ip by running `minikube ip`.
+4. Similarly to `port-forward` solution, you could also use the following 
command below to submit jobs to the cluster:
 
 {% highlight bash %}./bin/flink run -m : 
./examples/streaming/WordCount.jar{% endhighlight %}
 
-In order to terminate the Flink session cluster, use `kubectl`:
+In order to terminate the Flink cluster, terminate the specific 
[Session](#deploy-session-cluster) or [Job](#deploy-job-cluster) cluster 
components
+and use `kubectl` to terminate the common components:
 
-kubectl delete -f jobmanager-deployment.yaml
-kubectl delete -f taskmanager-deployment.yaml
+```sh
 kubectl delete -f jobmanager-service.yaml
 kubectl delete -f flink-configuration-configmap.yaml
+# if created then also the rest service
+kubectl delete -f jobmanager-rest-service.yaml
+```
+
+### Deploy Session Cluster
+
+A Flink Session cluster is executed as a long-running Kubernetes Deployment.
+Note that you can run multiple Flink jobs on a Session cluster.
+Each job needs to be submitted to the cluster after the cluster has been 
deployed.
 
-## Flink job cluster on Kubernetes
+A basic Flink Session cluster deployment in Kubernetes has three components:

Review comment:
   I think it implicitly means here that this is just a simple example to 
start from.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

2020-05-15 Thread GitBox



azagrebin commented on a change in pull request #12131:
URL: https://github.com/apache/flink/pull/12131#discussion_r425740724



##
File path: docs/ops/deployment/kubernetes.md
##
@@ -76,57 +74,92 @@ You can then access the Flink UI via different ways:
 {% highlight bash %}./bin/flink run -m localhost:8081 
./examples/streaming/WordCount.jar{% endhighlight %}
 
 *  Create a `NodePort` service on the rest service of jobmanager:
-1. Run `kubectl create -f jobmanager-rest-service.yaml` to create the 
`NodePort` service on jobmanager. The example of `jobmanager-rest-service.yaml` 
can be found in [appendix](#session-cluster-resource-definitions).
+1. Run `kubectl create -f jobmanager-rest-service.yaml` to create the 
`NodePort` service on jobmanager. The example of `jobmanager-rest-service.yaml` 
can be found in [appendix](#common-cluster-resource-definitions).
 2. Run `kubectl get svc flink-jobmanager-rest` to know the `node-port` of 
this service and navigate to 
[http://:;](http://:)
 in your browser.
-3. Similarly to `port-forward` solution, you could also use the following 
command below to submit jobs to the cluster:
+3. If you use minikube, you can get its public ip by running `minikube ip`.
+4. Similarly to `port-forward` solution, you could also use the following 
command below to submit jobs to the cluster:

Review comment:
   I think it is about point 3 of the previous `* kubectl port-forward:` 
option where you can also use cli but with `localhost` instead of 
``





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

2020-05-15 Thread GitBox



azagrebin commented on a change in pull request #12131:
URL: https://github.com/apache/flink/pull/12131#discussion_r425725545



##
File path: docs/ops/deployment/docker.md
##
@@ -24,119 +24,476 @@ under the License.
 -->
 
 [Docker](https://www.docker.com) is a popular container runtime. 
-There are Docker images for Apache Flink available on Docker Hub which can be 
used to deploy a session cluster.
-The Flink repository also contains tooling to create container images to 
deploy a job cluster.
+There are Docker images for Apache Flink available [on Docker 
Hub](https://hub.docker.com/_/flink).
+You can use the docker images to deploy a Session or Job cluster in a 
containerised environment, e.g.
+[standalone Kubernetes](kubernetes.html) or [native 
Kubernetes](native_kubernetes.html).
 
 * This will be replaced by the TOC
 {:toc}
 
-## Flink session cluster
-
-A Flink session cluster can be used to run multiple jobs. 
-Each job needs to be submitted to the cluster after it has been deployed. 
-
-### Docker images
+## Docker Hub Flink images
 
 The [Flink Docker repository](https://hub.docker.com/_/flink/) is hosted on
 Docker Hub and serves images of Flink version 1.2.1 and later.
 
-Images for each supported combination of Hadoop and Scala are available, and 
tag aliases are provided for convenience.
+### Image tags
 
-Beginning with Flink 1.5, image tags that omit a Hadoop version (e.g.
-`-hadoop28`) correspond to Hadoop-free releases of Flink that do not include a
-bundled Hadoop distribution.
+Images for each supported combination of Flink and Scala versions are 
available, and
+[tag aliases](https://hub.docker.com/_/flink?tab=tags) are provided for 
convenience.
 
-For example, the following aliases can be used: *(`1.5.y` indicates the latest
-release of Flink 1.5)*
+For example, you can use the following aliases: *(`1.11.y` indicates the 
latest release of Flink 1.11)*
 
 * `flink:latest` → `flink:-scala_`
-* `flink:1.5` → `flink:1.5.y-scala_2.11`
-* `flink:1.5-hadoop27` → `flink:1.5.y-hadoop27-scala_2.11`
+* `flink:1.11` → `flink:1.11.y-scala_2.11`
+
+Note Prio to Flink 1.5 version, Hadoop 
dependencies were always bundled with Flink.
+You can see that certain tags include the version of Hadoop, e.g. (e.g. 
`-hadoop28`).
+Beginning with Flink 1.5, image tags that omit the Hadoop version correspond 
to Hadoop-free releases of Flink
+that do not include a bundled Hadoop distribution.
+
+## How to run Flink image
+
+The Flink image contains a regular Flink distribution with its default 
configuration and a standard entry point script.
+You can run its standard entry point in the following modes:
+* Flink Master for [a Session cluster](#start-a-session-cluster)
+* Flink Master for [a Single Job cluster](#start-a-single-job-cluster)
+* TaskManager for any cluster
+
+This allows to deploy a standalone cluster (Session or Single Job) in any 
containerised environment, for example:
+* manually in a local docker setup,
+* [in a Kubernetes cluster](kubernetes.html),
+* [with Docker Compose](#flink-with-docker-compose),
+* [with Docker swarm](#flink-with-docker-swarm).
+
+Note [The native 
Kubernetes](native_kubernetes.html) also runs the same image by default
+and deploys TaskManagers on demand so that you do not have to do it.
+
+The next chapters describe how to start a single Flink docker container for 
various purposes.
+
+### Start a Session Cluster
+
+A Flink Session cluster can be used to run multiple jobs. Each job needs to be 
submitted to the cluster after it has been deployed.
+To deploy a Flink Session cluster with docker, you need to start a Flink 
Master container:
+
+```sh
+docker run flink:{% if site.is_stable 
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif 
%} jobmanager
+```
+
+and required number of TaskManager containers:
+
+```sh
+docker run flink:{% if site.is_stable 
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif 
%} taskmanager
+```
+
+### Start a Single Job Cluster
+
+A Flink Job cluster is a dedicated cluster which runs a single job.
+The job artifacts should be already available locally in the container and, 
thus, there is no extra job submission needed.
+To deploy a cluster for a single job with docker, you need to
+* make job artifacts available locally *in all containers* under 
`/opt/flink/usrlib`,
+* start a Flink Master container in the Single Job mode
+* and required number of TaskManager containers.
+
+To make the **job artifacts available** locally in the container, you can
+
+* **either mount a volume** (or multiple volumes) with the artifacts to 
`/opt/flink/usrlib` when you start
+the Flink image as a Single Job Master and the required number of TaskManagers:
+
+```sh
+docker run \
+--mount 
type=bind,src=/host/path/to/job/artifacts1,target=/opt/flink/usrlib/artifacts/1 
\
+--mount 
type=bind,src=/host/path/to/job/artifacts2,target=/opt/flink/usrlib/artifacts/2 
\

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

2020-05-15 Thread GitBox



azagrebin commented on a change in pull request #12131:
URL: https://github.com/apache/flink/pull/12131#discussion_r425712944



##
File path: docs/ops/deployment/docker.md
##
@@ -24,119 +24,476 @@ under the License.
 -->
 
 [Docker](https://www.docker.com) is a popular container runtime. 
-There are Docker images for Apache Flink available on Docker Hub which can be 
used to deploy a session cluster.
-The Flink repository also contains tooling to create container images to 
deploy a job cluster.
+There are Docker images for Apache Flink available [on Docker 
Hub](https://hub.docker.com/_/flink).
+You can use the docker images to deploy a Session or Job cluster in a 
containerised environment, e.g.
+[standalone Kubernetes](kubernetes.html) or [native 
Kubernetes](native_kubernetes.html).
 
 * This will be replaced by the TOC
 {:toc}
 
-## Flink session cluster
-
-A Flink session cluster can be used to run multiple jobs. 
-Each job needs to be submitted to the cluster after it has been deployed. 
-
-### Docker images
+## Docker Hub Flink images
 
 The [Flink Docker repository](https://hub.docker.com/_/flink/) is hosted on
 Docker Hub and serves images of Flink version 1.2.1 and later.
 
-Images for each supported combination of Hadoop and Scala are available, and 
tag aliases are provided for convenience.
+### Image tags
 
-Beginning with Flink 1.5, image tags that omit a Hadoop version (e.g.
-`-hadoop28`) correspond to Hadoop-free releases of Flink that do not include a
-bundled Hadoop distribution.
+Images for each supported combination of Flink and Scala versions are 
available, and
+[tag aliases](https://hub.docker.com/_/flink?tab=tags) are provided for 
convenience.
 
-For example, the following aliases can be used: *(`1.5.y` indicates the latest
-release of Flink 1.5)*
+For example, you can use the following aliases: *(`1.11.y` indicates the 
latest release of Flink 1.11)*
 
 * `flink:latest` → `flink:-scala_`
-* `flink:1.5` → `flink:1.5.y-scala_2.11`
-* `flink:1.5-hadoop27` → `flink:1.5.y-hadoop27-scala_2.11`
+* `flink:1.11` → `flink:1.11.y-scala_2.11`
+
+Note Prio to Flink 1.5 version, Hadoop 
dependencies were always bundled with Flink.
+You can see that certain tags include the version of Hadoop, e.g. (e.g. 
`-hadoop28`).
+Beginning with Flink 1.5, image tags that omit the Hadoop version correspond 
to Hadoop-free releases of Flink
+that do not include a bundled Hadoop distribution.
+
+## How to run Flink image
+
+The Flink image contains a regular Flink distribution with its default 
configuration and a standard entry point script.
+You can run its standard entry point in the following modes:
+* Flink Master for [a Session cluster](#start-a-session-cluster)
+* Flink Master for [a Single Job cluster](#start-a-single-job-cluster)
+* TaskManager for any cluster
+
+This allows to deploy a standalone cluster (Session or Single Job) in any 
containerised environment, for example:
+* manually in a local docker setup,
+* [in a Kubernetes cluster](kubernetes.html),
+* [with Docker Compose](#flink-with-docker-compose),
+* [with Docker swarm](#flink-with-docker-swarm).
+
+Note [The native 
Kubernetes](native_kubernetes.html) also runs the same image by default
+and deploys TaskManagers on demand so that you do not have to do it.
+
+The next chapters describe how to start a single Flink docker container for 
various purposes.
+
+### Start a Session Cluster
+
+A Flink Session cluster can be used to run multiple jobs. Each job needs to be 
submitted to the cluster after it has been deployed.
+To deploy a Flink Session cluster with docker, you need to start a Flink 
Master container:
+
+```sh
+docker run flink:{% if site.is_stable 
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif 
%} jobmanager
+```
+
+and required number of TaskManager containers:
+
+```sh
+docker run flink:{% if site.is_stable 
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif 
%} taskmanager
+```
+
+### Start a Single Job Cluster
+
+A Flink Job cluster is a dedicated cluster which runs a single job.
+The job artifacts should be already available locally in the container and, 
thus, there is no extra job submission needed.
+To deploy a cluster for a single job with docker, you need to
+* make job artifacts available locally *in all containers* under 
`/opt/flink/usrlib`,
+* start a Flink Master container in the Single Job mode
+* and required number of TaskManager containers.
+
+To make the **job artifacts available** locally in the container, you can
+
+* **either mount a volume** (or multiple volumes) with the artifacts to 
`/opt/flink/usrlib` when you start
+the Flink image as a Single Job Master and the required number of TaskManagers:
+
+```sh
+docker run \
+--mount 
type=bind,src=/host/path/to/job/artifacts1,target=/opt/flink/usrlib/artifacts/1 
\
+--mount 
type=bind,src=/host/path/to/job/artifacts2,target=/opt/flink/usrlib/artifacts/2 
\

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

2020-05-15 Thread GitBox



azagrebin commented on a change in pull request #12131:
URL: https://github.com/apache/flink/pull/12131#discussion_r425698567



##
File path: docs/ops/deployment/docker.md
##
@@ -24,119 +24,476 @@ under the License.
 -->
 
 [Docker](https://www.docker.com) is a popular container runtime. 
-There are Docker images for Apache Flink available on Docker Hub which can be 
used to deploy a session cluster.
-The Flink repository also contains tooling to create container images to 
deploy a job cluster.
+There are Docker images for Apache Flink available [on Docker 
Hub](https://hub.docker.com/_/flink).
+You can use the docker images to deploy a Session or Job cluster in a 
containerised environment, e.g.
+[standalone Kubernetes](kubernetes.html) or [native 
Kubernetes](native_kubernetes.html).
 
 * This will be replaced by the TOC
 {:toc}
 
-## Flink session cluster
-
-A Flink session cluster can be used to run multiple jobs. 
-Each job needs to be submitted to the cluster after it has been deployed. 
-
-### Docker images
+## Docker Hub Flink images
 
 The [Flink Docker repository](https://hub.docker.com/_/flink/) is hosted on
 Docker Hub and serves images of Flink version 1.2.1 and later.
 
-Images for each supported combination of Hadoop and Scala are available, and 
tag aliases are provided for convenience.
+### Image tags
 
-Beginning with Flink 1.5, image tags that omit a Hadoop version (e.g.
-`-hadoop28`) correspond to Hadoop-free releases of Flink that do not include a
-bundled Hadoop distribution.
+Images for each supported combination of Flink and Scala versions are 
available, and
+[tag aliases](https://hub.docker.com/_/flink?tab=tags) are provided for 
convenience.
 
-For example, the following aliases can be used: *(`1.5.y` indicates the latest
-release of Flink 1.5)*
+For example, you can use the following aliases: *(`1.11.y` indicates the 
latest release of Flink 1.11)*
 
 * `flink:latest` → `flink:-scala_`
-* `flink:1.5` → `flink:1.5.y-scala_2.11`
-* `flink:1.5-hadoop27` → `flink:1.5.y-hadoop27-scala_2.11`
+* `flink:1.11` → `flink:1.11.y-scala_2.11`
+
+Note Prio to Flink 1.5 version, Hadoop 
dependencies were always bundled with Flink.
+You can see that certain tags include the version of Hadoop, e.g. (e.g. 
`-hadoop28`).
+Beginning with Flink 1.5, image tags that omit the Hadoop version correspond 
to Hadoop-free releases of Flink
+that do not include a bundled Hadoop distribution.
+
+## How to run Flink image
+
+The Flink image contains a regular Flink distribution with its default 
configuration and a standard entry point script.
+You can run its standard entry point in the following modes:
+* Flink Master for [a Session cluster](#start-a-session-cluster)
+* Flink Master for [a Single Job cluster](#start-a-single-job-cluster)
+* TaskManager for any cluster
+
+This allows to deploy a standalone cluster (Session or Single Job) in any 
containerised environment, for example:
+* manually in a local docker setup,
+* [in a Kubernetes cluster](kubernetes.html),
+* [with Docker Compose](#flink-with-docker-compose),
+* [with Docker swarm](#flink-with-docker-swarm).
+
+Note [The native 
Kubernetes](native_kubernetes.html) also runs the same image by default
+and deploys TaskManagers on demand so that you do not have to do it.
+
+The next chapters describe how to start a single Flink docker container for 
various purposes.
+
+### Start a Session Cluster
+
+A Flink Session cluster can be used to run multiple jobs. Each job needs to be 
submitted to the cluster after it has been deployed.
+To deploy a Flink Session cluster with docker, you need to start a Flink 
Master container:
+
+```sh
+docker run flink:{% if site.is_stable 
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif 
%} jobmanager
+```
+
+and required number of TaskManager containers:

Review comment:
   no big worries, just trying to be more explicit, I think this reassures 
unexperienced users and no big harm/verbosity





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

2020-05-15 Thread GitBox



azagrebin commented on a change in pull request #12131:
URL: https://github.com/apache/flink/pull/12131#discussion_r425698567



##
File path: docs/ops/deployment/docker.md
##
@@ -24,119 +24,476 @@ under the License.
 -->
 
 [Docker](https://www.docker.com) is a popular container runtime. 
-There are Docker images for Apache Flink available on Docker Hub which can be 
used to deploy a session cluster.
-The Flink repository also contains tooling to create container images to 
deploy a job cluster.
+There are Docker images for Apache Flink available [on Docker 
Hub](https://hub.docker.com/_/flink).
+You can use the docker images to deploy a Session or Job cluster in a 
containerised environment, e.g.
+[standalone Kubernetes](kubernetes.html) or [native 
Kubernetes](native_kubernetes.html).
 
 * This will be replaced by the TOC
 {:toc}
 
-## Flink session cluster
-
-A Flink session cluster can be used to run multiple jobs. 
-Each job needs to be submitted to the cluster after it has been deployed. 
-
-### Docker images
+## Docker Hub Flink images
 
 The [Flink Docker repository](https://hub.docker.com/_/flink/) is hosted on
 Docker Hub and serves images of Flink version 1.2.1 and later.
 
-Images for each supported combination of Hadoop and Scala are available, and 
tag aliases are provided for convenience.
+### Image tags
 
-Beginning with Flink 1.5, image tags that omit a Hadoop version (e.g.
-`-hadoop28`) correspond to Hadoop-free releases of Flink that do not include a
-bundled Hadoop distribution.
+Images for each supported combination of Flink and Scala versions are 
available, and
+[tag aliases](https://hub.docker.com/_/flink?tab=tags) are provided for 
convenience.
 
-For example, the following aliases can be used: *(`1.5.y` indicates the latest
-release of Flink 1.5)*
+For example, you can use the following aliases: *(`1.11.y` indicates the 
latest release of Flink 1.11)*
 
 * `flink:latest` → `flink:-scala_`
-* `flink:1.5` → `flink:1.5.y-scala_2.11`
-* `flink:1.5-hadoop27` → `flink:1.5.y-hadoop27-scala_2.11`
+* `flink:1.11` → `flink:1.11.y-scala_2.11`
+
+Note Prio to Flink 1.5 version, Hadoop 
dependencies were always bundled with Flink.
+You can see that certain tags include the version of Hadoop, e.g. (e.g. 
`-hadoop28`).
+Beginning with Flink 1.5, image tags that omit the Hadoop version correspond 
to Hadoop-free releases of Flink
+that do not include a bundled Hadoop distribution.
+
+## How to run Flink image
+
+The Flink image contains a regular Flink distribution with its default 
configuration and a standard entry point script.
+You can run its standard entry point in the following modes:
+* Flink Master for [a Session cluster](#start-a-session-cluster)
+* Flink Master for [a Single Job cluster](#start-a-single-job-cluster)
+* TaskManager for any cluster
+
+This allows to deploy a standalone cluster (Session or Single Job) in any 
containerised environment, for example:
+* manually in a local docker setup,
+* [in a Kubernetes cluster](kubernetes.html),
+* [with Docker Compose](#flink-with-docker-compose),
+* [with Docker swarm](#flink-with-docker-swarm).
+
+Note [The native 
Kubernetes](native_kubernetes.html) also runs the same image by default
+and deploys TaskManagers on demand so that you do not have to do it.
+
+The next chapters describe how to start a single Flink docker container for 
various purposes.
+
+### Start a Session Cluster
+
+A Flink Session cluster can be used to run multiple jobs. Each job needs to be 
submitted to the cluster after it has been deployed.
+To deploy a Flink Session cluster with docker, you need to start a Flink 
Master container:
+
+```sh
+docker run flink:{% if site.is_stable 
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif 
%} jobmanager
+```
+
+and required number of TaskManager containers:

Review comment:
   no big worries, just trying to be more explicit, I think this reassures 
unexperienced users





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

2020-05-14 Thread GitBox



azagrebin commented on a change in pull request #12131:
URL: https://github.com/apache/flink/pull/12131#discussion_r425147269



##
File path: docs/ops/deployment/docker.md
##
@@ -24,119 +24,476 @@ under the License.
 -->
 
 [Docker](https://www.docker.com) is a popular container runtime. 
-There are Docker images for Apache Flink available on Docker Hub which can be 
used to deploy a session cluster.
-The Flink repository also contains tooling to create container images to 
deploy a job cluster.
+There are Docker images for Apache Flink available [on Docker 
Hub](https://hub.docker.com/_/flink).
+You can use the docker images to deploy a Session or Job cluster in a 
containerised environment, e.g.
+[standalone Kubernetes](kubernetes.html) or [native 
Kubernetes](native_kubernetes.html).
 
 * This will be replaced by the TOC
 {:toc}
 
-## Flink session cluster
-
-A Flink session cluster can be used to run multiple jobs. 
-Each job needs to be submitted to the cluster after it has been deployed. 
-
-### Docker images
+## Docker Hub Flink images
 
 The [Flink Docker repository](https://hub.docker.com/_/flink/) is hosted on
 Docker Hub and serves images of Flink version 1.2.1 and later.
 
-Images for each supported combination of Hadoop and Scala are available, and 
tag aliases are provided for convenience.
+### Image tags
 
-Beginning with Flink 1.5, image tags that omit a Hadoop version (e.g.
-`-hadoop28`) correspond to Hadoop-free releases of Flink that do not include a
-bundled Hadoop distribution.
+Images for each supported combination of Flink and Scala versions are 
available, and
+[tag aliases](https://hub.docker.com/_/flink?tab=tags) are provided for 
convenience.
 
-For example, the following aliases can be used: *(`1.5.y` indicates the latest
-release of Flink 1.5)*
+For example, you can use the following aliases: *(`1.11.y` indicates the 
latest release of Flink 1.11)*
 
 * `flink:latest` → `flink:-scala_`
-* `flink:1.5` → `flink:1.5.y-scala_2.11`
-* `flink:1.5-hadoop27` → `flink:1.5.y-hadoop27-scala_2.11`
+* `flink:1.11` → `flink:1.11.y-scala_2.11`
+
+Note Prio to Flink 1.5 version, Hadoop 
dependencies were always bundled with Flink.
+You can see that certain tags include the version of Hadoop, e.g. (e.g. 
`-hadoop28`).
+Beginning with Flink 1.5, image tags that omit the Hadoop version correspond 
to Hadoop-free releases of Flink
+that do not include a bundled Hadoop distribution.
+
+## How to run Flink image
+
+The Flink image contains a regular Flink distribution with its default 
configuration and a standard entry point script.
+You can run its standard entry point in the following modes:
+* Flink Master for [a Session cluster](#start-a-session-cluster)
+* Flink Master for [a Single Job cluster](#start-a-single-job-cluster)
+* TaskManager for any cluster
+
+This allows to deploy a standalone cluster (Session or Single Job) in any 
containerised environment, for example:
+* manually in a local docker setup,
+* [in a Kubernetes cluster](kubernetes.html),
+* [with Docker Compose](#flink-with-docker-compose),
+* [with Docker swarm](#flink-with-docker-swarm).
+
+Note [The native 
Kubernetes](native_kubernetes.html) also runs the same image by default
+and deploys TaskManagers on demand so that you do not have to do it.
+
+The next chapters describe how to start a single Flink docker container for 
various purposes.
+
+### Start a Session Cluster
+
+A Flink Session cluster can be used to run multiple jobs. Each job needs to be 
submitted to the cluster after it has been deployed.
+To deploy a Flink Session cluster with docker, you need to start a Flink 
Master container:
+
+```sh
+docker run flink:{% if site.is_stable 
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif 
%} jobmanager
+```
+
+and required number of TaskManager containers:
+
+```sh
+docker run flink:{% if site.is_stable 
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif 
%} taskmanager
+```
+
+### Start a Single Job Cluster
+
+A Flink Job cluster is a dedicated cluster which runs a single job.
+The job artifacts should be already available locally in the container and, 
thus, there is no extra job submission needed.
+To deploy a cluster for a single job with docker, you need to
+* make job artifacts available locally *in all containers* under 
`/opt/flink/usrlib`,
+* start a Flink Master container in the Single Job mode
+* and required number of TaskManager containers.
+
+To make the **job artifacts available** locally in the container, you can
+
+* **either mount a volume** (or multiple volumes) with the artifacts to 
`/opt/flink/usrlib` when you start
+the Flink image as a Single Job Master and the required number of TaskManagers:
+
+```sh
+docker run \
+--mount 
type=bind,src=/host/path/to/job/artifacts1,target=/opt/flink/usrlib/artifacts/1 
\
+--mount 
type=bind,src=/host/path/to/job/artifacts2,target=/opt/flink/usrlib/artifacts/2 
\

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

2020-05-14 Thread GitBox



azagrebin commented on a change in pull request #12131:
URL: https://github.com/apache/flink/pull/12131#discussion_r425144968



##
File path: docs/ops/deployment/docker.md
##
@@ -24,119 +24,476 @@ under the License.
 -->
 
 [Docker](https://www.docker.com) is a popular container runtime. 
-There are Docker images for Apache Flink available on Docker Hub which can be 
used to deploy a session cluster.
-The Flink repository also contains tooling to create container images to 
deploy a job cluster.
+There are Docker images for Apache Flink available [on Docker 
Hub](https://hub.docker.com/_/flink).
+You can use the docker images to deploy a Session or Job cluster in a 
containerised environment, e.g.
+[standalone Kubernetes](kubernetes.html) or [native 
Kubernetes](native_kubernetes.html).
 
 * This will be replaced by the TOC
 {:toc}
 
-## Flink session cluster
-
-A Flink session cluster can be used to run multiple jobs. 
-Each job needs to be submitted to the cluster after it has been deployed. 
-
-### Docker images
+## Docker Hub Flink images
 
 The [Flink Docker repository](https://hub.docker.com/_/flink/) is hosted on
 Docker Hub and serves images of Flink version 1.2.1 and later.
 
-Images for each supported combination of Hadoop and Scala are available, and 
tag aliases are provided for convenience.
+### Image tags
 
-Beginning with Flink 1.5, image tags that omit a Hadoop version (e.g.
-`-hadoop28`) correspond to Hadoop-free releases of Flink that do not include a
-bundled Hadoop distribution.
+Images for each supported combination of Flink and Scala versions are 
available, and
+[tag aliases](https://hub.docker.com/_/flink?tab=tags) are provided for 
convenience.
 
-For example, the following aliases can be used: *(`1.5.y` indicates the latest
-release of Flink 1.5)*
+For example, you can use the following aliases: *(`1.11.y` indicates the 
latest release of Flink 1.11)*
 
 * `flink:latest` → `flink:-scala_`
-* `flink:1.5` → `flink:1.5.y-scala_2.11`
-* `flink:1.5-hadoop27` → `flink:1.5.y-hadoop27-scala_2.11`
+* `flink:1.11` → `flink:1.11.y-scala_2.11`
+
+Note Prio to Flink 1.5 version, Hadoop 
dependencies were always bundled with Flink.
+You can see that certain tags include the version of Hadoop, e.g. (e.g. 
`-hadoop28`).
+Beginning with Flink 1.5, image tags that omit the Hadoop version correspond 
to Hadoop-free releases of Flink
+that do not include a bundled Hadoop distribution.
+
+## How to run Flink image
+
+The Flink image contains a regular Flink distribution with its default 
configuration and a standard entry point script.
+You can run its standard entry point in the following modes:
+* Flink Master for [a Session cluster](#start-a-session-cluster)
+* Flink Master for [a Single Job cluster](#start-a-single-job-cluster)
+* TaskManager for any cluster
+
+This allows to deploy a standalone cluster (Session or Single Job) in any 
containerised environment, for example:
+* manually in a local docker setup,
+* [in a Kubernetes cluster](kubernetes.html),
+* [with Docker Compose](#flink-with-docker-compose),
+* [with Docker swarm](#flink-with-docker-swarm).
+
+Note [The native 
Kubernetes](native_kubernetes.html) also runs the same image by default
+and deploys TaskManagers on demand so that you do not have to do it.
+
+The next chapters describe how to start a single Flink docker container for 
various purposes.
+
+### Start a Session Cluster
+
+A Flink Session cluster can be used to run multiple jobs. Each job needs to be 
submitted to the cluster after it has been deployed.
+To deploy a Flink Session cluster with docker, you need to start a Flink 
Master container:
+
+```sh
+docker run flink:{% if site.is_stable 
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif 
%} jobmanager
+```
+
+and required number of TaskManager containers:
+
+```sh
+docker run flink:{% if site.is_stable 
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif 
%} taskmanager
+```
+
+### Start a Single Job Cluster
+
+A Flink Job cluster is a dedicated cluster which runs a single job.
+The job artifacts should be already available locally in the container and, 
thus, there is no extra job submission needed.
+To deploy a cluster for a single job with docker, you need to
+* make job artifacts available locally *in all containers* under 
`/opt/flink/usrlib`,

Review comment:
   True, I have not found any general description of the job standalone 
cluster, independent from deployment environment. We should add it and then 
adjust docker docs to refer to it. For now, I can add e.g.:
   ```
   The *job artifacts* are included into the class path of Flink's JVM process 
within the container and consist of:
   * your job jar, which you would normally submit to a *Session cluster* and
   * all other necessary dependencies or resources, not included into Flink.
   ```





This is an automated me

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

2020-05-14 Thread GitBox



azagrebin commented on a change in pull request #12131:
URL: https://github.com/apache/flink/pull/12131#discussion_r425134658



##
File path: docs/ops/deployment/docker.md
##
@@ -24,119 +24,476 @@ under the License.
 -->
 
 [Docker](https://www.docker.com) is a popular container runtime. 
-There are Docker images for Apache Flink available on Docker Hub which can be 
used to deploy a session cluster.
-The Flink repository also contains tooling to create container images to 
deploy a job cluster.
+There are Docker images for Apache Flink available [on Docker 
Hub](https://hub.docker.com/_/flink).
+You can use the docker images to deploy a Session or Job cluster in a 
containerised environment, e.g.
+[standalone Kubernetes](kubernetes.html) or [native 
Kubernetes](native_kubernetes.html).
 
 * This will be replaced by the TOC
 {:toc}
 
-## Flink session cluster
-
-A Flink session cluster can be used to run multiple jobs. 
-Each job needs to be submitted to the cluster after it has been deployed. 
-
-### Docker images
+## Docker Hub Flink images
 
 The [Flink Docker repository](https://hub.docker.com/_/flink/) is hosted on
 Docker Hub and serves images of Flink version 1.2.1 and later.
 
-Images for each supported combination of Hadoop and Scala are available, and 
tag aliases are provided for convenience.
+### Image tags
 
-Beginning with Flink 1.5, image tags that omit a Hadoop version (e.g.
-`-hadoop28`) correspond to Hadoop-free releases of Flink that do not include a
-bundled Hadoop distribution.
+Images for each supported combination of Flink and Scala versions are 
available, and
+[tag aliases](https://hub.docker.com/_/flink?tab=tags) are provided for 
convenience.
 
-For example, the following aliases can be used: *(`1.5.y` indicates the latest
-release of Flink 1.5)*
+For example, you can use the following aliases: *(`1.11.y` indicates the 
latest release of Flink 1.11)*
 
 * `flink:latest` → `flink:-scala_`
-* `flink:1.5` → `flink:1.5.y-scala_2.11`
-* `flink:1.5-hadoop27` → `flink:1.5.y-hadoop27-scala_2.11`
+* `flink:1.11` → `flink:1.11.y-scala_2.11`
+
+Note Prio to Flink 1.5 version, Hadoop 
dependencies were always bundled with Flink.
+You can see that certain tags include the version of Hadoop, e.g. (e.g. 
`-hadoop28`).
+Beginning with Flink 1.5, image tags that omit the Hadoop version correspond 
to Hadoop-free releases of Flink
+that do not include a bundled Hadoop distribution.
+
+## How to run Flink image
+
+The Flink image contains a regular Flink distribution with its default 
configuration and a standard entry point script.
+You can run its standard entry point in the following modes:
+* Flink Master for [a Session cluster](#start-a-session-cluster)
+* Flink Master for [a Single Job cluster](#start-a-single-job-cluster)
+* TaskManager for any cluster
+
+This allows to deploy a standalone cluster (Session or Single Job) in any 
containerised environment, for example:
+* manually in a local docker setup,
+* [in a Kubernetes cluster](kubernetes.html),
+* [with Docker Compose](#flink-with-docker-compose),
+* [with Docker swarm](#flink-with-docker-swarm).
+
+Note [The native 
Kubernetes](native_kubernetes.html) also runs the same image by default
+and deploys TaskManagers on demand so that you do not have to do it.
+
+The next chapters describe how to start a single Flink docker container for 
various purposes.
+
+### Start a Session Cluster
+
+A Flink Session cluster can be used to run multiple jobs. Each job needs to be 
submitted to the cluster after it has been deployed.
+To deploy a Flink Session cluster with docker, you need to start a Flink 
Master container:
+
+```sh
+docker run flink:{% if site.is_stable 
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif 
%} jobmanager
+```
+
+and required number of TaskManager containers:
+
+```sh
+docker run flink:{% if site.is_stable 
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif 
%} taskmanager
+```
+
+### Start a Single Job Cluster
+
+A Flink Job cluster is a dedicated cluster which runs a single job.
+The job artifacts should be already available locally in the container and, 
thus, there is no extra job submission needed.

Review comment:
   ok, how about this:
   ```
   A *Flink Job cluster* is a dedicated cluster which runs a single job.
   In this case, you deploy the cluster with the job as one step, thus, there 
is no extra job submission needed.
   Therefore, the job artifacts should be already available locally in the 
container.
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

2020-05-14 Thread GitBox



azagrebin commented on a change in pull request #12131:
URL: https://github.com/apache/flink/pull/12131#discussion_r425132315



##
File path: docs/ops/deployment/docker.md
##
@@ -24,119 +24,476 @@ under the License.
 -->
 
 [Docker](https://www.docker.com) is a popular container runtime. 
-There are Docker images for Apache Flink available on Docker Hub which can be 
used to deploy a session cluster.
-The Flink repository also contains tooling to create container images to 
deploy a job cluster.
+There are Docker images for Apache Flink available [on Docker 
Hub](https://hub.docker.com/_/flink).
+You can use the docker images to deploy a Session or Job cluster in a 
containerised environment, e.g.
+[standalone Kubernetes](kubernetes.html) or [native 
Kubernetes](native_kubernetes.html).
 
 * This will be replaced by the TOC
 {:toc}
 
-## Flink session cluster
-
-A Flink session cluster can be used to run multiple jobs. 
-Each job needs to be submitted to the cluster after it has been deployed. 
-
-### Docker images
+## Docker Hub Flink images
 
 The [Flink Docker repository](https://hub.docker.com/_/flink/) is hosted on
 Docker Hub and serves images of Flink version 1.2.1 and later.
 
-Images for each supported combination of Hadoop and Scala are available, and 
tag aliases are provided for convenience.
+### Image tags
 
-Beginning with Flink 1.5, image tags that omit a Hadoop version (e.g.
-`-hadoop28`) correspond to Hadoop-free releases of Flink that do not include a
-bundled Hadoop distribution.
+Images for each supported combination of Flink and Scala versions are 
available, and
+[tag aliases](https://hub.docker.com/_/flink?tab=tags) are provided for 
convenience.
 
-For example, the following aliases can be used: *(`1.5.y` indicates the latest
-release of Flink 1.5)*
+For example, you can use the following aliases: *(`1.11.y` indicates the 
latest release of Flink 1.11)*
 
 * `flink:latest` → `flink:-scala_`
-* `flink:1.5` → `flink:1.5.y-scala_2.11`
-* `flink:1.5-hadoop27` → `flink:1.5.y-hadoop27-scala_2.11`
+* `flink:1.11` → `flink:1.11.y-scala_2.11`
+
+Note Prio to Flink 1.5 version, Hadoop 
dependencies were always bundled with Flink.
+You can see that certain tags include the version of Hadoop, e.g. (e.g. 
`-hadoop28`).
+Beginning with Flink 1.5, image tags that omit the Hadoop version correspond 
to Hadoop-free releases of Flink
+that do not include a bundled Hadoop distribution.
+
+## How to run Flink image
+
+The Flink image contains a regular Flink distribution with its default 
configuration and a standard entry point script.
+You can run its standard entry point in the following modes:
+* Flink Master for [a Session cluster](#start-a-session-cluster)
+* Flink Master for [a Single Job cluster](#start-a-single-job-cluster)
+* TaskManager for any cluster
+
+This allows to deploy a standalone cluster (Session or Single Job) in any 
containerised environment, for example:
+* manually in a local docker setup,
+* [in a Kubernetes cluster](kubernetes.html),
+* [with Docker Compose](#flink-with-docker-compose),
+* [with Docker swarm](#flink-with-docker-swarm).
+
+Note [The native 
Kubernetes](native_kubernetes.html) also runs the same image by default
+and deploys TaskManagers on demand so that you do not have to do it.
+
+The next chapters describe how to start a single Flink docker container for 
various purposes.
+
+### Start a Session Cluster
+
+A Flink Session cluster can be used to run multiple jobs. Each job needs to be 
submitted to the cluster after it has been deployed.
+To deploy a Flink Session cluster with docker, you need to start a Flink 
Master container:
+
+```sh
+docker run flink:{% if site.is_stable 
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif 
%} jobmanager
+```
+
+and required number of TaskManager containers:

Review comment:
   Do you mean you have a concern?
   I can change to:
   `and one or more *TaskManager* containers:`





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

2020-05-14 Thread GitBox



azagrebin commented on a change in pull request #12131:
URL: https://github.com/apache/flink/pull/12131#discussion_r425131551



##
File path: docs/ops/deployment/docker.md
##
@@ -24,119 +24,476 @@ under the License.
 -->
 
 [Docker](https://www.docker.com) is a popular container runtime. 
-There are Docker images for Apache Flink available on Docker Hub which can be 
used to deploy a session cluster.
-The Flink repository also contains tooling to create container images to 
deploy a job cluster.
+There are Docker images for Apache Flink available [on Docker 
Hub](https://hub.docker.com/_/flink).
+You can use the docker images to deploy a Session or Job cluster in a 
containerised environment, e.g.
+[standalone Kubernetes](kubernetes.html) or [native 
Kubernetes](native_kubernetes.html).
 
 * This will be replaced by the TOC
 {:toc}
 
-## Flink session cluster
-
-A Flink session cluster can be used to run multiple jobs. 
-Each job needs to be submitted to the cluster after it has been deployed. 
-
-### Docker images
+## Docker Hub Flink images
 
 The [Flink Docker repository](https://hub.docker.com/_/flink/) is hosted on
 Docker Hub and serves images of Flink version 1.2.1 and later.
 
-Images for each supported combination of Hadoop and Scala are available, and 
tag aliases are provided for convenience.
+### Image tags
 
-Beginning with Flink 1.5, image tags that omit a Hadoop version (e.g.
-`-hadoop28`) correspond to Hadoop-free releases of Flink that do not include a
-bundled Hadoop distribution.
+Images for each supported combination of Flink and Scala versions are 
available, and
+[tag aliases](https://hub.docker.com/_/flink?tab=tags) are provided for 
convenience.
 
-For example, the following aliases can be used: *(`1.5.y` indicates the latest
-release of Flink 1.5)*
+For example, you can use the following aliases: *(`1.11.y` indicates the 
latest release of Flink 1.11)*
 
 * `flink:latest` → `flink:-scala_`
-* `flink:1.5` → `flink:1.5.y-scala_2.11`
-* `flink:1.5-hadoop27` → `flink:1.5.y-hadoop27-scala_2.11`
+* `flink:1.11` → `flink:1.11.y-scala_2.11`
+
+Note Prio to Flink 1.5 version, Hadoop 
dependencies were always bundled with Flink.
+You can see that certain tags include the version of Hadoop, e.g. (e.g. 
`-hadoop28`).
+Beginning with Flink 1.5, image tags that omit the Hadoop version correspond 
to Hadoop-free releases of Flink
+that do not include a bundled Hadoop distribution.
+
+## How to run Flink image
+
+The Flink image contains a regular Flink distribution with its default 
configuration and a standard entry point script.
+You can run its standard entry point in the following modes:
+* Flink Master for [a Session cluster](#start-a-session-cluster)

Review comment:
   I have at least highlight all these concepts with \*Flink Master\*





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

2020-05-14 Thread GitBox



azagrebin commented on a change in pull request #12131:
URL: https://github.com/apache/flink/pull/12131#discussion_r425130785



##
File path: docs/ops/deployment/docker.md
##
@@ -24,119 +24,476 @@ under the License.
 -->
 
 [Docker](https://www.docker.com) is a popular container runtime. 
-There are Docker images for Apache Flink available on Docker Hub which can be 
used to deploy a session cluster.
-The Flink repository also contains tooling to create container images to 
deploy a job cluster.
+There are Docker images for Apache Flink available [on Docker 
Hub](https://hub.docker.com/_/flink).
+You can use the docker images to deploy a Session or Job cluster in a 
containerised environment, e.g.
+[standalone Kubernetes](kubernetes.html) or [native 
Kubernetes](native_kubernetes.html).
 
 * This will be replaced by the TOC
 {:toc}
 
-## Flink session cluster
-
-A Flink session cluster can be used to run multiple jobs. 
-Each job needs to be submitted to the cluster after it has been deployed. 
-
-### Docker images
+## Docker Hub Flink images
 
 The [Flink Docker repository](https://hub.docker.com/_/flink/) is hosted on
 Docker Hub and serves images of Flink version 1.2.1 and later.
 
-Images for each supported combination of Hadoop and Scala are available, and 
tag aliases are provided for convenience.
+### Image tags
 
-Beginning with Flink 1.5, image tags that omit a Hadoop version (e.g.
-`-hadoop28`) correspond to Hadoop-free releases of Flink that do not include a
-bundled Hadoop distribution.
+Images for each supported combination of Flink and Scala versions are 
available, and
+[tag aliases](https://hub.docker.com/_/flink?tab=tags) are provided for 
convenience.
 
-For example, the following aliases can be used: *(`1.5.y` indicates the latest
-release of Flink 1.5)*
+For example, you can use the following aliases: *(`1.11.y` indicates the 
latest release of Flink 1.11)*
 
 * `flink:latest` → `flink:-scala_`
-* `flink:1.5` → `flink:1.5.y-scala_2.11`
-* `flink:1.5-hadoop27` → `flink:1.5.y-hadoop27-scala_2.11`
+* `flink:1.11` → `flink:1.11.y-scala_2.11`
+
+Note Prio to Flink 1.5 version, Hadoop 
dependencies were always bundled with Flink.
+You can see that certain tags include the version of Hadoop, e.g. (e.g. 
`-hadoop28`).
+Beginning with Flink 1.5, image tags that omit the Hadoop version correspond 
to Hadoop-free releases of Flink
+that do not include a bundled Hadoop distribution.
+
+## How to run Flink image
+
+The Flink image contains a regular Flink distribution with its default 
configuration and a standard entry point script.
+You can run its standard entry point in the following modes:
+* Flink Master for [a Session cluster](#start-a-session-cluster)
+* Flink Master for [a Single Job cluster](#start-a-single-job-cluster)
+* TaskManager for any cluster
+
+This allows to deploy a standalone cluster (Session or Single Job) in any 
containerised environment, for example:
+* manually in a local docker setup,
+* [in a Kubernetes cluster](kubernetes.html),
+* [with Docker Compose](#flink-with-docker-compose),
+* [with Docker swarm](#flink-with-docker-swarm).
+
+Note [The native 
Kubernetes](native_kubernetes.html) also runs the same image by default
+and deploys TaskManagers on demand so that you do not have to do it.
+
+The next chapters describe how to start a single Flink docker container for 
various purposes.
+
+### Start a Session Cluster
+
+A Flink Session cluster can be used to run multiple jobs. Each job needs to be 
submitted to the cluster after it has been deployed.
+To deploy a Flink Session cluster with docker, you need to start a Flink 
Master container:
+
+```sh
+docker run flink:{% if site.is_stable 
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif 
%} jobmanager
+```
+
+and required number of TaskManager containers:
+
+```sh
+docker run flink:{% if site.is_stable 
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif 
%} taskmanager
+```
+
+### Start a Single Job Cluster
+
+A Flink Job cluster is a dedicated cluster which runs a single job.
+The job artifacts should be already available locally in the container and, 
thus, there is no extra job submission needed.
+To deploy a cluster for a single job with docker, you need to
+* make job artifacts available locally *in all containers* under 
`/opt/flink/usrlib`,
+* start a Flink Master container in the Single Job mode
+* and required number of TaskManager containers.
+
+To make the **job artifacts available** locally in the container, you can
+
+* **either mount a volume** (or multiple volumes) with the artifacts to 
`/opt/flink/usrlib` when you start
+the Flink image as a Single Job Master and the required number of TaskManagers:
+
+```sh
+docker run \
+--mount 
type=bind,src=/host/path/to/job/artifacts1,target=/opt/flink/usrlib/artifacts/1 
\
+--mount 
type=bind,src=/host/path/to/job/artifacts2,target=/opt/flink/usrlib/artifacts/2 
\

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

2020-05-14 Thread GitBox



azagrebin commented on a change in pull request #12131:
URL: https://github.com/apache/flink/pull/12131#discussion_r425117737



##
File path: docs/ops/deployment/docker.md
##
@@ -24,119 +24,476 @@ under the License.
 -->
 
 [Docker](https://www.docker.com) is a popular container runtime. 
-There are Docker images for Apache Flink available on Docker Hub which can be 
used to deploy a session cluster.
-The Flink repository also contains tooling to create container images to 
deploy a job cluster.
+There are Docker images for Apache Flink available [on Docker 
Hub](https://hub.docker.com/_/flink).
+You can use the docker images to deploy a Session or Job cluster in a 
containerised environment, e.g.
+[standalone Kubernetes](kubernetes.html) or [native 
Kubernetes](native_kubernetes.html).
 
 * This will be replaced by the TOC
 {:toc}
 
-## Flink session cluster
-
-A Flink session cluster can be used to run multiple jobs. 
-Each job needs to be submitted to the cluster after it has been deployed. 
-
-### Docker images
+## Docker Hub Flink images
 
 The [Flink Docker repository](https://hub.docker.com/_/flink/) is hosted on
 Docker Hub and serves images of Flink version 1.2.1 and later.
 
-Images for each supported combination of Hadoop and Scala are available, and 
tag aliases are provided for convenience.
+### Image tags
 
-Beginning with Flink 1.5, image tags that omit a Hadoop version (e.g.
-`-hadoop28`) correspond to Hadoop-free releases of Flink that do not include a
-bundled Hadoop distribution.
+Images for each supported combination of Flink and Scala versions are 
available, and
+[tag aliases](https://hub.docker.com/_/flink?tab=tags) are provided for 
convenience.
 
-For example, the following aliases can be used: *(`1.5.y` indicates the latest
-release of Flink 1.5)*
+For example, you can use the following aliases: *(`1.11.y` indicates the 
latest release of Flink 1.11)*
 
 * `flink:latest` → `flink:-scala_`
-* `flink:1.5` → `flink:1.5.y-scala_2.11`
-* `flink:1.5-hadoop27` → `flink:1.5.y-hadoop27-scala_2.11`
+* `flink:1.11` → `flink:1.11.y-scala_2.11`
+
+Note Prio to Flink 1.5 version, Hadoop 
dependencies were always bundled with Flink.
+You can see that certain tags include the version of Hadoop, e.g. (e.g. 
`-hadoop28`).
+Beginning with Flink 1.5, image tags that omit the Hadoop version correspond 
to Hadoop-free releases of Flink
+that do not include a bundled Hadoop distribution.
+
+## How to run Flink image
+
+The Flink image contains a regular Flink distribution with its default 
configuration and a standard entry point script.
+You can run its standard entry point in the following modes:
+* Flink Master for [a Session cluster](#start-a-session-cluster)
+* Flink Master for [a Single Job cluster](#start-a-single-job-cluster)
+* TaskManager for any cluster
+
+This allows to deploy a standalone cluster (Session or Single Job) in any 
containerised environment, for example:
+* manually in a local docker setup,
+* [in a Kubernetes cluster](kubernetes.html),
+* [with Docker Compose](#flink-with-docker-compose),
+* [with Docker swarm](#flink-with-docker-swarm).
+
+Note [The native 
Kubernetes](native_kubernetes.html) also runs the same image by default
+and deploys TaskManagers on demand so that you do not have to do it.
+
+The next chapters describe how to start a single Flink docker container for 
various purposes.
+
+### Start a Session Cluster
+
+A Flink Session cluster can be used to run multiple jobs. Each job needs to be 
submitted to the cluster after it has been deployed.
+To deploy a Flink Session cluster with docker, you need to start a Flink 
Master container:
+
+```sh
+docker run flink:{% if site.is_stable 
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif 
%} jobmanager
+```
+
+and required number of TaskManager containers:
+
+```sh
+docker run flink:{% if site.is_stable 
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif 
%} taskmanager
+```
+
+### Start a Single Job Cluster
+
+A Flink Job cluster is a dedicated cluster which runs a single job.
+The job artifacts should be already available locally in the container and, 
thus, there is no extra job submission needed.
+To deploy a cluster for a single job with docker, you need to
+* make job artifacts available locally *in all containers* under 
`/opt/flink/usrlib`,
+* start a Flink Master container in the Single Job mode
+* and required number of TaskManager containers.
+
+To make the **job artifacts available** locally in the container, you can
+
+* **either mount a volume** (or multiple volumes) with the artifacts to 
`/opt/flink/usrlib` when you start
+the Flink image as a Single Job Master and the required number of TaskManagers:
+
+```sh
+docker run \
+--mount 
type=bind,src=/host/path/to/job/artifacts1,target=/opt/flink/usrlib/artifacts/1 
\
+--mount 
type=bind,src=/host/path/to/job/artifacts2,target=/opt/flink/usrlib/artifacts/2 
\

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

2020-05-14 Thread GitBox



azagrebin commented on a change in pull request #12131:
URL: https://github.com/apache/flink/pull/12131#discussion_r425115061



##
File path: docs/ops/deployment/docker.md
##
@@ -24,119 +24,476 @@ under the License.
 -->
 
 [Docker](https://www.docker.com) is a popular container runtime. 
-There are Docker images for Apache Flink available on Docker Hub which can be 
used to deploy a session cluster.
-The Flink repository also contains tooling to create container images to 
deploy a job cluster.
+There are Docker images for Apache Flink available [on Docker 
Hub](https://hub.docker.com/_/flink).
+You can use the docker images to deploy a Session or Job cluster in a 
containerised environment, e.g.
+[standalone Kubernetes](kubernetes.html) or [native 
Kubernetes](native_kubernetes.html).
 
 * This will be replaced by the TOC
 {:toc}
 
-## Flink session cluster
-
-A Flink session cluster can be used to run multiple jobs. 
-Each job needs to be submitted to the cluster after it has been deployed. 
-
-### Docker images
+## Docker Hub Flink images
 
 The [Flink Docker repository](https://hub.docker.com/_/flink/) is hosted on
 Docker Hub and serves images of Flink version 1.2.1 and later.
 
-Images for each supported combination of Hadoop and Scala are available, and 
tag aliases are provided for convenience.
+### Image tags
 
-Beginning with Flink 1.5, image tags that omit a Hadoop version (e.g.
-`-hadoop28`) correspond to Hadoop-free releases of Flink that do not include a
-bundled Hadoop distribution.
+Images for each supported combination of Flink and Scala versions are 
available, and
+[tag aliases](https://hub.docker.com/_/flink?tab=tags) are provided for 
convenience.
 
-For example, the following aliases can be used: *(`1.5.y` indicates the latest
-release of Flink 1.5)*
+For example, you can use the following aliases: *(`1.11.y` indicates the 
latest release of Flink 1.11)*
 
 * `flink:latest` → `flink:-scala_`
-* `flink:1.5` → `flink:1.5.y-scala_2.11`
-* `flink:1.5-hadoop27` → `flink:1.5.y-hadoop27-scala_2.11`
+* `flink:1.11` → `flink:1.11.y-scala_2.11`
+
+Note Prio to Flink 1.5 version, Hadoop 
dependencies were always bundled with Flink.
+You can see that certain tags include the version of Hadoop, e.g. (e.g. 
`-hadoop28`).
+Beginning with Flink 1.5, image tags that omit the Hadoop version correspond 
to Hadoop-free releases of Flink
+that do not include a bundled Hadoop distribution.
+
+## How to run Flink image
+
+The Flink image contains a regular Flink distribution with its default 
configuration and a standard entry point script.
+You can run its standard entry point in the following modes:
+* Flink Master for [a Session cluster](#start-a-session-cluster)
+* Flink Master for [a Single Job cluster](#start-a-single-job-cluster)
+* TaskManager for any cluster
+
+This allows to deploy a standalone cluster (Session or Single Job) in any 
containerised environment, for example:
+* manually in a local docker setup,
+* [in a Kubernetes cluster](kubernetes.html),
+* [with Docker Compose](#flink-with-docker-compose),
+* [with Docker swarm](#flink-with-docker-swarm).
+
+Note [The native 
Kubernetes](native_kubernetes.html) also runs the same image by default
+and deploys TaskManagers on demand so that you do not have to do it.
+
+The next chapters describe how to start a single Flink docker container for 
various purposes.
+
+### Start a Session Cluster
+
+A Flink Session cluster can be used to run multiple jobs. Each job needs to be 
submitted to the cluster after it has been deployed.
+To deploy a Flink Session cluster with docker, you need to start a Flink 
Master container:
+
+```sh
+docker run flink:{% if site.is_stable 
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif 
%} jobmanager
+```
+
+and required number of TaskManager containers:
+
+```sh
+docker run flink:{% if site.is_stable 
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif 
%} taskmanager
+```
+
+### Start a Single Job Cluster
+
+A Flink Job cluster is a dedicated cluster which runs a single job.
+The job artifacts should be already available locally in the container and, 
thus, there is no extra job submission needed.
+To deploy a cluster for a single job with docker, you need to
+* make job artifacts available locally *in all containers* under 
`/opt/flink/usrlib`,
+* start a Flink Master container in the Single Job mode
+* and required number of TaskManager containers.
+
+To make the **job artifacts available** locally in the container, you can
+
+* **either mount a volume** (or multiple volumes) with the artifacts to 
`/opt/flink/usrlib` when you start
+the Flink image as a Single Job Master and the required number of TaskManagers:
+
+```sh
+docker run \
+--mount 
type=bind,src=/host/path/to/job/artifacts1,target=/opt/flink/usrlib/artifacts/1 
\
+--mount 
type=bind,src=/host/path/to/job/artifacts2,target=/opt/flink/usrlib/artifacts/2 
\

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

2020-05-14 Thread GitBox



azagrebin commented on a change in pull request #12131:
URL: https://github.com/apache/flink/pull/12131#discussion_r425114909



##
File path: docs/ops/deployment/docker.md
##
@@ -24,119 +24,476 @@ under the License.
 -->
 
 [Docker](https://www.docker.com) is a popular container runtime. 
-There are Docker images for Apache Flink available on Docker Hub which can be 
used to deploy a session cluster.
-The Flink repository also contains tooling to create container images to 
deploy a job cluster.
+There are Docker images for Apache Flink available [on Docker 
Hub](https://hub.docker.com/_/flink).
+You can use the docker images to deploy a Session or Job cluster in a 
containerised environment, e.g.
+[standalone Kubernetes](kubernetes.html) or [native 
Kubernetes](native_kubernetes.html).
 
 * This will be replaced by the TOC
 {:toc}
 
-## Flink session cluster
-
-A Flink session cluster can be used to run multiple jobs. 
-Each job needs to be submitted to the cluster after it has been deployed. 
-
-### Docker images
+## Docker Hub Flink images
 
 The [Flink Docker repository](https://hub.docker.com/_/flink/) is hosted on
 Docker Hub and serves images of Flink version 1.2.1 and later.
 
-Images for each supported combination of Hadoop and Scala are available, and 
tag aliases are provided for convenience.
+### Image tags
 
-Beginning with Flink 1.5, image tags that omit a Hadoop version (e.g.
-`-hadoop28`) correspond to Hadoop-free releases of Flink that do not include a
-bundled Hadoop distribution.
+Images for each supported combination of Flink and Scala versions are 
available, and
+[tag aliases](https://hub.docker.com/_/flink?tab=tags) are provided for 
convenience.
 
-For example, the following aliases can be used: *(`1.5.y` indicates the latest
-release of Flink 1.5)*
+For example, you can use the following aliases: *(`1.11.y` indicates the 
latest release of Flink 1.11)*
 
 * `flink:latest` → `flink:-scala_`
-* `flink:1.5` → `flink:1.5.y-scala_2.11`
-* `flink:1.5-hadoop27` → `flink:1.5.y-hadoop27-scala_2.11`
+* `flink:1.11` → `flink:1.11.y-scala_2.11`
+
+Note Prio to Flink 1.5 version, Hadoop 
dependencies were always bundled with Flink.
+You can see that certain tags include the version of Hadoop, e.g. (e.g. 
`-hadoop28`).
+Beginning with Flink 1.5, image tags that omit the Hadoop version correspond 
to Hadoop-free releases of Flink
+that do not include a bundled Hadoop distribution.
+
+## How to run Flink image
+
+The Flink image contains a regular Flink distribution with its default 
configuration and a standard entry point script.
+You can run its standard entry point in the following modes:
+* Flink Master for [a Session cluster](#start-a-session-cluster)
+* Flink Master for [a Single Job cluster](#start-a-single-job-cluster)
+* TaskManager for any cluster
+
+This allows to deploy a standalone cluster (Session or Single Job) in any 
containerised environment, for example:
+* manually in a local docker setup,
+* [in a Kubernetes cluster](kubernetes.html),
+* [with Docker Compose](#flink-with-docker-compose),
+* [with Docker swarm](#flink-with-docker-swarm).
+
+Note [The native 
Kubernetes](native_kubernetes.html) also runs the same image by default
+and deploys TaskManagers on demand so that you do not have to do it.
+
+The next chapters describe how to start a single Flink docker container for 
various purposes.
+
+### Start a Session Cluster
+
+A Flink Session cluster can be used to run multiple jobs. Each job needs to be 
submitted to the cluster after it has been deployed.
+To deploy a Flink Session cluster with docker, you need to start a Flink 
Master container:
+
+```sh
+docker run flink:{% if site.is_stable 
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif 
%} jobmanager
+```
+
+and required number of TaskManager containers:
+
+```sh
+docker run flink:{% if site.is_stable 
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif 
%} taskmanager
+```
+
+### Start a Single Job Cluster
+
+A Flink Job cluster is a dedicated cluster which runs a single job.
+The job artifacts should be already available locally in the container and, 
thus, there is no extra job submission needed.
+To deploy a cluster for a single job with docker, you need to
+* make job artifacts available locally *in all containers* under 
`/opt/flink/usrlib`,
+* start a Flink Master container in the Single Job mode
+* and required number of TaskManager containers.
+
+To make the **job artifacts available** locally in the container, you can
+
+* **either mount a volume** (or multiple volumes) with the artifacts to 
`/opt/flink/usrlib` when you start
+the Flink image as a Single Job Master and the required number of TaskManagers:
+
+```sh
+docker run \
+--mount 
type=bind,src=/host/path/to/job/artifacts1,target=/opt/flink/usrlib/artifacts/1 
\
+--mount 
type=bind,src=/host/path/to/job/artifacts2,target=/opt/flink/usrlib/artifacts/2 
\

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

2020-05-14 Thread GitBox



azagrebin commented on a change in pull request #12131:
URL: https://github.com/apache/flink/pull/12131#discussion_r425113095



##
File path: docs/ops/deployment/docker.md
##
@@ -24,119 +24,476 @@ under the License.
 -->
 
 [Docker](https://www.docker.com) is a popular container runtime. 
-There are Docker images for Apache Flink available on Docker Hub which can be 
used to deploy a session cluster.
-The Flink repository also contains tooling to create container images to 
deploy a job cluster.
+There are Docker images for Apache Flink available [on Docker 
Hub](https://hub.docker.com/_/flink).
+You can use the docker images to deploy a Session or Job cluster in a 
containerised environment, e.g.
+[standalone Kubernetes](kubernetes.html) or [native 
Kubernetes](native_kubernetes.html).
 
 * This will be replaced by the TOC
 {:toc}
 
-## Flink session cluster
-
-A Flink session cluster can be used to run multiple jobs. 
-Each job needs to be submitted to the cluster after it has been deployed. 
-
-### Docker images
+## Docker Hub Flink images
 
 The [Flink Docker repository](https://hub.docker.com/_/flink/) is hosted on
 Docker Hub and serves images of Flink version 1.2.1 and later.
 
-Images for each supported combination of Hadoop and Scala are available, and 
tag aliases are provided for convenience.
+### Image tags
 
-Beginning with Flink 1.5, image tags that omit a Hadoop version (e.g.
-`-hadoop28`) correspond to Hadoop-free releases of Flink that do not include a
-bundled Hadoop distribution.
+Images for each supported combination of Flink and Scala versions are 
available, and
+[tag aliases](https://hub.docker.com/_/flink?tab=tags) are provided for 
convenience.
 
-For example, the following aliases can be used: *(`1.5.y` indicates the latest
-release of Flink 1.5)*
+For example, you can use the following aliases: *(`1.11.y` indicates the 
latest release of Flink 1.11)*
 
 * `flink:latest` → `flink:-scala_`
-* `flink:1.5` → `flink:1.5.y-scala_2.11`
-* `flink:1.5-hadoop27` → `flink:1.5.y-hadoop27-scala_2.11`
+* `flink:1.11` → `flink:1.11.y-scala_2.11`
+
+Note Prio to Flink 1.5 version, Hadoop 
dependencies were always bundled with Flink.
+You can see that certain tags include the version of Hadoop, e.g. (e.g. 
`-hadoop28`).
+Beginning with Flink 1.5, image tags that omit the Hadoop version correspond 
to Hadoop-free releases of Flink
+that do not include a bundled Hadoop distribution.
+
+## How to run Flink image
+
+The Flink image contains a regular Flink distribution with its default 
configuration and a standard entry point script.
+You can run its standard entry point in the following modes:
+* Flink Master for [a Session cluster](#start-a-session-cluster)
+* Flink Master for [a Single Job cluster](#start-a-single-job-cluster)
+* TaskManager for any cluster
+
+This allows to deploy a standalone cluster (Session or Single Job) in any 
containerised environment, for example:
+* manually in a local docker setup,
+* [in a Kubernetes cluster](kubernetes.html),
+* [with Docker Compose](#flink-with-docker-compose),
+* [with Docker swarm](#flink-with-docker-swarm).
+
+Note [The native 
Kubernetes](native_kubernetes.html) also runs the same image by default
+and deploys TaskManagers on demand so that you do not have to do it.
+
+The next chapters describe how to start a single Flink docker container for 
various purposes.
+
+### Start a Session Cluster
+
+A Flink Session cluster can be used to run multiple jobs. Each job needs to be 
submitted to the cluster after it has been deployed.
+To deploy a Flink Session cluster with docker, you need to start a Flink 
Master container:
+
+```sh
+docker run flink:{% if site.is_stable 
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif 
%} jobmanager
+```
+
+and required number of TaskManager containers:
+
+```sh
+docker run flink:{% if site.is_stable 
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif 
%} taskmanager
+```
+
+### Start a Single Job Cluster
+
+A Flink Job cluster is a dedicated cluster which runs a single job.
+The job artifacts should be already available locally in the container and, 
thus, there is no extra job submission needed.
+To deploy a cluster for a single job with docker, you need to
+* make job artifacts available locally *in all containers* under 
`/opt/flink/usrlib`,
+* start a Flink Master container in the Single Job mode
+* and required number of TaskManager containers.
+
+To make the **job artifacts available** locally in the container, you can
+
+* **either mount a volume** (or multiple volumes) with the artifacts to 
`/opt/flink/usrlib` when you start
+the Flink image as a Single Job Master and the required number of TaskManagers:
+
+```sh
+docker run \
+--mount 
type=bind,src=/host/path/to/job/artifacts1,target=/opt/flink/usrlib/artifacts/1 
\
+--mount 
type=bind,src=/host/path/to/job/artifacts2,target=/opt/flink/usrlib/artifacts/2 
\

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

2020-05-14 Thread GitBox



azagrebin commented on a change in pull request #12131:
URL: https://github.com/apache/flink/pull/12131#discussion_r425108297



##
File path: docs/ops/deployment/docker.md
##
@@ -24,119 +24,476 @@ under the License.
 -->
 
 [Docker](https://www.docker.com) is a popular container runtime. 
-There are Docker images for Apache Flink available on Docker Hub which can be 
used to deploy a session cluster.
-The Flink repository also contains tooling to create container images to 
deploy a job cluster.
+There are Docker images for Apache Flink available [on Docker 
Hub](https://hub.docker.com/_/flink).
+You can use the docker images to deploy a Session or Job cluster in a 
containerised environment, e.g.
+[standalone Kubernetes](kubernetes.html) or [native 
Kubernetes](native_kubernetes.html).
 
 * This will be replaced by the TOC
 {:toc}
 
-## Flink session cluster
-
-A Flink session cluster can be used to run multiple jobs. 
-Each job needs to be submitted to the cluster after it has been deployed. 
-
-### Docker images
+## Docker Hub Flink images
 
 The [Flink Docker repository](https://hub.docker.com/_/flink/) is hosted on
 Docker Hub and serves images of Flink version 1.2.1 and later.
 
-Images for each supported combination of Hadoop and Scala are available, and 
tag aliases are provided for convenience.
+### Image tags
 
-Beginning with Flink 1.5, image tags that omit a Hadoop version (e.g.
-`-hadoop28`) correspond to Hadoop-free releases of Flink that do not include a
-bundled Hadoop distribution.
+Images for each supported combination of Flink and Scala versions are 
available, and
+[tag aliases](https://hub.docker.com/_/flink?tab=tags) are provided for 
convenience.
 
-For example, the following aliases can be used: *(`1.5.y` indicates the latest
-release of Flink 1.5)*
+For example, you can use the following aliases: *(`1.11.y` indicates the 
latest release of Flink 1.11)*
 
 * `flink:latest` → `flink:-scala_`
-* `flink:1.5` → `flink:1.5.y-scala_2.11`
-* `flink:1.5-hadoop27` → `flink:1.5.y-hadoop27-scala_2.11`
+* `flink:1.11` → `flink:1.11.y-scala_2.11`
+
+Note Prio to Flink 1.5 version, Hadoop 
dependencies were always bundled with Flink.
+You can see that certain tags include the version of Hadoop, e.g. (e.g. 
`-hadoop28`).
+Beginning with Flink 1.5, image tags that omit the Hadoop version correspond 
to Hadoop-free releases of Flink
+that do not include a bundled Hadoop distribution.
+
+## How to run Flink image
+
+The Flink image contains a regular Flink distribution with its default 
configuration and a standard entry point script.
+You can run its standard entry point in the following modes:
+* Flink Master for [a Session cluster](#start-a-session-cluster)
+* Flink Master for [a Single Job cluster](#start-a-single-job-cluster)
+* TaskManager for any cluster
+
+This allows to deploy a standalone cluster (Session or Single Job) in any 
containerised environment, for example:
+* manually in a local docker setup,
+* [in a Kubernetes cluster](kubernetes.html),
+* [with Docker Compose](#flink-with-docker-compose),
+* [with Docker swarm](#flink-with-docker-swarm).
+
+Note [The native 
Kubernetes](native_kubernetes.html) also runs the same image by default
+and deploys TaskManagers on demand so that you do not have to do it.
+
+The next chapters describe how to start a single Flink docker container for 
various purposes.
+
+### Start a Session Cluster
+
+A Flink Session cluster can be used to run multiple jobs. Each job needs to be 
submitted to the cluster after it has been deployed.
+To deploy a Flink Session cluster with docker, you need to start a Flink 
Master container:
+
+```sh
+docker run flink:{% if site.is_stable 
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif 
%} jobmanager
+```
+
+and required number of TaskManager containers:
+
+```sh
+docker run flink:{% if site.is_stable 
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif 
%} taskmanager
+```
+
+### Start a Single Job Cluster
+
+A Flink Job cluster is a dedicated cluster which runs a single job.
+The job artifacts should be already available locally in the container and, 
thus, there is no extra job submission needed.
+To deploy a cluster for a single job with docker, you need to
+* make job artifacts available locally *in all containers* under 
`/opt/flink/usrlib`,
+* start a Flink Master container in the Single Job mode
+* and required number of TaskManager containers.
+
+To make the **job artifacts available** locally in the container, you can
+
+* **either mount a volume** (or multiple volumes) with the artifacts to 
`/opt/flink/usrlib` when you start
+the Flink image as a Single Job Master and the required number of TaskManagers:
+
+```sh
+docker run \
+--mount 
type=bind,src=/host/path/to/job/artifacts1,target=/opt/flink/usrlib/artifacts/1 
\
+--mount 
type=bind,src=/host/path/to/job/artifacts2,target=/opt/flink/usrlib/artifacts/2 
\

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

2020-05-14 Thread GitBox



azagrebin commented on a change in pull request #12131:
URL: https://github.com/apache/flink/pull/12131#discussion_r425096153



##
File path: docs/ops/deployment/docker.md
##
@@ -24,119 +24,476 @@ under the License.
 -->
 
 [Docker](https://www.docker.com) is a popular container runtime. 
-There are Docker images for Apache Flink available on Docker Hub which can be 
used to deploy a session cluster.
-The Flink repository also contains tooling to create container images to 
deploy a job cluster.
+There are Docker images for Apache Flink available [on Docker 
Hub](https://hub.docker.com/_/flink).
+You can use the docker images to deploy a Session or Job cluster in a 
containerised environment, e.g.
+[standalone Kubernetes](kubernetes.html) or [native 
Kubernetes](native_kubernetes.html).
 
 * This will be replaced by the TOC
 {:toc}
 
-## Flink session cluster
-
-A Flink session cluster can be used to run multiple jobs. 
-Each job needs to be submitted to the cluster after it has been deployed. 
-
-### Docker images
+## Docker Hub Flink images
 
 The [Flink Docker repository](https://hub.docker.com/_/flink/) is hosted on
 Docker Hub and serves images of Flink version 1.2.1 and later.
 
-Images for each supported combination of Hadoop and Scala are available, and 
tag aliases are provided for convenience.
+### Image tags
 
-Beginning with Flink 1.5, image tags that omit a Hadoop version (e.g.
-`-hadoop28`) correspond to Hadoop-free releases of Flink that do not include a
-bundled Hadoop distribution.
+Images for each supported combination of Flink and Scala versions are 
available, and
+[tag aliases](https://hub.docker.com/_/flink?tab=tags) are provided for 
convenience.
 
-For example, the following aliases can be used: *(`1.5.y` indicates the latest
-release of Flink 1.5)*
+For example, you can use the following aliases: *(`1.11.y` indicates the 
latest release of Flink 1.11)*
 
 * `flink:latest` → `flink:-scala_`
-* `flink:1.5` → `flink:1.5.y-scala_2.11`
-* `flink:1.5-hadoop27` → `flink:1.5.y-hadoop27-scala_2.11`
+* `flink:1.11` → `flink:1.11.y-scala_2.11`
+
+Note Prio to Flink 1.5 version, Hadoop 
dependencies were always bundled with Flink.
+You can see that certain tags include the version of Hadoop, e.g. (e.g. 
`-hadoop28`).
+Beginning with Flink 1.5, image tags that omit the Hadoop version correspond 
to Hadoop-free releases of Flink
+that do not include a bundled Hadoop distribution.
+
+## How to run Flink image
+
+The Flink image contains a regular Flink distribution with its default 
configuration and a standard entry point script.
+You can run its standard entry point in the following modes:
+* Flink Master for [a Session cluster](#start-a-session-cluster)
+* Flink Master for [a Single Job cluster](#start-a-single-job-cluster)
+* TaskManager for any cluster
+
+This allows to deploy a standalone cluster (Session or Single Job) in any 
containerised environment, for example:
+* manually in a local docker setup,
+* [in a Kubernetes cluster](kubernetes.html),
+* [with Docker Compose](#flink-with-docker-compose),
+* [with Docker swarm](#flink-with-docker-swarm).
+
+Note [The native 
Kubernetes](native_kubernetes.html) also runs the same image by default
+and deploys TaskManagers on demand so that you do not have to do it.
+
+The next chapters describe how to start a single Flink docker container for 
various purposes.
+
+### Start a Session Cluster
+
+A Flink Session cluster can be used to run multiple jobs. Each job needs to be 
submitted to the cluster after it has been deployed.
+To deploy a Flink Session cluster with docker, you need to start a Flink 
Master container:
+
+```sh
+docker run flink:{% if site.is_stable 
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif 
%} jobmanager
+```
+
+and required number of TaskManager containers:
+
+```sh
+docker run flink:{% if site.is_stable 
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif 
%} taskmanager
+```
+
+### Start a Single Job Cluster
+
+A Flink Job cluster is a dedicated cluster which runs a single job.
+The job artifacts should be already available locally in the container and, 
thus, there is no extra job submission needed.
+To deploy a cluster for a single job with docker, you need to
+* make job artifacts available locally *in all containers* under 
`/opt/flink/usrlib`,
+* start a Flink Master container in the Single Job mode
+* and required number of TaskManager containers.
+
+To make the **job artifacts available** locally in the container, you can
+
+* **either mount a volume** (or multiple volumes) with the artifacts to 
`/opt/flink/usrlib` when you start
+the Flink image as a Single Job Master and the required number of TaskManagers:
+
+```sh
+docker run \
+--mount 
type=bind,src=/host/path/to/job/artifacts1,target=/opt/flink/usrlib/artifacts/1 
\
+--mount 
type=bind,src=/host/path/to/job/artifacts2,target=/opt/flink/usrlib/artifacts/2 
\

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

2020-05-14 Thread GitBox



azagrebin commented on a change in pull request #12131:
URL: https://github.com/apache/flink/pull/12131#discussion_r425096153



##
File path: docs/ops/deployment/docker.md
##
@@ -24,119 +24,476 @@ under the License.
 -->
 
 [Docker](https://www.docker.com) is a popular container runtime. 
-There are Docker images for Apache Flink available on Docker Hub which can be 
used to deploy a session cluster.
-The Flink repository also contains tooling to create container images to 
deploy a job cluster.
+There are Docker images for Apache Flink available [on Docker 
Hub](https://hub.docker.com/_/flink).
+You can use the docker images to deploy a Session or Job cluster in a 
containerised environment, e.g.
+[standalone Kubernetes](kubernetes.html) or [native 
Kubernetes](native_kubernetes.html).
 
 * This will be replaced by the TOC
 {:toc}
 
-## Flink session cluster
-
-A Flink session cluster can be used to run multiple jobs. 
-Each job needs to be submitted to the cluster after it has been deployed. 
-
-### Docker images
+## Docker Hub Flink images
 
 The [Flink Docker repository](https://hub.docker.com/_/flink/) is hosted on
 Docker Hub and serves images of Flink version 1.2.1 and later.
 
-Images for each supported combination of Hadoop and Scala are available, and 
tag aliases are provided for convenience.
+### Image tags
 
-Beginning with Flink 1.5, image tags that omit a Hadoop version (e.g.
-`-hadoop28`) correspond to Hadoop-free releases of Flink that do not include a
-bundled Hadoop distribution.
+Images for each supported combination of Flink and Scala versions are 
available, and
+[tag aliases](https://hub.docker.com/_/flink?tab=tags) are provided for 
convenience.
 
-For example, the following aliases can be used: *(`1.5.y` indicates the latest
-release of Flink 1.5)*
+For example, you can use the following aliases: *(`1.11.y` indicates the 
latest release of Flink 1.11)*
 
 * `flink:latest` → `flink:-scala_`
-* `flink:1.5` → `flink:1.5.y-scala_2.11`
-* `flink:1.5-hadoop27` → `flink:1.5.y-hadoop27-scala_2.11`
+* `flink:1.11` → `flink:1.11.y-scala_2.11`
+
+Note Prio to Flink 1.5 version, Hadoop 
dependencies were always bundled with Flink.
+You can see that certain tags include the version of Hadoop, e.g. (e.g. 
`-hadoop28`).
+Beginning with Flink 1.5, image tags that omit the Hadoop version correspond 
to Hadoop-free releases of Flink
+that do not include a bundled Hadoop distribution.
+
+## How to run Flink image
+
+The Flink image contains a regular Flink distribution with its default 
configuration and a standard entry point script.
+You can run its standard entry point in the following modes:
+* Flink Master for [a Session cluster](#start-a-session-cluster)
+* Flink Master for [a Single Job cluster](#start-a-single-job-cluster)
+* TaskManager for any cluster
+
+This allows to deploy a standalone cluster (Session or Single Job) in any 
containerised environment, for example:
+* manually in a local docker setup,
+* [in a Kubernetes cluster](kubernetes.html),
+* [with Docker Compose](#flink-with-docker-compose),
+* [with Docker swarm](#flink-with-docker-swarm).
+
+Note [The native 
Kubernetes](native_kubernetes.html) also runs the same image by default
+and deploys TaskManagers on demand so that you do not have to do it.
+
+The next chapters describe how to start a single Flink docker container for 
various purposes.
+
+### Start a Session Cluster
+
+A Flink Session cluster can be used to run multiple jobs. Each job needs to be 
submitted to the cluster after it has been deployed.
+To deploy a Flink Session cluster with docker, you need to start a Flink 
Master container:
+
+```sh
+docker run flink:{% if site.is_stable 
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif 
%} jobmanager
+```
+
+and required number of TaskManager containers:
+
+```sh
+docker run flink:{% if site.is_stable 
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif 
%} taskmanager
+```
+
+### Start a Single Job Cluster
+
+A Flink Job cluster is a dedicated cluster which runs a single job.
+The job artifacts should be already available locally in the container and, 
thus, there is no extra job submission needed.
+To deploy a cluster for a single job with docker, you need to
+* make job artifacts available locally *in all containers* under 
`/opt/flink/usrlib`,
+* start a Flink Master container in the Single Job mode
+* and required number of TaskManager containers.
+
+To make the **job artifacts available** locally in the container, you can
+
+* **either mount a volume** (or multiple volumes) with the artifacts to 
`/opt/flink/usrlib` when you start
+the Flink image as a Single Job Master and the required number of TaskManagers:
+
+```sh
+docker run \
+--mount 
type=bind,src=/host/path/to/job/artifacts1,target=/opt/flink/usrlib/artifacts/1 
\
+--mount 
type=bind,src=/host/path/to/job/artifacts2,target=/opt/flink/usrlib/artifacts/2 
\

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

2020-05-14 Thread GitBox



azagrebin commented on a change in pull request #12131:
URL: https://github.com/apache/flink/pull/12131#discussion_r425023459



##
File path: docs/ops/deployment/docker.md
##
@@ -24,119 +24,476 @@ under the License.
 -->
 
 [Docker](https://www.docker.com) is a popular container runtime. 
-There are Docker images for Apache Flink available on Docker Hub which can be 
used to deploy a session cluster.
-The Flink repository also contains tooling to create container images to 
deploy a job cluster.
+There are Docker images for Apache Flink available [on Docker 
Hub](https://hub.docker.com/_/flink).
+You can use the docker images to deploy a Session or Job cluster in a 
containerised environment, e.g.
+[standalone Kubernetes](kubernetes.html) or [native 
Kubernetes](native_kubernetes.html).
 
 * This will be replaced by the TOC
 {:toc}
 
-## Flink session cluster
-
-A Flink session cluster can be used to run multiple jobs. 
-Each job needs to be submitted to the cluster after it has been deployed. 
-
-### Docker images
+## Docker Hub Flink images
 
 The [Flink Docker repository](https://hub.docker.com/_/flink/) is hosted on
 Docker Hub and serves images of Flink version 1.2.1 and later.
 
-Images for each supported combination of Hadoop and Scala are available, and 
tag aliases are provided for convenience.
+### Image tags
 
-Beginning with Flink 1.5, image tags that omit a Hadoop version (e.g.
-`-hadoop28`) correspond to Hadoop-free releases of Flink that do not include a
-bundled Hadoop distribution.
+Images for each supported combination of Flink and Scala versions are 
available, and
+[tag aliases](https://hub.docker.com/_/flink?tab=tags) are provided for 
convenience.
 
-For example, the following aliases can be used: *(`1.5.y` indicates the latest
-release of Flink 1.5)*
+For example, you can use the following aliases: *(`1.11.y` indicates the 
latest release of Flink 1.11)*
 
 * `flink:latest` → `flink:-scala_`
-* `flink:1.5` → `flink:1.5.y-scala_2.11`
-* `flink:1.5-hadoop27` → `flink:1.5.y-hadoop27-scala_2.11`
+* `flink:1.11` → `flink:1.11.y-scala_2.11`
+
+Note Prio to Flink 1.5 version, Hadoop 
dependencies were always bundled with Flink.
+You can see that certain tags include the version of Hadoop, e.g. (e.g. 
`-hadoop28`).
+Beginning with Flink 1.5, image tags that omit the Hadoop version correspond 
to Hadoop-free releases of Flink
+that do not include a bundled Hadoop distribution.
+
+## How to run Flink image
+
+The Flink image contains a regular Flink distribution with its default 
configuration and a standard entry point script.
+You can run its standard entry point in the following modes:
+* Flink Master for [a Session cluster](#start-a-session-cluster)
+* Flink Master for [a Single Job cluster](#start-a-single-job-cluster)
+* TaskManager for any cluster
+
+This allows to deploy a standalone cluster (Session or Single Job) in any 
containerised environment, for example:
+* manually in a local docker setup,
+* [in a Kubernetes cluster](kubernetes.html),
+* [with Docker Compose](#flink-with-docker-compose),
+* [with Docker swarm](#flink-with-docker-swarm).
+
+Note [The native 
Kubernetes](native_kubernetes.html) also runs the same image by default
+and deploys TaskManagers on demand so that you do not have to do it.
+
+The next chapters describe how to start a single Flink docker container for 
various purposes.
+
+### Start a Session Cluster
+
+A Flink Session cluster can be used to run multiple jobs. Each job needs to be 
submitted to the cluster after it has been deployed.
+To deploy a Flink Session cluster with docker, you need to start a Flink 
Master container:
+
+```sh
+docker run flink:{% if site.is_stable 
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif 
%} jobmanager
+```
+
+and required number of TaskManager containers:
+
+```sh
+docker run flink:{% if site.is_stable 
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif 
%} taskmanager
+```
+
+### Start a Single Job Cluster
+
+A Flink Job cluster is a dedicated cluster which runs a single job.
+The job artifacts should be already available locally in the container and, 
thus, there is no extra job submission needed.
+To deploy a cluster for a single job with docker, you need to
+* make job artifacts available locally *in all containers* under 
`/opt/flink/usrlib`,
+* start a Flink Master container in the Single Job mode
+* and required number of TaskManager containers.
+
+To make the **job artifacts available** locally in the container, you can
+
+* **either mount a volume** (or multiple volumes) with the artifacts to 
`/opt/flink/usrlib` when you start
+the Flink image as a Single Job Master and the required number of TaskManagers:
+
+```sh
+docker run \
+--mount 
type=bind,src=/host/path/to/job/artifacts1,target=/opt/flink/usrlib/artifacts/1 
\
+--mount 
type=bind,src=/host/path/to/job/artifacts2,target=/opt/flink/usrlib/artifacts/2 
\

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

2020-05-14 Thread GitBox



azagrebin commented on a change in pull request #12131:
URL: https://github.com/apache/flink/pull/12131#discussion_r425019269



##
File path: docs/ops/deployment/docker.md
##
@@ -24,119 +24,476 @@ under the License.
 -->
 
 [Docker](https://www.docker.com) is a popular container runtime. 
-There are Docker images for Apache Flink available on Docker Hub which can be 
used to deploy a session cluster.
-The Flink repository also contains tooling to create container images to 
deploy a job cluster.
+There are Docker images for Apache Flink available [on Docker 
Hub](https://hub.docker.com/_/flink).
+You can use the docker images to deploy a Session or Job cluster in a 
containerised environment, e.g.
+[standalone Kubernetes](kubernetes.html) or [native 
Kubernetes](native_kubernetes.html).
 
 * This will be replaced by the TOC
 {:toc}
 
-## Flink session cluster
-
-A Flink session cluster can be used to run multiple jobs. 
-Each job needs to be submitted to the cluster after it has been deployed. 
-
-### Docker images
+## Docker Hub Flink images
 
 The [Flink Docker repository](https://hub.docker.com/_/flink/) is hosted on
 Docker Hub and serves images of Flink version 1.2.1 and later.
 
-Images for each supported combination of Hadoop and Scala are available, and 
tag aliases are provided for convenience.
+### Image tags
 
-Beginning with Flink 1.5, image tags that omit a Hadoop version (e.g.
-`-hadoop28`) correspond to Hadoop-free releases of Flink that do not include a
-bundled Hadoop distribution.
+Images for each supported combination of Flink and Scala versions are 
available, and
+[tag aliases](https://hub.docker.com/_/flink?tab=tags) are provided for 
convenience.
 
-For example, the following aliases can be used: *(`1.5.y` indicates the latest
-release of Flink 1.5)*
+For example, you can use the following aliases: *(`1.11.y` indicates the 
latest release of Flink 1.11)*
 
 * `flink:latest` → `flink:-scala_`
-* `flink:1.5` → `flink:1.5.y-scala_2.11`
-* `flink:1.5-hadoop27` → `flink:1.5.y-hadoop27-scala_2.11`
+* `flink:1.11` → `flink:1.11.y-scala_2.11`
+
+Note Prio to Flink 1.5 version, Hadoop 
dependencies were always bundled with Flink.
+You can see that certain tags include the version of Hadoop, e.g. (e.g. 
`-hadoop28`).
+Beginning with Flink 1.5, image tags that omit the Hadoop version correspond 
to Hadoop-free releases of Flink
+that do not include a bundled Hadoop distribution.
+
+## How to run Flink image
+
+The Flink image contains a regular Flink distribution with its default 
configuration and a standard entry point script.
+You can run its standard entry point in the following modes:
+* Flink Master for [a Session cluster](#start-a-session-cluster)
+* Flink Master for [a Single Job cluster](#start-a-single-job-cluster)
+* TaskManager for any cluster
+
+This allows to deploy a standalone cluster (Session or Single Job) in any 
containerised environment, for example:
+* manually in a local docker setup,
+* [in a Kubernetes cluster](kubernetes.html),
+* [with Docker Compose](#flink-with-docker-compose),
+* [with Docker swarm](#flink-with-docker-swarm).
+
+Note [The native 
Kubernetes](native_kubernetes.html) also runs the same image by default
+and deploys TaskManagers on demand so that you do not have to do it.
+
+The next chapters describe how to start a single Flink docker container for 
various purposes.
+
+### Start a Session Cluster
+
+A Flink Session cluster can be used to run multiple jobs. Each job needs to be 
submitted to the cluster after it has been deployed.
+To deploy a Flink Session cluster with docker, you need to start a Flink 
Master container:
+
+```sh
+docker run flink:{% if site.is_stable 
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif 
%} jobmanager
+```
+
+and required number of TaskManager containers:
+
+```sh
+docker run flink:{% if site.is_stable 
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif 
%} taskmanager
+```
+
+### Start a Single Job Cluster
+
+A Flink Job cluster is a dedicated cluster which runs a single job.
+The job artifacts should be already available locally in the container and, 
thus, there is no extra job submission needed.
+To deploy a cluster for a single job with docker, you need to
+* make job artifacts available locally *in all containers* under 
`/opt/flink/usrlib`,
+* start a Flink Master container in the Single Job mode
+* and required number of TaskManager containers.
+
+To make the **job artifacts available** locally in the container, you can
+
+* **either mount a volume** (or multiple volumes) with the artifacts to 
`/opt/flink/usrlib` when you start
+the Flink image as a Single Job Master and the required number of TaskManagers:
+
+```sh
+docker run \
+--mount 
type=bind,src=/host/path/to/job/artifacts1,target=/opt/flink/usrlib/artifacts/1 
\
+--mount 
type=bind,src=/host/path/to/job/artifacts2,target=/opt/flink/usrlib/artifacts/2 
\

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

2020-05-14 Thread GitBox



azagrebin commented on a change in pull request #12131:
URL: https://github.com/apache/flink/pull/12131#discussion_r425016548



##
File path: docs/ops/deployment/docker.md
##
@@ -24,119 +24,476 @@ under the License.
 -->
 
 [Docker](https://www.docker.com) is a popular container runtime. 
-There are Docker images for Apache Flink available on Docker Hub which can be 
used to deploy a session cluster.
-The Flink repository also contains tooling to create container images to 
deploy a job cluster.
+There are Docker images for Apache Flink available [on Docker 
Hub](https://hub.docker.com/_/flink).
+You can use the docker images to deploy a Session or Job cluster in a 
containerised environment, e.g.
+[standalone Kubernetes](kubernetes.html) or [native 
Kubernetes](native_kubernetes.html).
 
 * This will be replaced by the TOC
 {:toc}
 
-## Flink session cluster
-
-A Flink session cluster can be used to run multiple jobs. 
-Each job needs to be submitted to the cluster after it has been deployed. 
-
-### Docker images
+## Docker Hub Flink images
 
 The [Flink Docker repository](https://hub.docker.com/_/flink/) is hosted on
 Docker Hub and serves images of Flink version 1.2.1 and later.
 
-Images for each supported combination of Hadoop and Scala are available, and 
tag aliases are provided for convenience.
+### Image tags
 
-Beginning with Flink 1.5, image tags that omit a Hadoop version (e.g.
-`-hadoop28`) correspond to Hadoop-free releases of Flink that do not include a
-bundled Hadoop distribution.
+Images for each supported combination of Flink and Scala versions are 
available, and
+[tag aliases](https://hub.docker.com/_/flink?tab=tags) are provided for 
convenience.
 
-For example, the following aliases can be used: *(`1.5.y` indicates the latest
-release of Flink 1.5)*
+For example, you can use the following aliases: *(`1.11.y` indicates the 
latest release of Flink 1.11)*
 
 * `flink:latest` → `flink:-scala_`
-* `flink:1.5` → `flink:1.5.y-scala_2.11`
-* `flink:1.5-hadoop27` → `flink:1.5.y-hadoop27-scala_2.11`
+* `flink:1.11` → `flink:1.11.y-scala_2.11`
+
+Note Prio to Flink 1.5 version, Hadoop 
dependencies were always bundled with Flink.
+You can see that certain tags include the version of Hadoop, e.g. (e.g. 
`-hadoop28`).
+Beginning with Flink 1.5, image tags that omit the Hadoop version correspond 
to Hadoop-free releases of Flink
+that do not include a bundled Hadoop distribution.
+
+## How to run Flink image
+
+The Flink image contains a regular Flink distribution with its default 
configuration and a standard entry point script.
+You can run its standard entry point in the following modes:
+* Flink Master for [a Session cluster](#start-a-session-cluster)

Review comment:
   do you think we should link the glossary everywhere then or just once in 
the beginning? because the glossary is a crosscutting for all docs as I 
understand





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

2020-05-14 Thread GitBox



azagrebin commented on a change in pull request #12131:
URL: https://github.com/apache/flink/pull/12131#discussion_r425013111



##
File path: docs/ops/deployment/native_kubernetes.md
##
@@ -76,6 +76,22 @@ Please follow our [configuration guide]({{ site.baseurl 
}}/ops/config.html) if y
 
 If you do not specify a particular name for your session by 
`kubernetes.cluster-id`, the Flink client will generate a UUID name. 
 
+### Custom Flink docker image
+
+If you want to use a custom docker image to deploy Flink containers, check 
[the Flink docker image documentation](docker.html),
+[its tags](#image-tags), [how to customize the Flink docker 
image](docker.html#customize-flink-image) and [enable 
plugins](docker.html#using-plugins).
+If you created a custom docker image you can provide it by setting the 
[`kubernetes.container.image`](../config.html#kubernetes-container-image) 
configuration option:
+
+{% highlight bash %}
+./bin/kubernetes-session.sh \
+  -Dkubernetes.cluster-id= \
+  -Dtaskmanager.memory.process.size=4096m \

Review comment:
   it is copied from above, I would keep it and be verbose to avoid 
confusion





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

[GitHub] [flink] azagrebin commented on a change in pull request #12131: [FLINK-17161][docs][docker] Document the official docker hub image and examples of how to use it

35 matches

Site Navigation

Mail list logo

Footer information