[GitHub] flink issue #6232: [FLINK-9695] [mesos] Add option for Mesos executor to for...

[lishim]

Github user lishim commented on the issue:

https://github.com/apache/flink/pull/6232
  
Hi @tillrohrmann.  Thank you for your comment.  I have updated the code to 
conform to other configuration options.


---

[GitHub] flink pull request #6232: [FLINK-9695] [mesos] Add option for Mesos executor...

[lishim]

GitHub user lishim opened a pull request:

https://github.com/apache/flink/pull/6232

[FLINK-9695] [mesos] Add option for Mesos executor to forcefully pull 
Docker images

## What is the purpose of the change

* This pull request introduces an option for Mesos executor to forcefully 
pull Docker images

## Brief change log

* A new parameter 
```mesos.resourcemanager.tasks.container.docker.forcePullImage``` was introduced

## Verifying this change

This change added tests and can be verified as follows:

  - Added unit tests that verify correctness of behavior of new parameter
  - Manually verified behavior by running Flink in Mesos with Docker 
containerizer and 2 task managers.  Confirmed that when a task started on a 
node where the task image already resided, an existing cached image was used 
when ```mesos.resourcemanager.tasks.container.docker.forcePullImage``` was left 
out or set to ```false```.  Confirmed that fresh image was pulled and used over 
a cached image when 
```mesos.resourcemanager.tasks.container.docker.forcePullImage``` was set to 
```true```.

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): no
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: no
  - The serializers: no
  - The runtime per-record code paths (performance sensitive): no
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: yes
  - The S3 file system connector: no

## Documentation

  - Does this pull request introduce a new feature? yes
  - If yes, how is the feature documented? docs


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lishim/flink feature/forcePullImage

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/6232.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #6232


commit 2e6274b1f4f3692ae4bc57780ba3c4888e7123d6
Author: Leonid Ishimnikov 
Date:   2018-06-30T05:34:23Z

[FLINK-9695] Added 
mesos.resourcemanager.tasks.container.docker.forcePullImage option




---

[GitHub] flink pull request #6207: [FLINK-9660] [mesos] Allow passing custom artifact...

[lishim]

GitHub user lishim opened a pull request:

https://github.com/apache/flink/pull/6207

[FLINK-9660] [mesos] Allow passing custom artifacts to Mesos workers

## What is the purpose of the change

* This pull request introduces a feature that allows passing custom 
artifact URIs to Mesos workers

## Brief change log

* A new parameter ```mesos.resourcemanager.tasks.uris``` was introduced

## Verifying this change

This change added tests and can be verified as follows:

  - Added unit tests that verify correctness of behavior of new parameter
  - Manually verified behavior by running Flink in Mesos with Docker 
containerizer and 2 task managers.  A private Docker registry was used for 
taskmanager Docker image.  Confirmed that Mesos was unable to pull the image 
with default settings.  After that, added 
```mesos.resourcemanager.tasks.uris``` parameter with path to local archive 
containing Docker credentials configuration.  Confirmed the file was extracted 
to local sandbox of taskmanager Mesos task, and Docker image was successfully 
pulled.

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): no
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: no
  - The serializers: no
  - The runtime per-record code paths (performance sensitive): no
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: yes
  - The S3 file system connector: no

## Documentation

  - Does this pull request introduce a new feature? yes
  - If yes, how is the feature documented? docs


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lishim/flink master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/6207.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #6207


commit 2547f1b68e22da167bfa0264a72223fea3ed0d43
Author: Leonid Ishimnikov 
Date:   2018-06-21T19:13:09Z

[FLINK-9660] Allow passing custom artifacts to Mesos workers




---

[GitHub] flink issue #5346: [FLINK-8490] [mesos] Allow custom docker parameters for d...

[lishim]

Github user lishim commented on the issue:

https://github.com/apache/flink/pull/5346
  
Thanks @joerg84 for your review and comments.  I have updated the PR to 
address your concerns.


---

[GitHub] flink pull request #5346: [FLINK-8490] [mesos] Allow custom docker parameter...

[lishim]

Github user lishim commented on a diff in the pull request:

https://github.com/apache/flink/pull/5346#discussion_r163596583
  
--- Diff: 
flink-mesos/src/test/java/org/apache/flink/mesos/runtime/clusterframework/MesosTaskManagerParametersTest.java
 ---
@@ -84,6 +84,35 @@ public void testContainerVolumes() throws Exception {
assertEquals(Protos.Volume.Mode.RO, 
params.containerVolumes().get(0).getMode());
}
 
+   @Test
+   public void testContainerDockerParameter() throws Exception {
+   Configuration config = new Configuration();
+   
config.setString(MesosTaskManagerParameters.MESOS_RM_CONTAINER_DOCKER_PARAMETERS,
 "testKey=testValue");
+
+   MesosTaskManagerParameters params = 
MesosTaskManagerParameters.create(config);
+   assertEquals(params.dockerParameters().get(0).getKey(), 
"testKey");
--- End diff --

Added a size check.


---

[GitHub] flink pull request #5346: [FLINK-8490] [mesos] Allow custom docker parameter...

[lishim]

Github user lishim commented on a diff in the pull request:

https://github.com/apache/flink/pull/5346#discussion_r163596501
  
--- Diff: 
flink-mesos/src/main/java/org/apache/flink/mesos/runtime/clusterframework/MesosTaskManagerParameters.java
 ---
@@ -365,6 +386,31 @@ public String call(String s) {
}
}
 
+   public static List 
buildDockerParameters(Option dockerParameters) {
+   if (dockerParameters.isEmpty()) {
+   return Collections.emptyList();
+   } else {
+   String[] dockerParameterSpecifications = 
dockerParameters.get().split(",");
+
+   List parameters = new 
ArrayList<>(dockerParameterSpecifications.length);
+
+   for (String dockerParameterSpecification : 
dockerParameterSpecifications) {
+   if 
(!dockerParameterSpecification.trim().isEmpty()) {
+   String[] match = 
dockerParameterSpecification.split("=", 2);
--- End diff --

Some Docker parameters are key=value pairs themselves, so key2=val1=val3 is 
actually valid specification.  I added a comment to clarify this, and included 
a test case.


---

[GitHub] flink pull request #5346: [FLINK-8490] [mesos] Allow custom docker parameter...

[lishim]

Github user lishim commented on a diff in the pull request:

https://github.com/apache/flink/pull/5346#discussion_r163595840
  
--- Diff: docs/ops/config.md ---
@@ -508,6 +508,8 @@ May be set to -1 to disable this feature.
 
 - `mesos.resourcemanager.tasks.container.volumes`: A comma separated list 
of `[host_path:]`container_path`[:RO|RW]`. This allows for mounting additional 
volumes into your container. (**NO DEFAULT**)
 
+- `mesos.resourcemanager.tasks.container.docker.parameters`: Custom 
parameters to be passed into docker run command when using the docker 
containerizer.  Comma separated list of key=value pairs. (**NO DEFAULT**)
--- End diff --

Fixed


---

[GitHub] flink pull request #5346: [FLINK-8490] [mesos] Allow custom docker parameter...

[lishim]

Github user lishim commented on a diff in the pull request:

https://github.com/apache/flink/pull/5346#discussion_r163595893
  
--- Diff: docs/ops/deployment/mesos.md ---
@@ -264,6 +264,8 @@ May be set to -1 to disable this feature.
 
 `mesos.resourcemanager.tasks.container.volumes`: A comma separated list of 
`[host_path:]`container_path`[:RO|RW]`. This allows for mounting additional 
volumes into your container. (**NO DEFAULT**)
 
+`mesos.resourcemanager.tasks.container.docker.parameters`: Custom 
parameters to be passed into docker run command when using the docker 
containerizer.  Comma separated list of key=value pairs. (**NO DEFAULT**)
--- End diff --

Fixed


---

[GitHub] flink pull request #5346: [FLINK-8490] [mesos] Allow custom docker parameter...

[lishim]

GitHub user lishim opened a pull request:

https://github.com/apache/flink/pull/5346

[FLINK-8490] [mesos] Allow custom docker parameters for docker tasks on 
Mesos

## What is the purpose of the change
- This pull request introduces a feature that allows passing custom 
parameters to docker when using docker containerizer.

## Brief change log
- A new parameter mesos.resourcemanager.tasks.container.docker.parameters 
is introduced

## Verifying this change
This change added tests and can be verified as follows:
- Added unit tests that verify correct behavior of the new parameter
- Manually verified the change by running it in an Mesos cluster with 
Docker containerizer and 2 task managers, setting 
mesos.resourcemanager.tasks.container.docker.parameters=privileged=true and 
verifying that the task manager container is able to run a privileged command, 
then removing the parameter and verifying that the privileged command can no 
longer run.

## Does this pull request potentially affect one of the following parts:
  - Dependencies (does it add or upgrade a dependency): no
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: no
  - The serializers: no
  - The runtime per-record code paths (performance sensitive): no
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: yes
  - The S3 file system connector: no

## Documentation

  - Does this pull request introduce a new feature? yes
  - If yes, how is the feature documented? docs


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lishim/flink master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5346.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5346


commit bf392edcd446dee375d2e80c7597c13f1479696b
Author: Leonid Ishimnikov <lishim@...>
Date:   2018-01-23T18:55:17Z

[FLINK-8490] Allow custom docker parameters for docker tasks on Mesos




---