GitHub user Ethanlm opened a pull request:
https://github.com/apache/storm/pull/2901
[STORM-3271] Docker support: launch storm workers in docker containers
Spent a lot of effort on this.
This patch let storm supervisor launches the worker in a docker container
so that user code is isolated from each other and from the host machine. This
is a security enhancement.
Detailed explanation of this are available at docs/Docker-support.md in
this pr.
I tested it manually (only on `RHEL7`, didn't test it on other os systems)
and tested with secure cluster setup. Also tested with our own integration test
suites with docker support turned off and proved that this pr won't break storm.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Ethanlm/storm STORM-3271
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/2901.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2901
----
commit ba8b7cf80347fe077e097206023adb3326c8b4ad
Author: Ethan Li <ethanli@...>
Date: 2018-10-12T23:24:15Z
refactor supervisor source code
commit 2387da9ae554ed81573e35545992754bc1454c9d
Author: Ethan Li <ethanli@...>
Date: 2018-10-12T23:24:23Z
add docker support
commit b130bcb649f6adcb1fef588dc7033361999a9d0f
Author: Ethan Li <ethanli@...>
Date: 2018-10-15T15:27:39Z
add cgrouproot and nscd bind mounts
add user's groups and simply getGroupIdInfo and getUserIdInfo functions
commit be2efc833144004c126ff150f9676adc15575549
Author: Ethan Li <ethanli@...>
Date: 2018-10-16T13:48:16Z
address some review comments
commit 80f3296688fce2bec9692a2699ae396a32c8e02e
Author: Ethan Li <ethanli@...>
Date: 2018-10-16T22:04:15Z
fix worker-launcher for secure storm cluster
commit d4e2127ef9664c0398f2e85c26d0dde5734d3444
Author: Ethan Li <ethanli@...>
Date: 2018-10-17T14:35:19Z
launch docker container in attached mode
commit 8e4c3c7a3cfe2a38dc847ea2ebda2ce74f9a45ff
Author: Ethan Li <ethanli@...>
Date: 2018-10-17T16:05:18Z
check prerequisites in prepare method; bind mount workerArtifact more
precisely
commit e0131136fd864aa5bc4a3b8e40b538d04a1c36ae
Author: Ethan Li <ethanli@...>
Date: 2018-10-19T19:57:24Z
fix user could be null (e.g when supervisor restarts)
commit 5dfcd9ff8869f6a4334ccecdefb0f5c1f40ea9fb
Author: Ethan Li <ethanli@...>
Date: 2018-10-22T20:24:20Z
fix typo; make network type non-configurable
commit fa5e2999312258b4da0b1936591804780bd8ec8e
Author: Ethan Li <ethanli@...>
Date: 2018-10-23T19:55:45Z
make worker-launcher more secure
commit 0f08bcde454f7a2e613eb673ba697236d8122999
Author: Ethan Li <ethanli@...>
Date: 2018-10-24T16:28:24Z
make nsenter more secure
commit 721c6b85e38b87761f0a52c19cd9b684bf70cbba
Author: Ethan Li <ethanli@...>
Date: 2018-10-24T20:10:56Z
add more comments; add list of readonly bindmounts
commit be88fb7c174ab15e22ab99590530919499fe9d38
Author: Ethan Li <ethanli@...>
Date: 2018-10-24T20:53:10Z
making nsenter safer
commit 1d84bf7d86cfa7e3c603ef4f99e0e6c4e2ec9ede
Author: Ethan Li <ethanli@...>
Date: 2018-10-25T14:23:18Z
add a whitelist for docker images
commit 9c8ae38bdbc31d897eca9f1779d96d5b995fcec6
Author: Ethan Li <ethanli@...>
Date: 2018-10-25T21:20:39Z
check every image in the whitelist; prevent memory leark from
worker-launcher; fix format and trivial issues.
commit 52350f4363821ac14a5b7b2114bc1bddb204299a
Author: Ethan Li <ethanli@...>
Date: 2018-10-26T20:18:25Z
launch docker container in background; use docker-wait to get the exitCode
if the contaner exits
commit 6c80da77481cccabd047452b055ab3cd68a67c40
Author: Ethan Li <ethanli@...>
Date: 2018-10-30T20:46:26Z
add a cgroup sub path template to support various cgroup setup
commit 8b9c18f314c1f8bf12140631f7de052bfc35cff5
Author: Ethan Li <ethanli@...>
Date: 2018-10-31T20:19:13Z
add unit tests for docker commands
commit 05b31e875536f01b3fb61a378ce9de3c01d4a5d3
Author: Ethan Li <ethanli@...>
Date: 2018-11-01T13:51:00Z
use nsenter to do profiling; mount shared_by_topology/tmp to /tmp
commit a1ffe50265f446b68dac0350c8cb4c49bf05992b
Author: Ethan Li <ethanli@...>
Date: 2018-11-01T15:02:12Z
fix loging for storm.resource.isolation.plugin
commit c4647183092541dc5957441500f62719510689f5
Author: Ethan Li <ethanli@...>
Date: 2018-11-01T17:38:54Z
fix issues when DockerManager is not used
commit 190f06938a38f4225d94f86c2f1afbba068d1b22
Author: Ethan Li <ethanli@...>
Date: 2018-11-05T16:54:32Z
add documentation
----
---
