It’s command: /opt/Apache/beam/boot in json. U might be able to find some examples online. I’ll reply you when I get home to paste the actual command.
Sent from my iPhone > On 18 Sep 2019, at 9:35 PM, Yu Watanabe <[email protected]> wrote: > > Benjamin. > > Thank you for the reply. > Per your suggest, I read the design sheet and it states that harness > container is a mandatory settings for all TaskManger. > > https://s.apache.org/portable-flink-runner-overview > > > The Flink cluster itself is deployed as normal. For example, it might be > > deployed on a standalone Flink cluster, on Kubernetes, or on YARN. > > The only requirement is that TaskManager nodes must have access to Docker. > > I first tried installing docker on all data nodes on all data nodes in emr > cluster, however, I still get errors when flink deletes running container. > > ========================================================================================= > Caused by: java.lang.Exception: The user defined 'open()' method caused an > exception: java.io.IOException: Received exit code 1 for command 'docker > inspect -f {{.State.Running}} > 3e7e0d2d9d8362995d4b566ce1611834d56c5ca550ae89ae698a279271e4c33b'. stderr: > Error: No such object: > 3e7e0d2d9d8362995d4b566ce1611834d56c5ca550ae89ae698a279271e4c33b > at > org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:498) > at > org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:368) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711) > ... 1 more > Caused by: > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: > java.io.IOException: Received exit code 1 for command 'docker inspect -f > {{.State.Running}} > 3e7e0d2d9d8362995d4b566ce1611834d56c5ca550ae89ae698a279271e4c33b'. stderr: > Error: No such object: > 3e7e0d2d9d8362995d4b566ce1611834d56c5ca550ae89ae698a279271e4c33b > at > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4966) > at > org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$SimpleStageBundleFactory.<init>(DefaultJobBundleFactory.java:211) > at > org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$SimpleStageBundleFactory.<init>(DefaultJobBundleFactory.java:202) > at > org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory.forStage(DefaultJobBundleFactory.java:185) > at > org.apache.beam.runners.flink.translation.functions.FlinkDefaultExecutableStageContext.getStageBundleFactory(FlinkDefaultExecutableStageContext.java:49) > at > org.apache.beam.runners.flink.translation.functions.ReferenceCountingFlinkExecutableStageContextFactory$WrappedContext.getStageBundleFactory(ReferenceCountingFlinkExecutableStageContextFactory.java:203) > at > org.apache.beam.runners.flink.translation.functions.FlinkExecutableStageFunction.open(FlinkExecutableStageFunction.java:129) > at > org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36) > at > org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:494) > ... 3 more > Caused by: java.io.IOException: Received exit code 1 for command 'docker > inspect -f {{.State.Running}} > 3e7e0d2d9d8362995d4b566ce1611834d56c5ca550ae89ae698a279271e4c33b'. stderr: > Error: No such object: > 3e7e0d2d9d8362995d4b566ce1611834d56c5ca550ae89ae698a279271e4c33b > at > org.apache.beam.runners.fnexecution.environment.DockerCommand.runShortCommand(DockerCommand.java:179) > at > org.apache.beam.runners.fnexecution.environment.DockerCommand.isContainerRunning(DockerCommand.java:112) > at > org.apache.beam.runners.fnexecution.environment.DockerEnvironmentFactory.createEnvironment(DockerEnvironmentFactory.java:160) > at > org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$1.load(DefaultJobBundleFactory.java:178) > at > org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$1.load(DefaultJobBundleFactory.java:162) > at > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3528) > at > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2277) > at > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154) > at > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044) > at > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.get(LocalCache.java:3952) > at > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974) > at > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958) > at > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4964) > ... 11 more > Suppressed: java.io.IOException: Received exit code 1 for command > 'docker kill > 3e7e0d2d9d8362995d4b566ce1611834d56c5ca550ae89ae698a279271e4c33b'. stderr: > Error response from daemon: Cannot kill container: > 3e7e0d2d9d8362995d4b566ce1611834d56c5ca550ae89ae698a279271e4c33b: No such > container: 3e7e0d2d9d8362995d4b566ce1611834d56c5ca550ae89ae698a279271e4c33b > at > org.apache.beam.runners.fnexecution.environment.DockerCommand.runShortCommand(DockerCommand.java:179) > at > org.apache.beam.runners.fnexecution.environment.DockerCommand.killContainer(DockerCommand.java:134) > at > org.apache.beam.runners.fnexecution.environment.DockerEnvironmentFactory.createEnvironment(DockerEnvironmentFactory.java:174) > ... 21 more > > ========================================================================================= > > > Next , I'd like to try using environment type 'PROCESS' but it seems you need > external command for 'environment_config' > > https://github.com/apache/beam/blob/7931ec055e2da7214c82e368ef7d7fd679faaef1/sdks/python/apache_beam/runners/portability/portable_runner.py#L138 > > config = json.loads(portable_options.environment_config) > > May I ask what command I need to set as shell script for each task managers ? > > Best Regards, > Yu Watanabe > >> On Wed, Sep 18, 2019 at 9:39 PM Benjamin Tan <[email protected]> >> wrote: >> Seems like docker is not installed. Maybe run with PROCESS as the >> environment type ? Or install docker. >> >> Sent from my iPhone >> >>> On 18 Sep 2019, at 6:40 PM, Yu Watanabe <[email protected]> wrote: >>> >>> Hello. >>> >>> I am trying to run FlinkRunner (2.15.0) on AWS EC2 instance and submit job >>> to AWS EMR (5.26.0). >>> >>> However, I get below error when I run the pipeline and fail. >>> >>> ========================================================- >>> Caused by: java.lang.Exception: The user defined 'open()' method caused an >>> exception: java.io.IOException: Cannot run program "docker": error=2, No >>> such file or directory >>> at >>> org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:498) >>> at >>> org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:368) >>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711) >>> ... 1 more >>> Caused by: >>> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: >>> java.io.IOException: Cannot run program "docker": error=2, No such file or >>> directory >>> at >>> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4966) >>> at >>> org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$SimpleStageBundleFactory.<init>(DefaultJobBundleFactory.java:211) >>> at >>> org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$SimpleStageBundleFactory.<init>(DefaultJobBundleFactory.java:202) >>> at >>> org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory.forStage(DefaultJobBundleFactory.java:185) >>> at >>> org.apache.beam.runners.flink.translation.functions.FlinkDefaultExecutableStageContext.getStageBundleFactory(FlinkDefaultExecutableStageContext.java:49) >>> at >>> org.apache.beam.runners.flink.translation.functions.ReferenceCountingFlinkExecutableStageContextFactory$WrappedContext.getStageBundleFactory(ReferenceCountingFlinkExecutableStageContextFactory.java:203) >>> at >>> org.apache.beam.runners.flink.translation.functions.FlinkExecutableStageFunction.open(FlinkExecutableStageFunction.java:129) >>> at >>> org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36) >>> at >>> org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:494) >>> ... 3 more >>> Caused by: java.io.IOException: Cannot run program "docker": error=2, No >>> such file or directory >>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) >>> at >>> org.apache.beam.runners.fnexecution.environment.DockerCommand.runShortCommand(DockerCommand.java:141) >>> at >>> org.apache.beam.runners.fnexecution.environment.DockerCommand.runImage(DockerCommand.java:92) >>> at >>> org.apache.beam.runners.fnexecution.environment.DockerEnvironmentFactory.createEnvironment(DockerEnvironmentFactory.java:152) >>> at >>> org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$1.load(DefaultJobBundleFactory.java:178) >>> at >>> org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$1.load(DefaultJobBundleFactory.java:162) >>> at >>> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3528) >>> at >>> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2277) >>> at >>> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154) >>> at >>> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044) >>> at >>> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.get(LocalCache.java:3952) >>> at >>> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974) >>> at >>> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958) >>> at >>> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4964) >>> ... 11 more >>> Caused by: java.io.IOException: error=2, No such file or directory >>> at java.lang.UNIXProcess.forkAndExec(Native Method) >>> at java.lang.UNIXProcess.<init>(UNIXProcess.java:247) >>> at java.lang.ProcessImpl.start(ProcessImpl.java:134) >>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) >>> ... 24 more >>> ========================================================- >>> >>> Pipeline options are below. >>> ========================================================- >>> options = PipelineOptions([ >>> "--runner=FlinkRunner", >>> "--flink_version=1.8", >>> >>> "--flink_master_url=ip-172-31-1-84.ap-northeast-1.compute.internal:43581", >>> >>> "--environment_config=asia.gcr.io/PROJECTNAME/beam/python3", >>> "--experiments=beam_fn_api" >>> ]) >>> >>> >>> p = beam.Pipeline(options=options) >>> ========================================================- >>> >>> I am able to run docker info ec2-user on the server where script is >>> running.. >>> >>> ========================================================- >>> (python) [ec2-user@ip-172-31-2-121 ~]$ docker info >>> Containers: 0 >>> Running: 0 >>> Paused: 0 >>> Stopped: 0 >>> ... >>> ========================================================- >>> >>> I used "debian-stretch" . >>> >>> ========================================================- >>> debian-stretch-hvm-x86_64-gp2-2019-09-08-17994-572488bb-fc09-4638-8628-e1e1d26436f4-ami-0ed2d2283aa1466df.4 >>> (ami-06f16171199d98c63) >>> ========================================================- >>> >>> This seems to not happen when flink runs locally. >>> >>> ========================================================- >>> admin@ip-172-31-9-89:/opt/flink$ sudo ss -atunp | grep 8081 >>> tcp LISTEN 0 128 :::8081 :::* >>> users:(("java",pid=18420,fd=82)) >>> admin@ip-172-31-9-89:/opt/flink$ sudo ps -ef | grep java | head -1 >>> admin 17698 1 0 08:59 ? 00:00:12 java -jar >>> /home/admin/.apache_beam/cache/beam-runners-flink-1.8-job-server-2.15.0.jar >>> --flink-master-url ip-172-31-1-84.ap-northeast-1.compute.internal:43581 >>> --artifacts-dir /tmp/artifactskj47j8yn --job-port 48205 --artifact-port 0 >>> --expansion-port 0 >>> admin@ip-172-31-9-89:/opt/flink$ >>> ========================================================- >>> >>> Would there be any other setting I need to look for when running on EC2 >>> instance ? >>> >>> Thanks, >>> Yu Watanabe >>> >>> -- >>> Yu Watanabe >>> Weekend Freelancer who loves to challenge building data platform >>> [email protected] >>> > > > -- > Yu Watanabe > Weekend Freelancer who loves to challenge building data platform > [email protected] >
