Hi All, I am following the below guide to setup Docker container run_time but face some non-trivial errors at least for my level. Would you please comment if you have some idea about the root-cause? http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/DockerContainers.html
After the error, I have provided the config files and Dockerfile as well as Docker image inspect command results (should have null for Entry Point and CMS?). I have three nodes, 1 RM and 2 NMs and default LCE works fine. ********************submit job script vars="YARN_CONTAINER_RUNTIME_TYPE=docker,YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=hadoop-ubuntu,YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE=false,YARN_CONTAINER_RUNTIME_DOCKER_CONTAINER_NETWORK=host" #vars="YARN_CONTAINER_RUNTIME_TYPE=default" hadoop jar /home/ubuntu/hadoop-3.1.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.0.jar pi -Dyarn.app.mapreduce.am.env=$vars -Dmapreduce.map.env=$vars -Dmapreduce.reduce.env=$vars 2 10 ******************** AM Log in one the nodes 2018-04-19 18:55:46,311 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1524178188987_0001_000001 (auth:SIMPLE) 2018-04-19 18:55:46,515 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_1524178188987_0001_01_000001 by user ubuntu 2018-04-19 18:55:46,617 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1524178188987_0001 2018-04-19 18:55:46,634 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1524178188987_0001 transitioned from NEW to INITING 2018-04-19 18:55:46,634 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=ubuntu IP=130.245.127.176 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1524178188987_0001 CONTAINERID=container_1524178188987_0001_01_000001 2018-04-19 18:55:46,635 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_1524178188987_0001_01_000001 to application application_1524178188987_0001 2018-04-19 18:55:46,649 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1524178188987_0001 transitioned from INITING to RUNNING 2018-04-19 18:55:46,655 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1524178188987_0001_01_000001 transitioned from NEW to LOCALIZING 2018-04-19 18:55:46,655 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1524178188987_0001 2018-04-19 18:55:46,698 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_1524178188987_0001_01_000001 2018-04-19 18:55:46,898 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /tmp/hadoop-ubuntu/nm-local-dir/nmPrivate/container_1524178188987_0001_01_000001.tokens 2018-04-19 18:55:50,371 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1524178188987_0001_01_000001 transitioned from LOCALIZING to SCHEDULED 2018-04-19 18:55:50,374 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler.ContainerScheduler: Starting container [container_1524178188987_0001_01_000001] 2018-04-19 18:55:50,479 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1524178188987_0001_01_000001 transitioned from SCHEDULED to RUNNING 2018-04-19 18:55:50,481 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_1524178188987_0001_01_000001 2018-04-19 18:55:51,842 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container container_1524178188987_0001_01_000001 succeeded 2018-04-19 18:55:51,844 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1524178188987_0001_01_000001 transitioned from RUNNING to EXITED_WITH_SUCCESS 2018-04-19 18:55:51,844 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_1524178188987_0001_01_000001 2018-04-19 18:55:51,957 INFO org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Removing Docker container : container_1524178188987_0001_01_000001 2018-04-19 18:55:56,963 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Could not get pid for container_1524178188987_0001_01_000001. Waited for 5000 ms. 2018-04-19 18:55:56,963 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Unable to obtain pid, but docker container request detected. Attempting to reap container container_1524178188987_0001_01_000001 2018-04-19 18:55:59,395 INFO org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Deleting absolute path : /tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1524178188987_0001/container_1524178188987_0001_01_000001 2018-04-19 18:55:59,395 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=ubuntu OPERATION=Container Finished - Succeeded TARGET=ContainerImpl RESULT=SUCCESS APPID=application_1524178188987_0001 CONTAINERID=container_1524178188987_0001_01_000001 2018-04-19 18:55:59,403 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1524178188987_0001_01_000001 transitioned from EXITED_WITH_SUCCESS to DONE 2018-04-19 18:55:59,404 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_1524178188987_0001_01_000001 from application application_1524178188987_0001 2018-04-19 18:55:59,404 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_1524178188987_0001_01_000001 2018-04-19 18:55:59,405 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1524178188987_0001 2018-04-19 18:56:00,412 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_1524178188987_0001_01_000001] 2018-04-19 18:56:13,455 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1524178188987_0001 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2018-04-19 18:56:13,457 INFO org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Deleting absolute path : /tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1524178188987_0001 2018-04-19 18:56:13,459 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1524178188987_0001 2018-04-19 18:56:13,470 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1524178188987_0001 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2018-04-19 18:56:13,470 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler: Scheduling Log Deletion for application: application_1524178188987_0001, with delay of 10800 seconds ********************** yarn-site.xml: according to the above link container-executor.cfg: yarn.nodemanager.linux-container-executor.group=ubuntu min.user.id=0 #feature.tc.enabled=1 #feature.docker.enabled=1 allowed.system.users=ubuntu # The configs below deal with settings for Docker [docker] module.enabled=true docker.privileged-containers.enabled=true docker.binary=/usr/bin/docker docker.allowed.capabilities=SYS_CHROOT,MKNOD,SETFCAP,SETPCAP,FSETID,CHOWN,AUDIT_WRITE,SETGID,NET_RAW,FOWNER,SETUID,DAC_OVERRIDE,KILL,NET_BIND_SERVICE #docker.allowed.devices=## comma seperated list of devices that can be mounted into a container docker.allowed.networks=bridge,host,none docker.allowed.ro-mounts=/sys/fs/cgroup,/tmp/hadoop-ubuntu/nm-local-dir docker.privileged-containers.registries=local #docker.host-pid-namespace.enabled=false docker.allowed.rw-mounts=/home/ubuntu/hadoop-3.1.0,/home/ubuntu/hadoop-3.1.0/logs Dockerfile: FROM ubuntu:16.04 #RUN rm /bin/sh && ln -s /bin/bash /bin/sh SHELL ["/bin/bash", "-c"] RUN apt-get update && \ apt-get upgrade -y && \ apt-get install -y software-properties-common && \ # apt-get install -y --no-install-recommends apt-utils && \ # apt-get install -y curl && \ add-apt-repository ppa:webupd8team/java -y && \ apt-get update && \ echo oracle-java7-installer shared/accepted-oracle-license-v1-1 select true | /usr/bin/debconf-set-selections && \ apt-get install -y oracle-java8-installer && \ # apt-get install -y ssh && \ # apt-get install -y rsync && \ apt-get install -y vim && \ apt-get clean ENV JAVA_HOME /usr/lib/jvm/java-8-oracle ENV PATH $PATH:$JAVA_HOME/bin # HADOOP ARG HADOOP_ARCHIVE= http://mirror.cc.columbia.edu/pub/software/apache/hadoop/common/hadoop-3.1.0/hadoop-3.1.0.tar.gz ENV HADOOP_HOME /usr/local/hadoop ENV HADOOP_COMMON_PATH /usr/local/hadoop ENV HADOOP_HDFS_HOME /usr/local/hadoop ENV HADOOP_MAPRED_HOME /usr/local/hadoop ENV HADOOP_YARN_HOME /usr/local/hadoop ENV HADOOP_CONF_DIR /usr/local/hadoop/etc/hadoop # download and extract hadoop, set JAVA_HOME in hadoop-env.sh, update path RUN wget $HADOOP_ARCHIVE && \ tar -xzf hadoop-3.1.0.tar.gz && \ mv hadoop-3.1.0 $HADOOP_HOME ADD rm-hadoop-config/* $HADOOP_HOME/etc/hadoop/ ENV PATH $PATH:$HADOOP_COMMON_PATH/bin WORKDIR $HADOOP_COMMON_PATH # Declare user RUN groupadd -g 1000 ubuntu && \ useradd -r -u 1000 -g 1000 ubuntu USER ubuntu ~/hadoop-common$ docker inspect local/hadoop-ubuntu [ { "Id": "sha256:d8335693084b5823675056b7d649b13d04a7e3c3b63688f83e9807405506b088", "RepoTags": [ "hadoop-ubuntu:latest", "local/hadoop-ubuntu:latest" ], "RepoDigests": [], "Parent": "sha256:f9b92fa15eadd74b9e3712ee379b56bc30a73492db2fd4e7b7f4f1d74e5671f2", "Comment": "", "Created": "2018-04-19T15:16:52.903547395Z", "Container": "4fcff6fa65639fe0e2a9379a335a401a8cde0c0e1eaaa2dd98d35ced402c29e3", "ContainerConfig": { "Hostname": "4fcff6fa6563", "Domainname": "", "User": "ubuntu", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": [ "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/jvm/java-8-oracle/bin:/usr/local/hadoop/bin", "JAVA_HOME=/usr/lib/jvm/java-8-oracle", "HADOOP_HOME=/usr/local/hadoop", "HADOOP_COMMON_PATH=/usr/local/hadoop", "HADOOP_HDFS_HOME=/usr/local/hadoop", "HADOOP_MAPRED_HOME=/usr/local/hadoop", "HADOOP_YARN_HOME=/usr/local/hadoop", "HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop" ], "Cmd": [ "/bin/bash", "-c", "#(nop) ", "USER ubuntu" ], "ArgsEscaped": true, "Image": "sha256:f9b92fa15eadd74b9e3712ee379b56bc30a73492db2fd4e7b7f4f1d74e5671f2", "Volumes": null, "WorkingDir": "/usr/local/hadoop", "Entrypoint": null, "OnBuild": null, "Labels": {}, "Shell": [ "/bin/bash", "-c" ] }, "DockerVersion": "18.03.0-ce", "Author": "", "Config": { "Hostname": "", "Domainname": "", "User": "ubuntu", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": [ "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/jvm/java-8-oracle/bin:/usr/local/hadoop/bin", "JAVA_HOME=/usr/lib/jvm/java-8-oracle", "HADOOP_HOME=/usr/local/hadoop", "HADOOP_COMMON_PATH=/usr/local/hadoop", "HADOOP_HDFS_HOME=/usr/local/hadoop", "HADOOP_MAPRED_HOME=/usr/local/hadoop", "HADOOP_YARN_HOME=/usr/local/hadoop", "HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop" ], "Cmd": [ "/bin/bash" ], "ArgsEscaped": true, "Image": "sha256:f9b92fa15eadd74b9e3712ee379b56bc30a73492db2fd4e7b7f4f1d74e5671f2", "Volumes": null, "WorkingDir": "/usr/local/hadoop", "Entrypoint": null, "OnBuild": null, "Labels": null, "Shell": [ "/bin/bash", "-c" ] }, "Architecture": "amd64", "Os": "linux", "Size": 2058935914, "VirtualSize": 2058935914, "GraphDriver": { "Data": null, "Name": "aufs" }, "RootFS": { "Type": "layers", "Layers": [ "sha256:fccbfa2912f0cd6b9d13f91f288f112a2b825f3f758a4443aacb45bfc108cc74", "sha256:e1a9a6284d0d24d8194ac84b372619e75cd35a46866b74925b7274c7056561e4", "sha256:ac7299292f8b2f710d3b911c6a4e02ae8f06792e39822e097f9c4e9c2672b32d", "sha256:a5e66470b2812e91798db36eb103c1f1e135bbe167e4b2ad5ba425b8db98ee8d", "sha256:a8de0e025d94b33db3542e1e8ce58829144b30c6cd1fff057eec55b1491933c3", "sha256:7e9a788452589001d42e7995dc0583bcca1e6f7780a301066ee0d6668aaf9c91", "sha256:65fac5b99df0506e1d204d9024264105ab2fe142ea970f0eba630669dc606055", "sha256:cd0da20f97700ab8e2ec37c464fcb8864cba86672aa96e7fc40e2572228895e2", "sha256:21094366bc298af4a7e83680cf9d732ce69cb92b11a3b20005b12c15bac3e486" ] }, "Metadata": { "LastTagTime": "2018-04-19T17:20:41.054733026-04:00" } } ] Best, Ahmad