Hi Shane, Sorry I got that I missed your suggestion in my last test and later updated to local/hadoop-ubuntu:latest and now I am working on the following error. I confirmed that mapred-site.xml within the contianer image has the information this error asking for.
Do you think container is being launched but the aplication master/mp and reduce tasks have faced obstacles to run successfully wihtin the container? Many thanks, Ahmad *************************************RM node, submission************************* $ ./run_hadoop_job_docker_image_v1.sh rm: `/user/ubuntu/teraOutput': No such file or directory Number of Maps = 2 Samples per Map = 10 Wrote input for Map #0 Wrote input for Map #1 Starting Job 2018-04-19 21:38:38,757 INFO client.RMProxy: Connecting to ResourceManager at bay1-vm1/130.245.127.176:8032 2018-04-19 21:38:39,412 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/ubuntu/.staging/job_1524187975341_0003 2018-04-19 21:38:39,999 INFO input.FileInputFormat: Total input files to process : 2 2018-04-19 21:38:40,141 INFO mapreduce.JobSubmitter: number of splits:2 2018-04-19 21:38:40,205 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled 2018-04-19 21:38:40,411 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1524187975341_0003 2018-04-19 21:38:40,413 INFO mapreduce.JobSubmitter: Executing with tokens: [] 2018-04-19 21:38:40,665 INFO conf.Configuration: resource-types.xml not found 2018-04-19 21:38:40,666 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'. 2018-04-19 21:38:40,766 INFO impl.YarnClientImpl: Submitted application application_1524187975341_0003 2018-04-19 21:38:40,823 INFO mapreduce.Job: The url to track the job: http://bay1-vm1:8088/proxy/application_1524187975341_0003/ 2018-04-19 21:38:40,824 INFO mapreduce.Job: Running job: job_1524187975341_0003 2018-04-19 21:38:56,990 INFO mapreduce.Job: Job job_1524187975341_0003 running in uber mode : false 2018-04-19 21:38:56,993 INFO mapreduce.Job: map 0% reduce 0% 2018-04-19 21:38:57,013 INFO mapreduce.Job: Job job_1524187975341_0003 failed with state FAILED due to: Application application_1524187975341_0003 failed 2 times due to AM Container for appattempt_1524187975341_0003_000002 exited with exitCode: 1 Failing this attempt.Diagnostics: [2018-04-19 21:38:54.466]Exception from container-launch. Container id: container_1524187975341_0003_02_000001 Exit code: 1 Shell output: main : command provided 4 main : run as user is ubuntu main : requested yarn user is ubuntu Creating script paths... Creating local dirs... Getting exit code file... Changing effective user to root... Launching docker container... Docker run command: /usr/bin/docker --config='/tmp/hadoop-ubuntu/nm-local-dir/nmPrivate/application_1524187975341_0003/container_1524187975341_0003_02_000001' run --name='container_1524187975341_0003_02_000001' --user='1000:1000' -d --workdir='/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1524187975341_0003/container_1524187975341_0003_02_000001' --net='host' -v '/tmp/hadoop-ubuntu/nm-local-dir/filecache:/tmp/hadoop-ubuntu/nm-local-dir/filecache:ro' -v '/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/filecache:/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/filecache:ro' -v '/home/ubuntu/hadoop-3.1.0/logs/userlogs/application_1524187975341_0003/container_1524187975341_0003_02_000001:/home/ubuntu/hadoop-3.1.0/logs/userlogs/application_1524187975341_0003/container_1524187975341_0003_02_000001' -v '/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1524187975341_0003:/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1524187975341_0003' --cap-drop='ALL' --cap-add='SYS_CHROOT' --cap-add='MKNOD' --cap-add='SETFCAP' --cap-add='SETPCAP' --cap-add='FSETID' --cap-add='CHOWN' --cap-add='AUDIT_WRITE' --cap-add='SETGID' --cap-add='NET_RAW' --cap-add='FOWNER' --cap-add='SETUID' --cap-add='DAC_OVERRIDE' --cap-add='KILL' --cap-add='NET_BIND_SERVICE' --group-add '1000' --group-add '4' --group-add '24' --group-add '27' --group-add '30' --group-add '46' --group-add '110' --group-add '115' --group-add '116' --group-add '117' 'local/hadoop-ubuntu:latest' 'bash' '/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1524187975341_0003/container_1524187975341_0003_02_000001/launch_container.sh' Inspecting docker container... Docker inspect command: /usr/bin/docker inspect --format {{.State.Pid}} container_1524187975341_0003_02_000001 pid from docker inspect: 2519 Writing to cgroup task files... Writing pid file... Writing to tmp file /tmp/hadoop-ubuntu/nm-local-dir/nmPrivate/application_1524187975341_0003/container_1524187975341_0003_02_000001/container_1524187975341_0003_02_000001.pid.tmp Waiting for docker container to finish. Obtaining the exit code... Docker inspect command: /usr/bin/docker inspect --format {{.State.ExitCode}} container_1524187975341_0003_02_000001 Exit code from docker inspect: 1 Wrote the exit code 1 to /tmp/hadoop-ubuntu/nm-local-dir/nmPrivate/application_1524187975341_0003/container_1524187975341_0003_02_000001/container_1524187975341_0003_02_000001.pid.exitcode [2018-04-19 21:38:54.473]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster Please check whether your etc/hadoop/mapred-site.xml contains the below configuration: <property> <name>yarn.app.mapreduce.am.env</name> <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value> </property> <property> <name>mapreduce.map.env</name> <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value> </property> <property> <name>mapreduce.reduce.env</name> <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value> </property> [2018-04-19 21:38:54.475]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster Please check whether your etc/hadoop/mapred-site.xml contains the below configuration: <property> <name>yarn.app.mapreduce.am.env</name> <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value> </property> <property> <name>mapreduce.map.env</name> <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value> </property> <property> <name>mapreduce.reduce.env</name> <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value> </property> For more detailed output, check the application tracking page: http://bay1-vm1:8088/cluster/app/application_1524187975341_0003 Then click on links to logs of each attempt. . Failing the application. 2018-04-19 21:38:57,042 INFO mapreduce.Job: Counters: 0 Job job_1524187975341_0003 failed! runtime in seconds: 25 ************************************************** ***************************1 of the NM logs******************** 2018-04-19 21:38:49,056 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1524187975341_0003_000002 (auth:SIMPLE) 2018-04-19 21:38:49,068 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_1524187975341_0003_02_000001 by user ubuntu 2018-04-19 21:38:49,073 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1524187975341_0003 2018-04-19 21:38:49,074 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=ubuntu IP=130.245.127.176 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1524187975341_0003 CONTAINERID=container_1524187975341_0003_02_000001 2018-04-19 21:38:49,074 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1524187975341_0003 transitioned from NEW to INITING 2018-04-19 21:38:49,075 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_1524187975341_0003_02_000001 to application application_1524187975341_0003 2018-04-19 21:38:49,076 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1524187975341_0003 transitioned from INITING to RUNNING 2018-04-19 21:38:49,079 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1524187975341_0003_02_000001 transitioned from NEW to LOCALIZING 2018-04-19 21:38:49,079 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1524187975341_0003 2018-04-19 21:38:49,080 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_1524187975341_0003_02_000001 2018-04-19 21:38:49,101 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /tmp/hadoop-ubuntu/nm-local-dir/nmPrivate/container_1524187975341_0003_02_000001.tokens 2018-04-19 21:38:52,446 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1524187975341_0003_02_000001 transitioned from LOCALIZING to SCHEDULED 2018-04-19 21:38:52,446 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler.ContainerScheduler: Starting container [container_1524187975341_0003_02_000001] 2018-04-19 21:38:52,528 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1524187975341_0003_02_000001 transitioned from SCHEDULED to RUNNING 2018-04-19 21:38:52,529 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_1524187975341_0003_02_000001 2018-04-19 21:38:53,608 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime: Docker inspect output for container_1524187975341_0003_02_000001: ,bay14-vm1 2018-04-19 21:38:53,609 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: container_1524187975341_0003_02_000001's ip = 130.245.127.158, and hostname = bay14-vm1 2018-04-19 21:38:54,463 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor: Shell execution returned exit code: 1. Privileged Execution Operation Stderr: Stdout: main : command provided 4 main : run as user is ubuntu main : requested yarn user is ubuntu Creating script paths... Creating local dirs... Getting exit code file... Changing effective user to root... Launching docker container... Docker run command: /usr/bin/docker --config='/tmp/hadoop-ubuntu/nm-local-dir/nmPrivate/application_1524187975341_0003/container_1524187975341_0003_02_000001' run --name='container_1524187975341_0003_02_000001' --user='1000:1000' -d --workdir='/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1524187975341_0003/container_1524187975341_0003_02_000001' --net='host' -v '/tmp/hadoop-ubuntu/nm-local-dir/filecache:/tmp/hadoop-ubuntu/nm-local-dir/filecache:ro' -v '/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/filecache:/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/filecache:ro' -v '/home/ubuntu/hadoop-3.1.0/logs/userlogs/application_1524187975341_0003/container_1524187975341_0003_02_000001:/home/ubuntu/hadoop-3.1.0/logs/userlogs/application_1524187975341_0003/container_1524187975341_0003_02_000001' -v '/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1524187975341_0003:/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1524187975341_0003' --cap-drop='ALL' --cap-add='SYS_CHROOT' --cap-add='MKNOD' --cap-add='SETFCAP' --cap-add='SETPCAP' --cap-add='FSETID' --cap-add='CHOWN' --cap-add='AUDIT_WRITE' --cap-add='SETGID' --cap-add='NET_RAW' --cap-add='FOWNER' --cap-add='SETUID' --cap-add='DAC_OVERRIDE' --cap-add='KILL' --cap-add='NET_BIND_SERVICE' --group-add '1000' --group-add '4' --group-add '24' --group-add '27' --group-add '30' --group-add '46' --group-add '110' --group-add '115' --group-add '116' --group-add '117' 'local/hadoop-ubuntu:latest' 'bash' '/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1524187975341_0003/container_1524187975341_0003_02_000001/launch_container.sh' Inspecting docker container... Docker inspect command: /usr/bin/docker inspect --format {{.State.Pid}} container_1524187975341_0003_02_000001 pid from docker inspect: 2519 Writing to cgroup task files... Writing pid file... Writing to tmp file /tmp/hadoop-ubuntu/nm-local-dir/nmPrivate/application_1524187975341_0003/container_1524187975341_0003_02_000001/container_1524187975341_0003_02_000001.pid.tmp Waiting for docker container to finish. Obtaining the exit code... Docker inspect command: /usr/bin/docker inspect --format {{.State.ExitCode}} container_1524187975341_0003_02_000001 Exit code from docker inspect: 1 Wrote the exit code 1 to /tmp/hadoop-ubuntu/nm-local-dir/nmPrivate/application_1524187975341_0003/container_1524187975341_0003_02_000001/container_1524187975341_0003_02_000001.pid.exitcode Full command array for failed execution: [/home/ubuntu/hadoop-3.1.0/bin/container-executor, ubuntu, ubuntu, 4, application_1524187975341_0003, container_1524187975341_0003_02_000001, /tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1524187975341_0003/container_1524187975341_0003_02_000001, /tmp/hadoop-ubuntu/nm-local-dir/nmPrivate/application_1524187975341_0003/container_1524187975341_0003_02_000001/launch_container.sh, /tmp/hadoop-ubuntu/nm-local-dir/nmPrivate/application_1524187975341_0003/container_1524187975341_0003_02_000001/container_1524187975341_0003_02_000001.tokens, /tmp/hadoop-ubuntu/nm-local-dir/nmPrivate/application_1524187975341_0003/container_1524187975341_0003_02_000001/container_1524187975341_0003_02_000001.pid, /tmp/hadoop-ubuntu/nm-local-dir, /home/ubuntu/hadoop-3.1.0/logs/userlogs, /tmp/hadoop-ubuntu/nm-docker-cmds/docker.container_1524187975341_0003_02_0000017038281858825700163.cmd, cgroups=none] 2018-04-19 21:38:54,464 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime: Launch container failed. Exception: org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationException: ExitCodeException exitCode=1: at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:180) at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(DockerLinuxContainerRuntime.java:897) at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime.launchContainer(DelegatingLinuxContainerRuntime.java:141) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:545) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:511) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:304) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:101) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:1009) at org.apache.hadoop.util.Shell.run(Shell.java:902) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1227) at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:152) ... 10 more 2018-04-19 21:38:54,464 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime: Docker command used: run cap-add=SYS_CHROOT,MKNOD,SETFCAP,SETPCAP,FSETID,CHOWN,AUDIT_WRITE,SETGID,NET_RAW,FOWNER,SETUID,DAC_OVERRIDE,KILL,NET_BIND_SERVICE cap-drop=ALL detach=true docker-command=run docker-config=/tmp/hadoop-ubuntu/nm-local-dir/nmPrivate/application_1524187975341_0003/container_1524187975341_0003_02_000001 group-add=1000,4,24,27,30,46,110,115,116,117 image=local/hadoop-ubuntu:latest launch-command=bash,/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1524187975341_0003/container_1524187975341_0003_02_000001/launch_container.sh name=container_1524187975341_0003_02_000001 net=host ro-mounts=/tmp/hadoop-ubuntu/nm-local-dir/filecache:/tmp/hadoop-ubuntu/nm-local-dir/filecache,/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/filecache:/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/filecache rw-mounts=/home/ubuntu/hadoop-3.1.0/logs/userlogs/application_1524187975341_0003/container_1524187975341_0003_02_000001:/home/ubuntu/hadoop-3.1.0/logs/userlogs/application_1524187975341_0003/container_1524187975341_0003_02_000001,/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1524187975341_0003:/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1524187975341_0003 user=1000:1000 workdir=/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1524187975341_0003/container_1524187975341_0003_02_000001 2018-04-19 21:38:54,465 WARN org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code from container container_1524187975341_0003_02_000001 is : 1 2018-04-19 21:38:54,465 WARN org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exception from container-launch with container ID: container_1524187975341_0003_02_000001 and exit code: 1 org.apache.hadoop.yarn.server.nodemanager.containermanager.runtime.ContainerExecutionException: Launch container failed at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(DockerLinuxContainerRuntime.java:904) at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime.launchContainer(DelegatingLinuxContainerRuntime.java:141) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:545) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:511) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:304) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:101) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-04-19 21:38:54,465 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch. 2018-04-19 21:38:54,465 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_1524187975341_0003_02_000001 2018-04-19 21:38:54,465 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 1 2018-04-19 21:38:54,465 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Shell output: main : command provided 4 2018-04-19 21:38:54,465 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: main : run as user is ubuntu 2018-04-19 21:38:54,465 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: main : requested yarn user is ubuntu 2018-04-19 21:38:54,465 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Creating script paths... 2018-04-19 21:38:54,465 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Creating local dirs... 2018-04-19 21:38:54,465 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Getting exit code file... 2018-04-19 21:38:54,465 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Changing effective user to root... 2018-04-19 21:38:54,465 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Launching docker container... 2018-04-19 21:38:54,465 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Docker run command: /usr/bin/docker --config='/tmp/hadoop-ubuntu/nm-local-dir/nmPrivate/application_1524187975341_0003/container_1524187975341_0003_02_000001' run --name='container_1524187975341_0003_02_000001' --user='1000:1000' -d --workdir='/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1524187975341_0003/container_1524187975341_0003_02_000001' --net='host' -v '/tmp/hadoop-ubuntu/nm-local-dir/filecache:/tmp/hadoop-ubuntu/nm-local-dir/filecache:ro' -v '/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/filecache:/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/filecache:ro' -v '/home/ubuntu/hadoop-3.1.0/logs/userlogs/application_1524187975341_0003/container_1524187975341_0003_02_000001:/home/ubuntu/hadoop-3.1.0/logs/userlogs/application_1524187975341_0003/container_1524187975341_0003_02_000001' -v '/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1524187975341_0003:/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1524187975341_0003' --cap-drop='ALL' --cap-add='SYS_CHROOT' --cap-add='MKNOD' --cap-add='SETFCAP' --cap-add='SETPCAP' --cap-add='FSETID' --cap-add='CHOWN' --cap-add='AUDIT_WRITE' --cap-add='SETGID' --cap-add='NET_RAW' --cap-add='FOWNER' --cap-add='SETUID' --cap-add='DAC_OVERRIDE' --cap-add='KILL' --cap-add='NET_BIND_SERVICE' --group-add '1000' --group-add '4' --group-add '24' --group-add '27' --group-add '30' --group-add '46' --group-add '110' --group-add '115' --group-add '116' --group-add '117' 'local/hadoop-ubuntu:latest' 'bash' '/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1524187975341_0003/container_1524187975341_0003_02_000001/launch_container.sh' 2018-04-19 21:38:54,466 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Inspecting docker container... 2018-04-19 21:38:54,466 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Docker inspect command: /usr/bin/docker inspect --format {{.State.Pid}} container_1524187975341_0003_02_000001 2018-04-19 21:38:54,466 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: pid from docker inspect: 2519 2018-04-19 21:38:54,466 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Writing to cgroup task files... 2018-04-19 21:38:54,466 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Writing pid file... 2018-04-19 21:38:54,466 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Writing to tmp file /tmp/hadoop-ubuntu/nm-local-dir/nmPrivate/application_1524187975341_0003/container_1524187975341_0003_02_000001/container_1524187975341_0003_02_000001.pid.tmp 2018-04-19 21:38:54,466 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Waiting for docker container to finish. 2018-04-19 21:38:54,466 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Obtaining the exit code... 2018-04-19 21:38:54,466 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Docker inspect command: /usr/bin/docker inspect --format {{.State.ExitCode}} container_1524187975341_0003_02_000001 2018-04-19 21:38:54,466 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code from docker inspect: 1 2018-04-19 21:38:54,466 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Wrote the exit code 1 to /tmp/hadoop-ubuntu/nm-local-dir/nmPrivate/application_1524187975341_0003/container_1524187975341_0003_02_000001/container_1524187975341_0003_02_000001.pid.exitcode 2018-04-19 21:38:54,466 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container launch failed : Container exited with a non-zero exit code 1. 2018-04-19 21:38:54,475 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1524187975341_0003_02_000001 transitioned from RUNNING to EXITED_WITH_FAILURE 2018-04-19 21:38:54,476 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_1524187975341_0003_02_000001 2018-04-19 21:38:54,597 INFO org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Removing Docker container : container_1524187975341_0003_02_000001 2018-04-19 21:38:56,643 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Skipping monitoring container container_1524187975341_0003_02_000001 since CPU usage is not yet available. 2018-04-19 21:38:56,869 INFO org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Deleting absolute path : /tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1524187975341_0003/container_1524187975341_0003_02_000001 2018-04-19 21:38:56,869 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=ubuntu OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE APPID=application_1524187975341_0003 CONTAINERID=container_1524187975341_0003_02_000001 2018-04-19 21:38:56,871 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1524187975341_0003_02_000001 transitioned from EXITED_WITH_FAILURE to DONE 2018-04-19 21:38:56,872 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_1524187975341_0003_02_000001 from application application_1524187975341_0003 2018-04-19 21:38:56,877 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_1524187975341_0003_02_000001 2018-04-19 21:38:56,877 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1524187975341_0003 2018-04-19 21:38:57,888 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_1524187975341_0003_02_000001] 2018-04-19 21:38:57,888 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1524187975341_0003 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2018-04-19 21:38:57,889 INFO org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Deleting absolute path : /tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1524187975341_0003 2018-04-19 21:38:57,889 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1524187975341_0003 2018-04-19 21:38:57,890 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1524187975341_0003 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2018-04-19 21:38:57,890 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler: Scheduling Log Deletion for application: application_1524187975341_0003, with delay of 10800 seconds On Thu, Apr 19, 2018 at 9:28 PM, Shane Kumpf <shane.kumpf.apa...@gmail.com> wrote: > Hello Ahmad, > > The image being used is not privileged/untrusted based on the settings in > container-executor.cfg. In container-executor.cfg you have set > docker.privileged-containers.registries=local, but the image name > variable in the job is using "hadoop-ubuntu:latest". Based on that > setting, YARN is expecting the image to be in the "local" namespace. Can > you set YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=local/hadoop-ubuntu:latest > and see if that resolves the issue? > > Thanks, > -Shane > > On Thu, Apr 19, 2018 at 4:59 PM, SeyyedAhmad Javadi < > sjav...@cs.stonybrook.edu> wrote: > >> Hi All, >> >> I am following the below guide to setup Docker container run_time but >> face some non-trivial errors at least for my level. Would you please >> comment if you have some idea about the root-cause? >> http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yar >> n-site/DockerContainers.html >> >> After the error, I have provided the config files and Dockerfile as well >> as Docker image inspect command results (should have null for Entry Point >> and CMS?). >> >> I have three nodes, 1 RM and 2 NMs and default LCE works fine. >> >> ********************submit job script >> vars="YARN_CONTAINER_RUNTIME_TYPE=docker,YARN_CONTAINER_RUNT >> IME_DOCKER_IMAGE=hadoop-ubuntu,YARN_CONTAINER_RUNTIME_DOCKER >> _RUN_OVERRIDE_DISABLE=false,YARN_CONTAINER_RUNTIME_DOCKER_ >> CONTAINER_NETWORK=host" >> >> #vars="YARN_CONTAINER_RUNTIME_TYPE=default" >> hadoop jar >> /home/ubuntu/hadoop-3.1.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.0.jar >> pi -Dyarn.app.mapreduce.am.env=$vars -Dmapreduce.map.env=$vars >> -Dmapreduce.reduce.env=$vars 2 10 >> >> ******************** AM Log in one the nodes >> 2018-04-19 18:55:46,311 INFO SecurityLogger.org.apache.hadoop.ipc.Server: >> Auth successful for appattempt_1524178188987_0001_000001 (auth:SIMPLE) >> 2018-04-19 18:55:46,515 INFO org.apache.hadoop.yarn.server. >> nodemanager.containermanager.ContainerManagerImpl: Start request for >> container_1524178188987_0001_01_000001 by user ubuntu >> 2018-04-19 18:55:46,617 INFO org.apache.hadoop.yarn.server. >> nodemanager.containermanager.ContainerManagerImpl: Creating a new >> application reference for app application_1524178188987_0001 >> 2018-04-19 18:55:46,634 INFO org.apache.hadoop.yarn.server. >> nodemanager.containermanager.application.ApplicationImpl: Application >> application_1524178188987_0001 transitioned from NEW to INITING >> 2018-04-19 18:55:46,634 INFO >> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: >> USER=ubuntu IP=130.245.127.176 OPERATION=Start Container Request >> TARGET=ContainerManageImpl RESULT=SUCCESS >> APPID=application_1524178188987_0001 CONTAINERID=container_15241781 >> 88987_0001_01_000001 >> 2018-04-19 18:55:46,635 INFO org.apache.hadoop.yarn.server. >> nodemanager.containermanager.application.ApplicationImpl: Adding >> container_1524178188987_0001_01_000001 to application >> application_1524178188987_0001 >> 2018-04-19 18:55:46,649 INFO org.apache.hadoop.yarn.server. >> nodemanager.containermanager.application.ApplicationImpl: Application >> application_1524178188987_0001 transitioned from INITING to RUNNING >> 2018-04-19 18:55:46,655 INFO org.apache.hadoop.yarn.server. >> nodemanager.containermanager.container.ContainerImpl: Container >> container_1524178188987_0001_01_000001 transitioned from NEW to >> LOCALIZING >> 2018-04-19 18:55:46,655 INFO org.apache.hadoop.yarn.server. >> nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for >> appId application_1524178188987_0001 >> 2018-04-19 18:55:46,698 INFO org.apache.hadoop.yarn.server. >> nodemanager.containermanager.localizer.ResourceLocalizationService: >> Created localizer for container_1524178188987_0001_01_000001 >> 2018-04-19 18:55:46,898 INFO org.apache.hadoop.yarn.server. >> nodemanager.containermanager.localizer.ResourceLocalizationService: >> Writing credentials to the nmPrivate file /tmp/hadoop-ubuntu/nm-local-di >> r/nmPrivate/container_1524178188987_0001_01_000001.tokens >> 2018-04-19 18:55:50,371 INFO org.apache.hadoop.yarn.server. >> nodemanager.containermanager.container.ContainerImpl: Container >> container_1524178188987_0001_01_000001 transitioned from LOCALIZING to >> SCHEDULED >> 2018-04-19 18:55:50,374 INFO org.apache.hadoop.yarn.server. >> nodemanager.containermanager.scheduler.ContainerScheduler: Starting >> container [container_1524178188987_0001_01_000001] >> 2018-04-19 18:55:50,479 INFO org.apache.hadoop.yarn.server. >> nodemanager.containermanager.container.ContainerImpl: Container >> container_1524178188987_0001_01_000001 transitioned from SCHEDULED to >> RUNNING >> 2018-04-19 18:55:50,481 INFO org.apache.hadoop.yarn.server. >> nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting >> resource-monitoring for container_1524178188987_0001_01_000001 >> 2018-04-19 18:55:51,842 INFO org.apache.hadoop.yarn.server. >> nodemanager.containermanager.launcher.ContainerLaunch: Container >> container_1524178188987_0001_01_000001 succeeded >> 2018-04-19 18:55:51,844 INFO org.apache.hadoop.yarn.server. >> nodemanager.containermanager.container.ContainerImpl: Container >> container_1524178188987_0001_01_000001 transitioned from RUNNING to >> EXITED_WITH_SUCCESS >> 2018-04-19 18:55:51,844 INFO org.apache.hadoop.yarn.server. >> nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up >> container container_1524178188987_0001_01_000001 >> 2018-04-19 18:55:51,957 INFO org.apache.hadoop.yarn.server. >> nodemanager.LinuxContainerExecutor: Removing Docker container : >> container_1524178188987_0001_01_000001 >> 2018-04-19 18:55:56,963 INFO org.apache.hadoop.yarn.server. >> nodemanager.containermanager.launcher.ContainerLaunch: Could not get pid >> for container_1524178188987_0001_01_000001. Waited for 5000 ms. >> 2018-04-19 18:55:56,963 INFO org.apache.hadoop.yarn.server. >> nodemanager.containermanager.launcher.ContainerLaunch: Unable to obtain >> pid, but docker container request detected. Attempting to reap container >> container_1524178188987_0001_01_000001 >> 2018-04-19 18:55:59,395 INFO org.apache.hadoop.yarn.server. >> nodemanager.LinuxContainerExecutor: Deleting absolute path : >> /tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/ap >> plication_1524178188987_0001/container_1524178188987_0001_01_000001 >> 2018-04-19 18:55:59,395 INFO >> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: >> USER=ubuntu OPERATION=Container Finished - Succeeded >> TARGET=ContainerImpl RESULT=SUCCESS >> APPID=application_1524178188987_0001 >> CONTAINERID=container_1524178188987_0001_01_000001 >> 2018-04-19 18:55:59,403 INFO org.apache.hadoop.yarn.server. >> nodemanager.containermanager.container.ContainerImpl: Container >> container_1524178188987_0001_01_000001 transitioned from >> EXITED_WITH_SUCCESS to DONE >> 2018-04-19 18:55:59,404 INFO org.apache.hadoop.yarn.server. >> nodemanager.containermanager.application.ApplicationImpl: Removing >> container_1524178188987_0001_01_000001 from application >> application_1524178188987_0001 >> 2018-04-19 18:55:59,404 INFO org.apache.hadoop.yarn.server. >> nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping >> resource-monitoring for container_1524178188987_0001_01_000001 >> 2018-04-19 18:55:59,405 INFO org.apache.hadoop.yarn.server. >> nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for >> appId application_1524178188987_0001 >> 2018-04-19 18:56:00,412 INFO org.apache.hadoop.yarn.server. >> nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM >> context: [container_1524178188987_0001_01_000001] >> 2018-04-19 18:56:13,455 INFO org.apache.hadoop.yarn.server. >> nodemanager.containermanager.application.ApplicationImpl: Application >> application_1524178188987_0001 transitioned from RUNNING to >> APPLICATION_RESOURCES_CLEANINGUP >> 2018-04-19 18:56:13,457 INFO org.apache.hadoop.yarn.server. >> nodemanager.LinuxContainerExecutor: Deleting absolute path : >> /tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/ap >> plication_1524178188987_0001 >> 2018-04-19 18:56:13,459 INFO org.apache.hadoop.yarn.server. >> nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for >> appId application_1524178188987_0001 >> 2018-04-19 18:56:13,470 INFO org.apache.hadoop.yarn.server. >> nodemanager.containermanager.application.ApplicationImpl: Application >> application_1524178188987_0001 transitioned from >> APPLICATION_RESOURCES_CLEANINGUP to FINISHED >> 2018-04-19 18:56:13,470 INFO org.apache.hadoop.yarn.server. >> nodemanager.containermanager.loghandler.NonAggregatingLogHandler: >> Scheduling Log Deletion for application: application_1524178188987_0001, >> with delay of 10800 seconds >> >> ********************** >> >> >> >> >> yarn-site.xml: according to the above link >> container-executor.cfg: >> >> yarn.nodemanager.linux-container-executor.group=ubuntu >> min.user.id=0 >> #feature.tc.enabled=1 >> #feature.docker.enabled=1 >> allowed.system.users=ubuntu >> # The configs below deal with settings for Docker >> [docker] >> module.enabled=true >> docker.privileged-containers.enabled=true >> docker.binary=/usr/bin/docker >> docker.allowed.capabilities=SYS_CHROOT,MKNOD,SETFCAP,SETPCAP >> ,FSETID,CHOWN,AUDIT_WRITE,SETGID,NET_RAW,FOWNER,SETUID, >> DAC_OVERRIDE,KILL,NET_BIND_SERVICE >> #docker.allowed.devices=## comma seperated list of devices that can be >> mounted into a container >> docker.allowed.networks=bridge,host,none >> docker.allowed.ro-mounts=/sys/fs/cgroup,/tmp/hadoop-ubuntu/nm-local-dir >> docker.privileged-containers.registries=local >> #docker.host-pid-namespace.enabled=false >> docker.allowed.rw-mounts=/home/ubuntu/hadoop-3.1.0,/home/ >> ubuntu/hadoop-3.1.0/logs >> >> >> Dockerfile: >> FROM ubuntu:16.04 >> #RUN rm /bin/sh && ln -s /bin/bash /bin/sh >> SHELL ["/bin/bash", "-c"] >> >> RUN apt-get update && \ >> apt-get upgrade -y && \ >> apt-get install -y software-properties-common && \ >> # apt-get install -y --no-install-recommends apt-utils && \ >> # apt-get install -y curl && \ >> add-apt-repository ppa:webupd8team/java -y && \ >> apt-get update && \ >> echo oracle-java7-installer shared/accepted-oracle-license-v1-1 >> select true | /usr/bin/debconf-set-selections && \ >> apt-get install -y oracle-java8-installer && \ >> # apt-get install -y ssh && \ >> # apt-get install -y rsync && \ >> apt-get install -y vim && \ >> apt-get clean >> >> ENV JAVA_HOME /usr/lib/jvm/java-8-oracle >> ENV PATH $PATH:$JAVA_HOME/bin >> >> # HADOOP >> ARG HADOOP_ARCHIVE=http://mirror.cc.columbia.edu/pub/software/ap >> ache/hadoop/common/hadoop-3.1.0/hadoop-3.1.0.tar.gz >> >> ENV HADOOP_HOME /usr/local/hadoop >> ENV HADOOP_COMMON_PATH /usr/local/hadoop >> ENV HADOOP_HDFS_HOME /usr/local/hadoop >> ENV HADOOP_MAPRED_HOME /usr/local/hadoop >> ENV HADOOP_YARN_HOME /usr/local/hadoop >> ENV HADOOP_CONF_DIR /usr/local/hadoop/etc/hadoop >> >> # download and extract hadoop, set JAVA_HOME in hadoop-env.sh, update path >> RUN wget $HADOOP_ARCHIVE && \ >> tar -xzf hadoop-3.1.0.tar.gz && \ >> mv hadoop-3.1.0 $HADOOP_HOME >> >> ADD rm-hadoop-config/* $HADOOP_HOME/etc/hadoop/ >> >> ENV PATH $PATH:$HADOOP_COMMON_PATH/bin >> >> WORKDIR $HADOOP_COMMON_PATH >> >> # Declare user >> RUN groupadd -g 1000 ubuntu && \ >> useradd -r -u 1000 -g 1000 ubuntu >> USER ubuntu >> >> >> ~/hadoop-common$ docker inspect local/hadoop-ubuntu >> [ >> { >> "Id": "sha256:d8335693084b5823675056 >> b7d649b13d04a7e3c3b63688f83e9807405506b088", >> "RepoTags": [ >> "hadoop-ubuntu:latest", >> "local/hadoop-ubuntu:latest" >> ], >> "RepoDigests": [], >> "Parent": "sha256:f9b92fa15eadd74b9e3712 >> ee379b56bc30a73492db2fd4e7b7f4f1d74e5671f2", >> "Comment": "", >> "Created": "2018-04-19T15:16:52.903547395Z", >> "Container": "4fcff6fa65639fe0e2a9379a335a4 >> 01a8cde0c0e1eaaa2dd98d35ced402c29e3", >> "ContainerConfig": { >> "Hostname": "4fcff6fa6563", >> "Domainname": "", >> "User": "ubuntu", >> "AttachStdin": false, >> "AttachStdout": false, >> "AttachStderr": false, >> "Tty": false, >> "OpenStdin": false, >> "StdinOnce": false, >> "Env": [ >> "PATH=/usr/local/sbin:/usr/loc >> al/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/jvm/java-8-ora >> cle/bin:/usr/local/hadoop/bin", >> "JAVA_HOME=/usr/lib/jvm/java-8-oracle", >> "HADOOP_HOME=/usr/local/hadoop", >> "HADOOP_COMMON_PATH=/usr/local/hadoop", >> "HADOOP_HDFS_HOME=/usr/local/hadoop", >> "HADOOP_MAPRED_HOME=/usr/local/hadoop", >> "HADOOP_YARN_HOME=/usr/local/hadoop", >> "HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop" >> ], >> "Cmd": [ >> "/bin/bash", >> "-c", >> "#(nop) ", >> "USER ubuntu" >> ], >> "ArgsEscaped": true, >> "Image": "sha256:f9b92fa15eadd74b9e3712 >> ee379b56bc30a73492db2fd4e7b7f4f1d74e5671f2", >> "Volumes": null, >> "WorkingDir": "/usr/local/hadoop", >> "Entrypoint": null, >> "OnBuild": null, >> "Labels": {}, >> "Shell": [ >> "/bin/bash", >> "-c" >> ] >> }, >> "DockerVersion": "18.03.0-ce", >> "Author": "", >> "Config": { >> "Hostname": "", >> "Domainname": "", >> "User": "ubuntu", >> "AttachStdin": false, >> "AttachStdout": false, >> "AttachStderr": false, >> "Tty": false, >> "OpenStdin": false, >> "StdinOnce": false, >> "Env": [ >> "PATH=/usr/local/sbin:/usr/loc >> al/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/jvm/java-8-ora >> cle/bin:/usr/local/hadoop/bin", >> "JAVA_HOME=/usr/lib/jvm/java-8-oracle", >> "HADOOP_HOME=/usr/local/hadoop", >> "HADOOP_COMMON_PATH=/usr/local/hadoop", >> "HADOOP_HDFS_HOME=/usr/local/hadoop", >> "HADOOP_MAPRED_HOME=/usr/local/hadoop", >> "HADOOP_YARN_HOME=/usr/local/hadoop", >> "HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop" >> ], >> "Cmd": [ >> "/bin/bash" >> ], >> "ArgsEscaped": true, >> "Image": "sha256:f9b92fa15eadd74b9e3712 >> ee379b56bc30a73492db2fd4e7b7f4f1d74e5671f2", >> "Volumes": null, >> "WorkingDir": "/usr/local/hadoop", >> "Entrypoint": null, >> "OnBuild": null, >> "Labels": null, >> "Shell": [ >> "/bin/bash", >> "-c" >> ] >> }, >> "Architecture": "amd64", >> "Os": "linux", >> "Size": 2058935914, >> "VirtualSize": 2058935914, >> "GraphDriver": { >> "Data": null, >> "Name": "aufs" >> }, >> "RootFS": { >> "Type": "layers", >> "Layers": [ >> "sha256:fccbfa2912f0cd6b9d13f9 >> 1f288f112a2b825f3f758a4443aacb45bfc108cc74", >> "sha256:e1a9a6284d0d24d8194ac8 >> 4b372619e75cd35a46866b74925b7274c7056561e4", >> "sha256:ac7299292f8b2f710d3b91 >> 1c6a4e02ae8f06792e39822e097f9c4e9c2672b32d", >> "sha256:a5e66470b2812e91798db3 >> 6eb103c1f1e135bbe167e4b2ad5ba425b8db98ee8d", >> "sha256:a8de0e025d94b33db3542e >> 1e8ce58829144b30c6cd1fff057eec55b1491933c3", >> "sha256:7e9a788452589001d42e79 >> 95dc0583bcca1e6f7780a301066ee0d6668aaf9c91", >> "sha256:65fac5b99df0506e1d204d >> 9024264105ab2fe142ea970f0eba630669dc606055", >> "sha256:cd0da20f97700ab8e2ec37 >> c464fcb8864cba86672aa96e7fc40e2572228895e2", >> "sha256:21094366bc298af4a7e836 >> 80cf9d732ce69cb92b11a3b20005b12c15bac3e486" >> ] >> }, >> "Metadata": { >> "LastTagTime": "2018-04-19T17:20:41.054733026-04:00" >> } >> } >> ] >> >> >> >> >> Best, >> Ahmad >> >> >