[ https://issues.apache.org/jira/browse/YARN-9647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890394#comment-16890394 ]
Eric Badger commented on YARN-9647: ----------------------------------- [~eyang], [~Jim_Brennan], [~billie.rinaldi], any ideas on how to fix this in a clean way? > Docker launch fails when local-dirs or log-dirs is unhealthy. > ------------------------------------------------------------- > > Key: YARN-9647 > URL: https://issues.apache.org/jira/browse/YARN-9647 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager > Affects Versions: 3.1.2 > Reporter: KWON BYUNGCHANG > Priority: Major > Attachments: YARN-9647.001.patch, YARN-9647.002.patch > > > my /etc/hadoop/conf/container-executor.cfg > {code} > [docker] > docker.allowed.ro-mounts=/data1/hadoop/yarn/local,/data2/hadoop/yarn/local > docker.allowed.rw-mounts=/data1/hadoop/yarn/local,/data2/hadoop/yarn/local > {code} > if /data2 is unhealthy, docker launch fails although container can use > /data1 as local-dir, log-dir > error message is below > {code} > [2019-06-25 14:55:26.168]Exception from container-launch. Container id: > container_e50_1561100493387_5185_01_000597 Exit code: 29 Exception message: > Launch container failed Shell error output: Could not determine real path of > mount '/data2/hadoop/yarn/local' Could not determine real path of mount > '/data2/hadoop/yarn/local' Unable to find permitted docker mounts on disk > Error constructing docker command, docker error code=16, error message='Mount > access error' Shell output: main : command provided 4 main : run as user is > magnum main : requested yarn user is magnum Creating script paths... Creating > local dirs... [2019-06-25 14:55:26.189]Container exited with a non-zero exit > code 29. [2019-06-25 14:55:26.192]Container exited with a non-zero exit code > 29. > {code} > root cause is that normalize_mounts() in docker-util.c return -1 because it > cannot resolve real path of /data2/hadoop/yarn/local.(note that /data2 is > disk fault at this point) > however disk of nm local dirs and nm log dirs can fail at any time. > docker launch should succeed if there are available local dirs and log dirs. -- This message was sent by Atlassian JIRA (v7.6.14#76016) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org