[ https://issues.apache.org/jira/browse/YARN-7070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153633#comment-16153633 ]
Jason Lowe commented on YARN-7070: ---------------------------------- Those warnings are "normal" and are a result of the container being killed by YARN. Exit code 143 is a result of the bash process being killed by SIGTERM. SIGTERM is signal 15, and bash returns with an exit code of 128 + signalnum when killed by a signal. 128 + 15 = 143. So it sounds like this only happens when a container is getting killed? Does it always happen when a container is killed or only when a container is killed just as the application completes as well (e.g.: container is getting killed because application was killed)? I suspect this is only happening when apps are getting killed and the app dir delete is racing with the container delete, but that should have been fixed by YARN-6846. Are you sure there are no similar container executor warnings in the NM logs regarding the deletion command processing? If the container executor is failing to cleanup a directory I would expect it to exit with a non-zero exit code which should result in similar WARN logs in the NM as you posted above, although instead of being command 1 (the third argument to the executor) it would be command 3. > some of local cache files for yarn can't be deleted > --------------------------------------------------- > > Key: YARN-7070 > URL: https://issues.apache.org/jira/browse/YARN-7070 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager > Affects Versions: 2.8.1 > Environment: Hadoop 2.8.1 > Reporter: Changyao Ye > Attachments: application_1501810184023_55949.log, warn_logs.txt > > > We have found some of cache files(in > /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache) for yarn on > nodemanager cannot be deleted properly. The directories are like > following(blockmgr***) > ================= > # ls -ltr application_1501810184023_55949 > total 120 > drwx--x--- 2 hdfs yarn 4096 Aug 22 04:29 filecache > drwxr-s--- 2 hdfs yarn 4096 Aug 22 04:56 > blockmgr-881fab2c-fba4-4bb1-8dd9-5ab35a512df7 > drwxr-s--- 10 hdfs yarn 4096 Aug 22 04:56 > blockmgr-bf8a19f5-e9ae-4269-a0ef-b27d0f9c17e7 > drwxr-s--- 11 hdfs yarn 4096 Aug 22 04:58 > blockmgr-f3437e8d-9595-4898-8bda-92ebff3ada1d > drwxr-s--- 18 hdfs yarn 4096 Aug 22 05:01 > blockmgr-930c0cd8-1d31-4cdb-a244-f6ad4bf74bff > drwxr-s--- 12 hdfs yarn 4096 Aug 22 05:13 > blockmgr-83fc0702-ac40-4743-812a-7d488e92004e > drwxr-s--- 9 hdfs yarn 4096 Aug 22 05:13 > blockmgr-f6cfe045-12c3-41d6-b77e-aa5200daeb6a > drwxr-s--- 12 hdfs yarn 4096 Aug 22 05:13 > blockmgr-53dcb4ea-ba5d-4b8b-859b-805b9303a149 > drwxr-s--- 10 hdfs yarn 4096 Aug 22 05:13 > blockmgr-0c0c4bb9-ef5e-4ca1-8d23-ce5cd58d0a75 > drwxr-s--- 9 hdfs yarn 4096 Aug 22 05:13 > blockmgr-557d0f39-67d2-491a-9307-12fc1d724380 > drwxr-s--- 10 hdfs yarn 4096 Aug 22 05:13 > blockmgr-fbc87680-4df7-498e-bf6d-456a5aea4fc9 > drwxr-s--- 10 hdfs yarn 4096 Aug 22 05:13 > blockmgr-53ee8251-fac1-4f62-82c2-5e970f0d86ec > drwxr-s--- 9 hdfs yarn 4096 Aug 22 05:14 > blockmgr-5a8bc187-abcf-482d-9da5-e8c4647d4731 > drwxr-s--- 10 hdfs yarn 4096 Aug 22 05:14 > blockmgr-251c3a99-cd85-442a-8945-52c344c0d861 > drwxr-s--- 13 hdfs yarn 4096 Aug 22 05:14 > blockmgr-c352c1ad-15dc-456b-8b62-5b83b9950494 > drwxr-s--- 12 hdfs yarn 4096 Aug 22 05:15 > blockmgr-b4f01347-4b51-4b35-8146-2aa840084c2b > drwxr-s--- 14 hdfs yarn 4096 Aug 22 05:15 > blockmgr-0095d26c-c134-48b4-82a6-e8ae02f0189c > drwxr-s--- 13 hdfs yarn 4096 Aug 22 05:15 > blockmgr-28a31574-61ae-459f-be3a-8608892246d7 > drwxr-s--- 16 hdfs yarn 4096 Aug 22 05:15 > blockmgr-c0cd0df9-b355-4209-b6aa-b549a1fa36eb > drwxr-s--- 11 hdfs yarn 4096 Aug 22 05:15 > blockmgr-a2730abb-9517-461e-bedf-d9a2dcef373f > drwxr-s--- 14 hdfs yarn 4096 Aug 22 05:15 > blockmgr-91dd2e1a-6bc2-4429-8b71-2f4240987159 > drwxr-s--- 12 hdfs yarn 4096 Aug 22 05:15 > blockmgr-f4e3a586-8817-45ea-a197-9fdbb3d91946 > drwxr-s--- 15 hdfs yarn 4096 Aug 22 05:15 > blockmgr-ba2c605e-89d8-4f7c-b42c-6ed4ba6bf4ea > drwxr-s--- 16 hdfs yarn 4096 Aug 22 05:15 > blockmgr-2ae72383-5f72-4002-84a7-e6335b8c2b6c > drwxr-s--- 13 hdfs yarn 4096 Aug 22 05:15 > blockmgr-6c5e260f-d3c7-4af6-91c1-168c73343f2d > drwxr-s--- 16 hdfs yarn 4096 Aug 22 05:15 > blockmgr-2e9923b1-281c-4a9d-8069-6c5430bd5fc3 > drwxr-s--- 18 hdfs yarn 4096 Aug 22 05:15 > blockmgr-cc3f1406-d8a2-4bf5-a276-8f7aed75c513 > drwxr-s--- 11 hdfs yarn 4096 Aug 22 05:15 > blockmgr-975bcce0-84b2-4590-880b-bf182d76e319 > drwxr-s--- 11 hdfs yarn 4096 Aug 22 05:15 > blockmgr-ce82cb63-5998-4227-b85e-77f1c633db43 > drwxr-s--- 11 hdfs yarn 4096 Aug 22 05:15 > blockmgr-592af4aa-3c89-4081-8746-29b99f2220b1 > ================= > We also applied patches YARN-4594, YARN-4731, but nothing changed. > YARN-4594 https://issues.apache.org/jira/browse/YARN-4594 > YARN-4731 https://issues.apache.org/jira/browse/YARN-4731 > Any advice will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org