[ https://issues.apache.org/jira/browse/MESOS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14596823#comment-14596823 ]
Jie Yu commented on MESOS-2914: ------------------------------- https://reviews.apache.org/r/35749/ https://reviews.apache.org/r/35750/ > Port mapping isolator should cleanup unknown orphan containers after all > known orphan containers are recovered during recovery. > ------------------------------------------------------------------------------------------------------------------------------- > > Key: MESOS-2914 > URL: https://issues.apache.org/jira/browse/MESOS-2914 > Project: Mesos > Issue Type: Bug > Reporter: Jie Yu > Assignee: Jie Yu > > Otherwise, the icmp/arp filter on host eth0 might be removed as a result of > _cleanup if 'infos' is empty, causing subsequent '_cleanup' to fail on both > known/unknown orphan containers. > {noformat} > I0612 17:46:51.518501 16308 containerizer.cpp:314] Recovering containerizer > I0612 17:46:51.520612 16308 port_mapping.cpp:1567] Discovered network > namespace handle symlink ddcb8397-3552-44f9-bc99-b5b69aa72944 -> 31607 > I0612 17:46:51.521183 16308 port_mapping.cpp:1567] Discovered network > namespace handle symlink d8c48a4a-fdfb-47dd-b8d8-07188c21600d -> 41020 > I0612 17:46:51.521883 16308 port_mapping.cpp:1567] Discovered network > namespace handle symlink 8953fc7f-9fca-4931-b0cb-2f4959ddee74 -> 3302 > I0612 17:46:51.522542 16308 port_mapping.cpp:1567] Discovered network > namespace handle symlink 50f9986f-ebbc-440d-86a7-9fa1a7c55a75 -> 19805 > I0612 17:46:51.523643 16308 port_mapping.cpp:2597] Removing IP packet filters > with ports [33792,34815] for container with pid 52304 > I0612 17:46:51.525063 16308 port_mapping.cpp:2616] Freed ephemeral ports > [33792,34816) for container with pid 52304 > I0612 17:46:51.547696 16308 port_mapping.cpp:2762] Successfully performed > cleanup for pid 52304 > I0612 17:46:51.550027 16308 port_mapping.cpp:1698] Network isolator recovery > complete > I0612 17:46:51.550946 16329 containerizer.cpp:449] Removing orphan container > 111ea69c-6184-4da1-a0e9-c34e8c6deb30 > I0612 17:46:51.552686 16329 containerizer.cpp:449] Removing orphan container > ddcb8397-3552-44f9-bc99-b5b69aa72944 > I0612 17:46:51.552734 16309 cgroups.cpp:2377] Freezing cgroup > /sys/fs/cgroup/freezer/mesos/111ea69c-6184-4da1-a0e9-c34e8c6deb30 > I0612 17:46:51.554932 16329 containerizer.cpp:449] Removing orphan container > 8953fc7f-9fca-4931-b0cb-2f4959ddee74 > I0612 17:46:51.555032 16309 cgroups.cpp:2377] Freezing cgroup > /sys/fs/cgroup/freezer/mesos/ddcb8397-3552-44f9-bc99-b5b69aa72944 > I0612 17:46:51.555629 16308 cgroups.cpp:1420] Successfully froze cgroup > /sys/fs/cgroup/freezer/mesos/111ea69c-6184-4da1-a0e9-c34e8c6deb30 after > 1.730304ms > I0612 17:46:51.557507 16329 containerizer.cpp:449] Removing orphan container > 50f9986f-ebbc-440d-86a7-9fa1a7c55a75 > I0612 17:46:51.557611 16309 cgroups.cpp:2377] Freezing cgroup > /sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74 > I0612 17:46:51.557896 16313 cgroups.cpp:1420] Successfully froze cgroup > /sys/fs/cgroup/freezer/mesos/ddcb8397-3552-44f9-bc99-b5b69aa72944 after > 1.685248ms > I0612 17:46:51.559412 16310 cgroups.cpp:2394] Thawing cgroup > /sys/fs/cgroup/freezer/mesos/111ea69c-6184-4da1-a0e9-c34e8c6deb30 > I0612 17:46:51.561564 16329 containerizer.cpp:449] Removing orphan container > d8c48a4a-fdfb-47dd-b8d8-07188c21600d > I0612 17:46:51.562489 16315 cgroups.cpp:2377] Freezing cgroup > /sys/fs/cgroup/freezer/mesos/50f9986f-ebbc-440d-86a7-9fa1a7c55a75 > I0612 17:46:51.562988 16313 cgroups.cpp:2394] Thawing cgroup > /sys/fs/cgroup/freezer/mesos/ddcb8397-3552-44f9-bc99-b5b69aa72944 > I0612 17:46:51.563303 16310 cgroups.cpp:1449] Successfullly thawed cgroup > /sys/fs/cgroup/freezer/mesos/111ea69c-6184-4da1-a0e9-c34e8c6deb30 after > 2.076928ms > I0612 17:46:51.566052 16308 cgroups.cpp:2377] Freezing cgroup > /sys/fs/cgroup/freezer/mesos/d8c48a4a-fdfb-47dd-b8d8-07188c21600d > I0612 17:46:51.566102 16313 slave.cpp:3911] Finished recovery > W0612 17:46:51.566432 16323 disk.cpp:299] Ignoring cleanup for unknown > container 111ea69c-6184-4da1-a0e9-c34e8c6deb30 > I0612 17:46:51.566651 16317 cgroups.cpp:1449] Successfullly thawed cgroup > /sys/fs/cgroup/freezer/mesos/ddcb8397-3552-44f9-bc99-b5b69aa72944 after > 2.12096ms > I0612 17:46:51.566987 16313 slave.cpp:3944] Garbage collecting old slave > 20150319-213133-2080910346-5050-57551-S3314 > I0612 17:46:51.567777 16318 cgroups.cpp:1420] Successfully froze cgroup > /sys/fs/cgroup/freezer/mesos/d8c48a4a-fdfb-47dd-b8d8-07188c21600d after > 1.323008ms > W0612 17:46:51.568042 16323 port_mapping.cpp:2544] Ignoring cleanup for > unknown container 111ea69c-6184-4da1-a0e9-c34e8c6deb30 > I0612 17:46:51.569522 16311 gc.cpp:56] Scheduling > '/var/lib/mesos/slaves/20150319-213133-2080910346-5050-57551-S3314' for gc > 6.99999341503407days in the future > W0612 17:46:51.569725 16329 disk.cpp:299] Ignoring cleanup for unknown > container ddcb8397-3552-44f9-bc99-b5b69aa72944 > I0612 17:46:51.570911 16325 cgroups.cpp:2394] Thawing cgroup > /sys/fs/cgroup/freezer/mesos/d8c48a4a-fdfb-47dd-b8d8-07188c21600d > I0612 17:46:51.573581 16316 port_mapping.cpp:2597] Removing IP packet filters > with ports [35840,36863] for container with pid 31607 > I0612 17:46:51.575127 16316 port_mapping.cpp:2616] Freed ephemeral ports > [35840,36864) for container with pid 31607 > I0612 17:46:51.588284 16330 cgroups.cpp:1449] Successfullly thawed cgroup > /sys/fs/cgroup/freezer/mesos/d8c48a4a-fdfb-47dd-b8d8-07188c21600d after > 14.503936ms > E0612 17:46:51.622140 16310 containerizer.cpp:480] Failed to clean up an > isolator when destroying orphan container > ddcb8397-3552-44f9-bc99-b5b69aa72944: The ICMP packet filter on host eth0 > does not exist, The ARP packet filter on host eth0 does not exist > W0612 17:46:51.773123 16313 disk.cpp:299] Ignoring cleanup for unknown > container d8c48a4a-fdfb-47dd-b8d8-07188c21600d > I0612 17:46:51.774153 16325 port_mapping.cpp:2597] Removing IP packet filters > with ports [32768,33791] for container with pid 41020 > I0612 17:46:51.775167 16325 port_mapping.cpp:2616] Freed ephemeral ports > [32768,33792) for container with pid 41020 > E0612 17:46:51.817221 16323 containerizer.cpp:480] Failed to clean up an > isolator when destroying orphan container > d8c48a4a-fdfb-47dd-b8d8-07188c21600d: The ICMP packet filter on host eth0 > does not exist, The ARP packet filter on host eth0 does not exist > I0612 17:46:51.872231 16314 cgroups.cpp:1420] Successfully froze cgroup > /sys/fs/cgroup/freezer/mesos/50f9986f-ebbc-440d-86a7-9fa1a7c55a75 after > 308.33792ms > I0612 17:46:51.874572 16314 cgroups.cpp:2394] Thawing cgroup > /sys/fs/cgroup/freezer/mesos/50f9986f-ebbc-440d-86a7-9fa1a7c55a75 > I0612 17:46:51.876566 16314 cgroups.cpp:1449] Successfullly thawed cgroup > /sys/fs/cgroup/freezer/mesos/50f9986f-ebbc-440d-86a7-9fa1a7c55a75 after > 1.593344ms > 2015-06-12 > 17:46:54,833:16307(0x7f172eb07940):ZOO_INFO@auth_completion_func@1286: > Authentication scheme digest succeeded > I0612 17:46:54.835737 16321 group.cpp:385] Trying to create path > '/home/mesos/prod/master' in ZooKeeper > I0612 17:46:54.839110 16321 detector.cpp:138] Detected a new leader: (id='1') > I0612 17:46:54.840276 16330 group.cpp:659] Trying to get > '/home/mesos/prod/master/info_0000000001' in ZooKeeper > I0612 17:46:54.842350 16330 detector.cpp:452] A new leading master > (UPID=master@10.44.14.132:5050) is detected > I0612 17:46:54.843297 16330 slave.cpp:653] New master detected at > master@10.44.14.132:5050 > I0612 17:46:54.843298 16312 status_update_manager.cpp:171] Pausing sending > status updates > I0612 17:46:54.844091 16330 slave.cpp:678] No credentials provided. > Attempting to register without authentication > I0612 17:46:54.845087 16330 slave.cpp:689] Detecting new master > I0612 17:47:01.561920 16309 cgroups.cpp:2394] Thawing cgroup > /sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74 > I0612 17:47:01.564687 16309 cgroups.cpp:1449] Successfullly thawed cgroup > /sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74 after > 1.924096ms > I0612 17:47:01.565467 16309 cgroups.cpp:2377] Freezing cgroup > /sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74 > W0612 17:47:09.978946 16326 disk.cpp:299] Ignoring cleanup for unknown > container 50f9986f-ebbc-440d-86a7-9fa1a7c55a75 > I0612 17:47:09.979818 16327 port_mapping.cpp:2597] Removing IP packet filters > with ports [34816,35839] for container with pid 19805 > I0612 17:47:09.981474 16327 port_mapping.cpp:2616] Freed ephemeral ports > [34816,35840) for container with pid 19805 > E0612 17:47:10.278715 16325 containerizer.cpp:480] Failed to clean up an > isolator when destroying orphan container > 50f9986f-ebbc-440d-86a7-9fa1a7c55a75: The ICMP packet filter on host eth0 > does not exist, The ARP packet filter on host eth0 does not exist > I0612 17:47:11.568151 16326 cgroups.cpp:2394] Thawing cgroup > /sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74 > I0612 17:47:11.570915 16326 cgroups.cpp:1449] Successfullly thawed cgroup > /sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74 after > 1.987072ms > I0612 17:47:11.571728 16326 cgroups.cpp:2377] Freezing cgroup > /sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74 > I0612 17:47:14.549536 16316 slave.cpp:821] Registered with master > master@10.44.14.132:5050; given slave ID > 20150602-190100-2215521290-5050-39399-S23257 > I0612 17:47:14.550220 16318 status_update_manager.cpp:178] Resuming sending > status updates > I0612 17:47:21.574513 16319 cgroups.cpp:2394] Thawing cgroup > /sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74 > I0612 17:47:21.576817 16319 cgroups.cpp:1449] Successfullly thawed cgroup > /sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74 after > 1.587968ms > I0612 17:47:21.577466 16319 cgroups.cpp:2377] Freezing cgroup > /sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74 > I0612 17:47:31.580281 16310 cgroups.cpp:2394] Thawing cgroup > /sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74 > I0612 17:47:31.582365 16310 cgroups.cpp:1449] Successfullly thawed cgroup > /sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74 after > 1.410048ms > I0612 17:47:31.582895 16310 cgroups.cpp:2377] Freezing cgroup > /sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74 > I0612 17:47:41.585619 16322 cgroups.cpp:2394] Thawing cgroup > /sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74 > I0612 17:47:41.587703 16322 cgroups.cpp:1449] Successfullly thawed cgroup > /sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74 after > 1.418752ms > I0612 17:47:41.588436 16322 cgroups.cpp:2377] Freezing cgroup > /sys/fs/cgroup/freezer/mesos/8953fc7f-9fca-4931-b0cb-2f4959ddee74 > I0612 17:47:51.515689 16330 slave.cpp:3733] Current disk usage 11.43%. Max > allowed age: 5.499938588861215days > E0612 17:47:51.557831 16330 containerizer.cpp:468] Failed to destroy orphan > container 8953fc7f-9fca-4931-b0cb-2f4959ddee74: Timed out after 1mins > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)