[ https://issues.apache.org/jira/browse/MESOS-4999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15206475#comment-15206475 ]
Sergey Galkin commented on MESOS-4999: -------------------------------------- I can't delete tasks through Marathon API {code} ╰─➤ cat /tmp/mara { "ids": [ "task.66e562a95c285ce39f37693061a46c2e.394f52eb-f039-11e5-ace3-3cfdfe9c6364", "task.66e562a95c285ce39f37693061a46c2e.394e6888-f039-11e5-ace3-3cfdfe9c6364", "task.66e562a95c285ce39f37693061a46c2e.3950163c-f039-11e5-ace3-3cfdfe9c6364", "task.66e562a95c285ce39f37693061a46c2e.394eddb9-f039-11e5-ace3-3cfdfe9c6364", "task.66e562a95c285ce39f37693061a46c2e.394e1a67-f039-11e5-ace3-3cfdfe9c6364", "task.66e562a95c285ce39f37693061a46c2e.3c3e5c8d-f039-11e5-ace3-3cfdfe9c6364", "task.66e562a95c285ce39f37693061a46c2e.394df356-f039-11e5-ace3-3cfdfe9c6364", "task.66e562a95c285ce39f37693061a46c2e.394d7e24-f039-11e5-ace3-3cfdfe9c6364", "task.66e562a95c285ce39f37693061a46c2e.394dcc45-f039-11e5-ace3-3cfdfe9c6364", "task.66e562a95c285ce39f37693061a46c2e.394f04ca-f039-11e5-ace3-3cfdfe9c6364" ] } {code} {code} ╰─➤ curl -v -XPOST -d@/tmp/mara -H Content-Type:application/json http://172.20.8.34:8080/env-testing/marathon/v2/tasks/delete Note: Unnecessary use of -X or --request, POST is already inferred. * Trying 172.20.8.34... * Connected to 172.20.8.34 (172.20.8.34) port 8080 (#0) > POST /env-testing/marathon/v2/tasks/delete HTTP/1.1 > Host: 172.20.8.34:8080 > User-Agent: curl/7.47.1 > Accept: */* > Content-Type:application/json > Content-Length: 825 > * upload completely sent off: 825 out of 825 bytes < HTTP/1.1 200 OK < Server: nginx/1.4.6 (Ubuntu) < Date: Tue, 22 Mar 2016 14:51:26 GMT < Content-Type: application/json; qs=2 < Transfer-Encoding: chunked < Connection: keep-alive < X-Marathon-Leader: http://172.20.9.51:8080 < Cache-Control: no-cache, no-store, must-revalidate < Pragma: no-cache < Expires: 0 < * Connection #0 to host 172.20.8.34 left intact {"tasks":[]} {code} {code} ╰─➤ curl http://172.20.8.34:8080/env-testing/marathon/v2/tasks {"tasks":[{"id":"66e562a95c285ce39f37693061a46c2e.f2b796f3-f03c-11e5-ace3-3cfdfe9c6364","host":"172.20.9.200","ipAddresses":[],"ports":[5692],"startedAt":"2016-03-22T14:47:07.477Z","stagedAt":"2016-03-22T14:47:06.549Z","version":"2016-03-17T14:36:47.536Z","slaveId":"5445dbdc-c58a-4f78-aef2-9ab129a640fa-S98","appId":"/66e562a95c285ce39f37693061a46c2e","servicePorts":[10498]},{"id":"66e562a95c285ce39f37693061a46c2e.f2b7e516-f03c-11e5-ace3-3cfdfe9c6364","host":"172.20.9.231","ipAddresses":[],"ports":[23564],"startedAt":"2016-03-22T14:47:07.454Z","stagedAt":"2016-03-22T14:47:06.552Z","version":"2016-03-17T14:36:47.536Z","slaveId":"5445dbdc-c58a-4f78-aef2-9ab129a640fa-S162","appId":"/66e562a95c285ce39f37693061a46c2e","servicePorts":[10498]},{"id":"66e562a95c285ce39f37693061a46c2e.f2b76fe2-f03c-11e5-ace3-3cfdfe9c6364","host":"172.20.9.210","ipAddresses":[],"ports":[23968],"startedAt":"2016-03-22T14:47:07.395Z","stagedAt":"2016-03-22T14:47:06.548Z","version":"2016-03-17T14:36:47.536Z","slaveId":"5445dbdc-c58a-4f78-aef2-9ab129a640fa-S159","appId":"/66e562a95c285ce39f37693061a46c2e","servicePorts":[10498]},{"id":"66e562a95c285ce39f37693061a46c2e.f2b80c27-f03c-11e5-ace3-3cfdfe9c6364","host":"172.20.9.230","ipAddresses":[],"ports":[26219],"startedAt":"2016-03-22T14:47:07.478Z","stagedAt":"2016-03-22T14:47:06.552Z","version":"2016-03-17T14:36:47.536Z","slaveId":"5445dbdc-c58a-4f78-aef2-9ab129a640fa-S157","appId":"/66e562a95c285ce39f37693061a46c2e","servicePorts":[10498]},{"id":"66e562a95c285ce39f37693061a46c2e.f2b6ac8e-f03c-11e5-ace3-3cfdfe9c6364","host":"172.20.9.76","ipAddresses":[],"ports":[48380],"startedAt":"2016-03-22T14:47:07.405Z","stagedAt":"2016-03-22T14:47:06.544Z","version":"2016-03-17T14:36:47.536Z","slaveId":"5445dbdc-c58a-4f78-aef2-9ab129a640fa-S174","appId":"/66e562a95c285ce39f37693061a46c2e","servicePorts":[10498]},{"id":"66e562a95c285ce39f37693061a46c2e.f2b7be04-f03c-11e5-ace3-3cfdfe9c6364","host":"172.20.9.107","ipAddresses":[],"ports":[55007],"startedAt":"2016-03-22T14:47:07.392Z","stagedAt":"2016-03-22T14:47:06.550Z","version":"2016-03-17T14:36:47.536Z","slaveId":"5445dbdc-c58a-4f78-aef2-9ab129a640fa-S146","appId":"/66e562a95c285ce39f37693061a46c2e","servicePorts":[10498]},{"id":"66e562a95c285ce39f37693061a46c2e.f2b7be05-f03c-11e5-ace3-3cfdfe9c6364","host":"172.20.9.221","ipAddresses":[],"ports":[46224],"startedAt":"2016-03-22T14:47:07.498Z","stagedAt":"2016-03-22T14:47:06.551Z","version":"2016-03-17T14:36:47.536Z","slaveId":"5445dbdc-c58a-4f78-aef2-9ab129a640fa-S80","appId":"/66e562a95c285ce39f37693061a46c2e","servicePorts":[10498]},{"id":"66e562a95c285ce39f37693061a46c2e.f2b6d39f-f03c-11e5-ace3-3cfdfe9c6364","host":"172.20.9.140","ipAddresses":[],"ports":[41310],"startedAt":"2016-03-22T14:47:07.433Z","stagedAt":"2016-03-22T14:47:06.545Z","version":"2016-03-17T14:36:47.536Z","slaveId":"5445dbdc-c58a-4f78-aef2-9ab129a640fa-S158","appId":"/66e562a95c285ce39f37693061a46c2e","servicePorts":[10498]},{"id":"66e562a95c285ce39f37693061a46c2e.f2b721c1-f03c-11e5-ace3-3cfdfe9c6364","host":"172.20.9.110","ipAddresses":[],"ports":[26690],"startedAt":"2016-03-22T14:47:07.408Z","stagedAt":"2016-03-22T14:47:06.547Z","version":"2016-03-17T14:36:47.536Z","slaveId":"5445dbdc-c58a-4f78-aef2-9ab129a640fa-S7","appId":"/66e562a95c285ce39f37693061a46c2e","servicePorts":[10498]},{"id":"66e562a95c285ce39f37693061a46c2e.f2b6fab0-f03c-11e5-ace3-3cfdfe9c6364","host":"172.20.9.106","ipAddresses":[],"ports":[9516],"startedAt":"2016-03-22T14:47:07.449Z","stagedAt":"2016-03-22T14:47:06.546Z","version":"2016-03-17T14:36:47.536Z","slaveId":"5445dbdc-c58a-4f78-aef2-9ab129a640fa-S53","appId":"/66e562a95c285ce39f37693061a46c2e","servicePorts":[10498]}]} {code} > Mesos (or Marathon) lost tasks > ------------------------------ > > Key: MESOS-4999 > URL: https://issues.apache.org/jira/browse/MESOS-4999 > Project: Mesos > Issue Type: Bug > Affects Versions: 0.27.2 > Environment: mesos - 0.27.0 > marathon - 0.15.2 > 189 mesos slaves with Ubuntu 14.04.2 on HP ProLiant DL380 Gen9, > CPU - 2 x Intel(R) Xeon(R) CPU E5-2680 v3 @2.50GHz (48 cores (with > hyperthreading)) > RAM - 264G, > Storage - 3.0T on RAID on HP Smart Array P840 Controller, > HDD - 12 x HP EH0600JDYTL > Network - 2 x Intel Corporation Ethernet 10G 2P X710, > Reporter: Sergey Galkin > Attachments: mesos-nodes.png > > > After a lot of create/delete application with docker instances through > Marathon API I have a lot of lost tasks after last *deleting all application > in Marathon*. > They are divided into three types > 1. Tasks hangs in STAGED status. I don't see this tasks in 'docker ps' on the > slave and _service docker restart_ on mesos slave did not fix these tasks. > 2. RUNNING because docker hangs and can't delete these instances (a lot of > {code} > Killing docker task > Shutting down > Killing docker task > Shutting down > {code} > in stdout, > _docker stop ID_ hangs and these tasks can be fixed by _service docker > restart_ on mesos slave. > 3. RUNNING after _service docker restart_ on mesos slave. > Screenshot attached -- This message was sent by Atlassian JIRA (v6.3.4#6332)