Hi all, We’re running Mesos 0.23.0, Marathon 0.11.1 on Centos 6.7 and are seeing a crash in the mesos-slave process when we kill a task (a Docker container, in this case) via the Marathon UI.
What I get from the error logs is this: E0104 10:41:20.233595 9358 slave.cpp:2821] Failed to update resources for container 27115405-8136-4401-a124-2441da5d962b of executor app_bankaccountapp.a089113c-a4f8-11e5-ae6d-005056a02165 running task app_bankaccountapp.a089113c-a4f8-11e5-ae6d-005056a02165 on status update for terminal task, destroying container: Container '27115405-8136-4401-a124-2441da5d962b' not found Context W0104 10:41:20.233652 9358 composing.cpp:513] Container '27115405-8136-4401-a124-2441da5d962b' not found Context F0104 10:41:21.397953 9358 slave.cpp:3486] CHECK_SOME(os::touch(path)): Failed to open file: No such file or directory Context *** Check failure stack trace: *** Context @ 0x7f68edb9451d (unknown) Context @ 0x7f68edb9619d (unknown) Context @ 0x7f68edb940e2 (unknown) Context @ 0x7f68edb96b89 (unknown) Context @ 0x7f68ed4be979 (unknown) Context @ 0x7f68ed6e0abd (unknown) Context @ 0x7f68ed6f994e (unknown) Context @ 0x7f68edb452a4 (unknown) Context @ 0x7f68edb455bf (unknown) Context @ 0x3b39007a51 (unknown) Context @ 0x3b38ce893d (unknown) Context Ordinarily a mesos-slave restart doesn’t cause problems, but when this happens, it seems to be unable to recover the existing tasks and so it causes every process on that slave to restart. This can get tricky when we execute production deployments to multiple services. It’s not 100% reproducible, but we’ve seen it now on Mesos 0.22.1 and 0.23.0. Any help debugging this would be most helpful! Thanks, Scott SCOTT RANKIN VP, Technology Motus, LLC Two Financial Center, 60 South Street, Boston, MA 02111 617.467.1931 (W) | sran...@motus.com<mailto:rcaraf...@motus.com> Follow us on LinkedIn<https://www.linkedin.com/company/motus-llc/> | Visit us at motus.com<http://www.motus.com/> This email message contains information that Motus, LLC considers confidential and/or proprietary, or may later designate as confidential and proprietary. It is intended only for use of the individual or entity named above and should not be forwarded to any other persons or entities without the express consent of Motus, LLC, nor should it be used for any purpose other than in the course of any potential or actual business relationship with Motus, LLC. If the reader of this message is not the intended recipient, or the employee or agent responsible to deliver it to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify sender immediately and destroy the original message. Internal Revenue Service regulations require that certain types of written advice include a disclaimer. To the extent the preceding message contains advice relating to a Federal tax issue, unless expressly stated otherwise the advice is not intended or written to be used, and it cannot be used by the recipient or any other taxpayer, for the purpose of avoiding Federal tax penalties, and was not written to support the promotion or marketing of any transaction or matter discussed herein.
