Hello dev@ We still didn't found any solution, that's why I decided to prepare environment showing that problem: https://github.com/klocekPL/mesos-ec-debug and simpler External Containerizer Program that runs mesos-executor without any container to simplify code: https://github.com/klocekPL/mesos-ec-debug/blob/master/ecp/ecp.py but show general idea in our solution.
Provisioning git clone https://github.com/klocekPL/mesos-ec-debug cd mesos-ec-debug vagrant up Default configuration: 1 x mesos-master (1CPU, 512MB) + 1 x mesos-slave (1CPU, 1GB) Mesos Master WebUI http://192.168.255.10:5050 Marathon WebUI http://192.168.255.10:8080 Steps to reproduce problem 1. Add task curl -X POST -H "Content-Type: application/json" http://192.168.255.10:8080/v2/apps -d ' { "id": "sleep", "cmd": "sleep 1000000", "instances": 2, "cpus": 0.1, "mem": 16 }' 2. Check Mesos-master console: http://192.168.255.10:5050 Should work fine. 3. Scale up to 16+ instances (per node) 4. See how Mesos-slave will die :( * env tested on Vagrant 1.7.x and VirtualBox 5.x I'm open for any suggestions, thanks in advance! PS: I found that someone had the same problem: https://issues.apache.org/jira/browse/MESOS-1473 Regards, Łukasz On 28 October 2015 at 15:58, Łukasz Adamczyk <[email protected]> wrote: > First thing: External Containerizer will be deprecated. I know that and we > are open for moving that functionality to module, but at this moment we > need to use External Containerizer that should work, am I right? > > Second thing: We decided to make our External Containerizer and Executor > open source: https://github.com/AVSystem/mesoslxc > > Does anyone have any idea what can be wrong? > > Regards, > Łukasz > > On 21 October 2015 at 16:19, haosdent <[email protected]> wrote: > >> seems could not found the implemention of your External Containerizer. >> -> seems >> could not found the implemention of your External Containerizer in email. >> >> On Wed, Oct 21, 2015 at 10:19 PM, haosdent <[email protected]> wrote: >> >> > Sounds like your External Containerizer have problems and seems could >> not >> > found the implemention of your External Containerizer. By the way, do >> you >> > know External Containerizer is deprecated and we would let Containerizer >> > become module. http://search-hadoop.com/m/0Vlr6yasmiJToni >> > >> > On Wed, Oct 21, 2015 at 10:06 PM, Łukasz Adamczyk < >> > [email protected]> wrote: >> > >> >> Same problem with /help http://pastebin.com/2KUPDWv9 Of course when I >> use >> >> Mesos or Docker Containerizer it works well, but we need our custom LXC >> >> based containers that's why we need External Containerizer. >> >> >> >> Thanks for reply, >> >> Łukasz >> >> >> >> On 21 October 2015 at 15:38, haosdent <[email protected]> wrote: >> >> >> >> > And is it possible for you to use MesosContainerizer or >> >> DockerContainerizer >> >> > instead of ExternalContainerizer? >> >> > >> >> > On Wed, Oct 21, 2015 at 9:34 PM, haosdent <[email protected]> >> wrote: >> >> > >> >> > > Could "curl -vv http://10.0.0.184:5051/help"; success? >> >> > > >> >> > > On Wed, Oct 21, 2015 at 9:30 PM, Łukasz Adamczyk < >> >> > [email protected] >> >> > > > wrote: >> >> > > >> >> > >> Hello, >> >> > >> >> >> > >> Any ideas? Maybe someone can suggest me what I should check? IMHO >> >> > >> mesos-agent hangs, it accepts new TCP connections (port 5051), so >> >> > service >> >> > >> is still listening, but suddenly stopped sending HTTP responses: >> >> > >> http://pastebin.com/QsyMft7j >> >> > >> >> >> > >> Regards, >> >> > >> Łukasz >> >> > >> >> >> > >> On 19 October 2015 at 18:05, Łukasz Adamczyk < >> >> [email protected]> >> >> > >> wrote: >> >> > >> >> >> > >> > Hello, >> >> > >> > >> >> > >> > TL;DR; >> >> > >> > >> >> > >> > I have odd problem with Mesos Agent + External Containerizer: >> I'm >> >> able >> >> > >> to >> >> > >> > schedule only 14 tasks per agent (sic!), after reaching that >> limit >> >> my >> >> > >> > mesos-agent ignores mesos.internal.RunTaskMessage from >> >> mesos-master. >> >> > >> > >> >> > >> > More details: >> >> > >> > >> >> > >> > Web UI with 14/20 tasks with RUNNING status: >> >> > >> > http://foobar.krakow.pl/mesosproblem/mesos-master-ui1.png >> >> > >> > mesos-agent output with GLOG_V=20 (short version): >> >> > >> > http://foobar.krakow.pl/mesosproblem/mesos-slave-tldr.txt >> >> > >> > mesos-agent output with GLOG_V=20 (full version - 0.8MB): >> >> > >> > http://foobar.krakow.pl/mesosproblem/mesos-slave.txt >> >> > >> > mesos-master output with GLOG_V=20 (full version - 1.8MB): >> >> > >> > http://foobar.krakow.pl/mesosproblem/mesos-master.txt >> >> > >> > Web UI after mesos.internal.ShutdownMessage (due to health check >> >> > >> timeout): >> >> > >> > http://foobar.krakow.pl/mesosproblem/mesos-master-ui2.png >> >> > >> > >> >> > >> > Tested on mesos 0.25 and 0.23, with Marathon 0.10 + External >> >> > >> Containerizer >> >> > >> > + Custom Executor (libmesos 0.23). Can you suggest me what is >> >> wrong? >> >> > >> Thanks >> >> > >> > in advance! >> >> > >> > >> >> > >> > Regards, >> >> > >> > Łukasz >> >> > >> > >> >> > >> > -- >> >> > >> > Łukasz Adamczyk >> >> > >> > >> >> > >> >> >> > >> -- >> >> > >> Łukasz Adamczyk >> >> > >> >> >> > > >> >> > > >> >> > > >> >> > > -- >> >> > > Best Regards, >> >> > > Haosdent Huang >> >> > > >> >> > >> >> > >> >> > >> >> > -- >> >> > Best Regards, >> >> > Haosdent Huang >> >> > >> >> >> >> >> >> >> >> -- >> >> Łukasz Adamczyk >> >> >> > >> > >> > >> > -- >> > Best Regards, >> > Haosdent Huang >> > >> >> >> >> -- >> Best Regards, >> Haosdent Huang >> > > > > -- > Łukasz Adamczyk > -- Łukasz Adamczyk
