Re: Implementing an Executor

2014-11-20 Thread Tim Chen
Hi Janet, Can you elaborate more what you like to get back from the docker container that you launched? Thanks, Tim On Wed, Nov 19, 2014 at 5:22 PM, Tom Arnfeld t...@duedil.com wrote: Hi Janet, Oh sorry my mistake, I didn't read your email correctly, I thought you were using the

Re: Implementing an Executor

2014-11-20 Thread David Greenberg
One cool feature of the docker containerizer is that you can actually launch your executor inside the docker image, so that you can just layer the executor's custom logic on top of whatever container you desire. This way, you can more easily control what's happening in the docker image, and still

Re: Implementing an Executor

2014-11-20 Thread Janet Borschowa
Hi Tom, Thank you - this helps a lot! Janet On Wed, Nov 19, 2014 at 5:22 PM, Tom Arnfeld t...@duedil.com wrote: Hi Janet, Oh sorry my mistake, I didn't read your email correctly, I thought you were using the containerizer. What you're doing here is actually going to be quite difficult to

Re: Implementing an Executor

2014-11-20 Thread Janet Borschowa
Hi Tim, I'd like to get the NetworkSetting's IPAddress and Ports from the launched docker container right now. But there is other information that I'll probably need in the future such as the Volumes. What would be ideal is if one could get the information about the docker container that is

Re: Implementing an Executor

2014-11-20 Thread Janet Borschowa
Hi David, Thanks for your suggestion. This approach sounds promising for what I need to do. I'm going to have to try this out. Thanks! Janet -- Janet Borschowa CodeFutures Corporation On Thu, Nov 20, 2014 at 5:04 AM, David Greenberg dsg123456...@gmail.com wrote: One cool feature of the

Re: Issue with cgroups_limit_swap

2014-11-20 Thread Anton Lindstrom
Hi, I'm afraid we're not using it. The intention was to use the feature but we switched to docker instead, so there was never a production testing of the feature. On Wed, Nov 19, 2014 at 7:15 PM, Ian Downes idow...@twitter.com wrote: This code was added a few months ago by Anton Lindström who I

Re: Master memory usage

2014-11-20 Thread Benjamin Mahler
It shouldn't be that high, especially with the size of the cluster I see in your stats. Which scheduler(s) are you running, and do they create large TaskInfo objects? Just a hunch, as I do not recall any leaks in 0.19.1. On Tue, Nov 18, 2014 at 1:00 AM, Tom Arnfeld t...@duedil.com wrote: I've

Re: Master memory usage

2014-11-20 Thread Tom Arnfeld
That's what I thought. There around 2500 tasks launched with this master, most of which will be by our Hadoop JT. The Hadoop framework ships the configuration for the TT using the TaskInfo.data property, and that looks to be about 80K per task. Any debugging suggestions? -- Tom Arnfeld

Re: Master memory usage

2014-11-20 Thread Benjamin Mahler
Have you done the math on number of tasks * size of task? We didn't wipe the .data field in 0.19.1: https://issues.apache.org/jira/browse/MESOS-1746 On Thu, Nov 20, 2014 at 2:51 PM, Tom Arnfeld t...@duedil.com wrote: That's what I thought. There around 2500 tasks launched with this master,