Re: Mesos sometimes not allocating the entire cluster

[Klaus Ma]

ight configuration
>>> because I can see there are at least two agent has role with "dev"
>>> configured.
>>>
>>> 56 1363 I0219 18:08:26.284010 28810 hierarchical.hpp:1025] Filtered
>>> ports(dev):[3000-5000]; cpus(dev):10; mem(dev):63488; disk(dev):153600 on
>>> slave 20160112-165226-67375276-5050-22401-S300 for framework
>>> 20160219-164457-67375276-5050-28802-0015
>>> 57 1364 I0219 18:08:26.284162 28810 hierarchical.hpp:941] Allocating
>>> ports(dev):[3000-5000]; cpus(dev):10; mem(dev):63488; disk(dev):153600 on
>>> slave 20160112-165226-67375276-5050-22401-S300 to framework
>>> 20160219-164457-67375276-5050-28802-0014
>>> 58 1365 I0219 18:08:26.286725 28810 hierarchical.hpp:1025] Filtered
>>> ports(dev):[3000-5000]; cpus(dev):10; mem(dev):63488; disk(dev):153600 on
>>> slave 20160112-165226-67375276-5050-22401-S303 for framework
>>> 20160219-164457-67375276-5050-28802-0015
>>> 59 1366 I0219 18:08:26.286875 28810 hierarchical.hpp:941] Allocating
>>> ports(dev):[3000-5000]; cpus(dev):10; mem(dev):63488; disk(dev):153600 on
>>> slave 20160112-165226-67375276-5050-22401-S303 to framework
>>> 20160219-164457-67375276-5050-28802-0014
>>>
>>> Also I think that the framework 20160219-164457-67375276-5050-28802-0014
>>> and 20160219-164457-67375276-5050-28802-0015 may have a high weight cause I
>>> saw that framework  20160219-164457-67375276-5050-28802-0014 get 26 agents
>>> at 18:08:26.
>>>
>>> Another is that some other agents may also have role configured but no
>>> frameworks are configured with the agent role and this caused some agents
>>> have some static reserved resources cannot be allocated.
>>>
>>> I searched 20160112-174949-84152492-5050-19807-S316 in the log and found
>>> that it was allocating the following resources to frameworks:
>>>
>>> 974 2420 I0219 18:08:37.504587 28808 hierarchical.hpp:941] Allocating
>>> ports(*):[3000-5000]; cpus(*):0.5; mem(*):16384; disk(*):51200 on slave
>>> 20160112-174949-84152492-5050-19807-S316 to framework
>>> 20160219-164457-67375276-5050-28802-0014
>>>
>>> This agent should have another 9.5 cpus reserved by some role and no
>>> framework is configured using resources from this role, thus the resources
>>> on this role are wasting.  I think that the following agent may also have
>>> some reserved resources configured:
>>> 20160112-174949-84152492-5050-19807-S317,
>>> 20160112-174949-84152492-5050-19807-S322 and even more agents.
>>>
>>> So I would suggest that you check the master and each slave start
>>> command to see how does role configured. You can also check this via the
>>> command: < curl "http://master-ip:5050/master/state.json; 2>/dev/null|
>>> jq . >  (Note: There is a dot in the end of the command) to get all
>>> slave resources status: reserved, used, total resources etc.
>>>
>>> Thanks,
>>>
>>> Guangya
>>>
>>>
>>> On Mon, Feb 22, 2016 at 5:16 PM, Tom Arnfeld <t...@duedil.com> wrote:
>>>
>>>> No roles, no reservations.
>>>>
>>>> We're using the default filter options with all frameworks and default
>>>> allocation interval.
>>>>
>>>> On 21 Feb 2016, at 08:10, Guangya Liu <gyliu...@gmail.com> wrote:
>>>>
>>>> Hi Tom,
>>>>
>>>> I traced the agent of "20160112-165226-67375276-5050-22401-S199" and
>>>> found that it is keeps declining by many frameworks: once a framework got
>>>> it, the framework will decline it immediately. Does some your framework has
>>>> special offer filter logic?
>>>>
>>>> Also I want to get more for your cluster:
>>>> 1) What is the role for each framework and what is the weight for each
>>>> role?
>>>> 2) Do you start all agents without any reservation?
>>>>
>>>> Thanks,
>>>>
>>>> Guangya
>>>>
>>>> On Sun, Feb 21, 2016 at 9:23 AM, Klaus Ma <klaus1982...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Tom,
>>>>>
>>>>> What's the allocation interval, can you try to reduce filter's timeout
>>>>> of framework?
>>>>>
>>>>> According to the log, ~12 frameworks on cluster with ~42 agents; the
>>>>> filter duration is 5sec, and there're ~60 times filtered in each seconds
>>&g

Re: Mesos sometimes not allocating the entire cluster

[Guangya Liu]

-165226-67375276-5050-22401-S303 for framework
>> 20160219-164457-67375276-5050-28802-0015
>> 59 1366 I0219 18:08:26.286875 28810 hierarchical.hpp:941] Allocating
>> ports(dev):[3000-5000]; cpus(dev):10; mem(dev):63488; disk(dev):153600 on
>> slave 20160112-165226-67375276-5050-22401-S303 to framework
>> 20160219-164457-67375276-5050-28802-0014
>>
>> Also I think that the framework 20160219-164457-67375276-5050-28802-0014
>> and 20160219-164457-67375276-5050-28802-0015 may have a high weight cause I
>> saw that framework  20160219-164457-67375276-5050-28802-0014 get 26 agents
>> at 18:08:26.
>>
>> Another is that some other agents may also have role configured but no
>> frameworks are configured with the agent role and this caused some agents
>> have some static reserved resources cannot be allocated.
>>
>> I searched 20160112-174949-84152492-5050-19807-S316 in the log and found
>> that it was allocating the following resources to frameworks:
>>
>> 974 2420 I0219 18:08:37.504587 28808 hierarchical.hpp:941] Allocating
>> ports(*):[3000-5000]; cpus(*):0.5; mem(*):16384; disk(*):51200 on slave
>> 20160112-174949-84152492-5050-19807-S316 to framework
>> 20160219-164457-67375276-5050-28802-0014
>>
>> This agent should have another 9.5 cpus reserved by some role and no
>> framework is configured using resources from this role, thus the resources
>> on this role are wasting.  I think that the following agent may also have
>> some reserved resources configured:
>> 20160112-174949-84152492-5050-19807-S317,
>> 20160112-174949-84152492-5050-19807-S322 and even more agents.
>>
>> So I would suggest that you check the master and each slave start command
>> to see how does role configured. You can also check this via the command: < 
>> curl
>> "http://master-ip:5050/master/state.json; 2>/dev/null| jq . >  (Note:
>> There is a dot in the end of the command) to get all slave resources
>> status: reserved, used, total resources etc.
>>
>> Thanks,
>>
>> Guangya
>>
>>
>> On Mon, Feb 22, 2016 at 5:16 PM, Tom Arnfeld <t...@duedil.com> wrote:
>>
>>> No roles, no reservations.
>>>
>>> We're using the default filter options with all frameworks and default
>>> allocation interval.
>>>
>>> On 21 Feb 2016, at 08:10, Guangya Liu <gyliu...@gmail.com> wrote:
>>>
>>> Hi Tom,
>>>
>>> I traced the agent of "20160112-165226-67375276-5050-22401-S199" and
>>> found that it is keeps declining by many frameworks: once a framework got
>>> it, the framework will decline it immediately. Does some your framework has
>>> special offer filter logic?
>>>
>>> Also I want to get more for your cluster:
>>> 1) What is the role for each framework and what is the weight for each
>>> role?
>>> 2) Do you start all agents without any reservation?
>>>
>>> Thanks,
>>>
>>> Guangya
>>>
>>> On Sun, Feb 21, 2016 at 9:23 AM, Klaus Ma <klaus1982...@gmail.com>
>>> wrote:
>>>
>>>> Hi Tom,
>>>>
>>>> What's the allocation interval, can you try to reduce filter's timeout
>>>> of framework?
>>>>
>>>> According to the log, ~12 frameworks on cluster with ~42 agents; the
>>>> filter duration is 5sec, and there're ~60 times filtered in each seconds
>>>> (e.g. 65 in 18:08:34). For example, framework 
>>>> (20160219-164457-67375276-5050-28802-0015)
>>>> just get resources from 6 agents and filtered the other 36 agents at
>>>> 18:08:35 (egrep "Alloca|Filtered" mesos-master.log | grep
>>>> "20160219-164457-67375276-5050-28802-0015" | grep "18:08:35")
>>>>
>>>> Thanks
>>>> Klaus
>>>>
>>>> --
>>>> From: t...@duedil.com
>>>> Subject: Re: Mesos sometimes not allocating the entire cluster
>>>> Date: Sat, 20 Feb 2016 16:36:54 +
>>>> To: user@mesos.apache.org
>>>>
>>>> Hi Guangya,
>>>>
>>>> Indeed we have about ~45 agents. I’ve attached the log from the master…
>>>>
>>>>
>>>>
>>>> Hope there’s something here that highlights the issue, we can’t find
>>>> anything that we can’t explain.
>>>>
>>>> Cheers,
>>>>
>>>> Tom.
>>>>
>>>> On 19 Feb 2016, at 03:02, Guangya Liu <g

Re: Mesos sometimes not allocating the entire cluster

[Tom Arnfeld]

ents 
>> have some static reserved resources cannot be allocated.
>> 
>> I searched 20160112-174949-84152492-5050-19807-S316 in the log and found 
>> that it was allocating the following resources to frameworks:
>> 
>> 974 2420 I0219 18:08:37.504587 28808 hierarchical.hpp:941] Allocating 
>> ports(*):[3000-5000]; cpus(*):0.5; mem(*):16384; disk(*):51200 on slave 
>> 20160112-174949-84152492-5050-19807-S316 to framework 
>> 20160219-164457-67375276-5050-28802-0014
>> 
>> This agent should have another 9.5 cpus reserved by some role and no 
>> framework is configured using resources from this role, thus the resources 
>> on this role are wasting.  I think that the following agent may also have 
>> some reserved resources configured: 
>> 20160112-174949-84152492-5050-19807-S317, 
>> 20160112-174949-84152492-5050-19807-S322 and even more agents.
>>  
>> So I would suggest that you check the master and each slave start command to 
>> see how does role configured. You can also check this via the command: < 
>> curl "http://master-ip:5050/master/state.json 
>> <http://master-ip:5050/master/state.json>" 2>/dev/null| jq . >  (Note: There 
>> is a dot in the end of the command) to get all slave resources status: 
>> reserved, used, total resources etc.
>> 
>> Thanks,
>> 
>> Guangya
>> 
>> 
>> On Mon, Feb 22, 2016 at 5:16 PM, Tom Arnfeld <t...@duedil.com 
>> <mailto:t...@duedil.com>> wrote:
>> No roles, no reservations.
>> 
>> We're using the default filter options with all frameworks and default 
>> allocation interval.
>> 
>> On 21 Feb 2016, at 08:10, Guangya Liu <gyliu...@gmail.com 
>> <mailto:gyliu...@gmail.com>> wrote:
>> 
>>> Hi Tom,
>>> 
>>> I traced the agent of "20160112-165226-67375276-5050-22401-S199" and found 
>>> that it is keeps declining by many frameworks: once a framework got it, the 
>>> framework will decline it immediately. Does some your framework has special 
>>> offer filter logic?
>>> 
>>> Also I want to get more for your cluster:
>>> 1) What is the role for each framework and what is the weight for each role?
>>> 2) Do you start all agents without any reservation?
>>> 
>>> Thanks,
>>> 
>>> Guangya 
>>> 
>>> On Sun, Feb 21, 2016 at 9:23 AM, Klaus Ma <klaus1982...@gmail.com 
>>> <mailto:klaus1982...@gmail.com>> wrote:
>>> Hi Tom,
>>> 
>>> What's the allocation interval, can you try to reduce filter's timeout of 
>>> framework?
>>> 
>>> According to the log, ~12 frameworks on cluster with ~42 agents; the filter 
>>> duration is 5sec, and there're ~60 times filtered in each seconds (e.g. 65 
>>> in 18:08:34). For example, framework 
>>> (20160219-164457-67375276-5050-28802-0015) just get resources from 6 agents 
>>> and filtered the other 36 agents at 18:08:35 (egrep "Alloca|Filtered" 
>>> mesos-master.log | grep "20160219-164457-67375276-5050-28802-0015" | grep 
>>> "18:08:35")
>>> 
>>> Thanks
>>> Klaus
>>> 
>>> From: t...@duedil.com <mailto:t...@duedil.com>
>>> Subject: Re: Mesos sometimes not allocating the entire cluster
>>> Date: Sat, 20 Feb 2016 16:36:54 +
>>> To: user@mesos.apache.org <mailto:user@mesos.apache.org>
>>> 
>>> Hi Guangya,
>>> 
>>> Indeed we have about ~45 agents. I’ve attached the log from the master…
>>> 
>>> 
>>> 
>>> Hope there’s something here that highlights the issue, we can’t find 
>>> anything that we can’t explain.
>>> 
>>> Cheers,
>>> 
>>> Tom.
>>> 
>>> On 19 Feb 2016, at 03:02, Guangya Liu <gyliu...@gmail.com 
>>> <mailto:gyliu...@gmail.com>> wrote:
>>> 
>>> Hi Tom,
>>> 
>>> After the patch was applied, there is no need to restart framework but only 
>>> mesos master.
>>> 
>>> One question is that I saw from your log, seems your cluster has at least 
>>> 36 agents, right? I was asking this question because if there are more 
>>> frameworks than agents, frameworks with low weight may not able to get 
>>> resources sometimes.
>>> 
>>> Can you please enable GLOG_v=2 for mesos master for a while and put the log 
>>> somewhere for us to check (Do not enable this for a long time as you will 
>>> get log message flooded), this kind of

Re: Mesos sometimes not allocating the entire cluster

[Guangya Liu]

gt;>
>> We're using the default filter options with all frameworks and default
>> allocation interval.
>>
>> On 21 Feb 2016, at 08:10, Guangya Liu <gyliu...@gmail.com> wrote:
>>
>> Hi Tom,
>>
>> I traced the agent of "20160112-165226-67375276-5050-22401-S199" and
>> found that it is keeps declining by many frameworks: once a framework got
>> it, the framework will decline it immediately. Does some your framework has
>> special offer filter logic?
>>
>> Also I want to get more for your cluster:
>> 1) What is the role for each framework and what is the weight for each
>> role?
>> 2) Do you start all agents without any reservation?
>>
>> Thanks,
>>
>> Guangya
>>
>> On Sun, Feb 21, 2016 at 9:23 AM, Klaus Ma <klaus1982...@gmail.com> wrote:
>>
>>> Hi Tom,
>>>
>>> What's the allocation interval, can you try to reduce filter's timeout
>>> of framework?
>>>
>>> According to the log, ~12 frameworks on cluster with ~42 agents; the
>>> filter duration is 5sec, and there're ~60 times filtered in each seconds
>>> (e.g. 65 in 18:08:34). For example, framework 
>>> (20160219-164457-67375276-5050-28802-0015)
>>> just get resources from 6 agents and filtered the other 36 agents at
>>> 18:08:35 (egrep "Alloca|Filtered" mesos-master.log | grep
>>> "20160219-164457-67375276-5050-28802-0015" | grep "18:08:35")
>>>
>>> Thanks
>>> Klaus
>>>
>>> --
>>> From: t...@duedil.com
>>> Subject: Re: Mesos sometimes not allocating the entire cluster
>>> Date: Sat, 20 Feb 2016 16:36:54 +
>>> To: user@mesos.apache.org
>>>
>>> Hi Guangya,
>>>
>>> Indeed we have about ~45 agents. I’ve attached the log from the master…
>>>
>>>
>>>
>>> Hope there’s something here that highlights the issue, we can’t find
>>> anything that we can’t explain.
>>>
>>> Cheers,
>>>
>>> Tom.
>>>
>>> On 19 Feb 2016, at 03:02, Guangya Liu <gyliu...@gmail.com> wrote:
>>>
>>> Hi Tom,
>>>
>>> After the patch was applied, there is no need to restart framework but
>>> only mesos master.
>>>
>>> One question is that I saw from your log, seems your cluster has at
>>> least 36 agents, right? I was asking this question because if there are
>>> more frameworks than agents, frameworks with low weight may not able to get
>>> resources sometimes.
>>>
>>> Can you please enable GLOG_v=2 for mesos master for a while and put the
>>> log somewhere for us to check (Do not enable this for a long time as you
>>> will get log message flooded), this kind of log messages may give some help
>>> for your problem.
>>>
>>> Another is that there is another problem trying to fix another
>>> performance issue for allocator but may not help you much, but you can
>>> still take a look: https://issues.apache.org/jira/browse/MESOS-4694
>>>
>>> Thanks,
>>>
>>> Guangya
>>>
>>> On Fri, Feb 19, 2016 at 2:19 AM, Tom Arnfeld <t...@duedil.com> wrote:
>>>
>>> Hi Ben,
>>>
>>> We've rolled that patch out (applied over 0.23.1) on our production
>>> cluster and have seen little change, the master is still not sending any
>>> offers to those frameworks. We did this upgrade online, so would there be
>>> any reason the fix wouldn't have helped (other than it not being the
>>> cause)? Would we need to restart the frameworks (so they get new IDs) to
>>> see the effect?
>>>
>>> It's not that the master is never sending them offers, it's that it does
>>> it up to a certain point... for different types of frameworks (all using
>>> libmesos) but then no more, regardless of how much free resource is
>>> available... the free resources are offered to some frameworks, but not
>>> all. Is there any way for us to do more introspection into the state of the
>>> master / allocator to try and debug? Right now we're at a bit of a loss of
>>> where to start diving in...
>>>
>>> Much appreciated as always,
>>>
>>> Tom.
>>>
>>> On 18 February 2016 at 10:21, Tom Arnfeld <t...@duedil.com> wrote:
>>>
>>> Hi Ben,
>>>
>>> I've only just seen your email! Really appreciate the reply, that's
>>> certainly an interesting bu

Re: Mesos sometimes not allocating the entire cluster

[Tom Arnfeld]

ately. Does some your framework has special 
>> offer filter logic?
>> 
>> Also I want to get more for your cluster:
>> 1) What is the role for each framework and what is the weight for each role?
>> 2) Do you start all agents without any reservation?
>> 
>> Thanks,
>> 
>> Guangya 
>> 
>> On Sun, Feb 21, 2016 at 9:23 AM, Klaus Ma <klaus1982...@gmail.com 
>> <mailto:klaus1982...@gmail.com>> wrote:
>> Hi Tom,
>> 
>> What's the allocation interval, can you try to reduce filter's timeout of 
>> framework?
>> 
>> According to the log, ~12 frameworks on cluster with ~42 agents; the filter 
>> duration is 5sec, and there're ~60 times filtered in each seconds (e.g. 65 
>> in 18:08:34). For example, framework 
>> (20160219-164457-67375276-5050-28802-0015) just get resources from 6 agents 
>> and filtered the other 36 agents at 18:08:35 (egrep "Alloca|Filtered" 
>> mesos-master.log | grep "20160219-164457-67375276-5050-28802-0015" | grep 
>> "18:08:35")
>> 
>> Thanks
>> Klaus
>> 
>> From: t...@duedil.com <mailto:t...@duedil.com>
>> Subject: Re: Mesos sometimes not allocating the entire cluster
>> Date: Sat, 20 Feb 2016 16:36:54 +
>> To: user@mesos.apache.org <mailto:user@mesos.apache.org>
>> 
>> Hi Guangya,
>> 
>> Indeed we have about ~45 agents. I’ve attached the log from the master…
>> 
>> 
>> 
>> Hope there’s something here that highlights the issue, we can’t find 
>> anything that we can’t explain.
>> 
>> Cheers,
>> 
>> Tom.
>> 
>> On 19 Feb 2016, at 03:02, Guangya Liu <gyliu...@gmail.com 
>> <mailto:gyliu...@gmail.com>> wrote:
>> 
>> Hi Tom,
>> 
>> After the patch was applied, there is no need to restart framework but only 
>> mesos master.
>> 
>> One question is that I saw from your log, seems your cluster has at least 36 
>> agents, right? I was asking this question because if there are more 
>> frameworks than agents, frameworks with low weight may not able to get 
>> resources sometimes.
>> 
>> Can you please enable GLOG_v=2 for mesos master for a while and put the log 
>> somewhere for us to check (Do not enable this for a long time as you will 
>> get log message flooded), this kind of log messages may give some help for 
>> your problem.
>> 
>> Another is that there is another problem trying to fix another performance 
>> issue for allocator but may not help you much, but you can still take a 
>> look: https://issues.apache.org/jira/browse/MESOS-4694 
>> <https://issues.apache.org/jira/browse/MESOS-4694>
>> 
>> Thanks,
>> 
>> Guangya
>> 
>> On Fri, Feb 19, 2016 at 2:19 AM, Tom Arnfeld <t...@duedil.com 
>> <mailto:t...@duedil.com>> wrote:
>> Hi Ben,
>> 
>> We've rolled that patch out (applied over 0.23.1) on our production cluster 
>> and have seen little change, the master is still not sending any offers to 
>> those frameworks. We did this upgrade online, so would there be any reason 
>> the fix wouldn't have helped (other than it not being the cause)? Would we 
>> need to restart the frameworks (so they get new IDs) to see the effect?
>> 
>> It's not that the master is never sending them offers, it's that it does it 
>> up to a certain point... for different types of frameworks (all using 
>> libmesos) but then no more, regardless of how much free resource is 
>> available... the free resources are offered to some frameworks, but not all. 
>> Is there any way for us to do more introspection into the state of the 
>> master / allocator to try and debug? Right now we're at a bit of a loss of 
>> where to start diving in...
>> 
>> Much appreciated as always,
>> 
>> Tom.
>> 
>> On 18 February 2016 at 10:21, Tom Arnfeld <t...@duedil.com 
>> <mailto:t...@duedil.com>> wrote:
>> Hi Ben,
>> 
>> I've only just seen your email! Really appreciate the reply, that's 
>> certainly an interesting bug and we'll try that patch and see how we get on.
>> 
>> Cheers,
>> 
>> Tom.
>> 
>> On 29 January 2016 at 19:54, Benjamin Mahler <bmah...@apache.org 
>> <mailto:bmah...@apache.org>> wrote:
>> Hi Tom,
>> 
>> I suspect you may be tripping the following issue:
>> https://issues.apache.org/jira/browse/MESOS-4302 
>> <https://issues.apache.org/jira/browse/MESOS-4302>
>> 
>> Please have a read through this and see if it applies here.

Re: Mesos sometimes not allocating the entire cluster

[Guangya Liu]

Hi Tom,

I think that your cluster should have some role, weight configuration
because I can see there are at least two agent has role with "dev"
configured.

56 1363 I0219 18:08:26.284010 28810 hierarchical.hpp:1025] Filtered
ports(dev):[3000-5000]; cpus(dev):10; mem(dev):63488; disk(dev):153600 on
slave 20160112-165226-67375276-5050-22401-S300 for framework
20160219-164457-67375276-5050-28802-0015
57 1364 I0219 18:08:26.284162 28810 hierarchical.hpp:941] Allocating
ports(dev):[3000-5000]; cpus(dev):10; mem(dev):63488; disk(dev):153600 on
slave 20160112-165226-67375276-5050-22401-S300 to framework
20160219-164457-67375276-5050-28802-0014
58 1365 I0219 18:08:26.286725 28810 hierarchical.hpp:1025] Filtered
ports(dev):[3000-5000]; cpus(dev):10; mem(dev):63488; disk(dev):153600 on
slave 20160112-165226-67375276-5050-22401-S303 for framework
20160219-164457-67375276-5050-28802-0015
59 1366 I0219 18:08:26.286875 28810 hierarchical.hpp:941] Allocating
ports(dev):[3000-5000]; cpus(dev):10; mem(dev):63488; disk(dev):153600 on
slave 20160112-165226-67375276-5050-22401-S303 to framework
20160219-164457-67375276-5050-28802-0014

Also I think that the framework 20160219-164457-67375276-5050-28802-0014
and 20160219-164457-67375276-5050-28802-0015 may have a high weight cause I
saw that framework  20160219-164457-67375276-5050-28802-0014 get 26 agents
at 18:08:26.

Another is that some other agents may also have role configured but no
frameworks are configured with the agent role and this caused some agents
have some static reserved resources cannot be allocated.

I searched 20160112-174949-84152492-5050-19807-S316 in the log and found
that it was allocating the following resources to frameworks:

974 2420 I0219 18:08:37.504587 28808 hierarchical.hpp:941] Allocating
ports(*):[3000-5000]; cpus(*):0.5; mem(*):16384; disk(*):51200 on slave
20160112-174949-84152492-5050-19807-S316 to framework
20160219-164457-67375276-5050-28802-0014

This agent should have another 9.5 cpus reserved by some role and no
framework is configured using resources from this role, thus the resources
on this role are wasting.  I think that the following agent may also have
some reserved resources configured:
20160112-174949-84152492-5050-19807-S317,
20160112-174949-84152492-5050-19807-S322 and even more agents.

So I would suggest that you check the master and each slave start command
to see how does role configured. You can also check this via the
command: < curl
"http://master-ip:5050/master/state.json; 2>/dev/null| jq . >  (Note: There
is a dot in the end of the command) to get all slave resources status:
reserved, used, total resources etc.

Thanks,

Guangya


On Mon, Feb 22, 2016 at 5:16 PM, Tom Arnfeld <t...@duedil.com> wrote:

> No roles, no reservations.
>
> We're using the default filter options with all frameworks and default
> allocation interval.
>
> On 21 Feb 2016, at 08:10, Guangya Liu <gyliu...@gmail.com> wrote:
>
> Hi Tom,
>
> I traced the agent of "20160112-165226-67375276-5050-22401-S199" and found
> that it is keeps declining by many frameworks: once a framework got it, the
> framework will decline it immediately. Does some your framework has special
> offer filter logic?
>
> Also I want to get more for your cluster:
> 1) What is the role for each framework and what is the weight for each
> role?
> 2) Do you start all agents without any reservation?
>
> Thanks,
>
> Guangya
>
> On Sun, Feb 21, 2016 at 9:23 AM, Klaus Ma <klaus1982...@gmail.com> wrote:
>
>> Hi Tom,
>>
>> What's the allocation interval, can you try to reduce filter's timeout of
>> framework?
>>
>> According to the log, ~12 frameworks on cluster with ~42 agents; the
>> filter duration is 5sec, and there're ~60 times filtered in each seconds
>> (e.g. 65 in 18:08:34). For example, framework 
>> (20160219-164457-67375276-5050-28802-0015)
>> just get resources from 6 agents and filtered the other 36 agents at
>> 18:08:35 (egrep "Alloca|Filtered" mesos-master.log | grep
>> "20160219-164457-67375276-5050-28802-0015" | grep "18:08:35")
>>
>> Thanks
>> Klaus
>>
>> --
>> From: t...@duedil.com
>> Subject: Re: Mesos sometimes not allocating the entire cluster
>> Date: Sat, 20 Feb 2016 16:36:54 +
>> To: user@mesos.apache.org
>>
>> Hi Guangya,
>>
>> Indeed we have about ~45 agents. I’ve attached the log from the master…
>>
>>
>>
>> Hope there’s something here that highlights the issue, we can’t find
>> anything that we can’t explain.
>>
>> Cheers,
>>
>> Tom.
>>
>> On 19 Feb 2016, at 03:02, Guangya Liu <gyliu...@gmail.com> wrote:
>>
>> Hi Tom,
>>
>

Re: Mesos sometimes not allocating the entire cluster

[Guangya Liu]

Hi Tom,

I traced the agent of "20160112-165226-67375276-5050-22401-S199" and found
that it is keeps declining by many frameworks: once a framework got it, the
framework will decline it immediately. Does some your framework has special
offer filter logic?

Also I want to get more for your cluster:
1) What is the role for each framework and what is the weight for each role?
2) Do you start all agents without any reservation?

Thanks,

Guangya

On Sun, Feb 21, 2016 at 9:23 AM, Klaus Ma <klaus1982...@gmail.com> wrote:

> Hi Tom,
>
> What's the allocation interval, can you try to reduce filter's timeout of
> framework?
>
> According to the log, ~12 frameworks on cluster with ~42 agents; the
> filter duration is 5sec, and there're ~60 times filtered in each seconds
> (e.g. 65 in 18:08:34). For example, framework 
> (20160219-164457-67375276-5050-28802-0015)
> just get resources from 6 agents and filtered the other 36 agents at
> 18:08:35 (egrep "Alloca|Filtered" mesos-master.log | grep
> "20160219-164457-67375276-5050-28802-0015" | grep "18:08:35")
>
> Thanks
> Klaus
>
> ----------
> From: t...@duedil.com
> Subject: Re: Mesos sometimes not allocating the entire cluster
> Date: Sat, 20 Feb 2016 16:36:54 +
> To: user@mesos.apache.org
>
> Hi Guangya,
>
> Indeed we have about ~45 agents. I’ve attached the log from the master…
>
>
>
> Hope there’s something here that highlights the issue, we can’t find
> anything that we can’t explain.
>
> Cheers,
>
> Tom.
>
> On 19 Feb 2016, at 03:02, Guangya Liu <gyliu...@gmail.com> wrote:
>
> Hi Tom,
>
> After the patch was applied, there is no need to restart framework but
> only mesos master.
>
> One question is that I saw from your log, seems your cluster has at least
> 36 agents, right? I was asking this question because if there are more
> frameworks than agents, frameworks with low weight may not able to get
> resources sometimes.
>
> Can you please enable GLOG_v=2 for mesos master for a while and put the
> log somewhere for us to check (Do not enable this for a long time as you
> will get log message flooded), this kind of log messages may give some help
> for your problem.
>
> Another is that there is another problem trying to fix another performance
> issue for allocator but may not help you much, but you can still take a
> look: https://issues.apache.org/jira/browse/MESOS-4694
>
> Thanks,
>
> Guangya
>
> On Fri, Feb 19, 2016 at 2:19 AM, Tom Arnfeld <t...@duedil.com> wrote:
>
> Hi Ben,
>
> We've rolled that patch out (applied over 0.23.1) on our production
> cluster and have seen little change, the master is still not sending any
> offers to those frameworks. We did this upgrade online, so would there be
> any reason the fix wouldn't have helped (other than it not being the
> cause)? Would we need to restart the frameworks (so they get new IDs) to
> see the effect?
>
> It's not that the master is never sending them offers, it's that it does
> it up to a certain point... for different types of frameworks (all using
> libmesos) but then no more, regardless of how much free resource is
> available... the free resources are offered to some frameworks, but not
> all. Is there any way for us to do more introspection into the state of the
> master / allocator to try and debug? Right now we're at a bit of a loss of
> where to start diving in...
>
> Much appreciated as always,
>
> Tom.
>
> On 18 February 2016 at 10:21, Tom Arnfeld <t...@duedil.com> wrote:
>
> Hi Ben,
>
> I've only just seen your email! Really appreciate the reply, that's
> certainly an interesting bug and we'll try that patch and see how we get on.
>
> Cheers,
>
> Tom.
>
> On 29 January 2016 at 19:54, Benjamin Mahler <bmah...@apache.org> wrote:
>
> Hi Tom,
>
> I suspect you may be tripping the following issue:
> https://issues.apache.org/jira/browse/MESOS-4302
>
> Please have a read through this and see if it applies here. You may also
> be able to apply the fix to your cluster to see if that helps things.
>
> Ben
>
> On Wed, Jan 20, 2016 at 10:19 AM, Tom Arnfeld <t...@duedil.com> wrote:
>
> Hey,
>
> I've noticed some interesting behaviour recently when we have lots of
> different frameworks connected to our Mesos cluster at once, all using a
> variety of different shares. Some of the frameworks don't get offered more
> resources (for long periods of time, hours even) leaving the cluster under
> utilised.
>
> Here's an example state where we see this happen..
>
> Framework 1 - 13% (user A)
> Framework 2 - 22% (user B)
> Framework 3 - 4% (user C)
> Framew

RE: Mesos sometimes not allocating the entire cluster

[Klaus Ma]

Hi Tom,

What's the allocation interval, can you try to reduce filter's timeout of 
framework?
According to the log, ~12 frameworks on cluster with ~42 agents; the filter 
duration is 5sec, and there're ~60 times filtered in each seconds (e.g. 65 in 
18:08:34). For example, framework (20160219-164457-67375276-5050-28802-0015) 
just get resources from 6 agents and filtered the other 36 agents at 18:08:35 
(egrep "Alloca|Filtered" mesos-master.log | grep 
"20160219-164457-67375276-5050-28802-0015" | grep "18:08:35")
ThanksKlaus
From: t...@duedil.com
Subject: Re: Mesos sometimes not allocating the entire cluster
Date: Sat, 20 Feb 2016 16:36:54 +
To: user@mesos.apache.org

Hi Guangya,
Indeed we have about ~45 agents. I’ve attached the log from the master…


Hope there’s something here that highlights the issue, we can’t find anything 
that we can’t explain.
Cheers,
Tom.


On 19 Feb 2016, at 03:02, Guangya Liu <gyliu...@gmail.com> wrote:Hi Tom,
After the patch was applied, there is no need to restart framework but only 
mesos master.
One question is that I saw from your log, seems your cluster has at least 36 
agents, right? I was asking this question because if there are more frameworks 
than agents, frameworks with low weight may not able to get resources sometimes.
Can you please enable GLOG_v=2 for mesos master for a while and put the log 
somewhere for us to check (Do not enable this for a long time as you will get 
log message flooded), this kind of log messages may give some help for your 
problem.
Another is that there is another problem trying to fix another performance 
issue for allocator but may not help you much, but you can still take a look: 
https://issues.apache.org/jira/browse/MESOS-4694
Thanks,
Guangya
On Fri, Feb 19, 2016 at 2:19 AM, Tom Arnfeld <t...@duedil.com> wrote:
Hi Ben,
We've rolled that patch out (applied over 0.23.1) on our production cluster and 
have seen little change, the master is still not sending any offers to those 
frameworks. We did this upgrade online, so would there be any reason the fix 
wouldn't have helped (other than it not being the cause)? Would we need to 
restart the frameworks (so they get new IDs) to see the effect?
It's not that the master is never sending them offers, it's that it does it up 
to a certain point... for different types of frameworks (all using libmesos) 
but then no more, regardless of how much free resource is available... the free 
resources are offered to some frameworks, but not all. Is there any way for us 
to do more introspection into the state of the master / allocator to try and 
debug? Right now we're at a bit of a loss of where to start diving in...
Much appreciated as always,
Tom.
On 18 February 2016 at 10:21, Tom Arnfeld <t...@duedil.com> wrote:
Hi Ben,
I've only just seen your email! Really appreciate the reply, that's certainly 
an interesting bug and we'll try that patch and see how we get on.
Cheers,
Tom.
On 29 January 2016 at 19:54, Benjamin Mahler <bmah...@apache.org> wrote:
Hi Tom,
I suspect you may be tripping the following 
issue:https://issues.apache.org/jira/browse/MESOS-4302

Please have a read through this and see if it applies here. You may also be 
able to apply the fix to your cluster to see if that helps things.
Ben
On Wed, Jan 20, 2016 at 10:19 AM, Tom Arnfeld <t...@duedil.com> wrote:
Hey,
I've noticed some interesting behaviour recently when we have lots of different 
frameworks connected to our Mesos cluster at once, all using a variety of 
different shares. Some of the frameworks don't get offered more resources (for 
long periods of time, hours even) leaving the cluster under utilised.
Here's an example state where we see this happen..
Framework 1 - 13% (user A)Framework 2 - 22% (user B)Framework 3 - 4% (user 
C)Framework 4 - 0.5% (user C)
Framework 5 - 1% (user C)
Framework 6 - 1% (user C)
Framework 7 - 1% (user C)
Framework 8 - 0.8% (user C)
Framework 9 - 11% (user D)
Framework 10 - 7% (user C)Framework 11 - 1% (user C)Framework 12 - 1% (user C)
Framework 13 - 6% (user E)
In this example, there's another ~30% of the cluster that is unallocated, and 
it stays like this for a significant amount of time until something changes, 
perhaps another user joins and allocates the rest chunks of this spare 
resource is offered to some of the frameworks, but not all of them.
I had always assumed that when lots of frameworks were involved, eventually the 
frameworks that would keep accepting resources indefinitely would consume the 
remaining resource, as every other framework had rejected the offers.
Could someone elaborate a little on how the DRF allocator / sorter handles this 
situation, is this likely to be related to the different users being used? Is 
there a way to mitigate this?
We're running version 0.23.1.
Cheers,
Tom.








-- 
Guangya Liu (刘光亚)
Senior Software Engineer
DCOS and OpenStack Development
IBM Platform Computing
Systems and Technology Group




  

Re: Mesos sometimes not allocating the entire cluster

[Tom Arnfeld]

Hi Ben,

I've only just seen your email! Really appreciate the reply, that's
certainly an interesting bug and we'll try that patch and see how we get on.

Cheers,

Tom.

On 29 January 2016 at 19:54, Benjamin Mahler  wrote:

> Hi Tom,
>
> I suspect you may be tripping the following issue:
> https://issues.apache.org/jira/browse/MESOS-4302
>
> Please have a read through this and see if it applies here. You may also
> be able to apply the fix to your cluster to see if that helps things.
>
> Ben
>
> On Wed, Jan 20, 2016 at 10:19 AM, Tom Arnfeld  wrote:
>
>> Hey,
>>
>> I've noticed some interesting behaviour recently when we have lots of
>> different frameworks connected to our Mesos cluster at once, all using a
>> variety of different shares. Some of the frameworks don't get offered more
>> resources (for long periods of time, hours even) leaving the cluster under
>> utilised.
>>
>> Here's an example state where we see this happen..
>>
>> Framework 1 - 13% (user A)
>> Framework 2 - 22% (user B)
>> Framework 3 - 4% (user C)
>> Framework 4 - 0.5% (user C)
>> Framework 5 - 1% (user C)
>> Framework 6 - 1% (user C)
>> Framework 7 - 1% (user C)
>> Framework 8 - 0.8% (user C)
>> Framework 9 - 11% (user D)
>> Framework 10 - 7% (user C)
>> Framework 11 - 1% (user C)
>> Framework 12 - 1% (user C)
>> Framework 13 - 6% (user E)
>>
>> In this example, there's another ~30% of the cluster that is unallocated,
>> and it stays like this for a significant amount of time until something
>> changes, perhaps another user joins and allocates the rest chunks of
>> this spare resource is offered to some of the frameworks, but not all of
>> them.
>>
>> I had always assumed that when lots of frameworks were involved,
>> eventually the frameworks that would keep accepting resources indefinitely
>> would consume the remaining resource, as every other framework had rejected
>> the offers.
>>
>> Could someone elaborate a little on how the DRF allocator / sorter
>> handles this situation, is this likely to be related to the different users
>> being used? Is there a way to mitigate this?
>>
>> We're running version 0.23.1.
>>
>> Cheers,
>>
>> Tom.
>>
>
>

Re: Mesos sometimes not allocating the entire cluster

[Benjamin Mahler]

Hi Tom,

I suspect you may be tripping the following issue:
https://issues.apache.org/jira/browse/MESOS-4302

Please have a read through this and see if it applies here. You may also be
able to apply the fix to your cluster to see if that helps things.

Ben

On Wed, Jan 20, 2016 at 10:19 AM, Tom Arnfeld  wrote:

> Hey,
>
> I've noticed some interesting behaviour recently when we have lots of
> different frameworks connected to our Mesos cluster at once, all using a
> variety of different shares. Some of the frameworks don't get offered more
> resources (for long periods of time, hours even) leaving the cluster under
> utilised.
>
> Here's an example state where we see this happen..
>
> Framework 1 - 13% (user A)
> Framework 2 - 22% (user B)
> Framework 3 - 4% (user C)
> Framework 4 - 0.5% (user C)
> Framework 5 - 1% (user C)
> Framework 6 - 1% (user C)
> Framework 7 - 1% (user C)
> Framework 8 - 0.8% (user C)
> Framework 9 - 11% (user D)
> Framework 10 - 7% (user C)
> Framework 11 - 1% (user C)
> Framework 12 - 1% (user C)
> Framework 13 - 6% (user E)
>
> In this example, there's another ~30% of the cluster that is unallocated,
> and it stays like this for a significant amount of time until something
> changes, perhaps another user joins and allocates the rest chunks of
> this spare resource is offered to some of the frameworks, but not all of
> them.
>
> I had always assumed that when lots of frameworks were involved,
> eventually the frameworks that would keep accepting resources indefinitely
> would consume the remaining resource, as every other framework had rejected
> the offers.
>
> Could someone elaborate a little on how the DRF allocator / sorter handles
> this situation, is this likely to be related to the different users being
> used? Is there a way to mitigate this?
>
> We're running version 0.23.1.
>
> Cheers,
>
> Tom.
>

Re: Mesos sometimes not allocating the entire cluster

[Klaus Ma]

Can you share the whole log of master? I'll be helpful :).


Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform OpenSource Technology, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

On Thu, Jan 21, 2016 at 11:57 PM, Tom Arnfeld  wrote:

> Guangya - Nope, there's no outstanding offers for any frameworks, the ones
> that are getting offers are responding properly.
>
> Klaus - This was just a sample of logs for a single agent, the cluster has
> at  least ~40 agents at any one time.
>
> On 21 January 2016 at 15:20, Guangya Liu  wrote:
>
>> Can you please help check if some outstanding offers in cluster which
>> does not accept by any framework? You can check this via the endpoint of
>> /master/state.json endpoint.
>>
>> If there are some outstanding offers, you can start the master with a
>> offer_timeout flag to let master rescind some offers if those offers are
>> not accepted by framework.
>>
>> Cited from
>> https://github.com/apache/mesos/blob/master/docs/configuration.md
>>
>> --offer_timeout=VALUE Duration of time before an offer is rescinded from
>> a framework.
>>
>> This helps fairness when running frameworks that hold on to offers, or
>> frameworks that accidentally drop offers.
>>
>> Thanks,
>>
>> Guangya
>>
>> On Thu, Jan 21, 2016 at 9:44 PM, Tom Arnfeld  wrote:
>>
>>> Hi Klaus,
>>>
>>> Sorry I think I explained this badly, these are the logs for one slave
>>> (that's empty) and we can see that it is making offers to some frameworks.
>>> In this instance, the Hadoop framework (and others) are not among those
>>> getting any offers, they get offered nothing. The allocator is deciding to
>>> send offers in a loop to a certain set of frameworks, starving others.
>>>
>>> On 21 January 2016 at 13:17, Klaus Ma  wrote:
>>>
 Yes, it seems Hadoop framework did not consume all offered resources:
 if framework launch task (1 CPUs) on offer (10 CPUs), the other 9 CPUs will
 return back to master (recoverResouces).

 
 Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
 Platform OpenSource Technology, STG, IBM GCG
 +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

 On Thu, Jan 21, 2016 at 6:46 PM, Tom Arnfeld  wrote:

> Thanks everyone!
>
> Stephan - There's a couple of useful points there, will definitely
> give it a read.
>
> Klaus - Thanks, we're running a bunch of different frameworks, in that
> list there's Hadoop MRv1, Apache Spark, Marathon and a couple of home 
> grown
> frameworks we have. In this particular case the Hadoop framework is the
> major concern, as it's designed to continually accept offers until it has
> enough slots it needs. With the example I gave above, we observe that the
> master is never sending any sizeable offers to some of these frameworks
> (the ones with the larger shares), which is where my confusion stems from.
>
> I've attached a snippet of our active master logs which show the
> activity for a single slave (which has no active executors). We can see
> that it's cycling though sending and recovering declined offers from a
> selection of different frameworks (in order) but I can say that not all of
> the frameworks are receiving these offers, in this case that's the Hadoop
> framework.
>
>
> On 21 January 2016 at 00:26, Klaus Ma  wrote:
>
>> Hi Tom,
>>
>> Which framework are you using, e.g. Swarm, Marathon or something
>> else? and which language package are you using?
>>
>> DRF will sort role/framework by allocation ratio, and offer all
>> "available" resources by slave; but if the resources it too small (<
>> 0.1CPU) or the resources was reject/declined by framework, the resources
>> will not offer it until filter timeout. For example, in Swarm 1.0, the
>> default filter timeout 5s (because of go scheduler API); so here is case
>> that may impact the utilisation: the Swarm got one slave with 16 CPUS, 
>> but
>> only launch one container with 1 CPUS; the other 15 CPUS will return back
>>  to master and did not re-offer until filter timeout (5s).
>> I had pull a request to make Swarm's parameters configurable, refer
>> to https://github.com/docker/swarm/pull/1585. I think you can check
>> this case by master log.
>>
>> If any comments, please let me know.
>>
>> 
>> Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
>> Platform OpenSource Technology, STG, IBM GCG
>> +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
>>
>> On Thu, Jan 21, 2016 at 2:19 AM, Tom Arnfeld  wrote:
>>
>>> Hey,
>>>
>>> I've noticed some interesting behaviour recently when we have lots
>>> of different frameworks connected to our 

Re: Mesos sometimes not allocating the entire cluster

[Tom Arnfeld]

I can’t send the entire log as there’s a lot of activity on the cluster all the 
time, is there anything particular you’re looking for?

> On 22 Jan 2016, at 12:46, Klaus Ma  wrote:
> 
> Can you share the whole log of master? I'll be helpful :).
> 
> 
> Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer 
> Platform OpenSource Technology, STG, IBM GCG 
> +86-10-8245 4084 | klaus1982...@gmail.com  | 
> http://k82.me 
> On Thu, Jan 21, 2016 at 11:57 PM, Tom Arnfeld  > wrote:
> Guangya - Nope, there's no outstanding offers for any frameworks, the ones 
> that are getting offers are responding properly.
> 
> Klaus - This was just a sample of logs for a single agent, the cluster has at 
>  least ~40 agents at any one time.
> 
> On 21 January 2016 at 15:20, Guangya Liu  > wrote:
> Can you please help check if some outstanding offers in cluster which does 
> not accept by any framework? You can check this via the endpoint of 
> /master/state.json endpoint.
> 
> If there are some outstanding offers, you can start the master with a 
> offer_timeout flag to let master rescind some offers if those offers are not 
> accepted by framework.
> 
> Cited from https://github.com/apache/mesos/blob/master/docs/configuration.md 
> 
> 
> --offer_timeout=VALUE Duration of time before an offer is rescinded from a 
> framework.
> This helps fairness when running frameworks that hold on to offers, or 
> frameworks that accidentally drop offers.
> 
> 
> Thanks,
> 
> Guangya
> 
> On Thu, Jan 21, 2016 at 9:44 PM, Tom Arnfeld  > wrote:
> Hi Klaus,
> 
> Sorry I think I explained this badly, these are the logs for one slave 
> (that's empty) and we can see that it is making offers to some frameworks. In 
> this instance, the Hadoop framework (and others) are not among those getting 
> any offers, they get offered nothing. The allocator is deciding to send 
> offers in a loop to a certain set of frameworks, starving others.
> 
> On 21 January 2016 at 13:17, Klaus Ma  > wrote:
> Yes, it seems Hadoop framework did not consume all offered resources: if 
> framework launch task (1 CPUs) on offer (10 CPUs), the other 9 CPUs will 
> return back to master (recoverResouces).
> 
> 
> Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer 
> Platform OpenSource Technology, STG, IBM GCG 
> +86-10-8245 4084  | klaus1982...@gmail.com 
>  | http://k82.me 
> On Thu, Jan 21, 2016 at 6:46 PM, Tom Arnfeld  > wrote:
> Thanks everyone!
> 
> Stephan - There's a couple of useful points there, will definitely give it a 
> read.
> 
> Klaus - Thanks, we're running a bunch of different frameworks, in that list 
> there's Hadoop MRv1, Apache Spark, Marathon and a couple of home grown 
> frameworks we have. In this particular case the Hadoop framework is the major 
> concern, as it's designed to continually accept offers until it has enough 
> slots it needs. With the example I gave above, we observe that the master is 
> never sending any sizeable offers to some of these frameworks (the ones with 
> the larger shares), which is where my confusion stems from.
> 
> I've attached a snippet of our active master logs which show the activity for 
> a single slave (which has no active executors). We can see that it's cycling 
> though sending and recovering declined offers from a selection of different 
> frameworks (in order) but I can say that not all of the frameworks are 
> receiving these offers, in this case that's the Hadoop framework.
> 
> 
> On 21 January 2016 at 00:26, Klaus Ma  > wrote:
> Hi Tom,
> 
> Which framework are you using, e.g. Swarm, Marathon or something else? and 
> which language package are you using?
> 
> DRF will sort role/framework by allocation ratio, and offer all "available" 
> resources by slave; but if the resources it too small (< 0.1CPU) or the 
> resources was reject/declined by framework, the resources will not offer it 
> until filter timeout. For example, in Swarm 1.0, the default filter timeout 
> 5s (because of go scheduler API); so here is case that may impact the 
> utilisation: the Swarm got one slave with 16 CPUS, but only launch one 
> container with 1 CPUS; the other 15 CPUS will return back  to master and did 
> not re-offer until filter timeout (5s).
> I had pull a request to make Swarm's parameters configurable, refer to 
> https://github.com/docker/swarm/pull/1585 
> . I think you can check this case 
> by master log.
> 
> If any comments, please let me know.
> 
> 
> 

Re: Mesos sometimes not allocating the entire cluster

[Klaus Ma]

Yes, it seems Hadoop framework did not consume all offered resources: if
framework launch task (1 CPUs) on offer (10 CPUs), the other 9 CPUs will
return back to master (recoverResouces).


Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform OpenSource Technology, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

On Thu, Jan 21, 2016 at 6:46 PM, Tom Arnfeld  wrote:

> Thanks everyone!
>
> Stephan - There's a couple of useful points there, will definitely give it
> a read.
>
> Klaus - Thanks, we're running a bunch of different frameworks, in that
> list there's Hadoop MRv1, Apache Spark, Marathon and a couple of home grown
> frameworks we have. In this particular case the Hadoop framework is the
> major concern, as it's designed to continually accept offers until it has
> enough slots it needs. With the example I gave above, we observe that the
> master is never sending any sizeable offers to some of these frameworks
> (the ones with the larger shares), which is where my confusion stems from.
>
> I've attached a snippet of our active master logs which show the activity
> for a single slave (which has no active executors). We can see that it's
> cycling though sending and recovering declined offers from a selection of
> different frameworks (in order) but I can say that not all of the
> frameworks are receiving these offers, in this case that's the Hadoop
> framework.
>
>
> On 21 January 2016 at 00:26, Klaus Ma  wrote:
>
>> Hi Tom,
>>
>> Which framework are you using, e.g. Swarm, Marathon or something else?
>> and which language package are you using?
>>
>> DRF will sort role/framework by allocation ratio, and offer all
>> "available" resources by slave; but if the resources it too small (<
>> 0.1CPU) or the resources was reject/declined by framework, the resources
>> will not offer it until filter timeout. For example, in Swarm 1.0, the
>> default filter timeout 5s (because of go scheduler API); so here is case
>> that may impact the utilisation: the Swarm got one slave with 16 CPUS, but
>> only launch one container with 1 CPUS; the other 15 CPUS will return back
>>  to master and did not re-offer until filter timeout (5s).
>> I had pull a request to make Swarm's parameters configurable, refer to
>> https://github.com/docker/swarm/pull/1585. I think you can check this
>> case by master log.
>>
>> If any comments, please let me know.
>>
>> 
>> Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
>> Platform OpenSource Technology, STG, IBM GCG
>> +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
>>
>> On Thu, Jan 21, 2016 at 2:19 AM, Tom Arnfeld  wrote:
>>
>>> Hey,
>>>
>>> I've noticed some interesting behaviour recently when we have lots of
>>> different frameworks connected to our Mesos cluster at once, all using a
>>> variety of different shares. Some of the frameworks don't get offered more
>>> resources (for long periods of time, hours even) leaving the cluster under
>>> utilised.
>>>
>>> Here's an example state where we see this happen..
>>>
>>> Framework 1 - 13% (user A)
>>> Framework 2 - 22% (user B)
>>> Framework 3 - 4% (user C)
>>> Framework 4 - 0.5% (user C)
>>> Framework 5 - 1% (user C)
>>> Framework 6 - 1% (user C)
>>> Framework 7 - 1% (user C)
>>> Framework 8 - 0.8% (user C)
>>> Framework 9 - 11% (user D)
>>> Framework 10 - 7% (user C)
>>> Framework 11 - 1% (user C)
>>> Framework 12 - 1% (user C)
>>> Framework 13 - 6% (user E)
>>>
>>> In this example, there's another ~30% of the cluster that is
>>> unallocated, and it stays like this for a significant amount of time until
>>> something changes, perhaps another user joins and allocates the rest
>>> chunks of this spare resource is offered to some of the frameworks, but not
>>> all of them.
>>>
>>> I had always assumed that when lots of frameworks were involved,
>>> eventually the frameworks that would keep accepting resources indefinitely
>>> would consume the remaining resource, as every other framework had rejected
>>> the offers.
>>>
>>> Could someone elaborate a little on how the DRF allocator / sorter
>>> handles this situation, is this likely to be related to the different users
>>> being used? Is there a way to mitigate this?
>>>
>>> We're running version 0.23.1.
>>>
>>> Cheers,
>>>
>>> Tom.
>>>
>>
>>
>

Re: Mesos sometimes not allocating the entire cluster

[Tom Arnfeld]

Thanks everyone!

Stephan - There's a couple of useful points there, will definitely give it
a read.

Klaus - Thanks, we're running a bunch of different frameworks, in that list
there's Hadoop MRv1, Apache Spark, Marathon and a couple of home grown
frameworks we have. In this particular case the Hadoop framework is the
major concern, as it's designed to continually accept offers until it has
enough slots it needs. With the example I gave above, we observe that the
master is never sending any sizeable offers to some of these frameworks
(the ones with the larger shares), which is where my confusion stems from.

I've attached a snippet of our active master logs which show the activity
for a single slave (which has no active executors). We can see that it's
cycling though sending and recovering declined offers from a selection of
different frameworks (in order) but I can say that not all of the
frameworks are receiving these offers, in this case that's the Hadoop
framework.


On 21 January 2016 at 00:26, Klaus Ma  wrote:

> Hi Tom,
>
> Which framework are you using, e.g. Swarm, Marathon or something else? and
> which language package are you using?
>
> DRF will sort role/framework by allocation ratio, and offer all
> "available" resources by slave; but if the resources it too small (<
> 0.1CPU) or the resources was reject/declined by framework, the resources
> will not offer it until filter timeout. For example, in Swarm 1.0, the
> default filter timeout 5s (because of go scheduler API); so here is case
> that may impact the utilisation: the Swarm got one slave with 16 CPUS, but
> only launch one container with 1 CPUS; the other 15 CPUS will return back
>  to master and did not re-offer until filter timeout (5s).
> I had pull a request to make Swarm's parameters configurable, refer to
> https://github.com/docker/swarm/pull/1585. I think you can check this
> case by master log.
>
> If any comments, please let me know.
>
> 
> Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
> Platform OpenSource Technology, STG, IBM GCG
> +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
>
> On Thu, Jan 21, 2016 at 2:19 AM, Tom Arnfeld  wrote:
>
>> Hey,
>>
>> I've noticed some interesting behaviour recently when we have lots of
>> different frameworks connected to our Mesos cluster at once, all using a
>> variety of different shares. Some of the frameworks don't get offered more
>> resources (for long periods of time, hours even) leaving the cluster under
>> utilised.
>>
>> Here's an example state where we see this happen..
>>
>> Framework 1 - 13% (user A)
>> Framework 2 - 22% (user B)
>> Framework 3 - 4% (user C)
>> Framework 4 - 0.5% (user C)
>> Framework 5 - 1% (user C)
>> Framework 6 - 1% (user C)
>> Framework 7 - 1% (user C)
>> Framework 8 - 0.8% (user C)
>> Framework 9 - 11% (user D)
>> Framework 10 - 7% (user C)
>> Framework 11 - 1% (user C)
>> Framework 12 - 1% (user C)
>> Framework 13 - 6% (user E)
>>
>> In this example, there's another ~30% of the cluster that is unallocated,
>> and it stays like this for a significant amount of time until something
>> changes, perhaps another user joins and allocates the rest chunks of
>> this spare resource is offered to some of the frameworks, but not all of
>> them.
>>
>> I had always assumed that when lots of frameworks were involved,
>> eventually the frameworks that would keep accepting resources indefinitely
>> would consume the remaining resource, as every other framework had rejected
>> the offers.
>>
>> Could someone elaborate a little on how the DRF allocator / sorter
>> handles this situation, is this likely to be related to the different users
>> being used? Is there a way to mitigate this?
>>
>> We're running version 0.23.1.
>>
>> Cheers,
>>
>> Tom.
>>
>
>
0121 10:43:27.513950 22408 hierarchical.hpp:761] Recovered 
ports(*):[3000-5000]; cpus(*):9.5; mem(*):59392; disk(*):51200 (total: 
ports(*):[3000-5000]; cpus(*):9.5; mem(*):59392; disk(*):51200, allocated: ) on 
slave 20151103-233456-100929708-5050-865-S36 from framework 
20160112-165226-67375276-5050-22401-0626
I0121 10:43:28.546314 22409 hierarchical.hpp:761] Recovered 
ports(*):[3000-5000]; cpus(*):9.5; mem(*):59392; disk(*):51200 (total: 
ports(*):[3000-5000]; cpus(*):9.5; mem(*):59392; disk(*):51200, allocated: ) on 
slave 20151103-233456-100929708-5050-865-S36 from framework 
20160112-165226-67375276-5050-22401-0644
I0121 10:43:30.095793 22403 hierarchical.hpp:761] Recovered 
ports(*):[3000-5000]; cpus(*):9.5; mem(*):59392; disk(*):51200 (total: 
ports(*):[3000-5000]; cpus(*):9.5; mem(*):59392; disk(*):51200, allocated: ) on 
slave 20151103-233456-100929708-5050-865-S36 from framework 
20151103-233456-100929708-5050-865-4520
I0121 10:43:31.208264 22406 hierarchical.hpp:761] Recovered 
ports(*):[3000-5000]; cpus(*):9.5; mem(*):59392; disk(*):51200 (total: 
ports(*):[3000-5000]; cpus(*):9.5; mem(*):59392; disk(*):51200, allocated: ) on 
slave 

Re: Mesos sometimes not allocating the entire cluster

[Tom Arnfeld]

Guangya - Nope, there's no outstanding offers for any frameworks, the ones
that are getting offers are responding properly.

Klaus - This was just a sample of logs for a single agent, the cluster has
at  least ~40 agents at any one time.

On 21 January 2016 at 15:20, Guangya Liu  wrote:

> Can you please help check if some outstanding offers in cluster which does
> not accept by any framework? You can check this via the endpoint of
> /master/state.json endpoint.
>
> If there are some outstanding offers, you can start the master with a
> offer_timeout flag to let master rescind some offers if those offers are
> not accepted by framework.
>
> Cited from
> https://github.com/apache/mesos/blob/master/docs/configuration.md
>
> --offer_timeout=VALUE Duration of time before an offer is rescinded from
> a framework.
>
> This helps fairness when running frameworks that hold on to offers, or
> frameworks that accidentally drop offers.
>
> Thanks,
>
> Guangya
>
> On Thu, Jan 21, 2016 at 9:44 PM, Tom Arnfeld  wrote:
>
>> Hi Klaus,
>>
>> Sorry I think I explained this badly, these are the logs for one slave
>> (that's empty) and we can see that it is making offers to some frameworks.
>> In this instance, the Hadoop framework (and others) are not among those
>> getting any offers, they get offered nothing. The allocator is deciding to
>> send offers in a loop to a certain set of frameworks, starving others.
>>
>> On 21 January 2016 at 13:17, Klaus Ma  wrote:
>>
>>> Yes, it seems Hadoop framework did not consume all offered resources: if
>>> framework launch task (1 CPUs) on offer (10 CPUs), the other 9 CPUs will
>>> return back to master (recoverResouces).
>>>
>>> 
>>> Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
>>> Platform OpenSource Technology, STG, IBM GCG
>>> +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
>>>
>>> On Thu, Jan 21, 2016 at 6:46 PM, Tom Arnfeld  wrote:
>>>
 Thanks everyone!

 Stephan - There's a couple of useful points there, will definitely give
 it a read.

 Klaus - Thanks, we're running a bunch of different frameworks, in that
 list there's Hadoop MRv1, Apache Spark, Marathon and a couple of home grown
 frameworks we have. In this particular case the Hadoop framework is the
 major concern, as it's designed to continually accept offers until it has
 enough slots it needs. With the example I gave above, we observe that the
 master is never sending any sizeable offers to some of these frameworks
 (the ones with the larger shares), which is where my confusion stems from.

 I've attached a snippet of our active master logs which show the
 activity for a single slave (which has no active executors). We can see
 that it's cycling though sending and recovering declined offers from a
 selection of different frameworks (in order) but I can say that not all of
 the frameworks are receiving these offers, in this case that's the Hadoop
 framework.


 On 21 January 2016 at 00:26, Klaus Ma  wrote:

> Hi Tom,
>
> Which framework are you using, e.g. Swarm, Marathon or something else?
> and which language package are you using?
>
> DRF will sort role/framework by allocation ratio, and offer all
> "available" resources by slave; but if the resources it too small (<
> 0.1CPU) or the resources was reject/declined by framework, the resources
> will not offer it until filter timeout. For example, in Swarm 1.0, the
> default filter timeout 5s (because of go scheduler API); so here is case
> that may impact the utilisation: the Swarm got one slave with 16 CPUS, but
> only launch one container with 1 CPUS; the other 15 CPUS will return back
>  to master and did not re-offer until filter timeout (5s).
> I had pull a request to make Swarm's parameters configurable, refer to
> https://github.com/docker/swarm/pull/1585. I think you can check this
> case by master log.
>
> If any comments, please let me know.
>
> 
> Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
> Platform OpenSource Technology, STG, IBM GCG
> +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
>
> On Thu, Jan 21, 2016 at 2:19 AM, Tom Arnfeld  wrote:
>
>> Hey,
>>
>> I've noticed some interesting behaviour recently when we have lots of
>> different frameworks connected to our Mesos cluster at once, all using a
>> variety of different shares. Some of the frameworks don't get offered 
>> more
>> resources (for long periods of time, hours even) leaving the cluster 
>> under
>> utilised.
>>
>> Here's an example state where we see this happen..
>>
>> Framework 1 - 13% (user A)
>> Framework 2 - 22% (user B)
>> Framework 3 - 4% (user C)
>> Framework 4 - 

Re: Mesos sometimes not allocating the entire cluster

[Guangya Liu]

Can you please help check if some outstanding offers in cluster which does
not accept by any framework? You can check this via the endpoint of
/master/state.json endpoint.

If there are some outstanding offers, you can start the master with a
offer_timeout flag to let master rescind some offers if those offers are
not accepted by framework.

Cited from https://github.com/apache/mesos/blob/master/docs/configuration.md

--offer_timeout=VALUE Duration of time before an offer is rescinded from a
framework.

This helps fairness when running frameworks that hold on to offers, or
frameworks that accidentally drop offers.

Thanks,

Guangya

On Thu, Jan 21, 2016 at 9:44 PM, Tom Arnfeld  wrote:

> Hi Klaus,
>
> Sorry I think I explained this badly, these are the logs for one slave
> (that's empty) and we can see that it is making offers to some frameworks.
> In this instance, the Hadoop framework (and others) are not among those
> getting any offers, they get offered nothing. The allocator is deciding to
> send offers in a loop to a certain set of frameworks, starving others.
>
> On 21 January 2016 at 13:17, Klaus Ma  wrote:
>
>> Yes, it seems Hadoop framework did not consume all offered resources: if
>> framework launch task (1 CPUs) on offer (10 CPUs), the other 9 CPUs will
>> return back to master (recoverResouces).
>>
>> 
>> Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
>> Platform OpenSource Technology, STG, IBM GCG
>> +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
>>
>> On Thu, Jan 21, 2016 at 6:46 PM, Tom Arnfeld  wrote:
>>
>>> Thanks everyone!
>>>
>>> Stephan - There's a couple of useful points there, will definitely give
>>> it a read.
>>>
>>> Klaus - Thanks, we're running a bunch of different frameworks, in that
>>> list there's Hadoop MRv1, Apache Spark, Marathon and a couple of home grown
>>> frameworks we have. In this particular case the Hadoop framework is the
>>> major concern, as it's designed to continually accept offers until it has
>>> enough slots it needs. With the example I gave above, we observe that the
>>> master is never sending any sizeable offers to some of these frameworks
>>> (the ones with the larger shares), which is where my confusion stems from.
>>>
>>> I've attached a snippet of our active master logs which show the
>>> activity for a single slave (which has no active executors). We can see
>>> that it's cycling though sending and recovering declined offers from a
>>> selection of different frameworks (in order) but I can say that not all of
>>> the frameworks are receiving these offers, in this case that's the Hadoop
>>> framework.
>>>
>>>
>>> On 21 January 2016 at 00:26, Klaus Ma  wrote:
>>>
 Hi Tom,

 Which framework are you using, e.g. Swarm, Marathon or something else?
 and which language package are you using?

 DRF will sort role/framework by allocation ratio, and offer all
 "available" resources by slave; but if the resources it too small (<
 0.1CPU) or the resources was reject/declined by framework, the resources
 will not offer it until filter timeout. For example, in Swarm 1.0, the
 default filter timeout 5s (because of go scheduler API); so here is case
 that may impact the utilisation: the Swarm got one slave with 16 CPUS, but
 only launch one container with 1 CPUS; the other 15 CPUS will return back
  to master and did not re-offer until filter timeout (5s).
 I had pull a request to make Swarm's parameters configurable, refer to
 https://github.com/docker/swarm/pull/1585. I think you can check this
 case by master log.

 If any comments, please let me know.

 
 Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
 Platform OpenSource Technology, STG, IBM GCG
 +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

 On Thu, Jan 21, 2016 at 2:19 AM, Tom Arnfeld  wrote:

> Hey,
>
> I've noticed some interesting behaviour recently when we have lots of
> different frameworks connected to our Mesos cluster at once, all using a
> variety of different shares. Some of the frameworks don't get offered more
> resources (for long periods of time, hours even) leaving the cluster under
> utilised.
>
> Here's an example state where we see this happen..
>
> Framework 1 - 13% (user A)
> Framework 2 - 22% (user B)
> Framework 3 - 4% (user C)
> Framework 4 - 0.5% (user C)
> Framework 5 - 1% (user C)
> Framework 6 - 1% (user C)
> Framework 7 - 1% (user C)
> Framework 8 - 0.8% (user C)
> Framework 9 - 11% (user D)
> Framework 10 - 7% (user C)
> Framework 11 - 1% (user C)
> Framework 12 - 1% (user C)
> Framework 13 - 6% (user E)
>
> In this example, there's another ~30% of the cluster that is
> unallocated, and it stays like this for a significant 

Re: Mesos sometimes not allocating the entire cluster

[Klaus Ma]

Do you mean the only one slave is offered to some framework but the others
are starving?
Mesos allocator (DRF) offer resources by host; so if there's only one host,
the other framework can not get resources. We're have several JIRAs on how
to balance resources between frameworks.



Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform OpenSource Technology, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

On Thu, Jan 21, 2016 at 9:44 PM, Tom Arnfeld  wrote:

> Hi Klaus,
>
> Sorry I think I explained this badly, these are the logs for one slave
> (that's empty) and we can see that it is making offers to some frameworks.
> In this instance, the Hadoop framework (and others) are not among those
> getting any offers, they get offered nothing. The allocator is deciding to
> send offers in a loop to a certain set of frameworks, starving others.
>
> On 21 January 2016 at 13:17, Klaus Ma  wrote:
>
>> Yes, it seems Hadoop framework did not consume all offered resources: if
>> framework launch task (1 CPUs) on offer (10 CPUs), the other 9 CPUs will
>> return back to master (recoverResouces).
>>
>> 
>> Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
>> Platform OpenSource Technology, STG, IBM GCG
>> +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
>>
>> On Thu, Jan 21, 2016 at 6:46 PM, Tom Arnfeld  wrote:
>>
>>> Thanks everyone!
>>>
>>> Stephan - There's a couple of useful points there, will definitely give
>>> it a read.
>>>
>>> Klaus - Thanks, we're running a bunch of different frameworks, in that
>>> list there's Hadoop MRv1, Apache Spark, Marathon and a couple of home grown
>>> frameworks we have. In this particular case the Hadoop framework is the
>>> major concern, as it's designed to continually accept offers until it has
>>> enough slots it needs. With the example I gave above, we observe that the
>>> master is never sending any sizeable offers to some of these frameworks
>>> (the ones with the larger shares), which is where my confusion stems from.
>>>
>>> I've attached a snippet of our active master logs which show the
>>> activity for a single slave (which has no active executors). We can see
>>> that it's cycling though sending and recovering declined offers from a
>>> selection of different frameworks (in order) but I can say that not all of
>>> the frameworks are receiving these offers, in this case that's the Hadoop
>>> framework.
>>>
>>>
>>> On 21 January 2016 at 00:26, Klaus Ma  wrote:
>>>
 Hi Tom,

 Which framework are you using, e.g. Swarm, Marathon or something else?
 and which language package are you using?

 DRF will sort role/framework by allocation ratio, and offer all
 "available" resources by slave; but if the resources it too small (<
 0.1CPU) or the resources was reject/declined by framework, the resources
 will not offer it until filter timeout. For example, in Swarm 1.0, the
 default filter timeout 5s (because of go scheduler API); so here is case
 that may impact the utilisation: the Swarm got one slave with 16 CPUS, but
 only launch one container with 1 CPUS; the other 15 CPUS will return back
  to master and did not re-offer until filter timeout (5s).
 I had pull a request to make Swarm's parameters configurable, refer to
 https://github.com/docker/swarm/pull/1585. I think you can check this
 case by master log.

 If any comments, please let me know.

 
 Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
 Platform OpenSource Technology, STG, IBM GCG
 +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

 On Thu, Jan 21, 2016 at 2:19 AM, Tom Arnfeld  wrote:

> Hey,
>
> I've noticed some interesting behaviour recently when we have lots of
> different frameworks connected to our Mesos cluster at once, all using a
> variety of different shares. Some of the frameworks don't get offered more
> resources (for long periods of time, hours even) leaving the cluster under
> utilised.
>
> Here's an example state where we see this happen..
>
> Framework 1 - 13% (user A)
> Framework 2 - 22% (user B)
> Framework 3 - 4% (user C)
> Framework 4 - 0.5% (user C)
> Framework 5 - 1% (user C)
> Framework 6 - 1% (user C)
> Framework 7 - 1% (user C)
> Framework 8 - 0.8% (user C)
> Framework 9 - 11% (user D)
> Framework 10 - 7% (user C)
> Framework 11 - 1% (user C)
> Framework 12 - 1% (user C)
> Framework 13 - 6% (user E)
>
> In this example, there's another ~30% of the cluster that is
> unallocated, and it stays like this for a significant amount of time until
> something changes, perhaps another user joins and allocates the rest
> chunks of this spare resource is offered to some of the frameworks, but 
> not
> 

Re: Mesos sometimes not allocating the entire cluster

[Erb, Stephan]

Hi Tom,


have you checked if any of the starvation issues explained in the Ebay blog 
post applies to you as well?


http://www.ebaytechblog.com/2014/04/04/delivering-ebays-ci-solution-with-apache-mesos-part-i/?


Best Regards,

Stephan


From: Tom Arnfeld <t...@duedil.com>
Sent: Wednesday, January 20, 2016 7:19 PM
To: user@mesos.apache.org
Subject: Mesos sometimes not allocating the entire cluster

Hey,

I've noticed some interesting behaviour recently when we have lots of different 
frameworks connected to our Mesos cluster at once, all using a variety of 
different shares. Some of the frameworks don't get offered more resources (for 
long periods of time, hours even) leaving the cluster under utilised.

Here's an example state where we see this happen..

Framework 1 - 13% (user A)
Framework 2 - 22% (user B)
Framework 3 - 4% (user C)
Framework 4 - 0.5% (user C)
Framework 5 - 1% (user C)
Framework 6 - 1% (user C)
Framework 7 - 1% (user C)
Framework 8 - 0.8% (user C)
Framework 9 - 11% (user D)
Framework 10 - 7% (user C)
Framework 11 - 1% (user C)
Framework 12 - 1% (user C)
Framework 13 - 6% (user E)

In this example, there's another ~30% of the cluster that is unallocated, and 
it stays like this for a significant amount of time until something changes, 
perhaps another user joins and allocates the rest chunks of this spare 
resource is offered to some of the frameworks, but not all of them.

I had always assumed that when lots of frameworks were involved, eventually the 
frameworks that would keep accepting resources indefinitely would consume the 
remaining resource, as every other framework had rejected the offers.

Could someone elaborate a little on how the DRF allocator / sorter handles this 
situation, is this likely to be related to the different users being used? Is 
there a way to mitigate this?

We're running version 0.23.1.

Cheers,

Tom.

Mesos sometimes not allocating the entire cluster

[Tom Arnfeld]

Hey,

I've noticed some interesting behaviour recently when we have lots of
different frameworks connected to our Mesos cluster at once, all using a
variety of different shares. Some of the frameworks don't get offered more
resources (for long periods of time, hours even) leaving the cluster under
utilised.

Here's an example state where we see this happen..

Framework 1 - 13% (user A)
Framework 2 - 22% (user B)
Framework 3 - 4% (user C)
Framework 4 - 0.5% (user C)
Framework 5 - 1% (user C)
Framework 6 - 1% (user C)
Framework 7 - 1% (user C)
Framework 8 - 0.8% (user C)
Framework 9 - 11% (user D)
Framework 10 - 7% (user C)
Framework 11 - 1% (user C)
Framework 12 - 1% (user C)
Framework 13 - 6% (user E)

In this example, there's another ~30% of the cluster that is unallocated,
and it stays like this for a significant amount of time until something
changes, perhaps another user joins and allocates the rest chunks of
this spare resource is offered to some of the frameworks, but not all of
them.

I had always assumed that when lots of frameworks were involved, eventually
the frameworks that would keep accepting resources indefinitely would
consume the remaining resource, as every other framework had rejected the
offers.

Could someone elaborate a little on how the DRF allocator / sorter handles
this situation, is this likely to be related to the different users being
used? Is there a way to mitigate this?

We're running version 0.23.1.

Cheers,

Tom.

Re: Mesos sometimes not allocating the entire cluster

[Klaus Ma]

Hi Tom,

Which framework are you using, e.g. Swarm, Marathon or something else? and
which language package are you using?

DRF will sort role/framework by allocation ratio, and offer all "available"
resources by slave; but if the resources it too small (< 0.1CPU) or the
resources was reject/declined by framework, the resources will not offer it
until filter timeout. For example, in Swarm 1.0, the default filter timeout
5s (because of go scheduler API); so here is case that may impact the
utilisation: the Swarm got one slave with 16 CPUS, but only launch one
container with 1 CPUS; the other 15 CPUS will return back  to master and
did not re-offer until filter timeout (5s).
I had pull a request to make Swarm's parameters configurable, refer to
https://github.com/docker/swarm/pull/1585. I think you can check this case
by master log.

If any comments, please let me know.


Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform OpenSource Technology, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

On Thu, Jan 21, 2016 at 2:19 AM, Tom Arnfeld  wrote:

> Hey,
>
> I've noticed some interesting behaviour recently when we have lots of
> different frameworks connected to our Mesos cluster at once, all using a
> variety of different shares. Some of the frameworks don't get offered more
> resources (for long periods of time, hours even) leaving the cluster under
> utilised.
>
> Here's an example state where we see this happen..
>
> Framework 1 - 13% (user A)
> Framework 2 - 22% (user B)
> Framework 3 - 4% (user C)
> Framework 4 - 0.5% (user C)
> Framework 5 - 1% (user C)
> Framework 6 - 1% (user C)
> Framework 7 - 1% (user C)
> Framework 8 - 0.8% (user C)
> Framework 9 - 11% (user D)
> Framework 10 - 7% (user C)
> Framework 11 - 1% (user C)
> Framework 12 - 1% (user C)
> Framework 13 - 6% (user E)
>
> In this example, there's another ~30% of the cluster that is unallocated,
> and it stays like this for a significant amount of time until something
> changes, perhaps another user joins and allocates the rest chunks of
> this spare resource is offered to some of the frameworks, but not all of
> them.
>
> I had always assumed that when lots of frameworks were involved,
> eventually the frameworks that would keep accepting resources indefinitely
> would consume the remaining resource, as every other framework had rejected
> the offers.
>
> Could someone elaborate a little on how the DRF allocator / sorter handles
> this situation, is this likely to be related to the different users being
> used? Is there a way to mitigate this?
>
> We're running version 0.23.1.
>
> Cheers,
>
> Tom.
>