[GitHub] mesos pull request #265: Update presentations.md

[packtpartner]

Github user packtpartner commented on a diff in the pull request:

https://github.com/apache/mesos/pull/265#discussion_r173075337
  
--- Diff: docs/presentations.md ---
@@ -133,6 +133,10 @@ Presented Summer 2010 at [Yahoo](http://yahoo.com)
 # Apache Mesos - Training
 _(Listed in date descending order)_
 
+## Docker, Apache Mesos & DCOS: Run and manage cloud datacenter 
([Video](https://www.packtpub.com/networking-and-servers/docker-apache-mesos-dcos-run-and-manage-cloud-datacenter-video))
+Manuj Aggarwal, TetraTutorials Team
--- End diff --

I have created another PR without TetraTutorials Team


---

[GitHub] mesos pull request #270: Update presentations.md

[packtpartner]

GitHub user packtpartner opened a pull request:

https://github.com/apache/mesos/pull/270

Update presentations.md

Created another PR without "Tetra Tutorials Team"

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/packtpartner/mesos patch-6

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/mesos/pull/270.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #270


commit 943b5ff7eeea8c016859bf546cd134c506caf881
Author: Packt 
Date:   2018-03-08T06:40:43Z

Update presentations.md




---

MesosCon 2018

[Benjamin Hindman]

Fellow Mesos Community Members,

Many of you have probably seen the announcement for Mesosphere Advance, a
conference co-located at CloudNativeCon. Given the deep involvement that
Mesosphere has had with past MesosCons, this has left many of you asking us
if Mesosphere Advance replaces MesosCon.

No, it does not. Mesosphere Advance is the evolution of DC/OS Day which was
previously co-located with MesosCon and is focused on DC/OS and its
ecosystem. Having said that, Mesos is a big part of the DC/OS ecosystem and
we’ll be featuring Mesos and Mesos related projects prominently at
Mesosphere Advance. We’d love for you and your teams to consider submitting
talk there! Check out
https://www.surveymonkey.com/r/europe18-mesosphereadvance for more info.

So, where does that leave MesosCon? In the past Mesosphere has almost
exclusively managed the organization of MesosCon and going forward we want
it to be more community organized. To that end, we–a group of Mesos users
and developers (including Mesosphere)–are currently in the process of
organizing a MesosCon event and we should have an update soon, so please
stay tuned.

If you have questions, suggestions, or want to get involved in the planning
process, feel free to reach out to us personally!

Ben (PMC Chair) & Joerg (MesosCon Chair)

[GitHub] mesos issue #266: Tasks docs

[bmahler]

Github user bmahler commented on the issue:

https://github.com/apache/mesos/pull/266
  
@kohend can you close this PR based on my comment above?


---

[GitHub] mesos issue #266: Tasks docs

[bmahler]

Github user bmahler commented on the issue:

https://github.com/apache/mesos/pull/266
  
We've removed these docs entirely from the repo since they were very stale 
and they now get generated as part of the website publishing, you can find them 
here:

http://mesos.apache.org/documentation/latest/endpoints/

The way we will address what you're adding here is to provide versioned 
documentation, that work is ongoing here: https://reviews.apache.org/r/52064/ 
cc @tanderegg 


---

Re: Authorization Logging

[Benjamin Mahler]

Are you looking for a reviewer for these? Will Greg have time to review?

On Thu, Mar 1, 2018 at 3:19 AM, Alexander Rojas 
wrote:

> This is a good question on where to do the audit, should it happen in the
> authorization module itself, or in the caller. It doesn’t help that you can
> authorize using approvers or the authorizer or the not so long ago
> introuced acceptors. There are also function wrappers that help to do so.
>
> The feeling we have had in the past is that the authorizer interface was
> created to accomodate the needs of the people writing authorization modules
> but no so much its use inside our code base. That’s why I’ve been working
> in a set of patches to try to clean up a little bit the code that calls
> authorization based on ideas from BenH https://reviews.apache.org/r/65311/
> .
>
> Reviews/comments always welcomed
>
> Alexander Rojas
> alexander.ro...@gmail.com
>
>
>
>
> On 28. Feb 2018, at 23:52, Benjamin Mahler  wrote:
>
> When touching some code, I noticed that authorization logging is currently
> done rather inconsistently across the call-sites and many cases do not log
> the request:
>
> $ grep -R -A 3 'LOG.*Authorizing' src
>
> Should authorization logging be the concern of an authorizer
> implementation? For audit purposes I could imagine this also being part of
> a separate log that the authorizer maintains?
>
> Ben
>
>
>

Re: Reconsidering `allocatable` check in the allocator

[Benjamin Mahler]

+1 about it not being about network traffic.

I the direction we want to head towards is to express and enforce a minimum
granularity for scalar resources. For example:

CPU: 0.001, if we say that we can only deal with milli-cpus.
Disk: 1, if we say that we can only deal with the MB level of disk space
isolation.
GPU: 1, we can't let you consume a portion of a GPU.

Note that this issue is caused by the lack of an "Integer" resource,
because with an "Integer" resource we can just store the value based on the
minimum granularity (e.g. 1 milli-cpus, 1 byte disk, etc). Note also that
with Scalar resources, we currently only support three decimal points of
precision:
https://github.com/apache/mesos/blob/1.5.0/include/mesos/v1/mesos.proto#L1098-L1108

If you were to check the minimum granularity on the input side (e.g.
prevent frameworks from taking 4.0001 of 8 cpus in an offer), then you
don't technically need to prevent the allocator from violating the minimum
granularity (e.g. because we prevent 1.1 or 0.5 GPUs on the input side,
only whole numbers of GPUs will be available).

Re: Filtering discussion

I think this is a separate discussion, worth having, about how to let
frameworks give more information about what they want in order to tame the
churn of offers.

On Wed, Mar 7, 2018 at 9:53 AM, James Peach  wrote:

>
>
> > On Mar 7, 2018, at 5:52 AM, Benjamin Bannier <
> benjamin.bann...@mesosphere.io> wrote:
> >
> > Hi,
> >
> >> Chatted with BenM offline on this. There's another option what both of
> us
> >> agreed that it's probably better than any of the ones mentioned above.
> >>
> >> The idea is to make `allocable` return the portion of the input
> resources
> >> that are allocatable, and strip the unelectable portion.
> >>
> >> For example:
> >> 1) If the input resources are "cpus:0.001,gpus:1", the `allocatable`
> method
> >> will return "gpus:1".
> >> 2) If the input resources are "cpus:1,mem:1", the `allocatable` method
> will
> >> return "cpus:1".
> >> 3) If the input resources are "cpus:0.001,mem:1", the `allocatable`
> method
> >> will return an empty Resources object.
> >>
> >> Basically, the algorithm is like the following:
> >>
> >> allocatable = input
> >> foreach known resource type t: do
> >> r = resources of type t from the input
> >> if r is less than the min resource of type t; then
> >>   allocatable -= r
> >> fi
> >> done
> >> return allocatable
> >
> > I think that sounds like a faithful extension the current behavior to me
> (removing too small resources from the offerable pool), but I feel we
> should not just filter out any resource _kind_  below the minimum, but
> inside a kind all _addable_ subresources,
> >
> >allocatable : Resources = input
> >  for (resource: Resource) in input:
> >if resource < min(resource.kind):
> >  allocatable -= resource
> >
> >return allocatable
> >
> > This would have the effect of clumping together each distinguishable
> resource we care about instead of of accumulating say different disks which
> in sum are potentially not that more interesting to frameworks (they would
> prefer more of a particular disk than smaller pieces scattered across
> multiple disks).
> >
> > @alexr
> >> If we are about to offer some of the resources from a particular agent,
> why
> >> would we filter anything at all? I doubt we should be concerned about
> the
> >> size of the offer representation travelling through the network. If
> >> available resources are "cpus:0.001,gpus:1" and we want to allocate GPU,
> >> what is the benefit of filtering CPU?
> >>
> >> What about the following:
> >> allocatable(R)
> >> {
> >> return true
> >>   iff (there exists r in R for which size(r) > MIN(type(r)))
> >> }
> >
> > I think this is less about communication overhead, but more a tool to
> help to make sure that offered resources are actually useful to frameworks.
>
> I don't know whether there's a JIRA for this, but in the past we've
> proposed the idea of schedulers suppressing or filtering offers with a
> minimum resources specification, i.e. "don't bother me with offers that
> aren't at least X"
>
> J

[Containerization WG] Sync Agenda for March 8th, 2018

[Gilbert Song]

Hi folks,

Tomorrow's WG meeting starts at 9 am PST. We will have 3 agenda items to
discuss:

   - [Andrei, Greg, Gilbert]: Resolve the docker daemon hanging issue and
   do backport.
   - [zhitao] Persistent volume resize support (MESOS-4965
   ).
  - WIP Design Doc
  

   - [Harold] XFS Soft Limit Kill Support (MESOS-6575
   )
  - WIP Design Doc
  


Here is the link for Containerization WG:
https://docs.google.com/document/d/1z55a7tLZFoRWVuUxz1FZwgxkHeugtc2nHR89skFXSpU/edit#

Please join us via Zoom if you are interested: https://zoom.us/j/224160545

Thanks,
Gilbert

Re: Reconsidering `allocatable` check in the allocator

[James Peach]

> On Mar 7, 2018, at 5:52 AM, Benjamin Bannier  
> wrote:
> 
> Hi,
> 
>> Chatted with BenM offline on this. There's another option what both of us
>> agreed that it's probably better than any of the ones mentioned above.
>> 
>> The idea is to make `allocable` return the portion of the input resources
>> that are allocatable, and strip the unelectable portion.
>> 
>> For example:
>> 1) If the input resources are "cpus:0.001,gpus:1", the `allocatable` method
>> will return "gpus:1".
>> 2) If the input resources are "cpus:1,mem:1", the `allocatable` method will
>> return "cpus:1".
>> 3) If the input resources are "cpus:0.001,mem:1", the `allocatable` method
>> will return an empty Resources object.
>> 
>> Basically, the algorithm is like the following:
>> 
>> allocatable = input
>> foreach known resource type t: do
>> r = resources of type t from the input
>> if r is less than the min resource of type t; then
>>   allocatable -= r
>> fi
>> done
>> return allocatable
> 
> I think that sounds like a faithful extension the current behavior to me 
> (removing too small resources from the offerable pool), but I feel we should 
> not just filter out any resource _kind_  below the minimum, but inside a kind 
> all _addable_ subresources,
> 
>allocatable : Resources = input
>  for (resource: Resource) in input:
>if resource < min(resource.kind):
>  allocatable -= resource
> 
>return allocatable
> 
> This would have the effect of clumping together each distinguishable resource 
> we care about instead of of accumulating say different disks which in sum are 
> potentially not that more interesting to frameworks (they would prefer more 
> of a particular disk than smaller pieces scattered across multiple disks).
> 
> @alexr
>> If we are about to offer some of the resources from a particular agent, why
>> would we filter anything at all? I doubt we should be concerned about the
>> size of the offer representation travelling through the network. If
>> available resources are "cpus:0.001,gpus:1" and we want to allocate GPU,
>> what is the benefit of filtering CPU?
>> 
>> What about the following:
>> allocatable(R)
>> {
>> return true
>>   iff (there exists r in R for which size(r) > MIN(type(r)))
>> }
> 
> I think this is less about communication overhead, but more a tool to help to 
> make sure that offered resources are actually useful to frameworks. 

I don't know whether there's a JIRA for this, but in the past we've proposed 
the idea of schedulers suppressing or filtering offers with a minimum resources 
specification, i.e. "don't bother me with offers that aren't at least X"

J

Re: Reconsidering `allocatable` check in the allocator

[Benjamin Bannier]

Hi,

> Chatted with BenM offline on this. There's another option what both of us
> agreed that it's probably better than any of the ones mentioned above.
> 
> The idea is to make `allocable` return the portion of the input resources
> that are allocatable, and strip the unelectable portion.
> 
> For example:
> 1) If the input resources are "cpus:0.001,gpus:1", the `allocatable` method
> will return "gpus:1".
> 2) If the input resources are "cpus:1,mem:1", the `allocatable` method will
> return "cpus:1".
> 3) If the input resources are "cpus:0.001,mem:1", the `allocatable` method
> will return an empty Resources object.
> 
> Basically, the algorithm is like the following:
> 
> allocatable = input
> foreach known resource type t: do
>  r = resources of type t from the input
>  if r is less than the min resource of type t; then
>allocatable -= r
>  fi
> done
> return allocatable

I think that sounds like a faithful extension the current behavior to me 
(removing too small resources from the offerable pool), but I feel we should 
not just filter out any resource _kind_  below the minimum, but inside a kind 
all _addable_ subresources,

allocatable : Resources = input
  for (resource: Resource) in input:
if resource < min(resource.kind):
  allocatable -= resource

return allocatable

This would have the effect of clumping together each distinguishable resource 
we care about instead of of accumulating say different disks which in sum are 
potentially not that more interesting to frameworks (they would prefer more of 
a particular disk than smaller pieces scattered across multiple disks).

@alexr
> If we are about to offer some of the resources from a particular agent, why
> would we filter anything at all? I doubt we should be concerned about the
> size of the offer representation travelling through the network. If
> available resources are "cpus:0.001,gpus:1" and we want to allocate GPU,
> what is the benefit of filtering CPU?
> 
> What about the following:
> allocatable(R)
> {
>  return true
>iff (there exists r in R for which size(r) > MIN(type(r)))
> }

I think this is less about communication overhead, but more a tool to help to 
make sure that offered resources are actually useful to frameworks. If we would 
completely remove the current behavior of clumping resources it might take a 
long time for frameworks to actually receive sufficient interesting resources. 
While frameworks can use filters to prevent some offers, to filter out an offer 
we currently always require that the filtered resources are a superset of the 
resources we are about to offer. As the number of possible dimensions (e.g., 
resource kinds, labels, other fields) increases it becomes harder and harder 
for filters to be effective in this regard and the allocator needs to step in.

https://en.wikipedia.org/wiki/Curse_of_dimensionality


Cheers,

Benjamin

Re: Reconsidering `allocatable` check in the allocator

[Alex Rukletsov]

If we are about to offer some of the resources from a particular agent, why
would we filter anything at all? I doubt we should be concerned about the
size of the offer representation travelling through the network. If
available resources are "cpus:0.001,gpus:1" and we want to allocate GPU,
what is the benefit of filtering CPU?

What about the following:
allocatable(R)
{
  return true
iff (there exists r in R for which size(r) > MIN(type(r)))
}

On Wed, Mar 7, 2018 at 2:41 AM, Qian Zhang  wrote:

> So if the input resources are "cpus:0.001,disk:1024", the `allocatable`
> method will return "disk:1024"? This seems not compatible with the existing
> behavior: with the current implementation of `allocatable`, the same input
> resources will be just skipped because we think "cpus:0.001" is too small
> for framework to launch a task.
>
> allocatable = input
> > foreach known resource type t: do
> >   r = resources of type t from the input
> >   if r is less than the min resource of type t; then
> > allocatable -= r
> >   fi
> > done
> > return allocatable
> >
>
> Are we going to define min amount for each known resource type (including
> disk and gpu)?
>
>
> Regards,
> Qian Zhang
>
> On Wed, Mar 7, 2018 at 6:10 AM, Jie Yu  wrote:
>
> > Chatted with BenM offline on this. There's another option what both of us
> > agreed that it's probably better than any of the ones mentioned above.
> >
> > The idea is to make `allocable` return the portion of the input resources
> > that are allocatable, and strip the unelectable portion.
> >
> > For example:
> > 1) If the input resources are "cpus:0.001,gpus:1", the `allocatable`
> method
> > will return "gpus:1".
> > 2) If the input resources are "cpus:1,mem:1", the `allocatable` method
> will
> > return "cpus:1".
> > 3) If the input resources are "cpus:0.001,mem:1", the `allocatable`
> method
> > will return an empty Resources object.
> >
> > Basically, the algorithm is like the following:
> >
> > allocatable = input
> > foreach known resource type t: do
> >   r = resources of type t from the input
> >   if r is less than the min resource of type t; then
> > allocatable -= r
> >   fi
> > done
> > return allocatable
> >
> > Let me know what do you guys think!
> >
> > Thanks!
> > - Jie
> >
> > On Fri, Mar 2, 2018 at 4:44 PM, Benjamin Mahler 
> > wrote:
> >
> > > I think (2) would need to be:
> > >
> > > bool HierarchicalAllocatorProcess::allocatable(
> > > const Resources& resources)
> > > {
> > >   if (something outside {cpu, mem, disk} is present) return true
> > >   else return true iff at least one of {cpu, mem, disk} has >=
> {MIN_CPU,
> > > MIN_MEM, MIN_DISK}
> > > }
> > >
> > > Otherwise, 1 GPU would be offered but 1GPU + 0.001 CPU would not?
> > >
> > > On Fri, Mar 2, 2018 at 9:27 AM, Jie Yu  wrote:
> > >
> > > > Hi,
> > > >
> > > > The allocatable
> > > >  > > > ator/mesos/hierarchical.cpp#L2471-L2479>
> > > > check in the allocator (shown below) was originally introduced to
> help
> > > > alleviate the situation where a framework receives some resources,
> but
> > no
> > > > cpu/memory, thus cannot launch a task.
> > > >
> > > > bool HierarchicalAllocatorProcess::allocatable(
> > > > const Resources& resources)
> > > > {
> > > >   Option cpus = resources.cpus();
> > > >   Option mem = resources.mem();
> > > >
> > > >   return (cpus.isSome() && cpus.get() >= MIN_CPUS) ||
> > > >  (mem.isSome() && mem.get() >= MIN_MEM);
> > > > }
> > > >
> > > > As pointed by Benjamin in MESOS-7398
> > > > , it now seems to
> > > mainly
> > > > help to minimize the performance overhead from too many small offers
> > > > (instead too small resource amounts are kept out of the offer pool
> > until
> > > > they became accumulated into larger resources).
> > > >
> > > > This check does cause issues when new resources types are introduced.
> > For
> > > > instance, this check does prevent GPU resources alone from being
> > > allocated
> > > > to a framework. There are some other issues we discover MESOS-8626
> > > > .
> > > >
> > > > There are several proposals:
> > > >
> > > > (1) *Completely remove this check*. This check is a heuristic anyway,
> > and
> > > > only applies to a subset of resources (cpu/memory). However, there
> > might
> > > be
> > > > some implication of that change since it's also leveraged to prevent
> > too
> > > > many small offers. *If you are concerned about this approach, please
> > > raise
> > > > your voice.*
> > > >
> > > > (2) *Consider adjust the check to the following. *
> > > >
> > > > bool HierarchicalAllocatorProcess::allocatable(
> > > > const Resources& resources)
> > > > {
> > > >   Option cpus = resources.cpus();
> > > >   Option mem = resources.mem();
> > > >
> > > >   if