Re: Order of includes

[Benjamin Bannier]

Hi Yan,

I don’t feel too strongly about most of our style rules regarding include 
ordering since they are just about style.  

> For a cpp file foo.cpp, our style guide instructs folks to put the header
> foo.hpp at the top of the include list:
> https://github.com/apache/mesos/blob/master/docs/c%2B%2B-style-guide.md#order-of-includes
> 
> This is consistent with Google style guide but in reality most of the our
> files follow the rule of "treat foo.hpp the same way as other project
> headers”.

Among all our style rules regarding includes, this one actually does have a 
solid technical justification: It helps ensure that a header file always 
includes all symbols it requires (OK, possibly via discouraged transitive 
includes in the header itself). Not strictly following this rule has lead to 
broken header files making their way into the code base (both in the case of 
internal and public headers), see e.g.,

  https://reviews.apache.org/r/54083/
  https://reviews.apache.org/r/54084/
  https://reviews.apache.org/r/54083/

I’d rather have us follow a style that performs some automagic checking of 
header completeness than rely on humans to catch all issues.

Note that including `foo.hpp` first in `foo.cpp` is common practice, and I 
expect following this rule would lead to _less friction_ for newcomers to the 
Mesos code base, see e.g., (no particular order)

  http://llvm.org/docs/CodingStandards.html#include-style
  
https://github.com/bloomberg/bde/wiki/physical-code-organization#component-design-rules
  https://webkit.org/code-style-guidelines/#include-statements
  
https://github.com/facebook/hhvm/blob/master/hphp/doc/coding-conventions.md#what-to-include
  https://google.github.io/styleguide/cppguide.html#Names_and_Order_of_Includes


Cheers,

Benjamin

Re: Question about mesos authentication and authorization?

[Yu Wei]

Hi Greg,


Thanks for the information.


Jared


From: Greg Mann 
Sent: Wednesday, December 14, 2016 9:18:53 AM
To: dev
Subject: Re: Question about mesos authentication and authorization?

Hi Jared,
The default authenticator modules included with Mesos only support storing
the credentials in a file on disk. However, Mesos exposes an authenticator
module interface which can be used to implement more complex authentication
scenarios. Note that there are authenticator module interfaces for both the
V0 API [1] and the HTTP V1 API [2]. You can find an introduction to Mesos
modules in the docs [3].

Some Mesos users have implemented support for systems like LDAP and
Kerberos. DC/OS Enterprise is one example of a system with LDAP support,
but that functionality is proprietary so the module source-code isn't
freely available.

Cheers,
Greg

[1] https://github.com/apache/mesos/blob/master/include/meso
s/authentication/authenticator.hpp
[2] https://github.com/apache/mesos/blob/master/3rdparty/lib
process/include/process/authenticator.hpp
[3] http://mesos.apache.org/documentation/latest/modules/



On Mon, Dec 12, 2016 at 6:39 PM, Yu Wei  wrote:

> Hi folks,
>
>
> One question about mesos authentication and authorization.
>
> Where could I store the credentials? In documentation, it seems the
> credentials is stored in local disk.
>
> Is there any other approach to store and manage credentials via LDAP or
> other 3rd party software?
>
>
> Thanks,
>
> Jared, (??)
> Software developer
> Interested in open source software, big data, Linux
>

Re: [webui] Started show wrong time

[haosdent]

Hi, @Tomek. Thank you for your report. Create a ticket at
https://issues.apache.org/jira/browse/MESOS-6790, feel free to post a patch
if you have idea for it.

On Wed, Dec 14, 2016 at 7:25 AM, Alex Rukletsov  wrote:

> This looks like a bug. Tomek, could you please file a JIRA?
>
> On Tue, Dec 13, 2016 at 1:02 PM, Tomek Janiszewski 
> wrote:
>
> > Hi
> >
> > When task has enabled Mesos healthcheck start time in UI can show wrong
> > time. This happens because UI assumes that first status is task started
> > [0]. This is not always true because Mesos keeps only recent tasks
> statuses
> > [1] so when healthcheck updates tasks status it can override task start
> > time displayed in webui.
> >
> > Best
> > Tomek
> >
> > [0]
> > https://github.com/apache/mesos/blob/master/src/webui/
> > master/static/js/controllers.js#L140
> > [1]
> > https://github.com/apache/mesos/blob/f2adc8a95afda943f6a10e771aad64
> > 300da19047/src/common/protobuf_utils.cpp#L263-L265
> >
>



-- 
Best Regards,
Haosdent Huang

Re: Order of includes

[haosdent]

> "treat foo.hpp the same way as other project headers".
+1 This one may more easily to follow.

> 1) If you rely on symbols from bar.h, don't count on the fact that you 
> included
foo.h which (currently) includes bar.h: include bar.h yourself,
> 2) unless foo.h explicitly demonstrates its intent to provide you the symbols
of bar.h.
> 3) However, any includes present in the related header do not need to be 
> included
again in the related cc (i.e., foo.cc can rely on foo.h's includes).
I think most contributors have followed this for a long time?
+1 for "clarify this in the Mesos style guide"

On Wed, Dec 14, 2016 at 10:29 AM, Yan Xu  wrote:

> Another related practice we should standardize is that, Google style guide
> suggests here
>  Names_and_Order_of_Includes>
> :
>
> 1) If you rely on symbols from bar.h, don't count on the fact that you
> included foo.h which (currently) includes bar.h: include bar.h yourself,
> 2) unless foo.h explicitly demonstrates its intent to provide you the
> symbols of bar.h.
> 3) However, any includes present in the related header do not need to be
> included again in the related cc (i.e., foo.cc can rely on foo.h's
> includes).
>
> We obviously follow 1) but AFAIK we don't follow 2) and 3), instead we have
> been doing "include everything needed yourself, no exceptions".
>
> It would be great to clarify this in the Mesos style guide as well.
>
> ---
> @xujyan 
>
> On Wed, Dec 14, 2016 at 10:14 AM, Yan Xu  wrote:
>
> > For a cpp file foo.cpp, our style guide instructs folks to put the header
> > foo.hpp at the top of the include list: https://github.com/apache/meso
> > s/blob/master/docs/c%2B%2B-style-guide.md#order-of-includes
> >
> > This is consistent with Google style guide but in reality most of the our
> > files follow the rule of "treat foo.hpp the same way as other project
> > headers".
> >
> > Since this rule has been introduced to the Mesos style guide I haven't
> > seen much adoption. Most committers still observe the "old" rule when
> > creating new files and have been instructing contributors to do the same.
> >
> > Given the current status I'd suggest we revert the style guide on this?
> > It's only adding confusion and I don't see the need to do a sweeping
> change
> > to the codebase to comply with the new rule.
> >
> > Thoughts?
> >
> > Yan
> >
>



-- 
Best Regards,
Haosdent Huang

Re: Order of includes

[Yan Xu]

Another related practice we should standardize is that, Google style guide
suggests here

:

1) If you rely on symbols from bar.h, don't count on the fact that you
included foo.h which (currently) includes bar.h: include bar.h yourself,
2) unless foo.h explicitly demonstrates its intent to provide you the
symbols of bar.h.
3) However, any includes present in the related header do not need to be
included again in the related cc (i.e., foo.cc can rely on foo.h's
includes).

We obviously follow 1) but AFAIK we don't follow 2) and 3), instead we have
been doing "include everything needed yourself, no exceptions".

It would be great to clarify this in the Mesos style guide as well.

---
@xujyan 

On Wed, Dec 14, 2016 at 10:14 AM, Yan Xu  wrote:

> For a cpp file foo.cpp, our style guide instructs folks to put the header
> foo.hpp at the top of the include list: https://github.com/apache/meso
> s/blob/master/docs/c%2B%2B-style-guide.md#order-of-includes
>
> This is consistent with Google style guide but in reality most of the our
> files follow the rule of "treat foo.hpp the same way as other project
> headers".
>
> Since this rule has been introduced to the Mesos style guide I haven't
> seen much adoption. Most committers still observe the "old" rule when
> creating new files and have been instructing contributors to do the same.
>
> Given the current status I'd suggest we revert the style guide on this?
> It's only adding confusion and I don't see the need to do a sweeping change
> to the codebase to comply with the new rule.
>
> Thoughts?
>
> Yan
>

Order of includes

[Yan Xu]

For a cpp file foo.cpp, our style guide instructs folks to put the header
foo.hpp at the top of the include list:
https://github.com/apache/mesos/blob/master/docs/c%2B%2B-style-guide.md#order-of-includes

This is consistent with Google style guide but in reality most of the our
files follow the rule of "treat foo.hpp the same way as other project
headers".

Since this rule has been introduced to the Mesos style guide I haven't seen
much adoption. Most committers still observe the "old" rule when creating
new files and have been instructing contributors to do the same.

Given the current status I'd suggest we revert the style guide on this?
It's only adding confusion and I don't see the need to do a sweeping change
to the codebase to comply with the new rule.

Thoughts?

Yan

Re: Question about mesos authentication and authorization?

[Greg Mann]

Hi Jared,
The default authenticator modules included with Mesos only support storing
the credentials in a file on disk. However, Mesos exposes an authenticator
module interface which can be used to implement more complex authentication
scenarios. Note that there are authenticator module interfaces for both the
V0 API [1] and the HTTP V1 API [2]. You can find an introduction to Mesos
modules in the docs [3].

Some Mesos users have implemented support for systems like LDAP and
Kerberos. DC/OS Enterprise is one example of a system with LDAP support,
but that functionality is proprietary so the module source-code isn't
freely available.

Cheers,
Greg

[1] https://github.com/apache/mesos/blob/master/include/meso
s/authentication/authenticator.hpp
[2] https://github.com/apache/mesos/blob/master/3rdparty/lib
process/include/process/authenticator.hpp
[3] http://mesos.apache.org/documentation/latest/modules/



On Mon, Dec 12, 2016 at 6:39 PM, Yu Wei  wrote:

> Hi folks,
>
>
> One question about mesos authentication and authorization.
>
> Where could I store the credentials? In documentation, it seems the
> credentials is stored in local disk.
>
> Is there any other approach to store and manage credentials via LDAP or
> other 3rd party software?
>
>
> Thanks,
>
> Jared, (??)
> Software developer
> Interested in open source software, big data, Linux
>

Re: [webui] Started show wrong time

[Alex Rukletsov]

This looks like a bug. Tomek, could you please file a JIRA?

On Tue, Dec 13, 2016 at 1:02 PM, Tomek Janiszewski 
wrote:

> Hi
>
> When task has enabled Mesos healthcheck start time in UI can show wrong
> time. This happens because UI assumes that first status is task started
> [0]. This is not always true because Mesos keeps only recent tasks statuses
> [1] so when healthcheck updates tasks status it can override task start
> time displayed in webui.
>
> Best
> Tomek
>
> [0]
> https://github.com/apache/mesos/blob/master/src/webui/
> master/static/js/controllers.js#L140
> [1]
> https://github.com/apache/mesos/blob/f2adc8a95afda943f6a10e771aad64
> 300da19047/src/common/protobuf_utils.cpp#L263-L265
>

Re: Quota

[Vijay Srinivasaraghavan]

Hi Alex,
>>Granularity in the allocator is a single agent.
Does this mean if I have only one agent and the moment if I set any quota, the 
framework running on the agent will not be allocated with any resource?
>From the logs, I don't get to see much details for the scenario when the quota 
>is set following deploying a package through Marathon. However, when I remove 
>the quota, I see the following message in master log "Allocating 
>ports(*):[*]; disk(*):." "Sending 1 offers to framework XX (marathon) 
>at scheduler-XXX"
RegardsVijay 

On Sunday, December 11, 2016 9:05 AM, Alex Rukletsov  
wrote:
 

 Granularity in the allocator is a single agent. Hence even though you set 
quota for 0.0001 CPU, at least one agent is "blocked". This is probably the 
reason why marathon is not getting offers. You can turn verbose master logs and 
check allocator messages to confirm. Alex.
On 10 Dec 2016 2:14 am, "Vijay"  wrote:

The dispatcher needs 1cpu and 1G memory.

Regards,
Vijay

Sent from my iPhone

> On Dec 9, 2016, at 4:51 PM, Vinod Kone  wrote:
>
> And how many resources does spark need?
>
>> On Fri, Dec 9, 2016 at 4:05 PM, Vijay Srinivasaraghavan 
>>  wrote:
>> Here is the slave state info. I see marathon is registered as "slave_public" 
>> role and is configured with "default_accepted_resource_ roles" as "*"
>>
>> "slaves":[
>>       {
>>          "id":"69356344-e2c4-453d-baaf- 22df4a4cc430-S0",
>>          "pid":"slave(1)@xxx.xxx.xxx. 100:5051",
>>          "hostname":"xxx.xxx.xxx.100",
>>          "registered_time":1481267726. 19244,
>>          "resources":{
>>             "disk":12099.0,
>>             "mem":14863.0,
>>             "gpus":0.0,
>>             "cpus":4.0,
>>             "ports":"[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 
>>8182-32000]"
>>          },
>>          "used_resources":{
>>             "disk":0.0,
>>             "mem":0.0,
>>             "gpus":0.0,
>>             "cpus":0.0
>>          },
>>          "offered_resources":{
>>             "disk":0.0,
>>             "mem":0.0,
>>             "gpus":0.0,
>>             "cpus":0.0
>>          },
>>          "reserved_resources":{
>>
>>          },
>>          "unreserved_resources":{
>>             "disk":12099.0,
>>             "mem":14863.0,
>>             "gpus":0.0,
>>             "cpus":4.0,
>>             "ports":"[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 
>>8182-32000]"
>>          },
>>          "attributes":{
>>
>>          },
>>          "active":true,
>>          "version":"1.0.1"
>>       }
>>    ],
>>
>> Regards
>> Vijay
>> On Friday, December 9, 2016 3:48 PM, Vinod Kone  wrote:
>>
>>
>> How many resources does the agent register with the master? How many 
>> resources does spark task need?
>>
>> I'm guessing marathon is not registered with "test" role so it is only 
>> getting un-reserved resources which are not enough for spark task?
>>
>> On Fri, Dec 9, 2016 at 2:54 PM, Vijay Srinivasaraghavan 
>>  wrote:
>> I have a standalone DCOS setup (Single node Vagrant VM running DCOS 
>> v.1.9-dev build + Mesos 1.0.1 + Marathon 1.3.0). Both master and agent are 
>> running on same VM.
>>
>> Resource: 4 CPU, 16GB Memory, 20G Disk
>>
>> I have created a quota using new V1 API which creates a role "test" with 
>> resource constraints of 0.5 CPU and 1G Memory.
>>
>> When I try to deploy Spark package, Marathon receives the request but the 
>> task is in "waiting" state since it did not receive any offers from Master 
>> though I don't see any resource constraints from the hardware perspective.
>>
>> However, when I deleted the quota, Marathon is able to move forward with the 
>> deployment and Spark was deployed/up and running. I could see from the Mesos 
>> master logs that it had sent an offer to the Marathon framework.
>>
>> To debug the issue, I was trying to create a quota but this time did not 
>> provide any CPU and Memory (0 cpu and 0 mem). After this, when I try to 
>> deploy Spark from DCOS UI, I could see Marathon getting offer from Master 
>> and able to deploy Spark without the need to delete the quota this time.
>>
>> Did anyone notice similar behavior?
>>
>> Regards
>> Vijay
>>
>>
>>
>



   

Re: Command healthcheck failed but status KILLED

[Gastón Kleiman]

On Mon, Dec 12, 2016 at 12:47 PM, Tomek Janiszewski 
wrote:

> It there any information that kill is the result of failed healthcheck?
> TaskHealthStatus should have some details on what was wrong. When default
> executor is killing task it should add a reason and details to TaskStatus.
> What do you think?
>

I agree with you, so I created
https://issues.apache.org/jira/browse/MESOS-6786 with some ideas.

-Gastón

[webui] Started show wrong time

[Tomek Janiszewski]

Hi

When task has enabled Mesos healthcheck start time in UI can show wrong
time. This happens because UI assumes that first status is task started
[0]. This is not always true because Mesos keeps only recent tasks statuses
[1] so when healthcheck updates tasks status it can override task start
time displayed in webui.

Best
Tomek

[0]
https://github.com/apache/mesos/blob/master/src/webui/master/static/js/controllers.js#L140
[1]
https://github.com/apache/mesos/blob/f2adc8a95afda943f6a10e771aad64300da19047/src/common/protobuf_utils.cpp#L263-L265