Re: Review Request 32151: Add MESOS_{MAJOR|MINOR|PATCH}_VERSION to libmesos.

[Joris Van Remoortere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32151/
---

(Updated March 27, 2015, 12:01 a.m.)


Review request for mesos, Benjamin Hindman and Niklas Nielsen.


Changes
---

Fix Whitespace.
Remove extra using declaration.


Bugs: MESOS-1795 and MESOS-2161
https://issues.apache.org/jira/browse/MESOS-1795
https://issues.apache.org/jira/browse/MESOS-2161


Repository: mesos


Description
---

See summary.


Diffs (updated)
-

  include/mesos/version.hpp.in 524cebce5d84edac3ef5d0b72e3989f283164809 
  src/Makefile.am 7a06c7028eca8164b1f5fdea6a7ecd37ee6826bb 
  src/common/version.cpp PRE-CREATION 
  src/java/generated/org/apache/mesos/MesosNativeLibrary.java.in 
668647fdfc0e203fcde59263256659ba14e29960 
  src/java/jni/org_apache_mesos_MesosNativeLibrary.cpp PRE-CREATION 
  src/tests/common/version_tests.cpp PRE-CREATION 

Diff: https://reviews.apache.org/r/32151/diff/


Testing
---


Thanks,

Joris Van Remoortere

Re: Review Request 32550: Add major, minor, and patch accessors to stout's Version.

[Joris Van Remoortere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32550/
---

(Updated March 27, 2015, 12:01 a.m.)


Review request for mesos, Benjamin Hindman and Niklas Nielsen.


Bugs: MESOS-1795 and MESOS-2161
https://issues.apache.org/jira/browse/MESOS-1795
https://issues.apache.org/jira/browse/MESOS-2161


Repository: mesos


Description
---

We called them majorVersion, minorVersion, and patchVersion to avoid 
conflicting with major() and minor() (http://man.he.net/man3/gnu_dev_major)


Diffs
-

  3rdparty/libprocess/3rdparty/stout/include/stout/version.hpp 
090fcf09dd96538a8748cf4443d150911e2c0d27 

Diff: https://reviews.apache.org/r/32550/diff/


Testing
---


Thanks,

Joris Van Remoortere

Re: Review Request 32152: Fix memory corruption in AbstractState JNI bindings. MESOS-2161.

[Joris Van Remoortere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32152/
---

(Updated March 27, 2015, 12:02 a.m.)


Review request for mesos, Benjamin Hindman and Niklas Nielsen.


Bugs: MESOS-1795 and MESOS-2161
https://issues.apache.org/jira/browse/MESOS-1795
https://issues.apache.org/jira/browse/MESOS-2161


Repository: mesos


Description
---

See summary.


Diffs
-

  src/java/jni/org_apache_mesos_state_AbstractState.cpp 
1accc8a498a68b7cfd9e39dc1f3ce01c8bfd219f 
  src/java/src/org/apache/mesos/state/AbstractState.java 
c66bf0519e7fc671d1e167ccd1e778dc65d3d8e6 

Diff: https://reviews.apache.org/r/32152/diff/


Testing
---


Thanks,

Joris Van Remoortere

Re: Review Request 32151: Add MESOS_{MAJOR|MINOR|PATCH}_VERSION to libmesos.

[Joris Van Remoortere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32151/
---

(Updated March 27, 2015, 10:39 p.m.)


Review request for mesos, Benjamin Hindman and Niklas Nielsen.


Changes
---

-Clean up version macros
-Version implements Comparable
-Delegate default constructors for Version
-Add illegal argument checks for Version constructor
-Add some extra comments
-Change to javadoc comments where applicable
-Change version introduction barrier to 0.22.1


Bugs: MESOS-1795 and MESOS-2161
https://issues.apache.org/jira/browse/MESOS-1795
https://issues.apache.org/jira/browse/MESOS-2161


Repository: mesos


Description
---

See summary.


Diffs (updated)
-

  include/mesos/version.hpp.in 524cebce5d84edac3ef5d0b72e3989f283164809 
  src/Makefile.am 7a06c7028eca8164b1f5fdea6a7ecd37ee6826bb 
  src/common/version.cpp PRE-CREATION 
  src/java/generated/org/apache/mesos/MesosNativeLibrary.java.in 
668647fdfc0e203fcde59263256659ba14e29960 
  src/java/jni/org_apache_mesos_MesosNativeLibrary.cpp PRE-CREATION 
  src/tests/common/version_tests.cpp PRE-CREATION 

Diff: https://reviews.apache.org/r/32151/diff/


Testing
---


Thanks,

Joris Van Remoortere

Re: Review Request 32152: Fix memory corruption in AbstractState JNI bindings. MESOS-2161.

[Joris Van Remoortere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32152/
---

(Updated March 27, 2015, 10:43 p.m.)


Review request for mesos, Benjamin Hindman and Niklas Nielsen.


Changes
---

-Add global ref to jclass caching
-Fix / Add comments
-Change bug fix version to 0.22.1
-Rebase to get rid of LibraryNotLoadedException


Bugs: MESOS-1795 and MESOS-2161
https://issues.apache.org/jira/browse/MESOS-1795
https://issues.apache.org/jira/browse/MESOS-2161


Repository: mesos


Description
---

See summary.


Diffs (updated)
-

  src/java/jni/org_apache_mesos_state_AbstractState.cpp 
1accc8a498a68b7cfd9e39dc1f3ce01c8bfd219f 
  src/java/src/org/apache/mesos/state/AbstractState.java 
c66bf0519e7fc671d1e167ccd1e778dc65d3d8e6 

Diff: https://reviews.apache.org/r/32152/diff/


Testing
---


Thanks,

Joris Van Remoortere

Re: Review Request 32152: Fix memory corruption in AbstractState JNI bindings. MESOS-2161.

[Joris Van Remoortere]

> On March 27, 2015, 6:51 p.m., Benjamin Hindman wrote:
> > src/java/jni/org_apache_mesos_state_AbstractState.cpp, line 249
> > <https://reviews.apache.org/r/32152/diff/4/?file=907011#file907011line249>
> >
> > Can we confirm that we can make a jclass be static?
> 
> Michael Park wrote:
> I believe this can't be made `static` unless we make it a global like so:
> ```
> static jclass clazz = 
> (jclass)env->NewGlobalRef(env->GetObjectClass(thiz));
> ```
> I think it's the same situation as: 
> [MESOS-2414](https://issues.apache.org/jira/browse/MESOS-2414).

Indeed we can not. Using Mpark's strategy.


- Joris


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32152/#review78092
-----------


On March 27, 2015, 10:43 p.m., Joris Van Remoortere wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32152/
> ---
> 
> (Updated March 27, 2015, 10:43 p.m.)
> 
> 
> Review request for mesos, Benjamin Hindman and Niklas Nielsen.
> 
> 
> Bugs: MESOS-1795 and MESOS-2161
> https://issues.apache.org/jira/browse/MESOS-1795
> https://issues.apache.org/jira/browse/MESOS-2161
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> See summary.
> 
> 
> Diffs
> -
> 
>   src/java/jni/org_apache_mesos_state_AbstractState.cpp 
> 1accc8a498a68b7cfd9e39dc1f3ce01c8bfd219f 
>   src/java/src/org/apache/mesos/state/AbstractState.java 
> c66bf0519e7fc671d1e167ccd1e778dc65d3d8e6 
> 
> Diff: https://reviews.apache.org/r/32152/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Joris Van Remoortere
> 
>

Review Request 32630: Add style change proposal for constant reference to temporaries.

[Joris Van Remoortere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32630/
---

Review request for mesos, Benjamin Hindman and switched to 'mcypark'.


Repository: mesos


Description
---

Ben suggested we start pushing proposal markdowns to the repo, so they can go 
through a nice review process.
This proposal is to augment the style guide to disallow capturing temporaries 
by reference.


Diffs
-

  docs/proposals/style/const_ref_to_temporary.md PRE-CREATION 

Diff: https://reviews.apache.org/r/32630/diff/


Testing
---

The 'output' in the proposal shows the problem.


Thanks,

Joris Van Remoortere

[Style Proposal] Disallow Capture by Constant Reference to temporary

[Joris Van Remoortere]

I would like to propose we disallow capturing temporaries using a constant
reference:
const T& val = f();

*The reasons, and examples for this are outlined in the proposal here:*
[Review Board]: https://reviews.apache.org/r/32630/
[original markdown version]:
https://gist.github.com/jmlvanre/8a3de53ae88c2d19b375


Please feel free to comment here on the dev-list, though I also urge you to
comment on the Review Request as putting proposal markdowns on RB is
something new we're trying!

Joris

Re: Review Request 32558: Improve compile time of mesos by splitting flags

[Joris Van Remoortere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32558/#review78228
---


I know the code-base is currently inconsistent, but let's follow the include 
style rules for now so as not to make it more-so:

```c++
#include  // Includes from mesos public include 
directory.

#include "master/master.hpp" // Includes from mesos src directory.
```

If we want to change the overall include style rules, let's make that a 
seperate proposal / patch.

- Joris Van Remoortere


On March 27, 2015, 1:21 a.m., Cody Maloney wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32558/
> ---
> 
> (Updated March 27, 2015, 1:21 a.m.)
> 
> 
> Review request for mesos, Joris Van Remoortere and Michael Park.
> 
> 
> Bugs: MESOS-292
> https://issues.apache.org/jira/browse/MESOS-292
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Split the mesos master, slave flags into header + source file to 
> improve compile time significantly. Should be no functional changes.
> 
> Largely copy-paste, with a little reworking to remove unnecessary 
> headers from the .hpp, and order headers a little more reliably 
> (as well as remove duplicate includes) in the .cpp.
> 
> # Impact of these changes
> `time make check -j8`
> Before:
> make check  2732.93s user 103.89s system 514% cpu 9:11.63 total
> 
> After:
> make check  2421.18s user 96.60s system 506% cpu 8:16.67 total
> 
> 4 core machine, 8 hypter threads enabled. SSD, 16 GB RAM, 
> Intel i7-4790K.
> 
> The numbers aren't incredibly stable (Other software running, 
> overclocking enabled, etc). They are a good general measure though of
> speedup.
> 
> I do a `make check` rather than just `make` because that is what devs
> do in a day-to-day flow, and if runtime is significantly impacted, it
> will show up.
> 
> Test steps:
> ```
> # Warm cache
> ../configure --disable-python --disable-java
> make check
> # Timed build
> rm -rf *
> ../configure --disable-python --disable-java
> time make check
> ```
> 
> Note: Similar, likely greater improvements in compile time would happen
> if stout were made to not be header only. Specifically .
> 
> 
> Diffs
> -
> 
>   src/Makefile.am 7a06c7028eca8164b1f5fdea6a7ecd37ee6826bb 
>   src/logging/flags.hpp 4facb33201cee5a82451a13ca05607c875574752 
>   src/logging/flags.cpp PRE-CREATION 
>   src/master/allocator/allocator.hpp b67b8fddbd7a3fffc6fe24d5e77cd1db8cb6f69b 
>   src/master/flags.hpp 85ce3285731c11b464aca6eaaa0a9165c73afebd 
>   src/master/flags.cpp PRE-CREATION 
>   src/slave/flags.hpp 3da71afad38ae41adefab979dbed2ae0b10a98ef 
>   src/slave/flags.cpp PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/32558/diff/
> 
> 
> Testing
> ---
> 
> make check on ArchLinux with GCC 4.9.2
> make distcheck CentOS 7
> 
> 
> Thanks,
> 
> Cody Maloney
> 
>

Re: Review Request 32583: Marked RunTaskMessage::framework_id as optional.

[Joris Van Remoortere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32583/#review78704
---



src/messages/messages.proto
<https://reviews.apache.org/r/32583/#comment127658>

Add deprecated tag to indicate that no one should use this going forward. 
It doesn't affect the C++ code and only adds @deprecated for Java.

optional FrameworkID framework_id = 1 [deprecated = true];


- Joris Van Remoortere


On April 1, 2015, 7:34 p.m., Kapil Arya wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32583/
> ---
> 
> (Updated April 1, 2015, 7:34 p.m.)
> 
> 
> Review request for mesos, Adam B and Niklas Nielsen.
> 
> 
> Bugs: MESOS-2558
> https://issues.apache.org/jira/browse/MESOS-2558
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> The new preferred location for FrameworkID is FrameworkInfo.id. This patchset 
> achieves this goal by incrementally deprecating other locations for 
> FrameworkID.
> 
> Here is a plan to deal with the upgrade path:
> 
> For this release (N), we still keep setting RunTaskMessage::framework_id
>   - this would handle older Slaves with newer Master.
>   - added code to handle it being unset in the Slave (handles older
> Master with newer Slaves).
> 
> In the following release (N+1), stop reading/setting 
> RunTaskMessage::framework_id
>   - the previous version would handle the unset case.
> 
> In release N+2, remove the field altogether:
>   - the previous release is not setting/reading it.
> 
> 
> Diffs
> -
> 
>   src/messages/messages.proto 97c45c01dfcea38b1ae555c036d61e10c152c2c8 
>   src/slave/slave.cpp c7e65a6c095963feaa9d5fdbb519c68f8f761d16 
> 
> Diff: https://reviews.apache.org/r/32583/diff/
> 
> 
> Testing
> ---
> 
> make check.
> 
> TODO: Test for upgrade path.
> 
> 
> Thanks,
> 
> Kapil Arya
> 
>

Re: Review Request 32911: Fixed sandbox ownership bug for executors without URIs.

[Joris Van Remoortere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32911/#review79103
---



src/slave/containerizer/external_containerizer.cpp
<https://reviews.apache.org/r/32911/#comment128199>

Can we remove the capture by reference here? I know it's not in the style 
guide yet, but it will likely be accepted.


- Joris Van Remoortere


On April 7, 2015, 12:40 a.m., Niklas Nielsen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32911/
> ---
> 
> (Updated April 7, 2015, 12:40 a.m.)
> 
> 
> Review request for mesos, Benjamin Hindman and Ian Downes.
> 
> 
> Bugs: MESOS-2592
> https://issues.apache.org/jira/browse/MESOS-2592
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> During recent refactorings, executor directory ownership was delegated to the 
> fetcher. However, the fetcher is not invoked if no URIs are present in the 
> executor or task command. This left some of these tasks broken as the 
> directory ownership defaulted to the mesos-slave's (root).
> 
> 
> Diffs
> -
> 
>   src/slave/containerizer/external_containerizer.cpp 
> 1bbd61cb096771b7e4a1350079f79a20102e78f9 
>   src/slave/paths.hpp 1618439d728ded347ec75317ce8dd998acd7ee94 
>   src/slave/paths.cpp 01ea856aa2e628d4aee5fd31f7e49d147f740e8f 
>   src/slave/slave.cpp 521624c335b9110e12ee1ff21c3918e5af6a2bde 
> 
> Diff: https://reviews.apache.org/r/32911/diff/
> 
> 
> Testing
> ---
> 
> Functional tests with mesos-execute and make check. Have created JIRA's for 
> introduction of more permission/user tests.
> 
> 
> Thanks,
> 
> Niklas Nielsen
> 
>

Re: Review Request 32859: Add Camel-case libprocess variable and method names sample.

[Joris Van Remoortere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32859/#review79117
---


Hi haosdent,
I added some higher level comments in the JIRA ticket for you :-)

- Joris Van Remoortere


On April 6, 2015, 9:19 p.m., haosdent huang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32859/
> ---
> 
> (Updated April 6, 2015, 9:19 p.m.)
> 
> 
> Review request for mesos, Joris Van Remoortere and Niklas Nielsen.
> 
> 
> Bugs: MESOS-2576
> https://issues.apache.org/jira/browse/MESOS-2576
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Add Camel-case libprocess variable and method names sample.
> 
> 
> Diffs
> -
> 
>   3rdparty/libprocess/src/libev.hpp e4a403d9e769c13182f26034e0dd1c4db92b04cb 
>   3rdparty/libprocess/src/libev.cpp 610dfb896ed8f9c00f9cf8fc8dbfc7d434f7d1e5 
>   3rdparty/libprocess/src/libev_poll.cpp 
> 324e4dd950989f3717ca73fe48520ca3e518518f 
>   3rdparty/libprocess/src/process.cpp 
> cf4e36489be2c6aa01e838c1c71383f248deab5b 
> 
> Diff: https://reviews.apache.org/r/32859/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> haosdent huang
> 
>

Re: Review Request 27113: Libprocess benchmark cleanup

[Joris Van Remoortere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27113/
---

(Updated April 10, 2015, 8:59 p.m.)


Review request for Cody Maloney, Joerg Schad and Michael Park.


Changes
---

Cleaning up per BenM's review.


Bugs: MESOS-1980
https://issues.apache.org/jira/browse/MESOS-1980


Repository: mesos


Description (updated)
---

Clean up Libprocess for readability / extensibility.


Diffs (updated)
-

  3rdparty/libprocess/src/tests/benchmarks.cpp 
a927e4ecd8c8955cd9f85e716173a73a9a21c6cd 

Diff: https://reviews.apache.org/r/27113/diff/


Testing
---

make check


Thanks,

Joris Van Remoortere

Re: Changing Mesos Minimum Compiler Version

[Joris Van Remoortere]

+1

On Thu, Apr 9, 2015 at 2:14 PM, Cody Maloney  wrote:

> As discussed in the last community meeting, we'd like to bump the minimum
> required compiler version from GCC 4.4 to GCC 4.8.
>
> The overall goals are to make Mesos development safer, faster, and reduce
> the maintenance burden. Currently a lot of stout has different codepaths
> for Pre-C++11 and Post-C++11compilers.
>
> Progress will be tracked in the JIRA: MESOS-2604
>
> The resulting supported compiler versions will be:
> GCC 4.8, GCC 4.9
> Clang 3.5, Clang 3.6
>
> For reference
> Compilers by Distribution Version: http://goo.gl/p1t1ls
>
> C++11 features supported by each compiler:
> https://gcc.gnu.org/projects/cxx0x.html
> http://clang.llvm.org/cxx_status.html
>

Re: Review Request 27113: Libprocess benchmark cleanup

[Joris Van Remoortere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27113/
---

(Updated April 13, 2015, 5:43 p.m.)


Review request for Cody Maloney, Joerg Schad and Michael Park.


Changes
---

addressed Cody's comments.


Bugs: MESOS-1980
https://issues.apache.org/jira/browse/MESOS-1980


Repository: mesos


Description
---

Clean up Libprocess for readability / extensibility.


Diffs (updated)
-

  3rdparty/libprocess/src/tests/benchmarks.cpp 
a927e4ecd8c8955cd9f85e716173a73a9a21c6cd 

Diff: https://reviews.apache.org/r/27113/diff/


Testing
---

make check


Thanks,

Joris Van Remoortere

Re: Review Request 32961: WIP: Allow framework re-registeration to update master http fields.

[Joris Van Remoortere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32961/
---

(Updated April 14, 2015, 2:57 a.m.)


Review request for mesos, Benjamin Hindman and Niklas Nielsen.


Changes
---

Factor out allocator change.
Rebase on master.


Bugs: MESOS-1218, MESOS-2614 and MESOS-703
https://issues.apache.org/jira/browse/MESOS-1218
https://issues.apache.org/jira/browse/MESOS-2614
https://issues.apache.org/jira/browse/MESOS-703


Repository: mesos


Description
---

Fields: 'name', 'hostname', 'failover_timeout', 'webui_url'


Diffs (updated)
-

  src/master/master.hpp 6141917644b84edfed9836fa0a005d55a36880e3 
  src/master/master.cpp 44b0a0147f5354824d86332a67b30018634c9a36 

Diff: https://reviews.apache.org/r/32961/diff/


Testing
---

make check.
re-registered no_executor_framework with different 'name', 'hostname', 
'failover_timeout', and 'webui_url'


Thanks,

Joris Van Remoortere

Review Request 33159: Pump updateFramework through Allocator from Master.

[Joris Van Remoortere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33159/
---

Review request for mesos, Benjamin Hindman and Niklas Nielsen.


Bugs: MESOS-2615 and MESOS-703
https://issues.apache.org/jira/browse/MESOS-2615
https://issues.apache.org/jira/browse/MESOS-703


Repository: mesos


Description
---

Follow up of 32961 where we add the 'updateFramework' function to the allocator.


Diffs
-

  src/master/allocator/allocator.hpp 91f80501aa9bc733fd53f9cb1ac0f15949a74964 
  src/master/allocator/mesos/allocator.hpp 
fb898f1175b61b442204e6e38c69ccc2838a646f 
  src/master/allocator/mesos/hierarchical.hpp 
9f9a631ee52a3ab194e678f9be139e8dabc7f3db 
  src/master/master.cpp 44b0a0147f5354824d86332a67b30018634c9a36 
  src/tests/mesos.hpp 42e42ac425a448fcc5e93db1cef1112cbf5e67c4 

Diff: https://reviews.apache.org/r/33159/diff/


Testing
---

make check


Thanks,

Joris Van Remoortere

Re: Review Request 32961: Allow framework re-registeration to update master http fields.

[Joris Van Remoortere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32961/
---

(Updated April 14, 2015, 4:30 p.m.)


Review request for mesos, Benjamin Hindman and Niklas Nielsen.


Changes
---

add framework id to warning log.


Summary (updated)
-

Allow framework re-registeration to update master http fields.


Bugs: MESOS-1218, MESOS-2614 and MESOS-703
https://issues.apache.org/jira/browse/MESOS-1218
https://issues.apache.org/jira/browse/MESOS-2614
https://issues.apache.org/jira/browse/MESOS-703


Repository: mesos


Description
---

Fields: 'name', 'hostname', 'failover_timeout', 'webui_url'


Diffs (updated)
-

  src/master/master.hpp 6141917644b84edfed9836fa0a005d55a36880e3 
  src/master/master.cpp 44b0a0147f5354824d86332a67b30018634c9a36 

Diff: https://reviews.apache.org/r/32961/diff/


Testing
---

make check.
re-registered no_executor_framework with different 'name', 'hostname', 
'failover_timeout', and 'webui_url'


Thanks,

Joris Van Remoortere

Re: Oversubscription design doc

[Joris Van Remoortere]

We have a dedicated IRC channel for this: "mesos-oversubscription". Feel
free to drop in and chat.

On Tue, Apr 14, 2015 at 11:09 AM, Niklas Nielsen 
wrote:

> Hi everyone,
>
> In context of some of the recent discussions on 'Resource overcommittal',
> we are very interested and invested in enabling oversubscription in Mesos
> in a safe and efficient way. Some of the community members have been
> working on an architecture document and prototypes recently and feel we
> have something we can start working out of:
>
> https://docs.google.com/document/d/1pUnElxHy1uWfHY_FOvvRC73QaOGgdXE0OXN-gbxdXA0/edit#
>
> We will continue meeting up and discussing the approach, as it is a
> significant change to how workloads are run on Mesos.
> Feel free to jump in, add suggestions, raise concerns and participate in
> the development of this feature.
>
> Cheers,
> Niklas
>

Re: Review Request 27113: Libprocess benchmark cleanup

[Joris Van Remoortere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27113/
---

(Updated April 14, 2015, 7:46 p.m.)


Review request for mesos, Ben Mahler, Cody Maloney, Joerg Schad, and Michael 
Park.


Changes
---

minor fixes for Joerg.


Bugs: MESOS-1980
https://issues.apache.org/jira/browse/MESOS-1980


Repository: mesos


Description
---

Clean up Libprocess for readability / extensibility.


Diffs (updated)
-

  3rdparty/libprocess/src/tests/benchmarks.cpp 
a927e4ecd8c8955cd9f85e716173a73a9a21c6cd 

Diff: https://reviews.apache.org/r/27113/diff/


Testing
---

make check


Thanks,

Joris Van Remoortere

Re: Suggestion: Mesos 0.22.1 point release

[Joris Van Remoortere]

I think the plan is to cut a new RC by sometime tomorrow. The spreadsheet
is up-to-date, just need to cherry-pick and modify the change-log.

Joris

On Tue, Apr 14, 2015 at 5:37 PM, Benjamin Mahler 
wrote:

> Hey Nik, any progress on this? Is the spreadsheet up-to-date?
>
> On Wed, Apr 8, 2015 at 1:00 AM, Adam Bordelon  wrote:
>
> > Hi Adam,
> >
> > Yes, once we have finalized the scope of the point release, Niklas will
> > send out an announcement of Mesos 0.22.1-rc1 (release candidate) which we
> > would love you to test any way you can. The email will contain
> instructions
> > for building the release candidate and voting in the thread. See the vote
> > thread from 0.22.0-rc4 (became final):
> > http://www.mail-archive.com/dev%40mesos.apache.org/msg30668.html
> >
> > The current thread is to collect suggestions for bug fixes to include in
> > this point release.
> >
> > Cheers,
> > -Adam-
> >
> > On Tue, Apr 7, 2015 at 9:22 AM, Adam Avilla  wrote:
> >
> > > On Fri, Apr 3, 2015 at 3:47 PM, Niklas Nielsen 
> > > wrote:
> > >
> > > > Based on input from Vinod and Adam; I will reduce the scope on the
> > point
> > > > release to focus on MESOS-1795 and MESOS-2583.
> > > >
> > >
> > > Can I help test these in any way?
> > >
> > > --
> > > /adam
> > >
> >
>

Re: Review Request 29406: Introduce libevent ssl socket.

[Joris Van Remoortere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29406/
---

(Updated April 17, 2015, 4:05 p.m.)


Review request for Benjamin Hindman, Bernd Mathiske, Cody Maloney, Joerg Schad, 
Marco Massenzio, and Michael Park.


Bugs: MESOS-1913
https://issues.apache.org/jira/browse/MESOS-1913


Repository: mesos


Description (updated)
---

Requires:
configure --enable-libevent --enable-libevent-socket --enable-ssl
New environment variables:
USE_SSL=(0,1)
SSL_CERT=(path to certificate)
SSL_KEY=(path to key)
SSL_VERIFY_CERT=(0,1)
SSL_REQUIRE_CERT=(0,1)
SSL_CA_DIR=(path to CA directory)
SSL_CA_FILE=(path to CA file)

TODO:
Restrict SSL version more tightly
Track down leak in crypto from accept


Diffs
-

  3rdparty/libprocess/Makefile.am 8f96f49a386a70f14324d3a4744aa0b8bf3995f9 
  3rdparty/libprocess/include/process/socket.hpp 
ddb9e365fc1e65a568bdac4973964df1ab8cc05e 
  3rdparty/libprocess/src/libevent.hpp f6cc72178613a30446629532a773afccfd404212 
  3rdparty/libprocess/src/libevent.cpp 28c2cf7f49cc153158f2a470a1812e35f7d4b93a 
  3rdparty/libprocess/src/libevent_ssl_socket.hpp PRE-CREATION 
  3rdparty/libprocess/src/libevent_ssl_socket.cpp PRE-CREATION 
  3rdparty/libprocess/src/openssl.hpp PRE-CREATION 
  3rdparty/libprocess/src/openssl.cpp PRE-CREATION 
  3rdparty/libprocess/src/process.cpp 67b6b3b9c13d95fa1a24b48a12c5c831c7f249bf 
  3rdparty/libprocess/src/socket.cpp 4b0f6bec8051f938812dbc90a7312e4082ea203f 

Diff: https://reviews.apache.org/r/29406/diff/


Testing
---

make check (uses non-ssl socket)
benchmarks using ssl sockets
master, slave, framework, webui launch with ssl sockets


Thanks,

Joris Van Remoortere

Re: Review Request 27113: Libprocess benchmark cleanup

[Joris Van Remoortere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27113/
---

(Updated April 17, 2015, 5:28 p.m.)


Review request for mesos, Ben Mahler, Cody Maloney, Joerg Schad, and Michael 
Park.


Changes
---

Address bmahler's comments.
Rebased on 33315 which allows us to disable short-circuit passing of messages 
between actors. This means we can get rid of the forking code path, which 
simplifies the test significantly.


Bugs: MESOS-1980
https://issues.apache.org/jira/browse/MESOS-1980


Repository: mesos


Description
---

Clean up Libprocess for readability / extensibility.


Diffs (updated)
-

  3rdparty/libprocess/src/tests/benchmarks.cpp 
a927e4ecd8c8955cd9f85e716173a73a9a21c6cd 

Diff: https://reviews.apache.org/r/27113/diff/


Testing
---

make check


Thanks,

Joris Van Remoortere

Re: Review Request 27113: Libprocess benchmark cleanup

[Joris Van Remoortere]

> On April 14, 2015, 9:28 p.m., Ben Mahler wrote:
> > 3rdparty/libprocess/src/tests/benchmarks.cpp, lines 108-114
> > <https://reviews.apache.org/r/27113/diff/4/?file=925777#file925777line108>
> >
> > This seems to suggest support for multiple "runs" of the client. 
> > However, if a /run request arrives while the previous run is in progress, 
> > it looks like this will behave strangely. Did you want to deny requests 
> > while a run is in progress, or chain the runs after one another?

Caught this state and return an error message for now.


> On April 14, 2015, 9:28 p.m., Ben Mahler wrote:
> > 3rdparty/libprocess/src/tests/benchmarks.cpp, line 266
> > <https://reviews.apache.org/r/27113/diff/4/?file=925777#file925777line266>
> >
> > Could you do a self review of this test body, from the perspective of 
> > someone approaching this code for the first time?
> > 
> > I had a hard time understanding this, at a basic level what is the 
> > overall structure of the forking here? It's also a bit tricky to figure out 
> > the lower level details, like what the `requestPipes` and `resultPipes` are 
> > for, and why they're needed. What the vectors of pipes are used for, why 
> > they're needed. Etc.
> > 
> > In the process of being able to articulate what this does, we might 
> > figure out how we can make it more easily understandable :)

Take a look at the updated review. I'll let you mark this as fixed if you 
approve :-)


- Joris


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27113/#review80101
---


On April 17, 2015, 5:28 p.m., Joris Van Remoortere wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27113/
> ---
> 
> (Updated April 17, 2015, 5:28 p.m.)
> 
> 
> Review request for mesos, Ben Mahler, Cody Maloney, Joerg Schad, and Michael 
> Park.
> 
> 
> Bugs: MESOS-1980
> https://issues.apache.org/jira/browse/MESOS-1980
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Clean up Libprocess for readability / extensibility.
> 
> 
> Diffs
> -
> 
>   3rdparty/libprocess/src/tests/benchmarks.cpp 
> a927e4ecd8c8955cd9f85e716173a73a9a21c6cd 
> 
> Diff: https://reviews.apache.org/r/27113/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> 
> Thanks,
> 
> Joris Van Remoortere
> 
>

Re: Review Request 33376: MESOS-2633 Moved struct Framework methods to their own implementation class.

[Joris Van Remoortere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33376/#review80894
---


Hey Marco, great first patch!
Some high level comments:
- Can you check your editor settings to make sure you don't modify lines you 
don't intend?
- Just do a quick look over the 'diff' section before publishing a review so 
can you see if everything being reviewed is intended (i.e. eclipse template)
- If we're moving around large blocks of code, let's split out any refactoring 
/ semantic changes into a seperate review. This way it's easier to see just 
which lines you actually modified vs. moved. It will get the code commited 
faster. (i.e. logging refactor).

Can you make these changes and then I'll take a deeper look at framework.cpp :-)


.gitignore-template
<https://reviews.apache.org/r/33376/#comment131054>

I think this got added to your commit accidentaly? If you want to add this 
to the repository we should make a separate JIRA and review for this.



include/mesos/type_utils.hpp
<https://reviews.apache.org/r/33376/#comment131059>

Can you check the style guide / other files for how we arrange includes?



include/mesos/type_utils.hpp
<https://reviews.apache.org/r/33376/#comment131056>

I think we need an extra whitespace here. Can you check the style guide / 
other files?



include/mesos/type_utils.hpp
<https://reviews.apache.org/r/33376/#comment131055>

This comment doesn't seem to convey anything about what this code does, 
rather why it's part of this review request. Let's make it something more 
meaningful.

Also, comments end with a period.



include/mesos/type_utils.hpp
<https://reviews.apache.org/r/33376/#comment131058>

Do we still need the `internal` if we're already in the namespace? Is this 
required to disambiguate? Maybe this could be part of the more useful comment 
above!



include/mesos/type_utils.hpp
<https://reviews.apache.org/r/33376/#comment131057>

We need another newline here.
Also, can you check the style for where we close other namespaces? We have 
a specific pattern than we use :-)



src/Makefile.am
<https://reviews.apache.org/r/33376/#comment131061>

I don't think you meant to change the spacing of all these lines 
(highlighted and rest of file). Can you check your editor settings to see if 
you're modifying them accidentally?



src/master/framework.cpp
<https://reviews.apache.org/r/33376/#comment131062>

Can you check the style guide for includes here?



src/master/framework.cpp
<https://reviews.apache.org/r/33376/#comment131065>

Can we split out any code changes into a seperate review (with a JIRA)? 
Here and elsewhere. See my high-level comment at the top of the review.



src/master/master.hpp
<https://reviews.apache.org/r/33376/#comment131064>

We use a period at the end of comments.


- Joris Van Remoortere


On April 20, 2015, 10:38 p.m., Marco Massenzio wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33376/
> -------
> 
> (Updated April 20, 2015, 10:38 p.m.)
> 
> 
> Review request for mesos and Joris Van Remoortere.
> 
> 
> Bugs: MESOS-2633
> https://issues.apache.org/jira/browse/MESOS-2633
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Created new file framework.cpp containing all the methods' implementations 
> for the `Framework` class (declared in master/master.hpp)
> 
> Declared `operator ==` for Task in type_utils.hpp 
> (it was implemented before, but not declared in the header file);
> 
> Refactored all the LOG(WARNING) to a single utility method.
> 
> 
> Diffs
> -
> 
>   .gitignore-template 1a8e2a8677a29f06ba6386d56537a0013d38821c 
>   include/mesos/type_utils.hpp cdf5864389a72002b538c263d70bcade2bdffa45 
>   src/Makefile.am d15a37365bcdd5c3906160b46b389635b38b1673 
>   src/master/framework.cpp PRE-CREATION 
>   src/master/master.hpp c10e7c08c191acef9d31d98018a47a2a952a4dfc 
> 
> Diff: https://reviews.apache.org/r/33376/diff/
> 
> 
> Testing
> ---
> 
> All tests (make check) pass.
> 
> 
> Thanks,
> 
> Marco Massenzio
> 
>

Re: Changing Mesos Minimum Compiler Version

[Joris Van Remoortere]

Re: GCC 5.x, specifically section [2]
https://gcc.gnu.org/gcc-5/changes.html#offload

Although these changes are great, I'm not sure we currently need them for
Mesos itself.
I agree with you that they could make lots of frameworks rock, and I don't
think the gcc version for mesos prevents that:

   - We use protobufs to communicate between services which allows:
   - Executors can be compiled using a totally different compiler from
   Apache Mesos
   - Frameworks can be compiled using a totally different compiler from
   Apache Mesos
   - This means you can have a super optimized custom executor that takes
   advantage of all the benefits of GCC 5.X running on a Mesos built on GCC
   4.8 or Clang 3.5!

Hopefully this clarifies why this is not actually crucial, and why you
won't be missing out on any benefits!

Joris

On Tue, Apr 21, 2015 at 10:27 AM, Cody Maloney  wrote:

> The main holdup at the moment is simply cycles I have to convert our
> internal infrastructure for packaging mesos on all the distributions to use
> newer compilers on all those distributions. I want the infrastructure for
> supporting the change in place before we make it. I have about 1/3 of the
> work done (Can build on all distros except Debain Wheezy). I've gotta add
> the packaging steps (Shouldn't be too bad), and some glue code still
> though.
>
> On Tue, Apr 21, 2015 at 3:07 AM, Alex Rukletsov 
> wrote:
>
> > Folks, let's summarize and move on here.
> >
> > Proposal out on April 9, 2015. Current status (as of April 21, 2015):
> >
> >
> > +1 (Binding)
> > --
> > Vinod Kone
> > Timothy Chen
> > Yan Xu
> > Brenden Matthews
> >
> > +1 (Non-binding)
> > --
> > Cody Maloney
> > Joris Van Remoortere
> > Jeff Schroeder
> > Jörg Schad
> > Elizabeth Lingg
> > Alexander Rojas
> > Alex Rukletsov
> > Michael Park
> > Haosdent Huang
> > Bernd Mathiske
> >
> > 0 (Non-binding)
> > --
> > Nikolaos Ballas
> >
> > There were no -1 votes.
> >
> > Cody, let's convert MESOS-2604 to an epic and bump the version in 0.23.
> >
> > Thanks,
> > Alex
> >
> >
> > On Mon, Apr 13, 2015 at 12:46 PM, Bernd Mathiske 
> > wrote:
> >
> >> +1
> >>
> >> > On Apr 10, 2015, at 6:02 PM, Michael Park  wrote:
> >> >
> >> > +1
> >> >
> >> > On 9 April 2015 at 17:33, Alexander Gallego 
> >> wrote:
> >> >
> >> >> This is amazing for native devs/frameworks.
> >> >>
> >> >> Sent from my iPhone
> >> >>
> >> >>> On Apr 9, 2015, at 5:16 PM, Joris Van Remoortere <
> jo...@mesosphere.io
> >> >
> >> >> wrote:
> >> >>>
> >> >>> +1
> >> >>>
> >> >>>> On Thu, Apr 9, 2015 at 2:14 PM, Cody Maloney 
> >> >> wrote:
> >> >>>> As discussed in the last community meeting, we'd like to bump the
> >> >> minimum required compiler version from GCC 4.4 to GCC 4.8.
> >> >>>>
> >> >>>> The overall goals are to make Mesos development safer, faster, and
> >> >> reduce the maintenance burden. Currently a lot of stout has different
> >> >> codepaths for Pre-C++11 and Post-C++11compilers.
> >> >>>>
> >> >>>> Progress will be tracked in the JIRA: MESOS-2604
> >> >>>>
> >> >>>> The resulting supported compiler versions will be:
> >> >>>> GCC 4.8, GCC 4.9
> >> >>>> Clang 3.5, Clang 3.6
> >> >>>>
> >> >>>> For reference
> >> >>>> Compilers by Distribution Version: http://goo.gl/p1t1ls
> >> >>>>
> >> >>>> C++11 features supported by each compiler:
> >> >>>> https://gcc.gnu.org/projects/cxx0x.html
> >> >>>> http://clang.llvm.org/cxx_status.html
> >> >>>
> >> >>
> >>
> >>
> >
>

Re: Review Request 33376: MESOS-2633 Moved struct Framework methods to their own implementation class.

[Joris Van Remoortere]

> On April 21, 2015, 3:27 a.m., Joris Van Remoortere wrote:
> > Hey Marco, great first patch!
> > Some high level comments:
> > - Can you check your editor settings to make sure you don't modify lines 
> > you don't intend?
> > - Just do a quick look over the 'diff' section before publishing a review 
> > so can you see if everything being reviewed is intended (i.e. eclipse 
> > template)
> > - If we're moving around large blocks of code, let's split out any 
> > refactoring / semantic changes into a seperate review. This way it's easier 
> > to see just which lines you actually modified vs. moved. It will get the 
> > code commited faster. (i.e. logging refactor).
> > 
> > Can you make these changes and then I'll take a deeper look at 
> > framework.cpp :-)
> 
> Marco Massenzio wrote:
> 1. apart from removing tabs (which follows the "Boy Scout Principle") my 
> settings are just as you mention: I don't think my changes touched anything 
> other than intended changes?  
>The only areas I touched that were outside the scope was lining up the 
> continuation slashes (`\ `) on the .am file (which was, honestly, pretty 
> awful to look at :)
> 2. I did and it seemed to me to be ok? (Eclipse 'ignores' ought to be 
> part of the .gitignore IMO as other developers who may be using Eclips will 
> need it? why not?)
> 3. agreed - although it seemed to me such a minor non-functional change 
> that it did not warrant its own review: and the original code IMO should have 
> not gone past review anyway (violating DRY principle: so, as this was 
> technically "new" code of mine, I'd be on the 'blame' hook for it)
>  
> I will revert the check on the non-contains for the task, that is a 
> 'material' change and can go in its own review.
> Please advise on the .gitignore changes and why those should not be part 
> of the .gitignore.

1. I think you might have a different whitespace setting in eclipse. For 
everyone else the continuation slashes () in the .am file were lined up, and 
after your change will not be. We can sync on this together offline using vim 
or some other editor to verify this.
2. I think adding your eclipse ignore file is great for future developers. I'm 
just suggesting that we move it into a separate JIRA / review. This is just the 
way our project works. We try to have highly-specific, atomic reviews so that 
people can look through the commit log and see what they expect.
3. a) I wouldn't worry about having too many reviews, rather too few. Isolating 
the changes from the moves makes it easier for someone to review. Since the 
reviewer is also on the hook for the code, it's helpful to make it as easy as 
possibly on them to verify correctness.
   b) I totally understand your comment regarding the DRY principle. In the 
Mesos project sometimes we prefer to repeat ourselves, it's a style choice 
around ease of debugging. This is another discussion we can have offline as a 
team.
   c) Re: DRY, in this case the refactoring looks beneficial in my case, I 
would just suggest we make it a separate commit so that the cut between 
modification and movement of code is as clean as possible!

Thanks for reverting the 'material' change! That way we can have a separate 
discussion regarding the behavior change and why we may want to follow that 
pattern here, and elsewhere.


> On April 21, 2015, 3:27 a.m., Joris Van Remoortere wrote:
> > src/Makefile.am, lines 302-304
> > <https://reviews.apache.org/r/33376/diff/1/?file=937065#file937065line302>
> >
> > I don't think you meant to change the spacing of all these lines 
> > (highlighted and rest of file). Can you check your editor settings to see 
> > if you're modifying them accidentally?
> 
> Marco Massenzio wrote:
> the right-hand side continuation \ was all over the place and looked 
> honestly awful: I follow the "boyscout principle" and just lined up quite a 
> few of them around the *one* change I made (adding `framework.cpp`) - 
> hopefully this should upset no one?

I think we might have different whitespace modifiers in our editors. Let's sync 
up as in my high-level comment.


> On April 21, 2015, 3:27 a.m., Joris Van Remoortere wrote:
> > include/mesos/type_utils.hpp, line 184
> > <https://reviews.apache.org/r/33376/diff/1/?file=937064#file937064line184>
> >
> > I think we need an extra whitespace here. Can you check the style guide 
> > / other files?
> 
> Marco Massenzio wrote:
> I'm actually guessing you meant around the `==` in `operator == (...)`? 
> (you commented a blank line)
>

Re: constexpr and non-POD static variables

[Joris Van Remoortere]

We had an internal discussion about this. White-listing constexpr is fine
by us.
I think technically the google style guide allows c++11 features that are
not explicitly disallowed, but it doesn't hurt to add it to the style guide
:-)

Joris

On Thu, May 28, 2015 at 4:27 PM, Paul Brett 
wrote:

> Despite best efforts to avoid non-POD static variables, we appear to still
> have over 80 instances (see MESOS-2780).  We can, of course, use const
> inline functions to address these but this will change each constant
> reference at the calling site to a constant function reference.
>
> A more natural approach provided by C++11 is to use constexpr.  I have
> submitted for review (https://reviews.apache.org/r/34782/) an example to
> address the TC handle.
>
> What are the views on whitelisting constexpr?
>
> -- Paul Brett
>

Re: constexpr and non-POD static variables

[Joris Van Remoortere]

@Marco: If you think it hurts to add it to the white-list, then we need to
have a discussion about wiping the entire white-list.

I think having a white-list, and then not adding things makes it even more
confusing.

On Thu, May 28, 2015 at 10:15 PM, Marco Massenzio 
wrote:

> On Thu, May 28, 2015 at 4:45 PM, Joris Van Remoortere  >
> wrote:
>
> > We had an internal discussion about this. White-listing constexpr is fine
> > by us.
> > I think technically the google style guide allows c++11 features that are
> > not explicitly disallowed, but it doesn't hurt to add it to the style
> guide
> > :-)
> >
>
> I actually think that *adding* it to the guide does hurt :)
> (on the other hand, I'm absolutely fine with constexpr)
>
> I really like Google's approach of "if it's not explicitly forbidden, it's
> allowed" as it helps people from having to double-guess whether X is
> allowed or not.
>
> my twocent, anyway
>
> >
> > Joris
> >
> > On Thu, May 28, 2015 at 4:27 PM, Paul Brett 
> > wrote:
> >
> > > Despite best efforts to avoid non-POD static variables, we appear to
> > still
> > > have over 80 instances (see MESOS-2780).  We can, of course, use const
> > > inline functions to address these but this will change each constant
> > > reference at the calling site to a constant function reference.
> > >
> > > A more natural approach provided by C++11 is to use constexpr.  I have
> > > submitted for review (https://reviews.apache.org/r/34782/) an example
> to
> > > address the TC handle.
> > >
> > > What are the views on whitelisting constexpr?
> > >
> > > -- Paul Brett
> > >
> >
>

Re: Build failed in Jenkins: mesos-reviewbot #6201

[Joris Van Remoortere]

JPeach: That's correct. You had it right in the top level project, but the
comparison was wrong in the libprocess check. Probably just a copy paste
error, all fixed now! :-)
Thanks for helping us move to more general macros!
Joris

On Thu, Jun 4, 2015 at 10:51 PM, James Peach  wrote:

>
> > On Jun 4, 2015, at 12:50 PM, Vinod Kone  wrote:
> >
> > On Thu, Jun 4, 2015 at 7:48 AM, Apache Jenkins Server <
> > jenk...@builds.apache.org> wrote:
> >
> >> checking for C++ compiler version... 4.8.2
> >> checking for C++ compiler vendor... (cached) gnu
> >> configure: error: GCC 4.8 or higher required (found 4.8.2)
> >>
> >
> > hmm. this looks weird.
>
> Is this the bug in my autoconf patch that Ben fixed with
> 2b2618f855a9f377111ad0460fc4a6e9db4d3755?

Re: Introduction - Aditi

[Joris Van Remoortere]

Hi Aditi,

Please feel free to add me as a reviewer for these patches. I contributed
part of Phase 1, and ended up looking ahead as to what would be required
for the further phases. I'd be happy to review your patches!

Joris

On Thu, Jun 4, 2015 at 11:00 AM, Aditi Dixit  wrote:

> Hi all,
>
> I'm Aditi and I'm currently interning at Outreachy. Apache Mesos is an
> amazing piece of software and I'm really excited to work on it. This is my
> first time contributing to an open source software, so any advice you may
> have for me is welcome!
> The project that I would be working on is Updating the Framework Info. The
> goal of this project is to develop a way to allow frameworks to update
> their FrameworkInfo without having to restart all the slaves, tasks or
> master(s) and reconcile the FrameworkInfo. The project is quite nicely
> described by Vinod Kone in the Confluence link here
> <
> https://cwiki.apache.org/confluence/display/MESOS/Design+doc%3A+Updating+Framework+Info
> >.
>
> Right now, I'm working on starter tickets to get familiar with the Mesos
> codebase, but I hope to start working on the project soon.
>
> Regards,
> Aditi
>

Re: Apache Mesos Community Sync

[Joris Van Remoortere]

Reminder: The Mesos Community Developer Sync will be happening today at 3pm
Pacific.

To participate remotely, join the Google hangout:
https://plus.google.com/hangouts/_/twitter.com/mesos-sync

On Thu, Jun 18, 2015 at 7:22 AM, Adam Bordelon  wrote:

> Reminder: We're hosting a developer community sync at Mesosphere HQ this
> morning from 9-11am Pacific.
>
> The agenda is pretty bare, so please add more topics you would like to
> discuss:
>
> https://docs.google.com/document/d/153CUCj5LOJCFAVpdDZC7COJDwKh9RDjxaTA0S7lzwDA/edit
>
> If you want to join in person, just show up to 88 Stevenson St, ring the
> buzzer, take the elevator up to 2nd floor, and then you can take the stairs
> up to the 3rd floor dining room, or ask somebody to let you up the elevator
> to the 3rd floor.
>
> To participate remotely, join the Google hangout:
> https://plus.google.com/hangouts/_/mesosphere.io/mesos-developer
>
> On Mon, Jun 15, 2015 at 10:46 AM, Adam Bordelon 
> wrote:
>
>> As previously mentioned, we would like to host additional Mesos developer
>> syncs at our new Mesosphere HQ at 88 Stevenson St (tucked behind Market &
>> 2nd), starting this Thursday from 9-11am Pacific. We opted for an earlier
>> slot so that the European developer community can participate.
>>
>> Now that we are having these more frequently, it would be great to dive
>> deeper into designs for upcoming features as well as discuss longstanding
>> issues. While high-level status updates are useful, they should be a small
>> part of these meetings so that we can address issues currently facing our
>> developers.
>>
>> Please add agenda items to the same doc we've been using for previous
>> meetings' Agenda/Notes:
>>
>> https://docs.google.com/document/d/153CUCj5LOJCFAVpdDZC7COJDwKh9RDjxaTA0S7lzwDA/edit
>>
>> Join in person if you can, or join remotely via hangout:
>> https://plus.google.com/hangouts/_/mesosphere.io/mesos-developer
>>
>> Thanks,
>> -Adam-
>>
>>
>> On Thu, May 28, 2015 at 10:08 AM, Vinod Kone  wrote:
>>
>>> Cool.
>>>
>>> Here's the agenda doc
>>> <
>>> https://docs.google.com/document/d/153CUCj5LOJCFAVpdDZC7COJDwKh9RDjxaTA0S7lzwDA/edit#
>>> >
>>> for next week that folks can fill in.
>>>
>>> On Thu, May 28, 2015 at 9:52 AM, Adam Bordelon 
>>> wrote:
>>>
>>> > Looks like next week, Thursday June 4th on my calendar.
>>> > I thought it was always the first Thursday of the month.
>>> >
>>> > On Thu, May 28, 2015 at 9:33 AM, Vinod Kone 
>>> wrote:
>>> >
>>> > > Do we have community sync today or next week? I'm a bit confused.
>>> > >
>>> > > @vinodkone
>>> > >
>>> > > > On Apr 1, 2015, at 3:18 AM, Adam Bordelon 
>>> wrote:
>>> > > >
>>> > > > Reminder: We're having another Mesos Developer Community Sync this
>>> > > > Thursday, April 2nd from 3-5pm Pacific.
>>> > > >
>>> > > > Agenda:
>>> > > >
>>> > >
>>> >
>>> https://docs.google.com/document/d/153CUCj5LOJCFAVpdDZC7COJDwKh9RDjxaTA0S7lzwDA/edit?usp=sharing
>>> > > > To Join: follow the BlueJeans instructions from the recurring
>>> meeting
>>> > > > invite at the start of this thread.
>>> > > >
>>> > > >> On Fri, Mar 6, 2015 at 11:11 AM, Vinod Kone >> >
>>> > > wrote:
>>> > > >>
>>> > > >> Hi folks,
>>> > > >>
>>> > > >> We are planning to do monthly Mesos community meetings.
>>> Tentatively
>>> > > these
>>> > > >> are scheduled to occur on 1st Thursday of every month at 3 PM
>>> PST. See
>>> > > >> below for details to join the meeting remotely.
>>> > > >>
>>> > > >> This is a forum to ask questions/discuss about upcoming features,
>>> > > process
>>> > > >> etc. Everyone is welcome to join. Feel free to add items to the
>>> agenda
>>> > > for
>>> > > >> the next meeting here
>>> > > >> <
>>> > > >>
>>> > >
>>> >
>>> https://docs.google.com/document/d/153CUCj5LOJCFAVpdDZC7COJDwKh9RDjxaTA0S7lzwDA/edit?usp=sharing
>>> > > >> .
>>> > > >>
>>> > > >> Cheers,
>>> > > >>
>>> > > >> On Thu, Mar 5, 2015 at 11:23 AM, Vinod Kone via Blue Jeans
>>> Network <
>>> > > >> inv...@bluejeans.com> wrote:
>>> > > >>
>>> > > >>>[image: Blue Jeans]    Vinod Kone
>>> > > >>>  has invited you to a video meeting.
>>>  Meeting
>>> > > >> Title: Apache Mesos Community Sync
>>> > > >>>  Meeting Time: Every 4th week on Thursday • from March 5, 2015 •
>>> 3
>>> > p.m.
>>> > > >>> PST / 2 hrs  Join Meeting
>>> > > >>> <
>>> > > >>
>>> > >
>>> https://bluejeans.com/272369669?ll=en&g=mrsxmqdnmvzw64zomfygcy3imuxg64th
>>> > >
>>> > > >>> --
>>> > > >>>  Connecting directly from a room system?
>>> > > >>>
>>> > > >>> 1) Dial: 199.48.152.152 or bjn.vc
>>> > > >>> 2) Enter Meeting ID: 272369669 -or- use the pairing code
>>> > > >>>
>>> > > >>>
>>> > > >>> Just want to dial in? (all numbers <
>>> > > http://bluejeans.com/premium-numbers
>>> > > >>> )
>>> > > >>> 1) Direct-dial with my iPhone <+14087407256,,#272369669%23,%23>
>>> or
>>> > > >>> +1 408 740 7256 <+1%20408%20740%207256>+1 408 740 7256
>>> > > >>> +1 888 240 2560 <+1%20888%20240%202560>+1 888 240 2560 (US Toll

Re: Review board: diff update

[Joris Van Remoortere]

This is the worflow I usually share with people:

Updating a review after comments:
`git checkout master`
`git fetch apache`
`git rebase apache/master`
'git checkout `
`git rebase master`

`git rebase -i HEAD^` (number of `^` based on how far back you want to
pick




`git add `
'git rebase --continue`



where:
apache https://git-wip-us.apache.org/repos/asf/mesos.git


On Fri, Aug 7, 2015 at 10:09 AM, Khanduja, Vaibhav  wrote:

> Hi
>
> I am updating a diff on review board on an existing. I understand, the
> existing commits should be adjusted before executing post_review.py. I am
> having few issues here with adjusting the commits, and wondering if
> somebody could guide me with commands which should be used to adjust the
> commit?
>
> Thanks
>

Re: Mesos Style Guideline Adjustments

[Joris Van Remoortere]

I will volunteer to update all the comments to wrap at 80 if we agree to keep 
the code base consistent. 
Naturally that is also my vote ;-)
Joris

> On Aug 8, 2015, at 1:40 PM, Michael Park  wrote:
> 
> An update on this topic since we covered it at the community developer sync.
> 
>   1. We will adopt *Mozilla*'s *BreakBeforeBraces* style as their style is
>   equivalent to ours. The only change this entails for our codebase is to
>   consistently wrap the braces for *enum* definitions, as we're currently
>   inconsistent. I've taken on the work involved in this change:
>  - stout: https://reviews.apache.org/r/37258
>  - libprocess: https://reviews.apache.org/r/37259
>  - mesos: https://reviews.apache.org/r/37260
>  2. We will drop the rule for adding spaces around overloaded
>   operators. We'll simply do a sweep of the codebase to update all of them
>   consistently. Artem has kindly taken action on this already:
>  - stout: https://reviews.apache.org/r/37018/
>  - libprocess: https://reviews.apache.org/r/37017/
>  - mesos: https://reviews.apache.org/r/37013/
>  3. We will drop the rule for wrapping comments at 70 characters. We
>   have a few options to proceed here:
>  - Keep all the existing comments in tact, and simply allow new
>  comments to wrap at 80, this is less work.
>  - Update all instances of the comments wrapping at 70 to be wrapped
>  at 80, so that we can be consistent.
> 
> I proposed that we simply allow new comments to wrap at 80, but I have
> heard arguments to update the existing comments, so that we can be
> consistent across the codebase. If you have a suggestion/opinion on how we
> should proceed with (3), please share!
> 
> Thanks,
> 
> MPark.
> 
> On Mon, Aug 3, 2015 at 2:01 PM Alexander Rojas 
> wrote:
> 
>> I also vote up for that! I rather change our guidelines a little bit than
>> waiting for months
>> to get our changes into the clang-format source without any security that
>> it will actually land.
>> 
>>> On 31 Jul 2015, at 09:31, Alex Rukletsov  wrote:
>>> 
>>> I think automation is very important. If we should slightly change our
>>> style in order to set-up easily enforceable rules, I vote with both hands
>>> for that.
>>> 
 On Fri, Jul 31, 2015 at 3:25 AM, Michael Park  wrote:
 
 Oops, sorry I was so excited that this could just solve the issue that I
 forgot to answer your question.
 
 In general, the clang-format strives to adopt widely used styles, which
>> I'm
 not sure if we would be considered widely used. Aside from that, another
 concern was that it could take a while for our style proposals to make
>> it
 upstream and for it to be useful.
 
> On Thu, Jul 30, 2015, 6:12 PM Michael Park  wrote:
> 
> Is it worth adding our own style?
> 
> 
> 
> I noticed other have (LLVM, Google, Chromium, Mozilla, WebKit.). How
>> hard is it?
> 
> 
> I was just looking into this again and *Mozilla* was added as the
>> newest
> *BreakBeforeBraces* style. It breaks before braces on enum, function,
>> and
> record definitions (struct, class, union). I think we can actually use
 that
> one and be done with it. Having looked through the codebase, we wrap
>> the
> braces for *enum* for about half of the cases. It would be about 35
> instances that we have to fix from what I can see in our codebase. What
 do
> you think?
> 
> On Thu, Jul 30, 2015 at 5:14 PM Benjamin Mahler <
 benjamin.mah...@gmail.com>
> wrote:
> 
>> Is it worth adding our own style?
>> 
>> I noticed other have (LLVM, Google, Chromium, Mozilla, WebKit.). How
 hard
>> is it?
>> 
>> On Thu, Jul 30, 2015 at 4:23 PM, Michael Park 
 wrote:
>> 
>>> There are a few syntactical Mesos style guidelines that I would like
 to
>>> propose to drop. They are:
>>> 
>>>  1. Open braces for namespace should not be wrapped.
>>>  *NOTE*: The Google style guide does not wrap the brace after
>>> *namespace*,
>>>  and the Mesos style guide does not mention a rule at all. But it is
>>>  consistent throughout the codebase.
>>>  2. Overloaded operators should be padded with spaces.
>>>  3. Comments should be wrapped at 70 characters.
>>> 
>>> The main motivation is that as a community we would like to reduce
>> the
>>> discrepancy between what *clang-format* produces. This is a dual
>> effort, as
>>> we work on improving *clang-format* to support some of our style
>> which
>> is
>>> popular in the C++ community as well. Wrapping the function arguments
 to
>>> avoid "jaggedness" for example is a feature request which is being
>> tracked
>>> at: https://llvm.org/bugs/show_bug.cgi?id=23422
>>> 
>>> Going forward, the proposal is to update all of the instances of (1)
 and
>>> (2) at once. For (3), we can simply relax the const

Re: Mesos 0.25.0

[Joris Van Remoortere]

Hi Nik,

I'd like to co-manage with you as I am invested in Maintenance Primitives
:-)

Joris

On Thu, Aug 27, 2015 at 9:25 PM, Michael Park  wrote:

> Hey Niklas,
>
> As we discussed offline, I'm happy to co-manage or shadow this release.
>
> I would also like to add the beta follow-up features for persistence
> primitives (dynamic reservation + persistent volume):
>
>- Operator API
>- Authorization
>
> Thanks,
>
> MPark.
>
> On Thu, Aug 27, 2015 at 7:24 PM Niklas Nielsen 
> wrote:
>
> > Hi folks,
> >
> > While we have 0.24.0 in flight (voting and fixing is in progress), with a
> > release cadence of 1 month, we should consider to start thinking about
> > 0.25.0.
> >
> > I have cycles to release manage this one (and bring some of the newer
> > committers on for shadowing). If others want to take this on (or if
> people
> > have objections), feel free to jump in.
> >
> > Of larger items for the next release, we want to get (again, feel free to
> > add) are:
> >
> >  - Maintenance primitives in Alpha state
> >  - Networking plug-ability (demoed at MesosCon)
> >
> > We can wait until 0.24.0 lands, but we should be able to work on this in
> > parallel.
> >
> > Niklas
> >
>

Re: Mesos 0.25.0

[Joris Van Remoortere]

Following in the footsteps of the dashboard for the 0.23.0 release,
here is the Mesos 0.25.0 Release Dashboard


This dashboard was very successful at reducing the time required to triage
the first release candidate. Let's repeat that same success!

A special thanks to Adam for providing us with this great template.

Joris, Mpark, and Niklas

Re: Do we still need to add InverseOffer support to Scheduler API?

[Joris Van Remoortere]

Hi Qian,

There is no current plan to add this to the old API. Those tickets were
created pre-V1 API.
Currently the goal is to encourage developers to use the V1 API to have
access to new features such as maintenance primitives.

Joris

On Mon, Sep 14, 2015 at 10:22 AM, Qian AZ Zhang  wrote:

>
>
> Hi,
>
> In the maintenance epic (MESOS-1474), I see there are 3 tasks created to
> add InverseOffer support to Scheduler API:
> MESOS-2063  Add InverseOffer to C++ Scheduler API
> MESOS-2064  Add InverseOffer to Java Scheduler API
> MESOS-2065  Add InverseOffer to Python Scheduler API
>
> I think we have already supported Schedule HTTP API, so do we still need to
> update the C++ scheduler API (and the Java/Python binding) to support
> InverseOffer? If so, I think we may need to update all the example
> frameworks as well. Take C++ scheduler API as an example, we may need to
> add a new callback inverseResourceOffers() in the Scheduler class, and each
> example framework's scheduler needs to implement it.
>
>
> Regards,
> Qian Zhang

Re: Do we still need to add InverseOffer support to Scheduler API?

[Joris Van Remoortere]

The V1 API supports the full scheduler API. This means you can receive and
act upon offers and (once the patches are committed) inverse offers through
the V1 API.
Please see https://reviews.apache.org/r/37283/ as an example of how the V1
API can be used to deal with inverse offers.

Joris

On Mon, Sep 14, 2015 at 11:02 AM, Guangya Liu  wrote:

> Hi Joris,
>
> I think that those APIs are still needed as HTTP API is mainly initiated by
> operator, the current call for HTTP API including TEARDOWN, ACCEPT,
> DECLINE, REVIVE, KILL, SHUTDOWN etc, but the offer related operations such
> as offer and InverserOffers are initiatedby mesos master, the master need
> notify the framework for those offers via the callbacks. Comments?
>
> Thanks,
>
> Guangya
>
> On Mon, Sep 14, 2015 at 10:42 PM, Joris Van Remoortere <
> jo...@mesosphere.io>
> wrote:
>
> > Hi Qian,
> >
> > There is no current plan to add this to the old API. Those tickets were
> > created pre-V1 API.
> > Currently the goal is to encourage developers to use the V1 API to have
> > access to new features such as maintenance primitives.
> >
> > Joris
> >
> > On Mon, Sep 14, 2015 at 10:22 AM, Qian AZ Zhang 
> > wrote:
> >
> > >
> > >
> > > Hi,
> > >
> > > In the maintenance epic (MESOS-1474), I see there are 3 tasks created
> to
> > > add InverseOffer support to Scheduler API:
> > > MESOS-2063  Add InverseOffer to C++ Scheduler API
> > > MESOS-2064  Add InverseOffer to Java Scheduler API
> > > MESOS-2065  Add InverseOffer to Python Scheduler API
> > >
> > > I think we have already supported Schedule HTTP API, so do we still
> need
> > to
> > > update the C++ scheduler API (and the Java/Python binding) to support
> > > InverseOffer? If so, I think we may need to update all the example
> > > frameworks as well. Take C++ scheduler API as an example, we may need
> to
> > > add a new callback inverseResourceOffers() in the Scheduler class, and
> > each
> > > example framework's scheduler needs to implement it.
> > >
> > >
> > > Regards,
> > > Qian Zhang
> >
>

Re: [5/5] mesos git commit: Integer Precision for JSON <-> Protobuf conversions.

[Joris Van Remoortere]

Hi Jie,
Can you share your build info?
Did you rebootstrap?

We've been building this for over a week so we must have a different
configuration.

Thanks for letting us know! We will resolve this quickly :-)

Joris

On Wed, Sep 16, 2015 at 6:05 PM, Jie Yu  wrote:

> Joris,
>
> It breaks the build:
>
> picojson-1.3.0/picojson.h: In member function ‘std::string
> picojson::value::to_str() const’:
> picojson-1.3.0/picojson.h:370:38: error: expected ‘)’ before ‘PRId64’
>SNPRINTF(buf, sizeof(buf), "%" PRId64, u_.int64_);
>
>
>
> On Wed, Sep 16, 2015 at 2:56 PM,  wrote:
>
> > Integer Precision for JSON <-> Protobuf conversions.
> >
> > * Add TODO's for refactoring some JSON parsing in the docker code
> >   (See MESOS-3409).  Update how the JSON::Number is used.
> > * Tweak some tests to match changes to JSON::Number.
> > * Address a TODO on one test, which used a workaround for
> >   double-precision comparison.
> >
> > Review: https://reviews.apache.org/r/38077
> >
> >
> > Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
> > Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/2c277f1c
> > Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/2c277f1c
> > Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/2c277f1c
> >
> > Branch: refs/heads/master
> > Commit: 2c277f1c0e0dc0a6618ba930bb5f8d9dd753d4be
> > Parents: df9eacb
> > Author: Joseph Wu 
> > Authored: Wed Sep 16 13:44:47 2015 -0400
> > Committer: Joris Van Remoortere 
> > Committed: Wed Sep 16 17:48:43 2015 -0400
> >
> > --
> >  src/docker/docker.cpp   |   3 +-
> >  .../provisioners/docker/token_manager.cpp   |   3 +-
> >  src/tests/fault_tolerance_tests.cpp |   2 +-
> >  src/tests/master_tests.cpp  |   2 +-
> >  src/tests/monitor_tests.cpp |  50 +++--
> >  src/tests/rate_limiting_tests.cpp   | 106
> +--
> >  src/tests/slave_tests.cpp   |   2 +-
> >  7 files changed, 93 insertions(+), 75 deletions(-)
> > --
> >
> >
> >
> >
> http://git-wip-us.apache.org/repos/asf/mesos/blob/2c277f1c/src/docker/docker.cpp
> > --
> > diff --git a/src/docker/docker.cpp b/src/docker/docker.cpp
> > index c4c37cb..afcedf1 100755
> > --- a/src/docker/docker.cpp
> > +++ b/src/docker/docker.cpp
> > @@ -236,6 +236,7 @@ Try Docker::validateVersion(const Version&
> > minVersion) const
> >  }
> >
> >
> > +// TODO(josephw): Parse this string with a protobuf.
> >  Try Docker::Container::create(const string& output)
> >  {
> >Try parse = JSON::parse(output);
> > @@ -286,7 +287,7 @@ Try
> Docker::Container::create(const
> > string& output)
> >  return Error("Error finding Pid in State: " + pidValue.error());
> >}
> >
> > -  pid_t pid = pid_t(pidValue.get().value);
> > +  pid_t pid = pid_t(pidValue.get().as());
> >
> >Option optionalPid;
> >if (pid != 0) {
> >
> >
> >
> http://git-wip-us.apache.org/repos/asf/mesos/blob/2c277f1c/src/slave/containerizer/provisioners/docker/token_manager.cpp
> > --
> > diff --git
> a/src/slave/containerizer/provisioners/docker/token_manager.cpp
> > b/src/slave/containerizer/provisioners/docker/token_manager.cpp
> > index aec915f..95f459d 100644
> > --- a/src/slave/containerizer/provisioners/docker/token_manager.cpp
> > +++ b/src/slave/containerizer/provisioners/docker/token_manager.cpp
> > @@ -122,6 +122,7 @@ Token::Token(
> >  notBefore(_notBefore) {}
> >
> >
> > +// TODO(josephw): Parse this string with some protobufs.
> >  Try Token::create(const string& raw)
> >  {
> >auto decode = [](
> > @@ -196,7 +197,7 @@ Result Token::getTimeValue(const JSON::Object&
> > object, const string& key)
> >
> >// If expiration is provided, we will process it for future
> validations.
> >if (jsonValue.isSome()) {
> > -Try time = Time::create(jsonValue.get().value);
> > +Try time = Time::create(jsonValue.get().as());
> >  if (time.isError()) {
> >return Error("Failed to decode time: " + time.error());
> >  }
> >
> >

Re: [5/5] mesos git commit: Integer Precision for JSON <-> Protobuf conversions.

[Joris Van Remoortere]

Reverted for now while Joseph and I figure out a solution for this.
Thanks Jie!
Joris

On Wed, Sep 16, 2015 at 6:21 PM, Jie Yu  wrote:

> A different error on Ubuntu:
>
> In file included from ../../3rdparty/libprocess/include/process/http.hpp:39:0,
>  from ../../3rdparty/libprocess/include/process/event.hpp:21,
>  from 
> ../../3rdparty/libprocess/include/process/process.hpp:26,
>  from ../../src/docker/executor.cpp:26:
> ../../3rdparty/libprocess/3rdparty/stout/include/stout/json.hpp: In function 
> 'std::ostream& JSON::operator<<(std::ostream&, const JSON::Number&)':
> ../../3rdparty/libprocess/3rdparty/stout/include/stout/json.hpp:638:1: error: 
> control reaches end of non-void function [-Werror=return-type]
>  }
>  ^
> cc1plus: all warnings being treated as errors
>
>
> On Wed, Sep 16, 2015 at 3:16 PM, Jie Yu  wrote:
>
>> It breaks the CI as well:
>>
>> https://builds.apache.org/job/Mesos/COMPILER=gcc,CONFIGURATION=--verbose,OS=centos:7,label_exp=docker%7C%7CHadoop/807/console
>>
>> On Wed, Sep 16, 2015 at 3:15 PM, Jie Yu  wrote:
>>
>>> Joris, I am using CentOS6 with devtoolset-2
>>>
>>> gcc version 4.8.2 20140120 (Red Hat 4.8.2-15) (GCC)
>>>
>>> I did a clean build (rebootstrap, reconfigure).
>>>
>>> - Jie
>>>
>>>
>>> On Wed, Sep 16, 2015 at 3:14 PM, Joris Van Remoortere <
>>> jo...@mesosphere.io> wrote:
>>>
>>>> Hi Jie,
>>>> Can you share your build info?
>>>> Did you rebootstrap?
>>>>
>>>> We've been building this for over a week so we must have a different
>>>> configuration.
>>>>
>>>> Thanks for letting us know! We will resolve this quickly :-)
>>>>
>>>> Joris
>>>>
>>>> On Wed, Sep 16, 2015 at 6:05 PM, Jie Yu  wrote:
>>>>
>>>>> Joris,
>>>>>
>>>>> It breaks the build:
>>>>>
>>>>> picojson-1.3.0/picojson.h: In member function ‘std::string
>>>>> picojson::value::to_str() const’:
>>>>> picojson-1.3.0/picojson.h:370:38: error: expected ‘)’ before ‘PRId64’
>>>>>SNPRINTF(buf, sizeof(buf), "%" PRId64, u_.int64_);
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Sep 16, 2015 at 2:56 PM,  wrote:
>>>>>
>>>>> > Integer Precision for JSON <-> Protobuf conversions.
>>>>> >
>>>>> > * Add TODO's for refactoring some JSON parsing in the docker code
>>>>> >   (See MESOS-3409).  Update how the JSON::Number is used.
>>>>> > * Tweak some tests to match changes to JSON::Number.
>>>>> > * Address a TODO on one test, which used a workaround for
>>>>> >   double-precision comparison.
>>>>> >
>>>>> > Review: https://reviews.apache.org/r/38077
>>>>> >
>>>>> >
>>>>> > Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
>>>>> > Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/2c277f1c
>>>>> > Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/2c277f1c
>>>>> > Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/2c277f1c
>>>>> >
>>>>> > Branch: refs/heads/master
>>>>> > Commit: 2c277f1c0e0dc0a6618ba930bb5f8d9dd753d4be
>>>>> > Parents: df9eacb
>>>>> > Author: Joseph Wu 
>>>>> > Authored: Wed Sep 16 13:44:47 2015 -0400
>>>>> > Committer: Joris Van Remoortere 
>>>>> > Committed: Wed Sep 16 17:48:43 2015 -0400
>>>>> >
>>>>> >
>>>>> --
>>>>> >  src/docker/docker.cpp   |   3 +-
>>>>> >  .../provisioners/docker/token_manager.cpp   |   3 +-
>>>>> >  src/tests/fault_tolerance_tests.cpp |   2 +-
>>>>> >  src/tests/master_tests.cpp  |   2 +-
>>>>> >  src/tests/monitor_tests.cpp |  50 +++--
>>>>> >  src/tests/rate_limiting_tests.cpp   | 106
>>>>> +--
>>>>> >  src/tests/slave_tests.cpp   |   2 +-
>>>>> >  7 files changed, 93 insertions(+), 75 deletions(-)
>>

Re: Apache Mesos Community Sync

[Joris Van Remoortere]

Youtube on-air: http://youtu.be/ZQT6-fw8Ito
Speakers channel:
https://plus.google.com/hangouts/_/hoaevent/AP36tYd59qP_P4ac-NwOI7LztI_hBsku54gXqk1DhFGsKkne_cmByA

On Mon, Sep 14, 2015 at 7:02 PM, Adam Bordelon  wrote:

> We'll have the next community sync this Thursday (Sept. 17th) at 8:30am
> Pacific.
>
> Please add items to the agenda
> <https://docs.google.com/document/d/153CUCj5LOJCFAVpdDZC7COJDwKh9RDjxaTA0S7lzwDA/edit#heading=h.u1x3j7f3uixf>
> .
>
> We will try Hangouts on Air this time. We will post the video stream link
> shortly before the meeting, and only active participants (especially people
> on the agenda) should join the actual hangout. Others can watch the video
> stream and ask brief questions on #mesos on IRC. If you have something
> lengthier to discuss, put it on the agenda and ping us on email/IRC to get
> into the hangout. We hope this works better for everyone.
>
>
> On Wed, Sep 2, 2015 at 12:34 PM, Vinod Kone  wrote:
>
>> We'll have the next community sync tomorrow (Sept 3rd) at 3 PM PST.
>>
>> Please add items to agenda
>> <https://docs.google.com/document/d/153CUCj5LOJCFAVpdDZC7COJDwKh9RDjxaTA0S7lzwDA/edit#heading=h.u1x3j7f3uixf>
>> .
>>
>>
>> On Wed, Aug 5, 2015 at 4:12 PM, Vinod Kone  wrote:
>>
>>> We'll have the next community sync tomorrow at 3 PM PST.
>>>
>>> Please add items to agenda
>>> <https://docs.google.com/document/d/153CUCj5LOJCFAVpdDZC7COJDwKh9RDjxaTA0S7lzwDA/edit#heading=h.u1x3j7f3uixf>
>>> .
>>>
>>> Thanks,
>>>
>>> On Thu, Jul 2, 2015 at 11:24 AM, Joris Van Remoortere <
>>> jo...@mesosphere.io> wrote:
>>>
>>>> Reminder: The Mesos Community Developer Sync will be happening today at
>>>> 3pm Pacific.
>>>>
>>>> To participate remotely, join the Google hangout:
>>>> https://plus.google.com/hangouts/_/twitter.com/mesos-sync
>>>>
>>>> On Thu, Jun 18, 2015 at 7:22 AM, Adam Bordelon 
>>>> wrote:
>>>>
>>>>> Reminder: We're hosting a developer community sync at Mesosphere HQ
>>>>> this morning from 9-11am Pacific.
>>>>>
>>>>> The agenda is pretty bare, so please add more topics you would like to
>>>>> discuss:
>>>>>
>>>>> https://docs.google.com/document/d/153CUCj5LOJCFAVpdDZC7COJDwKh9RDjxaTA0S7lzwDA/edit
>>>>>
>>>>> If you want to join in person, just show up to 88 Stevenson St, ring
>>>>> the buzzer, take the elevator up to 2nd floor, and then you can take the
>>>>> stairs up to the 3rd floor dining room, or ask somebody to let you up the
>>>>> elevator to the 3rd floor.
>>>>>
>>>>> To participate remotely, join the Google hangout:
>>>>> https://plus.google.com/hangouts/_/mesosphere.io/mesos-developer
>>>>>
>>>>> On Mon, Jun 15, 2015 at 10:46 AM, Adam Bordelon 
>>>>> wrote:
>>>>>
>>>>>> As previously mentioned, we would like to host additional Mesos
>>>>>> developer syncs at our new Mesosphere HQ at 88 Stevenson St (tucked 
>>>>>> behind
>>>>>> Market & 2nd), starting this Thursday from 9-11am Pacific. We opted for 
>>>>>> an
>>>>>> earlier slot so that the European developer community can participate.
>>>>>>
>>>>>> Now that we are having these more frequently, it would be great to
>>>>>> dive deeper into designs for upcoming features as well as discuss
>>>>>> longstanding issues. While high-level status updates are useful, they
>>>>>> should be a small part of these meetings so that we can address issues
>>>>>> currently facing our developers.
>>>>>>
>>>>>> Please add agenda items to the same doc we've been using for previous
>>>>>> meetings' Agenda/Notes:
>>>>>>
>>>>>> https://docs.google.com/document/d/153CUCj5LOJCFAVpdDZC7COJDwKh9RDjxaTA0S7lzwDA/edit
>>>>>>
>>>>>> Join in person if you can, or join remotely via hangout:
>>>>>> https://plus.google.com/hangouts/_/mesosphere.io/mesos-developer
>>>>>>
>>>>>> Thanks,
>>>>>> -Adam-
>>>>>>
>>>>>>
>>>>>> On Thu, May 28, 2015 at 10:08 AM, Vinod Kone 
>>>>>> wrote:
>>>>>>
>>>>>

Re: Mesos 0.25.0

[Joris Van Remoortere]

As mentioned by Vinod in the Community Developer Meeting, we will be having
a 0.25.0 rc1 triage meeting Friday September 18th at 10:30am Pacific Time.
If you are interested in attending or are heavily engaged in the remaining
tickets please reach out to me if you would like to join the meeting. I
will try my best to accommodate everyone in the hangout.
Regardless, your shepherd will reach out to you if you have patches
outstanding targeted for 0.25.0.

The meeting will move quickly through the remaining issues, their
shepherds, and what is critical.

Joris

On Wed, Sep 16, 2015 at 7:56 PM, Michael Park  wrote:

> We have created the new target version for 0.26.0 and have cut out what
> we're planning to land for 0.25.0 on the 0.25.0 dashboard.
>
> Please set your target version to 0.26.0 unless it's a critical issue that
> needs to get into 0.25.0!
>
> Thanks,
>
> Joris, MPark, and Niklas
>
> On Mon, Sep 14, 2015, 9:23 AM Joris Van Remoortere 
> wrote:
>
> > Following in the footsteps of the dashboard for the 0.23.0 release,
> > here is the Mesos 0.25.0 Release Dashboard
> > <
> >
> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12326859
> > >
> >
> > This dashboard was very successful at reducing the time required to
> triage
> > the first release candidate. Let's repeat that same success!
> >
> > A special thanks to Adam for providing us with this great template.
> >
> > Joris, Mpark, and Niklas
> >
>

Re: Volunteer for next release

[Joris Van Remoortere]

+1 I can help triage with you :-)

On Thu, Oct 1, 2015 at 1:44 AM, Till Toenshoff  wrote:

> Hi folks,
>
> Bernd and I would like to volunteer for managing the next release 0.26.
>
> The scope and ETA of 0.26 will be discussed today at the Mesos community
> sync - hence more details on this will follow shortly.
>
> Thanks,
> Bernd and Till
>
>
>

Re: [VOTE] Release Apache Mesos 0.25.0 (rc2)

[Joris Van Remoortere]

+1 (binding)

On Mon, Oct 5, 2015 at 11:12 PM, Niklas Nielsen 
wrote:

> Hi all,
>
> Please vote on releasing the following candidate as Apache Mesos 0.25.0.
>
>
>
> 0.25.0 includes the following:
>
>
> 
>
>  * [MESOS-1474] - Experimental support for maintenance primitives.
>
>  * [MESOS-2600] - Added master endpoints /reserve and /unreserve for
> dynamic reservations.
>
>  * [MESOS-2044] - Extended Module APIs to enable IP per container
> assignment, isolation and resolution.
>
>
> ** Bug fixes
>
>   * [MESOS-2635] - Web UI Display Bug when starting lots of tasks with
> small cpu value.
>
>   * [MESOS-2986] - Docker version output is not compatible with Mesos.
>
>   * [MESOS-3046] - Stout's UUID re-seeds a new random generator during
> each call to UUID::random.
>
>   * [MESOS-3051] - performance issues with port ranges comparison.
>
>   * [MESOS-3052] - Allocator performance issue when using a large number
> of filters.
>
>   * [MESOS-3136] - COMMAND health checks with Marathon 0.10.0 are broken.
>
>   * [MESOS-3169] - FrameworkInfo should only be updated if the
> re-registration is valid.
>
>   * [MESOS-3185] - Refactor Subprocess logic in linux/perf.cpp to use
> common subroutine.
>
>   * [MESOS-3239] - Refactor master HTTP endpoints help messages such that
> they cannot be out of sync.
>
>   * [MESOS-3245] - The comments of DRFSorter::dirty is not correct.
>
>   * [MESOS-3254] - Cgroup CHECK fails test harness.
>
>   * [MESOS-3258] - Remove Frameworkinfo capabilities on re-registration.
>
>   * [MESOS-3261] - Move QoS plug-ins to a specified folder like
> resource_estimator.
>
>   * [MESOS-3269] - The comments of Master::updateSlave() is not correct.
>
>   * [MESOS-3282] - Web UI no longer shows Tasks information.
>
>   * [MESOS-3344] - Add more comments for strings::internal::fmt.
>
>   * [MESOS-3351] - duplicated slave id in master after master failover.
>
>   * [MESOS-3387] - Refactor MesosContainerizer to accept namespace
> dynamically.
>
>   * [MESOS-3408] - Labels field of FrameworkInfo should be added into v1
> mesos.proto.
>
>   * [MESOS-3411] - ReservationEndpointsTest.AvailableResources appears to
> be faulty.
>
>   * [MESOS-3423] - Perf event isolator stops performing sampling if a
> single timeout occurs.
>
>   * [MESOS-3426] - process::collect and process::await do not perform
> discard propagation.
>
>   * [MESOS-3430] -
> LinuxFilesystemIsolatorTest.ROOT_PersistentVolumeWithoutRootFilesystem
> fails on CentOS 7.1.
>
>   * [MESOS-3450] - Update Mesos C++ Style Guide for namespace usage.
>
>   * [MESOS-3451] - Failing tests after changes to
> Isolator/MesosContainerizer API.
>
>   * [MESOS-3458] - Segfault when accepting or declining inverse offers.
>
>   * [MESOS-3474] - ExamplesTest.{TestFramework, JavaFramework,
> PythonFramework} failed on CentOS 6.
>
>   * [MESOS-3489] - Add support for exposing Accept/Decline responses for
> inverse offers.
>
>   * [MESOS-3490] - Mesos UI fails to represent JSON entities.
>
>   * [MESOS-3512] - Don't retry close() on EINTR.
>
>   * [MESOS-3513] - Cgroups Test Filters aborts tests on Centos 6.6.
>
>   * [MESOS-3519] - Fix file descriptor leakage / double close in the code
> base.
>
>   * [MESOS-3538] -
> CgroupsNoHierarchyTest.ROOT_CGROUPS_NOHIERARCHY_MountUnmountHierarchy test
> is flaky.
>
>   * [MESOS-3575] - V1 API java/python protos are not generated.
>
>
> ** Improvements
>
>   * [MESOS-2719] - Deprecating '.json' extension in master endpoints urls.
>
>   * [MESOS-2757] - Add -> operator for Option, Try, Result,
> Future.
>
>   * [MESOS-2875] - Add containerId to ResourceUsage to enable QoS
> controller to target a container.
>
>   * [MESOS-2964] - libprocess io does not support peek().
>
>   * [MESOS-2983] - Deprecating '.json' extension in slave endpoints url.
>
>   * [MESOS-2984] - Deprecating '.json' extension in files endpoints url.
>
>   * [MESOS-3037] - Add a SUPPRESS call to the scheduler.
>
>   * [MESOS-3187] - Docker cli option support.
>
>   * [MESOS-3304] - Remove remnants of LIBPROCESS_STATISTICS_WINDOW.
>
>   * [MESOS-3312] - Factor out JSON to repeated protobuf conversion.
>
>   * [MESOS-3340] - Command-line flags should take precedence over OS Env
> variables.
>
>   * [MESOS-3347] - Remove dead code in src/linux/perf.cpp.
>
>   * [MESOS-3377] - mesos docker container with container_name as ENV
> variable.
>
>   * [MESOS-3457] - Add flag to disable hostname lookup.
>
>
> The full CHANGELOG for the release is available at:
>
>
> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=0.25.0-rc2
>
>
> 
>
>
> The candidate for Mesos 0.25.0 release is available at:
>
> https://dist.apache.org/repos/dist/dev/mesos/0.25.0-rc2/mesos-0.25.0.tar.gz
>
>
> The tag to be voted on is 0.25.0-rc2:
>
> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=

Re: [VOTE] Release Apache Mesos 0.25.0 (rc3)

[Joris Van Remoortere]

+1 (binding)

On Fri, Oct 9, 2015 at 5:36 PM, Niklas Nielsen  wrote:

> Hi all,
>
> Following up with an RC with the build fix suggested by Kapil:
>
> Please vote on releasing the following candidate as Apache Mesos 0.25.0.
>
>
>
> 0.25.0 includes the following:
>
>
> 
>
>  * [MESOS-1474] - Experimental support for maintenance primitives.
>
>  * [MESOS-2600] - Added master endpoints /reserve and /unreserve for
> dynamic reservations.
>
>  * [MESOS-2044] - Extended Module APIs to enable IP per container
> assignment, isolation and resolution.
>
>
> ** Bug fixes
>
>   * [MESOS-2635] - Web UI Display Bug when starting lots of tasks with
> small cpu value.
>
>   * [MESOS-2986] - Docker version output is not compatible with Mesos.
>
>   * [MESOS-3046] - Stout's UUID re-seeds a new random generator during
> each call to UUID::random.
>
>   * [MESOS-3051] - performance issues with port ranges comparison.
>
>   * [MESOS-3052] - Allocator performance issue when using a large number
> of filters.
>
>   * [MESOS-3136] - COMMAND health checks with Marathon 0.10.0 are broken.
>
>   * [MESOS-3169] - FrameworkInfo should only be updated if the
> re-registration is valid.
>
>   * [MESOS-3185] - Refactor Subprocess logic in linux/perf.cpp to use
> common subroutine.
>
>   * [MESOS-3239] - Refactor master HTTP endpoints help messages such that
> they cannot be out of sync.
>
>   * [MESOS-3245] - The comments of DRFSorter::dirty is not correct.
>
>   * [MESOS-3254] - Cgroup CHECK fails test harness.
>
>   * [MESOS-3258] - Remove Frameworkinfo capabilities on re-registration.
>
>   * [MESOS-3261] - Move QoS plug-ins to a specified folder like
> resource_estimator.
>
>   * [MESOS-3269] - The comments of Master::updateSlave() is not correct.
>
>   * [MESOS-3282] - Web UI no longer shows Tasks information.
>
>   * [MESOS-3344] - Add more comments for strings::internal::fmt.
>
>   * [MESOS-3351] - duplicated slave id in master after master failover.
>
>   * [MESOS-3387] - Refactor MesosContainerizer to accept namespace
> dynamically.
>
>   * [MESOS-3408] - Labels field of FrameworkInfo should be added into v1
> mesos.proto.
>
>   * [MESOS-3411] - ReservationEndpointsTest.AvailableResources appears to
> be faulty.
>
>   * [MESOS-3423] - Perf event isolator stops performing sampling if a
> single timeout occurs.
>
>   * [MESOS-3426] - process::collect and process::await do not perform
> discard propagation.
>
>   * [MESOS-3430] -
> LinuxFilesystemIsolatorTest.ROOT_PersistentVolumeWithoutRootFilesystem
> fails on CentOS 7.1.
>
>   * [MESOS-3450] - Update Mesos C++ Style Guide for namespace usage.
>
>   * [MESOS-3451] - Failing tests after changes to
> Isolator/MesosContainerizer API.
>
>   * [MESOS-3458] - Segfault when accepting or declining inverse offers.
>
>   * [MESOS-3474] - ExamplesTest.{TestFramework, JavaFramework,
> PythonFramework} failed on CentOS 6.
>
>   * [MESOS-3489] - Add support for exposing Accept/Decline responses for
> inverse offers.
>
>   * [MESOS-3490] - Mesos UI fails to represent JSON entities.
>
>   * [MESOS-3512] - Don't retry close() on EINTR.
>
>   * [MESOS-3513] - Cgroups Test Filters aborts tests on Centos 6.6.
>
>   * [MESOS-3519] - Fix file descriptor leakage / double close in the code
> base.
>
>   * [MESOS-3538] -
> CgroupsNoHierarchyTest.ROOT_CGROUPS_NOHIERARCHY_MountUnmountHierarchy test
> is flaky.
>
>   * [MESOS-3575] - V1 API java/python protos are not generated.
>
>
> ** Improvements
>
>   * [MESOS-2719] - Deprecating '.json' extension in master endpoints urls.
>
>   * [MESOS-2757] - Add -> operator for Option, Try, Result,
> Future.
>
>   * [MESOS-2875] - Add containerId to ResourceUsage to enable QoS
> controller to target a container.
>
>   * [MESOS-2964] - libprocess io does not support peek().
>
>   * [MESOS-2983] - Deprecating '.json' extension in slave endpoints url.
>
>   * [MESOS-2984] - Deprecating '.json' extension in files endpoints url.
>
>   * [MESOS-3037] - Add a SUPPRESS call to the scheduler.
>
>   * [MESOS-3187] - Docker cli option support.
>
>   * [MESOS-3304] - Remove remnants of LIBPROCESS_STATISTICS_WINDOW.
>
>   * [MESOS-3312] - Factor out JSON to repeated protobuf conversion.
>
>   * [MESOS-3340] - Command-line flags should take precedence over OS Env
> variables.
>
>   * [MESOS-3347] - Remove dead code in src/linux/perf.cpp.
>
>   * [MESOS-3377] - mesos docker container with container_name as ENV
> variable.
>
>   * [MESOS-3457] - Add flag to disable hostname lookup.
>
>
> The CHANGELOG for the release is available at:
>
>
> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=0.25.0-rc3
>
>
> 
>
>
> The candidate for Mesos 0.25.0 release is available at:
>
> https://dist.apache.org/repos/dist/dev/mesos/0.25.0-rc3/mesos-0.25.0.tar.gz
>
>
> The tag to be voted on is 0.25.0-rc3:
>
> ht

Re: More Project Structure in JIRA

[Joris Van Remoortere]

+1 sounds great!
I like the idea of making epics more manageable and the ability to file JIRAs 
based on a design doc while still separating the Epic for the MVP from the 
further phases.

> On Oct 15, 2015, at 10:11 AM, Bernd Mathiske  wrote:
> 
> Proposal: in extension of today’s limited two-level (epic, task) approach, 
> make full use of expressive power already available in JIRA to provide more 
> structure for larger projects to facilitate planning, tracking, and 
> reporting. This will facilitate dynamically planning of sub-projects, which 
> will make us more agile.
> 
> The general idea is to use links between epics to provide a recursive 
> hierarchical structure, with which one can span trees or DAGs of arbitrarily 
> large projects. This does not mean that we want to plan everything in minute 
> detail before starting to work. On the contrary.
> 
> You can start anywhere in the eventual tree and express part of the overall 
> effort, maybe say a short epic with a few task tickets. Then you can LATER 
> make this epic a dependency for a larger effort.
> 
> Conversely, you can subdivide a task in the epic into subtasks. However, this 
> does not mean that you have to literally use the feature “subtask” in JIRA 
> for this. Instead, staying recursive in our JIRA grammar, so to speak, 
> convert the task to an epic and then create ordinary tasks in it to represent 
> subtasks.
> 
> Now the task cannot be a task in its parent epic anymore. We fix this by 
> putting in a link of type "blocks" to the parent. When you then look at the 
> parent, it still holds a number of tasks, and it has one dependency on an 
> epic (to which you can add more).
> 
> Thus our dependency tree can grow in all directions. You can also rearrange 
> and update it in any shape or form if necessary.
> 
> Overall, we only use two JIRA elements: epics and tasks (of different flavors 
> such as bugs, improvements, etc.). Tasks are the leaves, everything else is 
> an epic. Review requests only ever happen for tasks.
> 
> The epics are there to provide a high level view and to allow dynamic (“more 
> agilish”, non-waterfall) planning. Granted, you’d also use a tree if you did 
> waterfall. The difference is that you’d spec it all out at once. My 
> observation is that not too few of us do exactly this - outside JIRA - and 
> then try to remember what tickets are where in their tree. Let’s make this 
> part of JIRA!
> 
> Why not use labels? Because they are in a flat name space and we want to 
> represent tree structure. How would you know that a label denotes a 
> subproject of another label? By memorizing or by depicting a tree outside 
> JIRA. Why not use components? Same problem as with labels: flat name space. 
> We can use labels and components these for many other purposes. Separate 
> discussion.
> 
> Aren’t we doing this already? Probably. I have not checked thoroughly. There 
> may occasionally be epics that link to other epics. If so, I would merely 
> like to encourage us to use this powerful expressive means more often.
> 
> Bernd
> 

Re: RFC: license headers interfere with doxygen documentation (MESOS-3581)

[Joris Van Remoortere]

+1 for (a).


—
*Joris Van Remoortere*
Mesosphere

On Tue, Oct 20, 2015 at 3:02 PM, Benjamin Mahler 
wrote:

> +1 for (a), in this case the wide sweep only touches the license comments,
> so it won't be disruptive to history.
>
> On Tue, Oct 20, 2015 at 11:59 AM, James Peach  wrote:
>
> >
> > > On Oct 20, 2015, at 8:55 AM, Bernd Mathiske 
> wrote:
> > >
> > > All, is changing every source code file prohibitive or not?
> > >
> > >> On Oct 20, 2015, at 10:01 AM, Benjamin Bannier <
> > benjamin.bann...@mesosphere.io> wrote:
> > >>
> > >> Hi,
> > >>
> > >> I would like to ask for input on how we plan to fix (both short- and
> > longterm) the interference of the license headers and Doxygen
> documentation
> > (https://issues.apache.org/jira/browse/MESOS-3581).
> > >>
> > >> Currently, and in line with the respective guidelines, license blocks
> > are wrapped in Javadoc-style comments which are also used for Doxygen
> > documentation. This leads to Doxygen interpreting license headers as
> > documentation for whatever entity follows them in the code, and heavily
> > clutters the generated documentation (see e.g.
> > http://mesos.apache.org/api/latest/c++/annotated.html). Given that
> > considerable effort is done to improve the documentation this
> unfortunate.
> > >>
> > >> * * *
> > >>
> > >> For a TLDR; of the Jira issue, there are two ways to fix this:
> > >>
> > >> (a) change *all* license headers to be wrapped in e.g. `/* .. */`,
> also
> > update the coding guidelines, or
> > >> (b) perform some preprocessor-like magic in the Doxygen layer.
> > >>
> > >> Option (a) is very noise but obvious and stable; option (b) OTOH
> > employs a simple but stupid text replacement under the covers codified in
> > the Doxygen config; it might produce some artifacts and be surprising
> since
> > the code Doxygen sees will be different from what is in the source.
> > >>
> > >> I personally believe option (a) is superior for purely technical
> reasons
> >
> > +1 for (a); there's no value in showing license headers to doxygen or
> > tooling workarounds
> >
> > >> with option (b) a possible temporary workaround.
> > >>
> > >>
> > >> To make sure that the generated documentation shows actual
> > documentation content in overviews like
> > http://mesos.apache.org/api/latest/c++/annotated.html and elsewhere we
> > should fix this. Please comment in the Jira issue (
> > https://issues.apache.org/jira/browse/MESOS-3581) your input on how you
> > think this should be fixed (short- and longterm).
> > >>
> > >>
> > >> Cheers,
> > >>
> > >> Benjamin
> > >
> >
> >
>

Re: minor doc fixes - do I need to have a JIRA ticket and a shepherd and a compass and a map of the pacific NW?

[Joris Van Remoortere]

Hi Erik!
For a simple patch like spelling / grammar mistakes you don't need a
shepherd or a JIRA.
Just submit your patch, and add a committer as a reviewer.
If you don't have a relationship yet with one, you can add me
(jvanremoortere), or in general just ask in IRC and someone will volunteer.

I'm sorry the current impression of the contribution process is laborious.
Mpark and I gave a talk about this at MesosCon EU. Hopefully the material
will be available soon!

Thanks for contributing!
Joris

—
*Joris Van Remoortere*
Mesosphere

On Tue, Oct 20, 2015 at 11:36 PM, Erik Weathers 
wrote:

> i.e., the process for making contributions to the Mesos code base is, uh,
> how shall I say it... *involved*.
>
>- https://mesos.apache.org/documentation/latest/submitting-a-patch/
>
> Hoping I don't have to jump through all the hoops to make some minor doc
> fixes.
>
> Appreciate if anyone can tell me how to easily submit some doc fixes
> without doing all of the possible 51 (sub)steps on that page.  Notably, my
> fixes include a spelling/grammar fix to the doc I just linked above (how
> meta).
>
> Thanks!!
>
> - Erik
>

Re: Proposal: move towards #pragma and away from #include guards

[Joris Van Remoortere]

+1 with all-at-once

—
*Joris Van Remoortere*
Mesosphere

On Thu, Nov 5, 2015 at 9:37 AM, Kapil Arya  wrote:

> +1.
>
> I think we should do it all at once as Artem mentioned. This doesn't really
> affect the history (git-blame, etc.) because we are not touching code per
> se.
>
> On Thu, Nov 5, 2015 at 12:29 PM, Artem Harutyunyan 
> wrote:
>
> > Hi Alex,
> >
> > While I agree with the idea in general, I strongly believe that we should
> > either leave it as it is or fix everything in one go (i.e. three
> > consecutive commits). Having both #include guards and #pragmas in the
> > codebase will be confusing and untidy. We have done code sweeps like this
> > in the past when we had to introduce changes to the style guide, so if
> > folks agree you just need to find a shepherd and do it :).
> >
> > Cheers,
> > Artem.
> >
> > On Wed, Nov 4, 2015 at 9:36 PM, Alex Clemmer <
> clemmer.alexan...@gmail.com>
> > wrote:
> >
> > > Hey folks.
> > >
> > > In r/39803[1], Mike Hopcroft (in quintessential MSFT style, heh)
> > > brought up the issue of moving away from #include guards and towards
> > > `#pragma once`.
> > >
> > > As this has been brought up before, I will be brief: we think it's
> > > revisiting because the primary objection in previous threads appears
> > > to be that, though `#pragma once` is a cleaner solution to the
> > > multiple-include problem, it's not so much better that it's worth the
> > > code churn. However, the ongoing Windows integration work means we
> > > have to touch these files anyway, so if we agree this is cleaner and
> > > desirable, then this is an opportunity to obtain that additional code
> > > clarity, without the cost of the churn.
> > >
> > > For the remainder of the email, I will summarize the history of our
> > > discussion of this issue, who will do the work, and what the next
> > > steps are.
> > >
> > > PROPOSAL: We propose that all new code use `#pragma once` instead of
> > > #include guards; for existing files, we propose that you change
> > > #include guards when you touch them.
> > >
> > > HISTORY: This has been discussed before, most recently a year ago on
> > > the mailing list[2]. There is a relevant JIRA[3] and discarded
> > > review[4] that changes style guide's recommendation on the matter.
> > >
> > > SUMMARIZED OBJECTIONS:
> > > 1. The Google style guide explicitly forbids `#pragma once`.
> > > 2. This results in a lot of code churn, but is only marginally better.
> > > 3. It's not C++ standardized/it's platform dependent/IBM's compiler
> > > doesn't support it.
> > > 4. Popular projects like Chrome don't do `#pragma once` because of
> > > history clutter.
> > > 5. Intermediate state of inconsistency as we transition to `#pragma
> > > once` from #include guards.
> > >
> > > OUR RESPONSE:
> > > Objections (1), (2), and (4) are essentially the same -- Dominic Hamon
> > > points out in a previous thread that the Google style guide was
> > > canonized when `#pragma once` was Windows-only, and the guidance has
> > > not changed since because of the history churn problem. As noted
> > > above, we think the history churn problem is minimized by the fact
> > > that it can be wrapped up into the Windows integration work.
> > >
> > > For objection (3), the consensus seems to be that the vast majority of
> > > compilers we care about (in particular, the ones supporting C++ 11) do
> > > support it.
> > >
> > > For objection (5) we believe the inconsistent state is likely to not
> > > be long lived, as long as we commit to wrapping this work up into the
> > > Windows integration work.
> > >
> > > SUMMARIZED ADVANTAGES:
> > > * Basically fool-proof. Communicates simply what its function is (you
> > > include this file once). Semantically it is "the right tool for the
> > > job".
> > > * No need for namespacing conventions for #include guards.
> > > * No conflicts with reserved identifiers[5].
> > > * No internal conflicts between include guards in Stout, Process
> > > library, and Mesos (this is one reason we need the namespacing
> > > conventions)
> > > * Reduces preprocessor definition clutter (we should rely on #define
> > > as little as humanly possible).
> > > * Optimized to be easy to read and reason 

Re: Mesos Style Guideline Adjustments

[Joris Van Remoortere]

@ben Is "jaggedness" the only *formatting* influence on the readability of
the comments? If so, would it be possible to make a simple github.io page
where we can paste comments, and they get formatted for minimal jaggedness
based on some simple math? This would help provide consistency between
contributions, and likely better results than humans trying to optimize the
jaggedness equation.

This way we can focus on the content of the comments when reviewing, rather
than the format. Later one we might even be able to incorporate this in
more intelligent editor friendly formatting tools.

If there is more than some simple math involved, this may not be a viable
solution.

Joris

—
*Joris Van Remoortere*
Mesosphere

On Fri, Nov 6, 2015 at 11:10 AM, Benjamin Mahler 
wrote:

> I raise this because there is a ton of value in keeping our codebase as
> readable as possible. Having had to navigate and maintain the code base for
> the past few years, this _is_ a "real issue" that deserves the contributors
> spending their mental resources on. I realize this seems very subtle, but
> it is of critical importance for the maintainability of the project that
> folks are paying attention to how their code and comments will be read by
> others.
>
> I think about this as follows: we'll always have "soft" rules that cannot
> be enforced by these style checkers but require mentorship to communicate
> as we grow the community. These tools cannot tell you how to structure your
> code in a simple form. They also can't tell you how to write a comment in a
> way that is clear to others.
>
> To Alex's example, two comments:
>
> (1) The second comment is poorly written, did no one even notice this??
> That's a "real issue" :(
> (2) It's important not to isolate the comments from the code. The comments
> live with the code and a major reason why long line length paragraphs are
> irregular is because the majority of code lines do not approach 80
> characters.
> (3) We tend to separate "multiple logical parts" of a comment with an empty
> // line, which helps readability.
> (4) I'm not saying wrap at 70 or at 80, but (a) write and wrap to reduce
> "jaggedness" (or to increase "regularity") and (b) long line lengths (80)
> for large paragraphs are harder to read.
>
> I just don't want the formatter to become a crutch that causes folks to
> think less about the implications of how they write their comments. Do the
> soft rules in (a) and (b) seem reasonable? We already need to be diligent
> in reviews to ensure that comments are well-written.
>
> On Fri, Nov 6, 2015 at 8:44 AM, Benjamin Bannier <
> benjamin.bann...@mesosphere.io> wrote:
>
> > Hi,
> >
> > just to echo Alexander’s point, for newbies like me being able to
> delegate
> > formatting decisions to tools as much as possible frees up a lot of
> mental
> > resources for tackling the real issues.
> >
> >
> > Cheers,
> >
> > Benjamin
> >
> > ps. Also looking forward to an updated and expanded clang-format config.
> >
> >
> > > On Nov 6, 2015, at 1:44 PM, Alexander Rojas 
> > wrote:
> > >
> > > I think one of the main reasons we move to having 80 as the limit for
> > both code and comments is the ability it gives us to use tools (e.g.
> > clang-format) to enforce formatting rules, so personally I rather have us
> > putting effort towards that goal. On that note, the developer branch of
> > clang-format allows a much closer formatting options to the ones we use.
> On
> > OS X it can be installed using `brew install --HEAD clang-format`.
> > >
> > > Right now I’m working on setting the config file to be as close as
> > possible to our style.
> > >
> > >> On 06 Nov 2015, at 10:09, Alex Rukletsov  wrote:
> > >>
> > >> I think jaggedness in the example you provide comes mainly from the
> fact
> > >> that the second comment has multiple logical blocks. I have formatted
> > both
> > >> comments at 70 and at 80, here is the outcome:
> > http://pastebin.com/nRQB0nCD
> > >>
> > >> While the first comment indeed looks better when wrapped at 70, I
> can't
> > say
> > >> the same about the second one.
> > >>
> > >> I would say, that the longer a line could be, the less jagged the
> > comment
> > >> block is. The ratio (`averageWordLength` / `maxLineLength`) approaches
> > 0 as
> > >> `maxLineLenght` approaches infinity, which means wrapping a long word
> > right
> > >> before the line end shou

Re: [2/2] mesos git commit: Added JSON parsing for Resources.

[Joris Van Remoortere]

fixed

—
*Joris Van Remoortere*
Mesosphere

On Tue, Nov 10, 2015 at 9:53 PM, James Peach  wrote:

>
> > On Nov 9, 2015, at 10:25 AM, m...@apache.org wrote:
> >
> > Added JSON parsing for Resources.
> >
> >
>
> [snip]
>
> >
> http://git-wip-us.apache.org/repos/asf/mesos/blob/62f17731/src/tests/resources_tests.cpp
> > --
> > diff --git a/src/tests/resources_tests.cpp
> b/src/tests/resources_tests.cpp
> > index 209d708..e663f5e 100644
> > --- a/src/tests/resources_tests.cpp
> > +++ b/src/tests/resources_tests.cpp
> > @@ -160,6 +160,577 @@ TEST(ResourcesTest, ParsingFromJSON)
> >   ASSERT_SOME(parse);
> >
> >
>
> [snip]
>
> > +  EXPECT_EQ(Value::RANGES, ports->type());
> > +  EXPECT_EQ(2, ports->ranges().range_size());
> > +
> > +  // Do not specify the ordering of ranges, only check the values.
> > +  if (1 != ports->ranges().range(0).begin()) {
> > +EXPECT_EQ(3, ports->ranges().range(0).begin());
> > +EXPECT_EQ(1, ports->ranges().range(1).begin());
> > +  } else {
> > +EXPECT_EQ(3, ports->ranges().range(1).begin());
> > +  }
>
> GCC 4.9.2 really wants unsigned constants here ...
>
>   CXX  tests/mesos_tests-resources_tests.o
> In file included from
> /opt/home/src/mesos.git/src/tests/resources_tests.cpp:23:0:
> ../3rdparty/libprocess/3rdparty/gmock-1.7.0/gtest/include/gtest/gtest.h:
> In instantiation of ‘testing::AssertionResult
> testing::internal::CmpHelperEQ(const char*, const char*, const T1&, const
> T2&) [with T1 = int; T2 = long unsigned int]’:
> ../3rdparty/libprocess/3rdparty/gmock-1.7.0/gtest/include/gtest/gtest.h:1485:30:
>  required from ‘static testing::AssertionResult
> testing::internal::EqHelper::Compare(const char*,
> const char*, const T1&, const T2&) [with T1 = int; T2 = long unsigned int;
> bool lhs_is_null_literal = false]’
> /opt/home/src/mesos.git/src/tests/resources_tests.cpp:219:5:   required
> from here
> ../3rdparty/libprocess/3rdparty/gmock-1.7.0/gtest/include/gtest/gtest.h:1448:16:
> error: comparison between signed and unsigned integer expressions
> [-Werror=sign-compare]
>if (expected == actual) {
> ^
>
> J
>
>

Re: `F()` vs `F(void)`

[Joris Van Remoortere]

+1

—
*Joris Van Remoortere*
Mesosphere

On Sun, Dec 13, 2015 at 11:39 AM, Benjamin Mahler  wrote:

> std::function becomes std::function?
>
> +1
>
> On Sun, Dec 13, 2015 at 10:55 AM, Jie Yu  wrote:
>
> > +1
> >
> > On Sun, Dec 13, 2015 at 10:46 AM, Michael Park  wrote:
> >
> > > Hello,
> > >
> > > In the C++ world, the *void* parameter is considered to be only there
> > for C
> > > compatibility reasons.
> > >
> > > We do a good job of not using *void *parameters in function
> declarations,
> > > e.g., *void F();*. On the other hand, we're *not* so good doing so for
> > > function types, e.g., *std::function*.
> > >
> > > I would like to see the codebase converge to *not* use *void* as a
> > > parameter type, and would like feedback if anyone holds strong opposing
> > > opinions.
> > >
> > > Thanks,
> > >
> > > MPark.
> > >
> >
>

Fwd: [VOTE] Release Apache Mesos 0.26.0 (rc5)

[Joris Van Remoortere]

+1 (binding)

From: Till Toenshoff 
Date: Thu, Dec 10, 2015 at 2:55 PM
Subject: [VOTE] Release Apache Mesos 0.26.0 (rc5)
To: u...@mesos.apache.org, dev 


Hi friends,

we did unfortunately, once again run into an issue that needed immediate
attention (see vote on rc4), hence we have to ask for another round of
testing and voting of this newest release-candidate.

The issue leading to this new release candidate was
https://issues.apache.org/jira/browse/MESOS-4106 <
https://issues.apache.org/jira/browse/MESOS-4106>. Apart from that, we also
pulled in a fix for https://issues.apache.org/jira/browse/MESOS-4015 <
https://issues.apache.org/jira/browse/MESOS-4015> as we believe it has
minimal additional risk while being very useful for some of us.

Please vote on releasing the following candidate as Apache Mesos 0.26.0.

The CHANGELOG for the release is available at:
https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=0.26.0-rc5


The candidate for Mesos 0.26.0 release is available at:
https://dist.apache.org/repos/dist/dev/mesos/0.26.0-rc5/mesos-0.26.0.tar.gz

The tag to be voted on is 0.26.0-rc5:
https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=0.26.0-rc5

The MD5 checksum of the tarball can be found at:
https://dist.apache.org/repos/dist/dev/mesos/0.26.0-rc5/mesos-0.26.0.tar.gz.md5

The signature of the tarball can be found at:
https://dist.apache.org/repos/dist/dev/mesos/0.26.0-rc5/mesos-0.26.0.tar.gz.asc

The PGP key used to sign the release is here:
https://dist.apache.org/repos/dist/release/mesos/KEYS

The JAR is up in Maven in a staging repository here:
https://repository.apache.org/content/repositories/orgapachemesos-1095

Please vote on releasing this package as Apache Mesos 0.26.0!

The vote is open until Tue Dec 15 22:35:22 CET 2015 and passes if a
majority of at least 3 +1 PMC votes are cast.

[ ] +1 Release this package as Apache Mesos 0.26.0
[ ] -1 Do not release this package because ...

Thanks,
Till & Bernd

Re: shepherd for MESOS-3909

[Joris Van Remoortere]

I will. I think Joseph can (and already has) help review this.

—
*Joris Van Remoortere*
Mesosphere

On Tue, Dec 22, 2015 at 11:02 AM, James Peach  wrote:

> Hi all,
>
> Can anyone shepherd https://issues.apache.org/jira/browse/MESOS-3909?
>
> Mesos modules depend on picojson, which AFAICT is not included in any
> distributions. This change installs picojson.h if Mesos was built with the
> bundled picojson (99% of cases IMHO). Even if you bring your own
> piconjson.h, I think this is required for correct builds since it would
> otherwise be possible for the version you use in your module to be
> mismatched with the version compiled into Mesos.
>
> Since picojson is an implementation detail of stout/json.hpp, I don't
> really think it should be exposed at all, but that would require stout to
> be a full library (not header-only).
>
> thanks,
> James

Re: Change minimum supported version for GCC to 4.8.1

[Joris Van Remoortere]

+1 (binding)

I would like to propose that we bump our minimum supported version for gcc
> from 4.8.0 to 4.8.1. The main motivation behind this is that there are at
> least 2 outstanding reviews on RB that want to use ref-qualifiers <
> http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2439.htm>
> introduced in GCC 4.8.1. The reviews in question are r41870 <
> https://reviews.apache.org/r/41870> and r41593 <
> https://reviews.apache.org/r/41593/diff/2?file=1173648#file1173648line58>.
>
> Also, our Getting Started document <
> http://mesos.apache.org/gettingstarted/> for Mesos already lists the
> minimum gcc version as > 4.8. Looking at the release timeline <
> https://gcc.gnu.org/gcc-4.8/> for GCC, it seems that 4.8.0/4.8.1 were
> released within a week of each other.
>
> Does anyone have a strong opinion against this change ?
>
> -anand
>

Re: Mesos 0.27.0

[Joris Van Remoortere]

+1

On Wed, Jan 13, 2016 at 1:27 PM, Timothy Chen  wrote:

> Hi all,
>
> As we continue with the monthly release cadence, we should be cutting
> a new release a month from the last one (12/16/15), which the next
> version will be 0.27.0.
>
> MPark, Kapil and I are volunteering to be the release manager for
> 0.27.0, let us know if there are any questions and definitely want and
> welcome the community help.
>
> MPark has created a dashboard for 0.27.0 release:
> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12327632
>
> We plan to try to cut a preview next monday (01/18) and a RC on
> wednesday (01/20).
>
> Please start updating your resolved jira tickets to either add target
> version to 0.27.0 or remove open jira tickets if it's not going to
> land before the weekend, which will just roll into the next release
> next month.
>
> Thanks!
>

Re: Links in documentation

[Joris Van Remoortere]

>
> *In fact it seems that all links ending with .md are interpreted as
> relative links on the webpage, i.e. [label](https://test.com/foo.md) is
> rendered into https://test.com/foo/
> ">label.


I think this should be fixed. We shouldn't be restricted from linking to
external documentation.

—
*Joris Van Remoortere*
Mesosphere

On Thu, Jan 14, 2016 at 4:44 AM, Jörg Schad  wrote:

> Hi,
> just a short note about links in our documentation.
> In the current documentation we use three different ways to link between
> different *.md pages:
> 1. [label](/documentation/latest/foo/)
> 2. [label](foo.md)
> 3. [label](https://github.com/apache/mesos/blob/master/docs/foo.md)
>
> First of all, option 3 should *not* be used as it is rendered incorrectly
> onto website* and rather long.
>
> Between option 1 and 2 Neil and myself discussed (MESOS-4295) and are
> favoring option 2 as it
> - previews better on github
> - is shorter
> - is easier to maintain multiple versions of the same doc.
>
> Any comments or objections?
>
> Thanks for your feedback!
>
> *In fact it seems that all links ending with .md are interpreted as
> relative links on the webpage, i.e. [label](https://test.com/foo.md) is
> rendered into https://test.com/foo/
> ">label.
>

JIRA Shepherds

[Joris Van Remoortere]

Hello Mesos developers,

You may have noticed some churn in Jira recently around the shepherd
assignment. Specifically, we have unassigned the shepherds for a bunch of
projects. We did this in order to get a better sense of which projects are
being actively shepherded versus having gone stale, and to identify for
which projects we need to find a new shepherd who has sufficient time to
dedicate to it.

This is not a statement that the un-assigned tickets are not important,
rather, we want to ensure that the people working on them have a shepherd
with sufficient resources.

We ask that you communicate (and agree!) with your shepherd before
assigning them in Jira, so that they are not surprised when you reviews
start getting posted.

The benefit for the developer community should be that it will be more
clear when working on a ticket whether there are sufficient resources in
the community to iterate on it in a timely manner.

Joris

Re: [VOTE] Release Apache Mesos 0.27.0 (rc1)

[Joris Van Remoortere]

-1 (binding)
There is a bug with mutable resource math that surfaced in the allocator
changes, but is also happening elsehwere.
Mpark is currently working on a patch to disallow this behavior in general,
and fix the current issues.

https://issues.apache.org/jira/browse/MESOS-4030


On Wed, Jan 27, 2016 at 6:51 AM, Alexander Rojas 
wrote:

> -1 (non binding)
>
> Test on Debian 8 under a VirtualBox debian machine. While doing
>
> `configure && sudo make -j4 check`
>
> and then:
>
> `sudo ./bin/mesos-tests.sh
> --gtest_filter=“-*.ROOT_CGROUPS_Listen:*.ROOT_IncreaseRSS"
> --gtest_repeat=10 --gtest_break_on_failure`
>
> I got failures on DockerContainerizerTest.ROOT_DOCKER_NC_PortMapping and
> NetClsIsolatorTest.ROOT_CGROUPS_NetClsIsolate.
>
> The first had a JIRA entry but was marked as resolved in 0.26.0 with a
> cannot reproduce. The second didn’t have any issues open.
>
> My -1 vote will stand until the failing tests are decidedly non blockers
> for the release.
>
> > On 27 Jan 2016, at 07:53, Timothy Chen  wrote:
> >
> > Hi all,
> >
> > Please vote on releasing the following candidate as Apache Mesos 0.27.0.
> >
> > 0.27.0 includes the following:
> >
> 
> > We added major features such as Implicit Roles, Quota, Multiple Disks
> and more.
> >
> > We also added major bug fixes such as performance improvements to
> > state.json requests and GLOG.
> >
> > The CHANGELOG for the release is available at:
> >
> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=0.27.0-rc1
> >
> 
> >
> > The candidate for Mesos 0.27.0 release is available at:
> >
> https://dist.apache.org/repos/dist/dev/mesos/0.27.0-rc1/mesos-0.27.0.tar.gz
> >
> > The tag to be voted on is 0.27.0-rc1:
> >
> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=0.27.0-rc1
> >
> > The MD5 checksum of the tarball can be found at:
> >
> https://dist.apache.org/repos/dist/dev/mesos/0.27.0-rc1/mesos-0.27.0.tar.gz.md5
> >
> > The signature of the tarball can be found at:
> >
> https://dist.apache.org/repos/dist/dev/mesos/0.27.0-rc1/mesos-0.27.0.tar.gz.asc
> >
> > The PGP key used to sign the release is here:
> > https://dist.apache.org/repos/dist/release/mesos/KEYS
> >
> > The JAR is up in Maven in a staging repository here:
> > https://repository.apache.org/content/repositories/orgapachemesos-1097
> >
> > Please vote on releasing this package as Apache Mesos 0.27.0!
> >
> > The vote is open until Fri Jan 29 23:59:59 PST 2016 and passes if a
> > majority of at least 3 +1 PMC votes are cast.
> >
> > [ ] +1 Release this package as Apache Mesos 0.27.0
> > [ ] -1 Do not release this package because ...
> >
> > Thanks,
> >
> > Tim, MPark and Kapil
>
>

Re: [VOTE] Release Apache Mesos 0.27.0 (rc1)

[Joris Van Remoortere]

Correction to my previous e-mail.
The mutable resource bug is here:
https://issues.apache.org/jira/browse/MESOS-4534

On Wed, Jan 27, 2016 at 11:57 AM, Joris Van Remoortere <
joris.van.remoort...@gmail.com> wrote:

> -1 (binding)
> There is a bug with mutable resource math that surfaced in the allocator
> changes, but is also happening elsehwere.
> Mpark is currently working on a patch to disallow this behavior in
> general, and fix the current issues.
>
> https://issues.apache.org/jira/browse/MESOS-4030
>
>
> On Wed, Jan 27, 2016 at 6:51 AM, Alexander Rojas 
> wrote:
>
>> -1 (non binding)
>>
>> Test on Debian 8 under a VirtualBox debian machine. While doing
>>
>> `configure && sudo make -j4 check`
>>
>> and then:
>>
>> `sudo ./bin/mesos-tests.sh
>> --gtest_filter=“-*.ROOT_CGROUPS_Listen:*.ROOT_IncreaseRSS"
>> --gtest_repeat=10 --gtest_break_on_failure`
>>
>> I got failures on DockerContainerizerTest.ROOT_DOCKER_NC_PortMapping and
>> NetClsIsolatorTest.ROOT_CGROUPS_NetClsIsolate.
>>
>> The first had a JIRA entry but was marked as resolved in 0.26.0 with a
>> cannot reproduce. The second didn’t have any issues open.
>>
>> My -1 vote will stand until the failing tests are decidedly non blockers
>> for the release.
>>
>> > On 27 Jan 2016, at 07:53, Timothy Chen  wrote:
>> >
>> > Hi all,
>> >
>> > Please vote on releasing the following candidate as Apache Mesos 0.27.0.
>> >
>> > 0.27.0 includes the following:
>> >
>> 
>> > We added major features such as Implicit Roles, Quota, Multiple Disks
>> and more.
>> >
>> > We also added major bug fixes such as performance improvements to
>> > state.json requests and GLOG.
>> >
>> > The CHANGELOG for the release is available at:
>> >
>> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=0.27.0-rc1
>> >
>> 
>> >
>> > The candidate for Mesos 0.27.0 release is available at:
>> >
>> https://dist.apache.org/repos/dist/dev/mesos/0.27.0-rc1/mesos-0.27.0.tar.gz
>> >
>> > The tag to be voted on is 0.27.0-rc1:
>> >
>> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=0.27.0-rc1
>> >
>> > The MD5 checksum of the tarball can be found at:
>> >
>> https://dist.apache.org/repos/dist/dev/mesos/0.27.0-rc1/mesos-0.27.0.tar.gz.md5
>> >
>> > The signature of the tarball can be found at:
>> >
>> https://dist.apache.org/repos/dist/dev/mesos/0.27.0-rc1/mesos-0.27.0.tar.gz.asc
>> >
>> > The PGP key used to sign the release is here:
>> > https://dist.apache.org/repos/dist/release/mesos/KEYS
>> >
>> > The JAR is up in Maven in a staging repository here:
>> > https://repository.apache.org/content/repositories/orgapachemesos-1097
>> >
>> > Please vote on releasing this package as Apache Mesos 0.27.0!
>> >
>> > The vote is open until Fri Jan 29 23:59:59 PST 2016 and passes if a
>> > majority of at least 3 +1 PMC votes are cast.
>> >
>> > [ ] +1 Release this package as Apache Mesos 0.27.0
>> > [ ] -1 Do not release this package because ...
>> >
>> > Thanks,
>> >
>> > Tim, MPark and Kapil
>>
>>
>

Re: [VOTE] Release Apache Mesos 0.27.0 (rc2)

[Joris Van Remoortere]

+1 (binding)

Tested on  Ubuntu 15.04:

   - Built with clang-3.6 and gcc-4.8.
   - Launched 1K agents.
   - Launched 100K tasks (slowly) on 100 agents.
  - Validated virtual memory set did not grow.
  - Validate UI was responsive.
   - Validate task recovery with 0.26 Master -> 0.27 Master with 0.27 Agent
   constant.
   - Validate task recovery with 0.26 Master -> 0.26 Master with 0.26 Agent
   constant.
   - Validate task recovery with 0.26 Agent -> 0.27 Agent with 0.26 Master
   constant.
   - Validate task recovery with 0.26 Agent -> 0.27 Agent with 0.27 Master
   constant.
   - Validated constrained framework when quota set.
   - Validated framework received remaining offers if quota set on a
   separate role.

Broken not considered blockers (Feel free to decide they are a blocker):

   - Web UI task count is off (MESOS-4562)

I was not able to verify the docker containerizer tests in my environment.

On Fri, Jan 29, 2016 at 1:45 PM, Vinod Kone  wrote:

> +1 (binding)
>
> Tested (non-root tests with ./support/docker_build.sh script) on ASF CI
> with
> the following configurations:
>
> NOTE: clang is intentionally disabled on centos and "clang + ubuntu +
> --verbose" build failed because of a docker image building issue (not mesos
> issue).
>
> Configuration Matrix gcc clang
> centos:7 --verbose --enable-libevent --enable-ssl
> [image: Success]
> <
> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/4/COMPILER=gcc,CONFIGURATION=--verbose%20--enable-libevent%20--enable-ssl,OS=centos%3A7,label_exp=docker%7C%7CHadoop/
> >
> [image: Not run]
> --verbose
> [image: Success]
> <
> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/4/COMPILER=gcc,CONFIGURATION=--verbose,OS=centos%3A7,label_exp=docker%7C%7CHadoop/
> >
> [image: Not run]
> ubuntu:14.04 --verbose --enable-libevent --enable-ssl
> [image: Success]
> <
> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/4/COMPILER=gcc,CONFIGURATION=--verbose%20--enable-libevent%20--enable-ssl,OS=ubuntu%3A14.04,label_exp=docker%7C%7CHadoop/
> >
> [image: Success]
> <
> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/4/COMPILER=clang,CONFIGURATION=--verbose%20--enable-libevent%20--enable-ssl,OS=ubuntu%3A14.04,label_exp=docker%7C%7CHadoop/
> >
> --verbose
> [image: Success]
> <
> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/4/COMPILER=gcc,CONFIGURATION=--verbose,OS=ubuntu%3A14.04,label_exp=docker%7C%7CHadoop/
> >
> [image: Failed]
> <
> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/4/COMPILER=clang,CONFIGURATION=--verbose,OS=ubuntu%3A14.04,label_exp=docker%7C%7CHadoop/
> >
>
> On Fri, Jan 29, 2016 at 10:43 AM, Jörg Schad  wrote:
>
> > [+1] (non-binding)
> >
> > I tested on Ubuntu 12.04 and OSX 10.10.5 without SSL.
> >
> > *make check* works on both platforms with no issues.
> >
> > With *sudo make check *some tests fail.
> > The ubuntu failure also occurs with other tests and seems to be related
> to
> > the cgroups configuration on that system. Currently looking into the
> > details here (thanks to Avinash for his support!) and would open a Jira
> if
> > necessary.
> > The 3 failures on OSX are known issues (see below for the respective Jira
> > tickets).
> >
> > *Ubuntu 12.04*
> >
> > [ RUN  ] AuthenticationTest.UnauthenticatedSlave
> >
> > F0129 15:55:48.241842 11049 cluster.cpp:369] CHECK_SOME(containerizer):
> > Could not create MesosContainerizer: Failed to create launcher: Failed to
> > create Linux launcher: Failed to mount cgroups hierarchy at
> > '/sys/fs/cgroup/freezer': Failed to create directory
> > '/sys/fs/cgroup/freezer': No such file or directory
> >
> > *** Check failure stack trace: ***
> >
> > @ 0x2b969832bd0e  google::LogMessage::Fail()
> >
> > @ 0x2b969832bc48  google::LogMessage::SendToLog()
> >
> > @ 0x2b969832b64a  google::LogMessage::Flush()
> >
> > @ 0x2b969832e562  google::LogMessageFatal::~LogMessageFatal()
> >
> > @   0x97fe2a  _CheckFatal::~_CheckFatal()
> >
> > @   0xa5376f
> mesos::internal::tests::Cluster::Slaves::start()
> >
> > @   0xffc286  mesos::internal::tests::MesosTest::StartSlave()
> >
> > @   0x9bd542
> >
> mesos::internal::tests::AuthenticationTest_UnauthenticatedSlave_Test::TestBody()
> >
> > @  0x164b8fc
> > testing::internal::HandleSehExceptionsInMethodIfSupported<>()
> >
> > @  0x164678a
> > testing::internal::HandleExceptionsInMethodIfSupported<>()
> >
> > @  0x1627919  testing::Test::Run()
> >
> > @  0x162809e  testing::TestInfo::Run()
> >
> > @  0x16286ee  testing::TestCase::Run()
> >
> > @  0x162ef36  testing::internal::UnitTestImpl::RunAllTests()
> >
> > @  0x164c521
> > testing::internal::HandleSehExceptionsInMethodIfSupported<>()
> >
> > @  0x164732a
> > testing::intern

Re: [RESULT][VOTE] Release Apache Mesos 0.27.0 (rc2)

[Joris Van Remoortere]

If we did feature based releases, we could hold the release for the
documentation.
Since we do time based releases, it so happened that the release went
through before we could get the docs done.

I'll be working on the docs shortly :-)

—
*Joris Van Remoortere*
Mesosphere

On Thu, Feb 4, 2016 at 8:59 AM, Kapil Arya  wrote:

> That's a good point, Niklas!
>
> IMO, we should be able to ship a major feature as long as we have an
> assurance that the user doc will land in the next few days (not
> weeks).
>
> On Thu, Feb 4, 2016 at 11:55 AM, Niklas Nielsen  wrote:
> > Thanks Jie and Kapil! Looking forward to it.
> >
> > I did read the proposal, but playing the devils advocate for a bit; how
> can
> > we ship a 'major feature' without bundled user documentation? Referring
> to
> > tickets and design docs (which may/or may not be the way things ended up
> > getting implemented) or to code is not good enough, in my mind.
> >
> > Should we line up some expectations to available user docs, example
> > framework code, etc. before announcing the availability of a feature?
> >
> > Niklas
> >
> > On Thu, Feb 4, 2016 at 5:41 PM, Kapil Arya  wrote:
> >>
> >> And here is the ticket tracking the user doc:
> >> https://issues.apache.org/jira/browse/MESOS-4531. Will link to the
> >> blog post once the doc is ready :-).
> >>
> >> On Thu, Feb 4, 2016 at 11:38 AM, Jie Yu  wrote:
> >> > Niklas,
> >> >
> >> > I think Joris is still working on the user doc for multi-disk support
> in
> >> > Mesos.
> >> >
> >> > - Jie
> >> >
> >> > On Thu, Feb 4, 2016 at 1:22 AM, Niklas Nielsen  wrote:
> >> >
> >> >> Awesome guys!
> >> >>
> >> >> Kapil, we usually linked to the user documentation in the blog to the
> >> >> new
> >> >> features. Do you have a link to the docs on multiple disk resources?
> >> >>
> >> >> On Wed, Feb 3, 2016 at 11:27 PM, Kapil Arya 
> >> >> wrote:
> >> >>
> >> >> > And here is the blog post:
> >> >> > http://mesos.apache.org/blog/mesos-0-27-0-released.
> >> >> >
> >> >> > On Wed, Feb 3, 2016 at 4:48 PM, Michael Park 
> >> >> > wrote:
> >> >> > > Kapil is currently working on it. We'll publish it shortly :)
> >> >> > >
> >> >> > > On 3 February 2016 at 13:41, Benjamin Mahler  >
> >> >> wrote:
> >> >> > >>
> >> >> > >> Great! Is a blog post on the way?
> >> >> > >>
> >> >> > >> On Sun, Jan 31, 2016 at 5:39 PM, Michael Park  >
> >> >> wrote:
> >> >> > >>
> >> >> > >> > Hi all,
> >> >> > >> >
> >> >> > >> > The vote for Mesos 0.27.0 (rc2) has passed with the
> >> >> > >> > following votes.
> >> >> > >> >
> >> >> > >> > +1 (Binding)
> >> >> > >> > --
> >> >> > >> > Vinod Kone
> >> >> > >> > Joris Van Remoortere
> >> >> > >> > Till Toenshoff
> >> >> > >> >
> >> >> > >> > +1 (Non-binding)
> >> >> > >> > --
> >> >> > >> > Jörg Schad
> >> >> > >> > Marco Massenzio
> >> >> > >> > Greg Mann
> >> >> > >> >
> >> >> > >> > There were no 0 or -1 votes.
> >> >> > >> >
> >> >> > >> > Please find the release at:
> >> >> > >> > https://dist.apache.org/repos/dist/release/mesos/0.27.0
> >> >> > >> >
> >> >> > >> > It is recommended to use a mirror to download the release:
> >> >> > >> > http://www.apache.org/dyn/closer.cgi
> >> >> > >> >
> >> >> > >> > The CHANGELOG for the release is available at:
> >> >> > >> >
> >> >> > >> >
> >> >> > >> >
> >> >> >
> >> >>
> >> >>
> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=0.27.0
> >> >> > >> >
> >> >> > >> > The mesos-0.27.0.jar has been released to:
> >> >> > >> > https://repository.apache.org
> >> >> > >> >
> >> >> > >> > The website (http://mesos.apache.org) will be updated
> shortly to
> >> >> > reflect
> >> >> > >> > this release.
> >> >> > >> >
> >> >> > >> > Thanks,
> >> >> > >> >
> >> >> > >> > Tim, Kapil, MPark
> >> >> > >> >
> >> >> > >
> >> >> > >
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Niklas
> >> >>
> >
> >
> >
> >
> > --
> > Niklas
>

Re: Need shepherd for MESOS-4353 [Limit the number of processes created by libprocess]

[Joris Van Remoortere]

Hi Maged,

I will shepherd this; however, I don't think it makes sense to make this a
maximum. Rather, it is just the number of libprocess_worker_threads.
The patch posted by James seems simpler and more to the point.
I would like to see something closer to that, addressing the concerns such
as not having silent failure.

—
*Joris Van Remoortere*
Mesosphere

On Tue, Feb 9, 2016 at 9:02 PM, Maged Michael 
wrote:

> https://issues.apache.org/jira/browse/MESOS-4353
>
> Thanks.
>

Re: Precision of scalar resources

[Joris Van Remoortere]

+1
Thanks for taking this on Neil!

—
*Joris Van Remoortere*
Mesosphere

On Fri, Feb 12, 2016 at 11:25 PM, Neil Conway  wrote:

> tl;dr:
>
> If you use resource values with more than three decimal digits of
> precision (e.g., you are launching a task that uses 2.5001 CPUs),
> please speak up!
>
> 
>
> Mesos uses floating point to represent scalar resource values, such as
> the number of CPUs in a resource offer or dynamic reservation. The
> master does resource math in floating point, which leads to a few
> problems:
>
> * due to roundoff error, frameworks can receive offers that have
> unexpected resource values (e.g., MESOS-3990)
> * various internal assertions in the master can fail due to roundoff
> error (e.g., MESOS-3552).
>
> In the long term, we can solve these problems by switching to a
> fixed-point representation for scalar values. However, that will
> require a long deprecation cycle.
>
> In the short term, we should make floating point behavior more
> reliable. To do that, I propose:
>
> (1) Resource values will support AT MOST three decimal digits of
> precision. Additional precision in resource values will be discarded
> (via rounding).
>
> (2) The master will internally used a fixed-point representation to
> avoid unpredictable roundoff behavior.
>
> For more details, please see the design doc here:
>
> https://docs.google.com/document/d/14qLxjZsfIpfynbx0USLJR0GELSq8hdZJUWw6kaY_DXc
> -- comments welcome!
>
> Thanks,
> Neil
>

Re: [VOTE] Release Apache Mesos 0.27.1 (rc1)

[Joris Van Remoortere]

+1 (binding)

   - systemd integration testing
   - upgrade path testing
   - launching and killing 10K tasks



On Wed, Feb 17, 2016 at 6:52 PM, Jörg Schad  wrote:

> + 1 (non-binding)
>
> Mac OS: non-root:  DockerFetcherPluginTest.INTERNET_CURL_FetchManifest
> flaky, already tracked by
> MESOS-4570
>root:  ExamplesTest.JavaFramework failed tracked
> already by MESOS-   3582
> FetcherCacheHttpTest.HttpMixed failed, already
> tracked by MESOS-   2858
> FetcherCacheTest.LocalUncachedExtract failed,
> tracked by MESOS-   3579
>upgraded cluster from 0.26.
>
>
> On Wed, Feb 17, 2016 at 2:50 PM, Zhitao Li  wrote:
> > +1 (non binding)
> >
> > Debian 8 (jessie) plain non root OK.
> > Mac OS X plain non root OK.
> > Ubuntu 14.04 plain/SSL with root and docker-engine 1.10: Only flaky test
> I observed on Ubuntu 14.04 is HealthCheckTest.HealthStatusChange, which is
> tracked in MESOS-1802 already.
> >
> >> On Feb 17, 2016, at 5:21 AM, Bernd Mathiske 
> wrote:
> >>
> >> +1 (binding)
> >>
> >> Test failures look a lot like with 0.27.0. Not clean, but nothing
> deemed too drastic yet.
> >>
> >> CentOS 7 plain:
> >> FetcherCacheHttpTest.HttpCachedSerialized flaky again, filed MESOS-4692
> >> LinuxFilesystemIsolatorTest.ROOT_ImageInVolumeWithRootFilesystem known
> as flaky: MESOS-4674
> >> LinuxFilesystemIsolatorTest.ROOT_MultipleContainers known as flaky:
> MESOS-4674
> >>
> >> CentOS 7 SSL-enabled:
> >> LinuxFilesystemIsolatorTest.ROOT_ImageInVolumeWithRootFilesystem,
> >> LinuxFilesystemIsolatorTest.ROOT_MultipleContainers
> >> both known as flaky: MESOS-4674
> >>
> >> CentOS 6 plain:  OK
> >>
> >> CentOS 6 SSL-enabled:
> >> MemoryPressureMesosTest.CGROUPS_ROOT_SlaveRecovery
> >> flaky as often observed before, probably MESOS-4053
> >>
> >> Ubuntu 14.04 plain/SSL, Ubuntu 12.04 plain/SSL, Ubuntu 15 plain: OK,
> >>
> >> Ubuntu 15 SSL-enabled:
> >> DockerContainerizerTest.ROOT_DOCKER_Logs known as flaky: MESOS-4676
> >>
> >> Other known frequently flaky tests that have not been tested this time
> (filtered out):
> >> HealthCheckTest.ROOT_DOCKER_DockerHealthyTask
> >> HealthCheckTest.ROOT_DOCKER_DockerHealthStatusChange
> >> HookTest.ROOT_DOCKER_VerifySlavePreLaunchDockerHook
> >> DockerContainerizerTest.ROOT_DOCKER_Launch_Executor
> >>
> >> Bernd
> >>
> >>> On Feb 17, 2016, at 1:52 AM, Michael Park  wrote:
> >>>
> >>> Hi all,
> >>>
> >>> Please vote on releasing the following candidate as Apache Mesos
> 0.27.1.
> >>>
> >>>
> >>> 0.27.1 includes the following:
> >>>
> 
> >>> * Improved `systemd` integration.
> >>> * Ability to disable `systemd` integration.
> >>>
> >>> * Additional performance improvements to /state endpoint.
> >>> * Removed duplicate "active" keys from the /state endpoint.
> >>>
> >>> The CHANGELOG for the release is available at:
> >>>
> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=0.27.1-rc1
> >>>
> 
> >>>
> >>> The candidate for Mesos 0.27.1 release is available at:
> >>>
> https://dist.apache.org/repos/dist/dev/mesos/0.27.1-rc1/mesos-0.27.1.tar.gz
> >>>
> >>> The tag to be voted on is 0.27.1-rc1:
> >>>
> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=0.27.1-rc1
> >>>
> >>> The MD5 checksum of the tarball can be found at:
> >>>
> https://dist.apache.org/repos/dist/dev/mesos/0.27.1-rc1/mesos-0.27.1.tar.gz.md5
> >>>
> >>> The signature of the tarball can be found at:
> >>>
> https://dist.apache.org/repos/dist/dev/mesos/0.27.1-rc1/mesos-0.27.1.tar.gz.asc
> >>>
> >>> The PGP key used to sign the release is here:
> >>> https://dist.apache.org/repos/dist/release/mesos/KEYS
> >>>
> >>> The JAR is up in Maven in a staging repository here:
> >>> https://repository.apache.org/content/repositories/orgapachemesos-1102
> >>>
> >>> Please vote on releasing this package as Apache Mesos 0.27.1!
> >>>
> >>> The vote is open until Fri Feb 19 17:00:00 PST 2016 and passes if a
> majority of at least 3 +1 PMC votes are cast.
> >>>
> >>> [ ] +1 Release this package as Apache Mesos 0.27.1
> >>> [ ] -1 Do not release this package because ...
> >>>
> >>> Thanks,
> >>>
> >>> Joris, MPark
> >>
> >
>

Re: 答复: Question about "Framework directly access Meso agent"

[Joris Van Remoortere]

If the main reason for contemplating this design is the size of the task
payload, have you considered a content addressable storage design?
For example: why can the task not be launched with URIs that allow the
agent to download the payload before launching the task? This fans out the
network load evenly among agents.

—
*Joris Van Remoortere*
Mesosphere

On Wed, Feb 17, 2016 at 9:35 PM, Suteng  wrote:

> Alex,
> We don't have test the performance of mesos. But we have develop a
> framework in house, which is like a simplified mesos, use to schedule a
> large number fine grain computation tasks.  We find that master will be a
> bottleneck. One reason is our task contain several KB data, and task number
> is quite huge.
> If we use mesos to replace it, maybe master still be a bottleneck.
>
> Master still do the resource bookkeeping, we can decompose launch task to
> two steps, firstly scheduler tell master which offer he wants, then master
> tell scheduler the address of agent. Secondly, scheduler can directly
> launch task to the agent, and also can directly send message to agent.
> Maybe I can do some test about the mesos master launch task throughput,
> with different number task data.
>
>
> -邮件原件-
> 发件人: C Rukletsov [mailto:a...@mesosphere.com]
> 发送时间: 2016年2月17日 18:04
> 收件人: dev
> 主题: Re: Question about "Framework directly access Meso agent"
>
> Suteng—
>
> such optimization makes sense in certain cases (e.g. sending a framework
> message), but it can be rather tricky in general, because the master has to
> maintain bookkeeping. Moreover, with the upcoming HTTP API it becomes
> harder for a framework to determine where to send messages to reach a
> specific agent.
>
> Have you done any performance tests and seen master becoming a bottleneck?
>
> On Wed, Feb 17, 2016 at 5:14 AM, Suteng  wrote:
>
> > Hi,
> >
> >
> >
> > Currently, Mesos framework’s task related operations lauchTask,
> > updateStatus and executorSendMessage etc., and resource related
> > operations resourceOffer etc., all operations are pass through Mesos
> Master.
> >
> > When the cluster and task number become huge, or with optimistic
> > resource offer, multi-framework concurrently launchTask, maybe Mesos
> > Master will be a bottleneck.
> >
> > Is possible for framework scheduler directly access Mesos agent,
> > launchTask, updateStatus and SendMessage2Executore to Mesos Agent
> > directly, bypass the Master?
> >
> > Will invoke big conflict with current mechanism?
> >
> >
> >
> > Looking forward to your comments and opinions.
> >
> >
> >
> > Best Regards,
> >
> > Teng
> >
> >
> >
> >
> >
> >
> >
> > Su Teng  00241668
> >
> >
> >
> > Distributed and Parallel Software Lab
> >
> > Huawei Technologies Co., Ltd.
> >
> > Email:sut...@huawei.com
> >
> >
> >
> >
> >
>

Re: Enable compiler optimization by default?

[Joris Van Remoortere]

+1 (binding)
with the condition that we fix the flag mixing problem between CXXFLAGS and
--enable-optimize --enable-debug, even if it is to disallow it for now.
I want to avoid surprising behavior with implicit flags such as optimize:
passing irrelevant CXXFLAGS magically turns it into a debug build.

On Thu, Feb 18, 2016 at 1:27 PM, Neil Conway  wrote:

> Great! I created https://issues.apache.org/jira/browse/MESOS-4709 to
> track this issue.
>
> Neil
>
> On Thu, Feb 18, 2016 at 12:43 AM, Jan Schlicht  wrote:
> > +1
> >
> > On Thu, Feb 18, 2016 at 2:34 AM, Klaus Ma 
> wrote:
> >
> >> +1;
> >>
> >> So our CI will also update to use optimisation flags, right?  We need to
> >> highlight this in upgrade document to our user; I used to meet so
> strange
> >> behaviour after changing -O level.
> >>
> >> On Thu, Feb 18, 2016 at 8:51 AM James DeFelice <
> james.defel...@gmail.com>
> >> wrote:
> >>
> >> > +1
> >> > On Feb 17, 2016 7:24 PM, "Neil Conway"  wrote:
> >> >
> >> > > Hi folks,
> >> > >
> >> > > At present, Mesos defaults to compiling with "-O0"; to enable
> compiler
> >> > > optimizations, the user needs to specify "--enable-optimize".
> >> > >
> >> > > I'd like to propose we change the default, for a few reasons:
> >> > >
> >> > > (1) The autoconf default for CFLAGS/CXXFLAGS is "-O2 -g".
> Anecdotally,
> >> > > I think most software packages compile with a reasonable level of
> >> > > optimizations enabled by default.
> >> > >
> >> > > (2) I think we should make the default configure flags appropriate
> for
> >> > > end-users (rather than Mesos developers): developers will be
> familiar
> >> > > enough with Mesos to tune the configure flags according to their own
> >> > > preferences.
> >> > >
> >> > > (3) The performance consequences of not enabling compiler
> >> > > optimizations can be pretty severe: 5x in a benchmark I just ran,
> and
> >> > > we've seen between 2x and 30x (!) performance differences for some
> >> > > real-world workloads.
> >> > >
> >> > > Neil
> >> > >
> >> >
> >> --
> >>
> >> Regards,
> >> 
> >> Da (Klaus), Ma (马达), PMP® | Advisory Software Engineer
> >> IBM Platform Development & Support, STG, IBM GCG
> >> +86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me
> >>
> >
> >
> >
> > --
> > *Jan Schlicht*
> > Distributed Systems Engineer, Mesosphere
>

Re: Anonymous modules running context - Another attempt

[Joris Van Remoortere]

Hi Marco,

It seems Kapil took a look at your reviews, and likely has context to
understand whether this change makes sense. I would suggest that you clean
up the style errors in the reviews, as there are still quite a few.

Thanks!

—
*Joris Van Remoortere*
Mesosphere

On Fri, Feb 26, 2016 at 9:58 PM, Marco Massenzio 
wrote:

> Hey folks,
>
> I'm again pleading some (minimal) amount of feedback, before I give up on
> the following two reviews:
>
> https://reviews.apache.org/r/41760/
> https://reviews.apache.org/r/41854/
>
> Honestly, they are tiny, have been out exactly two months today, I've even
> addressed all comments, really, all it's missing is someone with committer
> power to give their blessing...
> (or, tell me what further needs fixing).
>
> Many thanks in advance.
>
> --
> *Marco Massenzio*
> http://codetrips.com
>

Re: [VOTE] Release Apache Mesos 0.27.2 (rc1)

[Joris Van Remoortere]

@Michael Browning:
>
> MasterTest.MaxCompletedTasksPerFrameworkFlag [flaky, tracked in
> MESOS-4518]

This is supposed to be fixed in this release. It is concerning that this
came up.
Can you verify this and provide logs to Kevin Klues?


—
*Joris Van Remoortere*
Mesosphere

On Tue, Mar 1, 2016 at 2:00 PM, Michael Browning 
wrote:

> +1 (non-binding)
>
> Fedora 23: `make check` non-root OK
> OS X: `make check` non-root OK
> Ubuntu 14.04: `make check` non-root, three failures:
> ContainerLoggerTest.DefaultToSandbox [flaky, tracked in MESOS-4615]
> MasterQuotaTest.AvailableResourcesAfterRescinding [flaky, tracked in
> MESOS-4542]
> MasterTest.MaxCompletedTasksPerFrameworkFlag [flaky, tracked in MESOS-4518]
>
> On Mon, Feb 29, 2016 at 10:40 PM, Greg Mann  wrote:
>
> > +1 (non-binding)
> >
> > `sudo make check` on Ubuntu 14.04 using gcc, with libevent and SSL
> enabled.
> >
> > All tests pass except MemoryPressureMesosTest.CGROUPS_ROOT_Statistics,
> > which seems to be due to the issue found here:
> > https://issues.apache.org/jira/browse/MESOS-4053
> >
> >
> > On Mon, Feb 29, 2016 at 2:17 PM, Michael Park  wrote:
> >
> > > Vinod, we've only committed the CHANGELOGs to the specific tags. I
> didn't
> > > realize that I should commit those to master as well, but it makes
> total
> > > sense to do so. I'll do that. Thanks.
> > >
> > > On 29 February 2016 at 13:50, Vinod Kone  wrote:
> > >
> > >> I don't see CHANGELOGs for these versions on the master branch?
> > >>
> > >> On Mon, Feb 29, 2016 at 1:39 PM, Neil Conway 
> > >> wrote:
> > >>
> > >> > As described (briefly) in the release emails, 0.27.2, 0.26.1,
> 0.25.1,
> > >> > and 0.24.2 contains a new feature: "reliable floating point for
> scalar
> > >> > resources" (MESOS-4687).
> > >> >
> > >> > To elaborate on that slightly, Mesos now only supports scalar
> resource
> > >> > values with three decimal digits of precision (e.g., reserving
> "5.001
> > >> > CPUs" for a task). As a result of this change, frameworks that do
> > >> > their own resource math may see slightly different results;
> > >> > furthermore, if any frameworks were trying to manage extremely
> > >> > fine-grained resource values (> 3 decimal digits of precision), that
> > >> > will no longer be supported.
> > >> >
> > >> > For more information, please see:
> > >> >
> > >> >
> > >> >
> > >>
> >
> https://mail-archives.apache.org/mod_mbox/mesos-user/201602.mbox/%3CCAOW5sYZJn5caBOwZyPV008JgL1F2FYFxL_bM5CtYA2PF2OG7Bw%40mail.gmail.com%3E
> > >> >
> > >> >
> > >>
> >
> https://docs.google.com/document/d/14qLxjZsfIpfynbx0USLJR0GELSq8hdZJUWw6kaY_DXc/edit?usp=sharing
> > >> > https://issues.apache.org/jira/browse/MESOS-4687
> > >> >
> > >> > Neil
> > >> >
> > >> >
> > >> > On Fri, Feb 26, 2016 at 8:54 PM, Michael Park 
> > >> wrote:
> > >> > > Hi all,
> > >> > >
> > >> > > Please vote on releasing the following candidate as Apache Mesos
> > >> 0.27.2.
> > >> > >
> > >> > >
> > >> > > 0.27.2 includes the following:
> > >> > >
> > >> >
> > >>
> >
> 
> > >> > >
> > >> > > MESOS-4693 - Variable shadowing in
> > >> HookManager::slavePreLaunchDockerHook.
> > >> > > MESOS-4711 - Race condition in libevent poll implementation causes
> > >> crash.
> > >> > > MESOS-4754 - The "executors" field is exposed under a backwards
> > >> > incompatible
> > >> > > schema.
> > >> > > MESOS-4687 - Implement reliable floating point for scalar
> resources.
> > >> > >
> > >> > >
> > >> > > The CHANGELOG for the release is available at:
> > >> > >
> > >> >
> > >>
> >
> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=0.27.2-rc1
> > >> > >
> > >> >
> > >>
> >
> 

Re: [VOTE] Release Apache Mesos 0.26.1 (rc1)

[Joris Van Remoortere]

+1 (binding)
Greg's upgrade scripts & CI results

On Fri, Mar 4, 2016 at 11:30 AM, Vinod Kone  wrote:

> +1 (binding)
>
> On Tue, Mar 1, 2016 at 5:03 PM, Kevin Klues  wrote:
>
> > I committed a fix for this in:
> >
> >
> https://github.com/apache/mesos/commit/42f746937233349660c687ea7a66cc0a78871663
> >
> > Looks like that's post 0.26 though, so maybe it should be included in the
> > .1 rc
> >
> > On Mon, Feb 29, 2016 at 2:27 PM, Vinod Kone 
> wrote:
> >
> >> Looks like the ASF CI builds for CentOS7 are failing because they are
> >> unable to find JAVA_HOME. Couldn't tell if it's an issue with the docker
> >> build script or something in the configure script.
> >>
> >>
> >> checking for svn_txdelta in -lsvn_delta-1... yes
> >> checking for sasl_done in -lsasl2... yes
> >> checking SASL CRAM-MD5 support... yes
> >> checking for javac... /usr/bin/javac
> >> checking for java... /usr/bin/java
> >> checking value of Java system property 'java.home'...
> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/jre
> >> configure: error: could not guess JAVA_HOME
> >>
> >>
> >>
> >> *Revision*: a05261dbed1c2577676b11235380de95d586aeeb
> >>
> >>- refs/tags/0.26.1-rc1
> >>
> >> Configuration Matrix gcc clang
> >> centos:7 --verbose --enable-libevent --enable-ssl
> >> [image: Failed]
> >> <
> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/8/COMPILER=gcc,CONFIGURATION=--verbose%20--enable-libevent%20--enable-ssl,OS=centos%3A7,label_exp=(docker%7C%7CHadoop)&&(!ubuntu-us1)/
> >
> >> [image: Not run]
> >> --verbose
> >> [image: Failed]
> >> <
> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/8/COMPILER=gcc,CONFIGURATION=--verbose,OS=centos%3A7,label_exp=(docker%7C%7CHadoop)&&(!ubuntu-us1)/
> >
> >> [image: Not run]
> >> ubuntu:14.04 --verbose --enable-libevent --enable-ssl
> >> [image: Success]
> >> <
> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/8/COMPILER=gcc,CONFIGURATION=--verbose%20--enable-libevent%20--enable-ssl,OS=ubuntu%3A14.04,label_exp=(docker%7C%7CHadoop)&&(!ubuntu-us1)/
> >
> >> [image: Success]
> >> <
> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/8/COMPILER=clang,CONFIGURATION=--verbose%20--enable-libevent%20--enable-ssl,OS=ubuntu%3A14.04,label_exp=(docker%7C%7CHadoop)&&(!ubuntu-us1)/
> >
> >> --verbose
> >> [image: Success]
> >> <
> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/8/COMPILER=gcc,CONFIGURATION=--verbose,OS=ubuntu%3A14.04,label_exp=(docker%7C%7CHadoop)&&(!ubuntu-us1)/
> >
> >> [image: Success]
> >> <
> https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/8/COMPILER=clang,CONFIGURATION=--verbose,OS=ubuntu%3A14.04,label_exp=(docker%7C%7CHadoop)&&(!ubuntu-us1)/
> >
> >>
> >> On Mon, Feb 29, 2016 at 11:21 AM, Kapil Arya 
> wrote:
> >>
> >>> +1 (binding)
> >>>
> >>> Successful CI builds for the following distros:
> >>>
> >>> amd64/centos/6
> >>> amd64/centos/7
> >>> amd64/debian/jessie
> >>> amd64/ubuntu/precise
> >>> amd64/ubuntu/trusty
> >>> amd64/ubuntu/vivid
> >>>
> >>> Kapil
> >>>
> >>> On Sat, Feb 27, 2016 at 12:26 AM, Michael Park 
> wrote:
> >>>
> >>> > Hi all,
> >>> >
> >>> > Please vote on releasing the following candidate as Apache Mesos
> >>> 0.26.1.
> >>> >
> >>> >
> >>> > 0.26.1 includes the following:
> >>> >
> >>> >
> >>>
> 
> >>> >
> >>> >- Improvements
> >>> >   - `/state` endpoint performance
> >>> >   - systemd integration
> >>> >   - GLOG performance
> >>> >   - Configurable task/framework history
> >>> >   - Offer filter timeout fix for backlogged allocator
> >>> >
> >>> >
> >>> >- Bugs
> >>> >- SSL
> >>> >   - Libevent
> >>> >   - Fixed point resources math
> >>> >- HDFS
> >>> >   - Agent upgrade compatibility
> >>> >
> >>> > The CHANGELOG for the release is available at:
> >>> >
> >>> >
> >>>
> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=0.26.1-rc1
> >>> >
> >>> >
> >>>
> 
> >>> >
> >>> > The candidate for Mesos 0.26.1 release is available at:
> >>> >
> >>>
> https://dist.apache.org/repos/dist/dev/mesos/0.26.1-rc1/mesos-0.26.1.tar.gz
> >>> >
> >>> > The tag to be voted on is 0.26.1-rc1:
> >>> >
> >>>
> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=0.26.1-rc1
> >>> >
> >>> > The MD5 checksum of the tarball can be found at:
> >>> >
> >>> >
> >>>
> https://dist.apache.org/repos/dist/dev/mesos/0.26.1-rc1/mesos-0.26.1.tar.gz.md5
> >>> >
> >>> > The signature of the tarball can be found at:
> >>> >
> >>> >
> >>>
> https://dist.apache.org/repos/dist/dev/mesos/0.26.1-rc1/mesos-0.26.1.tar.gz.asc
> >>> >
> >>> > The PGP key used to sign the release is here:
> >>> > https://dist.apache.org/repos/dist/release/mesos/KEYS
> >>> >
> >>> > The JAR is up in Maven in a staging repository here:
> >>> >
> https://repository.apache

Re: [VOTE] Release Apache Mesos 0.25.1 (rc1)

[Joris Van Remoortere]

+1 (binding)
Greg's upgrade scripts & CI results

On Tue, Mar 1, 2016 at 4:37 PM, Greg Mann  wrote:

> I was also able to successfully test a simple upgrade scenario between
> 0.24.2-rc1 and 0.25.1-rc1 using Niklas's upgrade testing script, which I've
> modified slightly and reposted here: https://reviews.apache.org/r/44229/
>
> On Tue, Mar 1, 2016 at 9:29 AM, Greg Mann  wrote:
>
> > +1 (non-binding)
> >
> > `sudo make check` on Ubuntu 14.04 using gcc, with libevent and SSL
> enabled.
> >
> > All tests pass except:
> >
> > PerfEventIsolatorTest.ROOT_CGROUPS_Sample, which is covered here:
> > https://issues.apache.org/jira/browse/MESOS-4655
> >
> > CgroupsAnyHierarchyWithPerfEventTest.ROOT_CGROUPS_Perf,
> > MemoryPressureMesosTest.CGROUPS_ROOT_Statistics, and
> > MemoryPressureMesosTest.CGROUPS_ROOT_SlaveRecovery, due to the issue
> here:
> > https://issues.apache.org/jira/browse/MESOS-3215
> >
> > Cheers,
> > Greg
> >
> >
> > On Mon, Feb 29, 2016 at 11:21 AM, Kapil Arya 
> wrote:
> >
> >> +1 (binding)
> >>
> >> Successful CI builds for the following distros:
> >>
> >> amd64/centos/6
> >> amd64/centos/7
> >> amd64/debian/jessie
> >> amd64/ubuntu/precise
> >> amd64/ubuntu/trusty
> >> amd64/ubuntu/vivid
> >>
> >> Kapil
> >>
> >> On Sat, Feb 27, 2016 at 12:53 AM, Michael Park 
> wrote:
> >>
> >> > Hi all,
> >> >
> >> > Please vote on releasing the following candidate as Apache Mesos
> 0.25.1.
> >> >
> >> >
> >> > 0.25.1 includes the following:
> >> >
> >> >
> >>
> 
> >> >
> >> >- Improvements
> >> >   - `/state` endpoint performance
> >> >   - systemd integration
> >> >   - GLOG performance
> >> >   - Configurable task/framework history
> >> >   - Offer filter timeout fix for backlogged allocator
> >> >
> >> >
> >> >- Bugs
> >> >- SSL
> >> >   - Libevent
> >> >   - Fixed point resources math
> >> >- HDFS
> >> >   - Agent upgrade compatibility
> >> >   - Health checks
> >> >
> >> > The CHANGELOG for the release is available at:
> >> >
> >> >
> >>
> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=0.25.1-rc1
> >> >
> >> >
> >>
> 
> >> >
> >> > The candidate for Mesos 0.25.1 release is available at:
> >> >
> >>
> https://dist.apache.org/repos/dist/dev/mesos/0.25.1-rc1/mesos-0.25.1.tar.gz
> >> >
> >> > The tag to be voted on is 0.25.1-rc1:
> >> >
> >>
> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=0.25.1-rc1
> >> >
> >> > The MD5 checksum of the tarball can be found at:
> >> >
> >> >
> >>
> https://dist.apache.org/repos/dist/dev/mesos/0.25.1-rc1/mesos-0.25.1.tar.gz.md5
> >> >
> >> > The signature of the tarball can be found at:
> >> >
> >> >
> >>
> https://dist.apache.org/repos/dist/dev/mesos/0.25.1-rc1/mesos-0.25.1.tar.gz.asc
> >> >
> >> > The PGP key used to sign the release is here:
> >> > https://dist.apache.org/repos/dist/release/mesos/KEYS
> >> >
> >> > The JAR is up in Maven in a staging repository here:
> >> >
> https://repository.apache.org/content/repositories/orgapachemesos-1108
> >> >
> >> > Please vote on releasing this package as Apache Mesos 0.25.1!
> >> >
> >> > The vote is open until Wed Mar 2 23:59:59 PST 2016 and passes if a
> >> majority
> >> > of at least 3 +1 PMC votes are cast.
> >> >
> >> > [ ] +1 Release this package as Apache Mesos 0.25.1
> >> > [ ] -1 Do not release this package because ...
> >> >
> >> > Thanks,
> >> >
> >> > Joris, Kapil, MPark
> >> >
> >>
> >
> >
>

Re: [VOTE] Release Apache Mesos 0.24.2 (rc1)

[Joris Van Remoortere]

+1 (binding)

On Mon, Feb 29, 2016 at 3:24 PM, Greg Mann  wrote:

> +1 (non-binding)
>
> `sudo make check` on Ubuntu 14.04, using gcc with libevent and SSL enabled.
> All tests pass except MemoryPressureMesosTest.CGROUPS_ROOT_Statistics,
> which is a known failure in 0.24.
>
> Cheers,
> Greg
>
>
> On Mon, Feb 29, 2016 at 11:20 AM, Kapil Arya  wrote:
>
> > +1 (binding)
> >
> > Successful CI builds for the following distros:
> >
> > amd64/centos/6
> > amd64/centos/7
> > amd64/debian/jessie
> > amd64/ubuntu/precise
> > amd64/ubuntu/trusty
> > amd64/ubuntu/vivid
> >
> > Kapil
> >
> > On Sat, Feb 27, 2016 at 1:12 AM, Michael Park  wrote:
> >
> >> Hi all,
> >>
> >> Please vote on releasing the following candidate as Apache Mesos 0.24.2.
> >>
> >>
> >> 0.24.2 includes the following:
> >>
> >>
> 
> >>
> >>- Improvements
> >>   - Allocator filter performance
> >>   - Port Ranges performance
> >>   - UUID performance
> >>   - `/state` endpoint performance
> >>   - GLOG performance
> >>   - Configurable task/framework history
> >>   - Offer filter timeout fix for backlogged allocator
> >>
> >>
> >>- Bugs
> >>- SSL
> >>   - Libevent
> >>   - Fixed point resources math
> >>- HDFS
> >>   - Agent upgrade compatibility
> >>   - Health checks
> >>
> >> The CHANGELOG for the release is available at:
> >>
> >>
> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=0.24.2-rc1
> >>
> >>
> 
> >>
> >> The candidate for Mesos 0.24.2 release is available at:
> >>
> >>
> https://dist.apache.org/repos/dist/dev/mesos/0.24.2-rc1/mesos-0.24.2.tar.gz
> >>
> >> The tag to be voted on is 0.24.2-rc1:
> >>
> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=0.24.2-rc1
> >>
> >> The MD5 checksum of the tarball can be found at:
> >>
> >>
> https://dist.apache.org/repos/dist/dev/mesos/0.24.2-rc1/mesos-0.24.2.tar.gz.md5
> >>
> >> The signature of the tarball can be found at:
> >>
> >>
> https://dist.apache.org/repos/dist/dev/mesos/0.24.2-rc1/mesos-0.24.2.tar.gz.asc
> >>
> >> The PGP key used to sign the release is here:
> >> https://dist.apache.org/repos/dist/release/mesos/KEYS
> >>
> >> The JAR is up in Maven in a staging repository here:
> >> https://repository.apache.org/content/repositories/orgapachemesos-1110
> >>
> >> Please vote on releasing this package as Apache Mesos 0.24.2!
> >>
> >> The vote is open until Wed Mar 2 23:59:59 PST 2016 and passes if a
> >> majority of at least 3 +1 PMC votes are cast.
> >>
> >> [ ] +1 Release this package as Apache Mesos 0.24.2
> >> [ ] -1 Do not release this package because ...
> >>
> >> Thanks,
> >>
> >> Joris, Kapil, MPark
> >>
> >
> >
>

Re: [VOTE] Release Apache Mesos 0.27.2 (rc1)

[Joris Van Remoortere]

+1 (binding)
Greg's upgrade scripts & CI results

The missing commit is for a flaky test which doesn't influence the
production binaries.
Unless we need to cut another RC for a bug, I suggest we move ahead.

On Wed, Mar 2, 2016 at 10:36 AM, Jörg Schad  wrote:

> Except the missing fix for Mesos-4518, if we consider cutting a rc2
> for that, maybe we could include the fix for MESOS-4677 as well (see
> failing ROOT_CGROUPS_Pids_and_Tids test below).
> +1 (non-binding)
>
> All the failing tests I encountered seem to be known.
>
> Centos 7
> * LimitedCpuIsolatorTest.ROOT_CGROUPS_Pids_and_Tids  (fixed with
> MESOS-4677 for 0.28 )
> * LinuxFilesystemIsolatorTest.ROOT_MultipleContainers (open ticket
> MESOS-4423)
>
> Centos 7 - SSL
> All green
>
> Centos 6 (+/- SSL)
> * MemoryPressureMesosTest.CGROUPS_ROOT_SlaveRecovery (reopened MESOS-4047)
>
> Debian 8 (+/- SSL)
> * DockerContainerizerTest.ROOT_DOCKER_Kill (seems the same issue as
> MESOS-3937)
>
> Ubuntu 15 (+/- SSL)
> Green
>
> Ubuntu 14 (+/- SSL)
> Green
>
> Ubuntu 12 (+/- SSL)
> Green
>
> On Tue, Mar 1, 2016 at 10:18 PM, Kevin Klues  wrote:
> > -1 (non-binding)
> >
> > This release
> > candidate
> > should have included the backport to re
> > s
> > olv
> > e
> > MESOS-4518 <https://issues.apache.org/jira/browse/MESOS-4518>.
> > All of the other release candidates that came out as backports recently
> > have included this, but somehow this one was overlooked.
> >
> >
> >
> >
> > On Tue, Mar 1, 2016 at 4:35 PM, Greg Mann  wrote:
> >
> >> I was able to successfully test a simple upgrade scenario between
> >> 0.26.1-rc1 and 0.27.2-rc1 using Niklas's upgrade testing script, which
> I've
> >> modified slightly and reposted here:
> https://reviews.apache.org/r/44229/
> >>
> >> On Tue, Mar 1, 2016 at 2:22 PM, Kevin Klues  wrote:
> >>
> >> > The others all seem to have them though:
> >> >
> >> >
> >> >
> >>
> https://github.com/apache/mesos/commits/0.26.1-rc1/src/tests/master_tests.cpp
> >> >
> >> >
> >>
> https://github.com/apache/mesos/commits/0.25.1-rc1/src/tests/master_tests.cpp
> >> >
> >> >
> >>
> https://github.com/apache/mesos/commits/0.24.2-rc1/src/tests/master_tests.cpp
> >> >
> >> > Just not:
> >> >
> >> >
> >>
> https://github.com/apache/mesos/commits/0.27.2-rc1/src/tests/master_tests.cpp
> >> >
> >> > On Tue, Mar 1, 2016 at 2:17 PM, Kevin Klues 
> wrote:
> >> > > Looks like this rc is missing this commit:
> >> > >
> >> > >
> >> >
> >>
> https://github.com/apache/mesos/commit/d3108d776b6f7121e37176eda686ecc7245be4cd
> >> > >
> >> > > On Tue, Mar 1, 2016 at 2:08 PM, Joris Van Remoortere
> >> > >  wrote:
> >> > >> @Michael Browning:
> >> > >>>
> >> > >>> MasterTest.MaxCompletedTasksPerFrameworkFlag [flaky, tracked in
> >> > >>> MESOS-4518]
> >> > >>
> >> > >> This is supposed to be fixed in this release. It is concerning that
> >> this
> >> > >> came up.
> >> > >> Can you verify this and provide logs to Kevin Klues?
> >> > >>
> >> > >>
> >> > >> —
> >> > >> Joris Van Remoortere
> >> > >> Mesosphere
> >> > >>
> >> > >> On Tue, Mar 1, 2016 at 2:00 PM, Michael Browning <
> >> > invitapri...@gmail.com>
> >> > >> wrote:
> >> > >>>
> >> > >>> +1 (non-binding)
> >> > >>>
> >> > >>> Fedora 23: `make check` non-root OK
> >> > >>> OS X: `make check` non-root OK
> >> > >>> Ubuntu 14.04: `make check` non-root, three failures:
> >> > >>> ContainerLoggerTest.DefaultToSandbox [flaky, tracked in
> MESOS-4615]
> >> > >>> MasterQuotaTest.AvailableResourcesAfterRescinding [flaky, tracked
> in
> >> > >>> MESOS-4542]
> >> > >>> MasterTest.MaxCompletedTasksPerFrameworkFlag [flaky, tracked in
> >> > >>> MESOS-4518]
> >> > >>>
> >> > >>> On Mon, Feb 29, 2016 at 10:40 PM, Greg Mann 
> >> > wrote:
> >> > >>>
> >> > >>> >

Re: Compile failure

[Joris Van Remoortere]

Hello Walter,

Did you see this section on the Getting Started
<http://mesos.apache.org/gettingstarted/> page?

# 'Mesos > 0.21.0' requires 'subversion > 1.8' devel package,
# which is not available in the default repositories.
# Create a WANdisco SVN repo file to install the correct version:
$ sudo cat > /etc/yum.repos.d/wandisco-svn.repo <http://opensource.wandisco.com/centos/7/svn-1.9/RPMS/$basearch/
gpgcheck=1
gpgkey=http://opensource.wandisco.com/RPM-GPG-KEY-WANdisco
EOF


—
*Joris Van Remoortere*
Mesosphere

On Mon, Mar 7, 2016 at 7:01 AM, Walter Heestermans (TME) <
walter.heesterm...@external.toyota-europe.com> wrote:

> Hi,
>
> Trying to build the mesos apache-mesos-0.27.1 on Oracle Linux 7
>
> configure: error: cannot find libsvn_subr-1 headers
> ---
> libsubversion-1 is required for mesos to build.
>
> I search for packages to install , and found the subversion-devel one, but
> doesn't seems to exist when I yum
>
> sh-4.2# yum install subversion-devel
> Loaded plugins: ulninfo
> No package subversion-devel available.
> Error: Nothing to do
>
> Can somebody advise what to install?
>
> Walter
>
>
>
> This e-mail may contain confidential information. If you are not an
> addressee or otherwise authorised to receive this message, you should not
> use, copy, disclose or take any action based on this e-mail. If you have
> received this e-mail in error, please inform the sender promptly and delete
> this message and any attachments immediately.
>

Re: Negative durations

[Joris Van Remoortere]

Duration and TimePoint were introduced to map to the types in C++ (
http://en.cppreference.com/w/cpp/chrono).

Duration is negative so as not to surprise users when they do TimePoints A
- B and end up with a positive duration when B > A. You could imagine a
user of these writing code to check if their maintenance window is in the
past:

Duration remaining = Now - Scheduled;
if (remaining < 0) {
  // Don't use offer.
} else {
  // Still `remaining` time to use offer.
}

This logic would compile correctly and likely pass review because it is
intuitive; however, if duration was an absolute value, then it would be a
bug.

To represent infinity we currently wrap the Duration with an Option. The
None state represents infinity. You can find this in the protobufs.

—
*Joris Van Remoortere*
Mesosphere

On Wed, Mar 9, 2016 at 11:29 PM, Alexander Rojas 
wrote:

> IMO durations should not be negative. The problem with having them define
> an infinite period is the following:
>
> Lets say you want to compute the duration between two durations A and B:
>   C = A-B;
> if A < B, C is negative and it will be infinite.
>
> I one want’s to check for the case mentioned in [1] about sleeping 0
> marked with a negative duration, I think adding an explicit `if` is better
> since it reflects the intention.
>
> So in to conclude:
>
> 1. Duration operations should always be positive so A-B is effectively
> |A.underlying_type - B.underlying_type|.
> 2. We do need a constant or an specific trait to represent an infinite
> duration.
>
> > On 09 Mar 2016, at 17:20, Klaus Ma  wrote:
> >
> > One case I can image is to use negative value for forever duration?
> >
> > 
> > Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
> > Platform OpenSource Technology, STG, IBM GCG
> > +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
> >
> > On Wed, Mar 9, 2016 at 8:21 PM, Alex Rukletsov 
> wrote:
> >
> >> Folks,
> >>
> >> I've recently realized that durations we use in mesos (stout's
> `Duration`
> >> and `DurationInfo` protobuf) are based on signed integers. Negative
> >> duration concept is a bit strange to me, so I googled around a bit and
> >> found an interesting thread [1].
> >>
> >> Was it an explicit intention to allow `Duration`s to be negative? Do we
> use
> >> this feature? If yes, maybe we can introduce a class representing time
> >> delta (can be negative) and base `Duration` on top of it guaranteeing
> it is
> >> always non-negative?
> >>
> >> My ultimate intention is to avoid boilerplate code that validates every
> >> single instance of `Duration` in the codebase. I'd rather have a class
> with
> >> guarantees certain invariants.
> >>
> >> [1]
> https://internals.rust-lang.org/t/unsigned-version-of-duration/893/2
> >>
>
>

Re: [RESULT][VOTE] Release Apache Mesos 0.27.2 (rc1)

[Joris Van Remoortere]

The SHAs rarely exist on HEAD because we tend to cherry-pick bug fixes into
release candidates.

I'm all for using branches in the repository to put together the release
candidates, as well as have committers consider their bug fixes for some N
number of backport (future LTS) releases.

As Kevin and Michael mentioned, tags and branches are different concepts.
We can use them together. We still want to create immutable tags to point
at releases so that we don't accidentally add new patches to releases by
updating a branch.

I think building up the releases in public branches is a good step towards
improved visibility. I hope this will also increase the accountability of
the community to ensure the patches they contribute are committed to the
right branches.

On Fri, Mar 18, 2016 at 8:10 PM, Erik Weathers <
eweath...@groupon.com.invalid> wrote:

> BTW, if the tag is created against a commit that *doesn't* become
> "unreachable" from HEAD [1], then `git pull` is sufficient to also pull
> down the tags.
>
> The only time I've needed to do `git fetch --tags` is when the tagged
> commit SHA gets merged away.  So presumably the process being followed by
> the core committers / releasers is resulting in these "unreachable" tags.
> Not sure if that is preventable though.
>
> - Erik
>
> [1] http://eddiemoya.com/2013/02/21/better-git-git-fetch-not-getting-tags/
>
> From the git manual (“git help fetch”): [1]
>
> -t, –tags Most of the tags are fetched automatically as branch heads are
> downloaded, but tags that do not point at objects reachable from the branch
> heads that are being tracked will not be fetched by this mechanism. This
> flag lets all tags and their associated objects be downloaded. The default
> behavior for a remote may be specified with the remote..tagopt
> setting. See git-config(1).
>
>
>
> On Fri, Mar 18, 2016 at 6:22 PM, Michael Browning 
> wrote:
>
> > I agree with Kevin -- tags are immutable, so they're naturally suited
> > for labeling releases, which ought to be immutable too.
> >
> > On Fri, Mar 18, 2016 at 4:59 PM, Kevin Klues  wrote:
> > > I respectfully disagree.
> > >
> > > The whole purpose of tags is to mark permanent things like releases,
> > > whereas branches are designed as temporary lines of development that
> > > come and go (and grow and shrink) dynamically all the time.
> > >
> > > On Fri, Mar 18, 2016 at 4:04 PM, Jie Yu  wrote:
> > >> I like the idea of using branches to manage releases.
> > >>
> > >> We can use that to manage point releases and backports as well.
> > >>
> > >> Say we want to cut 0.29.0 now, we fork a branch 0.29.0 and tag RCs in
> > that
> > >> branch. Once the RC is accepted, the head of that branch will become
> the
> > >> release.
> > >>
> > >> Then, we immediate fork that branch and create 0.29.1 branch.
> > >>
> > >> When a new bug fix is committed on the trunk, the committer will
> decide
> > >> whether it'll affect the old releases (a bounded number, we can decide
> > that
> > >> later). If it does, the committer of that patch should also
> cherry-pick
> > >> that patch to the point releases (e.g., 0.29.1 in this case). We can
> do
> > a
> > >> timely based point releases.
> > >>
> > >> - Jie
> > >>
> > >> On Fri, Mar 18, 2016 at 1:35 PM, Cong Wang 
> > wrote:
> > >>
> > >>> On Wed, Mar 16, 2016 at 11:56 AM, Joseph Wu 
> > wrote:
> > >>> > Cong Wang,
> > >>> >
> > >>> > The tags are sync'd.  See:
> https://github.com/apache/mesos/releases
> > >>> >
> > >>> > You might not have done: git pull --tags
> > >>>
> > >>>
> > >>> Yeah, I figured it out by myself too. This is why I hate tags
> > personally,
> > >>> branches are better since they are fetched without additional
> > parameters.
> > >>>
> > >>> Any reason why Mesos maintainers picked tags over branches to manage
> > >>> releases? Just curious...
> > >>>
> > >
> > >
> > >
> > > --
> > > ~Kevin
> >
>

Re: RFC: RevocableInfo Changes

[Joris Van Remoortere]

@klaus:
I think @connor's question is whether we are absolutely sure we never want
to support throttle-able but non-revocable resources.
It's clear from the protos that this is not supported, the question is
whether we are sure that is what we want. If so, can you elaborate as to
*why* we would never want that concept in Mesos.

—
*Joris Van Remoortere*
Mesosphere

On Sun, Mar 20, 2016 at 8:33 PM, Klaus Ma  wrote:

> Here's some input :).
>
> If throttling is tolerable but preemption is not, how would that be
> expressed? (Is that supported?)
> [Klaus]: It's not supported; only revocable resources has this attribute:
> non-throttleable or throttleable. The throttleable revocable resources is
> reported by ResourceEstimator which means the resources maybe throttled by
> its original owner.
>
> How does this work with the QoS controller? Will there be a new correction
> type to indicate throttling, or does throttling happen "behind the agent's
> back"?
> [Klaus]: The QoSController/ResourceEstimator only manages throttleable
> revocable resources; the others resources (regular resources and
> non-throttleable revocable resources) are managed by allocator. The
> "manage" means generation and destroy/eviction. Regarding "throttling
> happen", good question. I think the throttling will dependent on
> containers, let me double check it :).
>
> If any comments, please let me know.
>
> 
> Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
> Platform OpenSource Technology, STG, IBM GCG
> +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
>
> On Sat, Mar 19, 2016 at 11:15 PM,  wrote:
>
>> Thanks for the good explanations so far Ben and Klaus.  Apologies if you
>> guys already covered these questions in the meeting:
>>
>> If throttling is tolerable but preemption is not, how would that be
>> expressed? (Is that supported?)
>>
>> How does this work with the QoS controller? Will there be a new
>> correction type to indicate throttling, or does throttling happen "behind
>> the agent's back"?
>>
>> Thanks,
>> --
>> Connor
>>
>> > On Mar 19, 2016, at 04:01, Klaus Ma  wrote:
>> >
>> > @team, in the latest meeting, we agree to keep current name
>> ThrottleInfo.
>> >
>> > If any more comments, please let me know.
>> >
>> >> On Wednesday, March 16, 2016 at 9:32:37 PM UTC+8, Guangya Liu wrote:
>> >> Also please show your comments if any for the name here, the current
>> name is ThrottleInfo, in Kubernetes resources qos design document, they are
>> using scavenging as the key work for such behaviour, so a possible name
>> here could be ScavengeInfo , please show your comments if any for those two
>> names or even if you want to propose a new name here.
>> >>
>> >> message RevocableInfo {
>> >> message ThrottleInfo {}
>> >>
>> >> // If set, indicates that the resources may be throttled at
>> >> // any time. Throttle-able resoruces can be used for tasks
>> >> // that do not have strict performance requirements and are
>> >> // capable of handling being throttled.
>> >> optional ThrottleInfo throttle_info = 1;
>> >>   }
>> >>
>> >> 在 2016年3月16日星期三 UTC+8上午10:24:14，Klaus Ma写道：
>> >>>
>> >>> The patches are updated accordingly; JIRA: MESOS-3888 , RR:
>> https://reviews.apache.org/r/40375/ .
>> >>>
>> >>> Thanks
>> >>> klaus
>> >>>
>> >>>> On Saturday, March 12, 2016 at 11:09:46 AM UTC+8, Benjamin Mahler
>> wrote:
>> >>>> Hey folks,
>> >>>>
>> >>>> In the resource allocation working group we've been looking into a
>> few projects that will make the allocator able to offer out resources as
>> revocable. For example:
>> >>>>
>> >>>> -We'll want to eventually allocate resources as revocable _by
>> default_, only allowing non-revocable when there are guarantees put in
>> place (static reservations or quota).
>> >>>>
>> >>>> -On the path to revocable by default, we can incrementally start to
>> offer certain resources as revocable. Consider when quota is set but the
>> role isn't using all of the quota. The unallocated quota can be offered to
>> other roles, but it should be revocable because we may revoke them should
>> the quota'ed role want to use the resources. Unused reservations fall into
>> a

Re: Filing bugs and flaky test tickets

[Joris Van Remoortere]

If you are filing a bug, please do set the `affected version` for all the
versions that you are sure it affects.

This will assist us in identifying potential backport targets, as well as
identify which things are critical to fix before the next release cycle.

Starting with 0.28 we added a new gadget to the JIRA release board that
lists all issues that are unresolved and affecting the previous release, in
order of severity.

—
*Joris Van Remoortere*
Mesosphere

On Tue, Mar 22, 2016 at 7:15 AM, Benjamin Mahler  wrote:

> Folks,
>
> Our JIRA volume is very high and I'm finding that important bugs and flaky
> tests are getting lost in the noise. When filing a bug or a flaky test
> ticket as a dev in the project, it's critical to 'git blame' the code to
> determine who is responsible for it, in order to notify them of the issue.
>
> If you're finding that folks are not following up on fixing test flakiness
> and keeping their components bug-free then please surface that to the list,
> sometimes things slip through the cracks and we definitely want to follow
> up and keep the quality high.
>
> I excluded the user list here because this is likely too much to ask of
> users. We've been thinking about how to ensure that issues coming in from
> users are correctly triaged and dispatched to the folks that should be made
> aware of them.
>
> If anyone knows of ways we could potentially use JIRA better here (e.g.
> fine-grained email notifications (i.e per component, per issue type, etc),
> periodic reports, etc) that would be great.
>
> Ben
>

Re: [RESULT][VOTE] Release Apache Mesos 0.27.2 (rc1)

[Joris Van Remoortere]

+1 for branch per RC (if we go with branches). I like your argument against
re-writing history if we make a mistake.

I think the 2 issues that have come up are:
1) visibility into the release process
2) pain / lack of context for the release manager of backports to resolve
merge conflicts for bugs

For 2), Jie's suggestion was that if we had branches set up for future
backports (even if it were something like 0.24.2-WIP) then committers would
both:
a) help amortize the cost of cherry-picking / merge conflicts
b) have current context about the patch when considering whether it is a
critical bug fix, as well as backwards compatible

These are likely issues that Vinod will also address in his 1.0 and LTS
proposals.

On Mon, Mar 21, 2016 at 1:52 PM, Benjamin Mahler  wrote:

> A single branch for a release seems brittle because it assumes that RC tag
> N shares the same lineage as RC tag N-1. This is mostly true, but not
> always. The branching approach that would mirror how I've put together
> releases would be to have a branch for each RC. The RC N branch is usually
> on top of the RC N-1 branch, but not always.
>
> This allows both of the following scenarios:
>
> (1)
> master > RC1 -> RC2
>
> (2)
> master > RC1
>  \> RC2
>
> Scenario (2) is not possible with a single release branch. The other nice
> thing about 1 branch per RC is that if we make mistakes, we just create a
> new branch, rather than re-write history or push out branch removals.
> Also, we should avoid naming the branch exactly the same as the tag, as
> that will lead to confusion.
>
> But my thought here is that we should take a step back and look at what
> problems we want to solve with the current tag-only approach. Joris and Jie
> seem to be hinting at there being a lack of visibility into the progress of
> RCs. Is that the issue we're trying to solve? Collaboration on RC's? I
> haven't done a release in quite some time so it would be great to
> understand the current pain points.
>
> On Mon, Mar 21, 2016 at 1:20 AM, Joris Van Remoortere 
> wrote:
>
> > The SHAs rarely exist on HEAD because we tend to cherry-pick bug fixes
> into
> > release candidates.
> >
> > I'm all for using branches in the repository to put together the release
> > candidates, as well as have committers consider their bug fixes for some
> N
> > number of backport (future LTS) releases.
> >
> > As Kevin and Michael mentioned, tags and branches are different concepts.
> > We can use them together. We still want to create immutable tags to point
> > at releases so that we don't accidentally add new patches to releases by
> > updating a branch.
> >
> > I think building up the releases in public branches is a good step
> towards
> > improved visibility. I hope this will also increase the accountability of
> > the community to ensure the patches they contribute are committed to the
> > right branches.
> >
> > On Fri, Mar 18, 2016 at 8:10 PM, Erik Weathers <
> > eweath...@groupon.com.invalid> wrote:
> >
> > > BTW, if the tag is created against a commit that *doesn't* become
> > > "unreachable" from HEAD [1], then `git pull` is sufficient to also pull
> > > down the tags.
> > >
> > > The only time I've needed to do `git fetch --tags` is when the tagged
> > > commit SHA gets merged away.  So presumably the process being followed
> by
> > > the core committers / releasers is resulting in these "unreachable"
> tags.
> > > Not sure if that is preventable though.
> > >
> > > - Erik
> > >
> > > [1]
> > http://eddiemoya.com/2013/02/21/better-git-git-fetch-not-getting-tags/
> > >
> > > From the git manual (“git help fetch”): [1]
> > >
> > > -t, –tags Most of the tags are fetched automatically as branch heads
> are
> > > downloaded, but tags that do not point at objects reachable from the
> > branch
> > > heads that are being tracked will not be fetched by this mechanism.
> This
> > > flag lets all tags and their associated objects be downloaded. The
> > default
> > > behavior for a remote may be specified with the remote..tagopt
> > > setting. See git-config(1).
> > >
> > >
> > >
> > > On Fri, Mar 18, 2016 at 6:22 PM, Michael Browning <
> > invitapri...@gmail.com>
> > > wrote:
> > >
> > > > I agree with Kevin -- tags are immutable, so they're naturally suited
> > > > for labeling releases, which ought to be immutable too.
> > >

Re: Ordering guarantee of future.onAny callbacks

[Joris Van Remoortere]

Rather than saying we shouldn't make this assumption (which I assume has
already been made in places, and will likely be made again), can we fix the
problem?
There's a `TODO` already suggesting running the callback in a separate
execution environment.
We could also synchronize with thread setting the future to true.

I'd rather discuss and fix the problem, than try to enforce avoiding this
case.

Thoughts?

—
*Joris Van Remoortere*
Mesosphere

On Tue, Mar 29, 2016 at 2:50 AM, Jie Yu  wrote:

> Hi,
>
> While digging a bug reported in, I realized an assumption we shouldn't make
> in our code.
> https://issues.apache.org/jira/browse/MESOS-5023
>
> Say you have the following code:
>
> void some_func()
> {
>   future
> .onAny(callback_A)
> .onAny(callback_B);
> }
>
> Question: will callback_A already be executed before callback_B?
>
> The answer is NO. We should never assume that. Under the following
> interleaving, callback_B can be invoked first:
>
> Thread-1   Thread-2
>
> onAny(callback_A) {
>   onAnyCallbacks.push_back(callback_A);
> }
>   set() {
> lock()
> if (state ==
> PENDING) {
>   state = READY;
>   result = true;
> }
> unlock();
>
> onAny(callback_B) {
>   lock()
>   if (state != PENDING) {
> run = true
>   }
>   unlock()
>
>   if (run) {
> callback_B()
>   }
>
>  if (result) {
>
> internal::run(data->onAnyCallbacks,
> *this);
>  }
>
> - Jie
>

Re: Ordering guarantee of future.onAny callbacks

[Joris Van Remoortere]

I think the API of future intuitively suggests parts of the chain will
execute in a specific order. The type system even enforces some of these
IIUC.

For example:
func().then([](T) -> X {...}).repair([](Future) -> X {...}).then([](X)
-> Y {...}).onAny([](Future){...})

As you can see, the .onAny(Future) can't just start executing on
the T returned
by func().

I think it would be confusing and error prone to remember that sibling `.
onAny` pairs could execute in arbitrary order, but members along the chain
such as `.then` and `.repair` could not.

I'm not sure the benefits of executing multiple `.onAny` calls in parallel
in the future (if we ever need to) outweigh the inconsistency in the API
regarding ordered execution of certain chains (At least as the API is
currently designed).

—
*Joris Van Remoortere*
Mesosphere

On Wed, Mar 30, 2016 at 12:25 AM, Benjamin Mahler 
wrote:

> I don't believe we have made this assumption often, and I'm not convinced
> it's a problem in Future.
>
> Why would one make such an assumption in the first place? It seems brittle,
> especially when callbacks are executed in deferred contexts that Future is
> unaware of. It also limits our flexibility within Future if we add these
> ordering semantics.
>
> On Tue, Mar 29, 2016 at 2:41 AM, Joris Van Remoortere  >
> wrote:
>
> > Rather than saying we shouldn't make this assumption (which I assume has
> > already been made in places, and will likely be made again), can we fix
> the
> > problem?
> > There's a `TODO` already suggesting running the callback in a separate
> > execution environment.
> > We could also synchronize with thread setting the future to true.
> >
> > I'd rather discuss and fix the problem, than try to enforce avoiding this
> > case.
> >
> > Thoughts?
> >
> > —
> > *Joris Van Remoortere*
> > Mesosphere
> >
> > On Tue, Mar 29, 2016 at 2:50 AM, Jie Yu  wrote:
> >
> > > Hi,
> > >
> > > While digging a bug reported in, I realized an assumption we shouldn't
> > make
> > > in our code.
> > > https://issues.apache.org/jira/browse/MESOS-5023
> > >
> > > Say you have the following code:
> > >
> > > void some_func()
> > > {
> > >   future
> > > .onAny(callback_A)
> > > .onAny(callback_B);
> > > }
> > >
> > > Question: will callback_A already be executed before callback_B?
> > >
> > > The answer is NO. We should never assume that. Under the following
> > > interleaving, callback_B can be invoked first:
> > >
> > > Thread-1   Thread-2
> > >
> > > onAny(callback_A) {
> > >   onAnyCallbacks.push_back(callback_A);
> > > }
> > >   set() {
> > > lock()
> > > if (state ==
> > > PENDING) {
> > >   state =
> READY;
> > >   result =
> true;
> > > }
> > > unlock();
> > >
> > > onAny(callback_B) {
> > >   lock()
> > >   if (state != PENDING) {
> > > run = true
> > >   }
> > >   unlock()
> > >
> > >   if (run) {
> > > callback_B()
> > >   }
> > >
> > >  if (result) {
> > >
> > > internal::run(data->onAnyCallbacks,
> > > *this);
> > >  }
> > >
> > > - Jie
> > >
> >
>

Re: docker, systemd, and cfs_quota

[Joris Van Remoortere]

Hey Michael,

There are a bunch of issues related to running with older versions of
systemd. See MESOS-3425.

Are you running with systemd disabled? If not, you likely have a category
of other problems mentioned in the ticket to worry about.

What is preventing you from upgrading the version of systemd?

Joris

—
*Joris Van Remoortere*
Mesosphere

On Fri, Apr 15, 2016 at 6:00 PM, Michael Browning 
wrote:

> Hi all,
>
> We've run into an interesting issue with Docker and systemd that's
> hampering our attempts to use cfs_quota for CPU isolation with Docker
> containers. The issue is that Docker doesn't tell systemd when it
> changes cpu.cfs_quota_us, so when our Puppet run triggers a `systemctl
> daemon-reload` invocation, the quota is reset to -1.
>
> I assume this could be solved with the `Delegate=true` in the dockerd unit
> file, but unfortunately we're not running a recent enough version of
> systemd to take advantage. Has anyone else ran into this when using
> DockerContainerizer with cfs_quota, or found a workaround?
>
> Regards,
> Michael
>

Re: Mesos Community Sync Notes

[Joris Van Remoortere]

That is the plan, I think it just hasn't been done yet :-)

—
*Joris Van Remoortere*
Mesosphere

On Thu, Apr 21, 2016 at 9:20 PM, Qian Zhang  wrote:

> In that wiki, can we also put the corresponding JIRA ticket link there so
> that folks can take a close look at the projects they are interested in?
>
>
> Thanks,
> Qian Zhang
>
> On Fri, Apr 22, 2016 at 8:20 AM, haosdent  wrote:
>
> > I would like to know more details about
> >
> > >Deprecate Docker containerizer (in favor of Unified containerizer w/
> > Docker support)
> >
> > Is there any jira issue about this?
> >
> > On Fri, Apr 22, 2016 at 7:48 AM, Greg Mann  wrote:
> >
> > > Hello all!
> > > Today we had another Mesos community sync meeting; find the notes
> pasted
> > > below. Of particular interest is our new roadmap document
> > > <https://cwiki.apache.org/confluence/display/MESOS/%5BDRAFT%5D+Roadmap
> >,
> > > which outlines items planned for development within the next six
> months.
> > >
> > > For future sync meeting times, see the community calendar
> > > <http://mesos.apache.org/community/>on the Mesos website.
> > >
> > > Cheers,
> > > Greg
> > >
> > >
> > >
> > > April 21, 2016
> > >
> > > Time: 3pm PST
> > >
> > > Location: Mesosphere HQ
> > >
> > > Attendees:
> > >
> > >-
> > >
> > >Mesosphere: Greg, Vinod, Joris, Joseph, Jie, Gilbert, Avinash,
> Kevin,
> > >MPark, Kapil, Jojy, Artem
> > >-
> > >
> > >Apple: Yan Xu
> > >
> > >
> > > Agenda/Notes:
> > >
> > >-
> > >
> > >Review of action items from last meeting
> > >-
> > >
> > >   MPark set up a system to automatically notify him before
> community
> > >   syncs, and he can forward this message to the mailing lists.
> Don’t
> > yet have
> > >   a fully automatic method for sending these notifications.
> > >   -
> > >
> > >   Jie: Wiki page of release features has been updated
> > >   -
> > >
> > >   Vinod: draft roadmap has been created, see link below
> > >
> > >
> > >-
> > >
> > >3rdparty flattening and module dependency installation
> > >-
> > >
> > >   Review chain starting at https://reviews.apache.org/r/46514/
> > >   -
> > >
> > >   Key points for 3rdparty flattening:
> > >   -
> > >
> > >  Added --with-stout to libprocess configure
> > >  -
> > >
> > >  Added --with-libprocess/stout to top-level configure
> > >  -
> > >
> > >  Moved 3rdparty/libprocess/3rdparty/* to 3rdparty/
> > >  -
> > >
> > >  Stop using libprocess/stout configure scripts; use top-level
> > >  configure instead
> > >  -
> > >
> > >   Module dependency installation:
> > >   -
> > >
> > >  Added --enable-install-module-dependencies
> > >  -
> > >
> > >  Boost, glog, protobuf, and picojson are installed in
> > >  ${libdir}/mesos/3rdparty
> > >  -
> > >
> > >  Appropriate flags added mesos.pc
> > >  -
> > >
> > >   No direct effect on distro packaging
> > >   -
> > >
> > >Roadmap!!
> > >-
> > >
> > >   https://cwiki.apache.org/confluence/display/MESOS/Roadmap
> > >   -
> > >
> > >Docker image issues: using Alpine images in the tests causes
> > >difficulties for some users on the Power platform.
> > >-
> > >
> > >   Additionally, it would be great to eliminate the dependency on
> > >   internet access for running the tests
> > >   -
> > >
> > >   One solution: use `docker load` to load a small image into the
> > >   local cache, from which it can be used by the Docker runtime
> > >   -
> > >
> > >   Another solution: establish a Mesos account on the Docker Hub and
> > >   place images for testing there. This would require internet
> access
> > for
> > >   tests.
> > >
> > >
> > > Action Item
> > >
> > >-
> > >
> > >Joris will review Kapil’s 3rdparty patches
> > >-
> > >
> > >MPark, Joris and AlexC will help with the CMake path
> > >
> > >
> >
> >
> > --
> > Best Regards,
> > Haosdent Huang
> >
>

Re: How to record time elapsed of a set of async functions operation?

[Joris Van Remoortere]

To be clear, BenM's example will result in the total wall-clock time for
the chain to complete, rather than the sum of time actually spent in those
functions.


—
*Joris Van Remoortere*
Mesosphere

On Fri, Apr 22, 2016 at 4:26 PM, Benjamin Mahler  wrote:

> Here's an example of using Stopwatch for this:
>
> Future f = Nothing();
> Stopwatch s;
>
> s.start();
> f.then(a)
>  .then(b)
>  .then(c)
>  .onAny([s](){ cout << "a->b->c took: " << s.elapsed() << endl; });
>
> On Fri, Apr 22, 2016 at 3:16 AM, Jian Qiu  wrote:
>
> > Hi folks,
> >
> > We are trying to do some performance tests for Mesos, and intend to add
> log
> > to output the time duration for different kinds of operation. There is a
> > Stopwatch class that can calculate the time elapsed for a blocking call.
> > However, I am wondering if there is some utility to record the time
> elapsed
> > for a async call chain. For instance, Mesos always have a chain like
> >  future.then(xxx).onAny.  Is there anyway to calculate the whole time
> > elapsed of this kind of chain instead of printing out the time stamp?
> >
> > Thanks.
> >
> > Regards
> > Qiu Jian
> >
>

Re: Default value of Filter.refuse_seconds

[Joris Van Remoortere]

After
https://github.com/apache/mesos/commit/447d814ac80e67f30a0ffe2ee6047d85dc8fc383
the refuse_seconds should not be as dangerous as it used to be.
This was backported to 0.24 and above recently. Without the backported
patch the filter may never take effect when the allocation takes longer
than the actual filter.

On Tue, Apr 26, 2016 at 3:34 PM, Erb, Stephan 
wrote:

> Hi everyone,
>
> within this Aurora review request https://reviews.apache.org/r/46603/ we
> are wondering about the current Filter.refuse_seconds default value of 5
> seconds [1].
>
> Is there any reason why this particular value was chosen? Would we have to
> expect increased load for the leading Mesos master of extremely large
> clusters if we set it to 0?
>
> Thanks for your feedback!,
> Stephan
>
>
> [1]
> https://github.com/apache/mesos/blob/3d2d3edb96c9c428f988653cdd4428a05690e747/include/mesos/mesos.proto#L1423

Re: Rack awareness support for Mesos

[Joris Van Remoortere]

+dev.

@Fan, I responded on the JIRA with some next steps.
Thanks for bringing this up!

—
*Joris Van Remoortere*
Mesosphere

On Tue, Jun 7, 2016 at 12:58 PM, james  wrote:

> On 06/07/2016 09:57 AM, Du, Fan wrote:
>
>>
>>
>> On 2016/6/6 21:27, james wrote:
>>
>>> Hello,
>>>
>>>
>>> @Stephen::I guess Stephen is bringing up the 'security' aspect of who
>>> get's access to the information, particularly cluster/cloud devops,
>>> customers or interlopers?
>>>
>>
>> ACLs should play in this part to address security concern.
>>
>
> YES, and so much more! I know folks that their primary (in house cluster)
> usage is deep packet inspection on  the cluster
> With a cluster (inside) there is no limit to new tools that can be
> judiciously altered to benefit from cluster codes
>
>
>>
>>> @Fan:: As a consultant, most of my customers either have  or are
>>> planning hybrid installations, where some codes run on a local cluster
>>> or using 'the cloud' for dynamic load requirements. I would think your
>>> proposed scheme needs to be very flexible, both in application to a
>>> campus or Metropolitan Area Network, if not massively distributed around
>>> the globe. What about different resouce types (racks of arm64, gpu
>>> centric hardware, DSPs, FPGA etc etc. Hardware diversity bring many
>>> benefits to the cluster/cloud capabilities.
>>>
>>>
>>> This also begs the quesion of hardware management (boot/config/online)
>>> of the various hardware, such as is built into coreOS. Are several
>>> applications going to be supported? Standards track? Just Mesos DC/OS
>>> centric?
>>>
>>
>> It depends whether this proposal is accepted by Mesos, if you think
>> this feature is useful, let's discuss detailed requirement under
>> MESOS-5545.
>>
>
> OK. Take a look at 'Rackview' on sourceforge::
> 'http://rackview.sourceforge.net/'
>
>
> Do I have access to the jira system by default joining this list,
> or do I have to request permission somewhere? (sorry jira is new to me
> so recommendations on jira, per mesos, in a document, would be keen.)
>
>
>> btw, I have limited knowledge of CoreOS, will look into it.
>>
>
> CoreOS has some great ideas. But many of their codes are not current
> (when compared to the gentoo portage tree) and thus many are suspect
> for security/function.
>
> I thought the purpose was to get more folks involved here in discussions
> and then better formulated ideas  can migrate to the ticket (5545)  and
> repos.
>
>
>>
>>> TIMING DATA:: This is the main issue I see. Once you start 'vectoring
>>> in resources' you need to add timing (latency) data to encourage robust
>>> and diversified use of of this data. For HPC, this could be very
>>> valuable for rDMA abusive algorithms where memory constrained workloads
>>> not only need the knowledge of additional nearby memory resources, but
>>> the approximated (based on previous data collected) latency and
>>> bandwidth constraints to use those additional resources.
>>>
>>
>> Out of curiosity, which open sourced Mesos framework do you/your
>> customer run MPI?
>>
>
> Easy dude.Most of this work in tightly help and nothing to publish
> or open up yet. It's a mess (my professional opinion) right now and
> I'm testing a variety of tools just be able to have better instrumentation
> on these codes. Still rDMA is very attractive so it does warrant much
> attention and extreme, internal, excitement.
>
>
>
>
> Mesos can support MPI framework, but AFIK, it's immature [1][2].
>>
>
> YEP.
>
> I think this part of work should be investigated in future.
>>
>> [1]: https://github.com/apache/mesos/tree/master/mpi   <- mpd ring
>> version
>> [2]：https://github.com/mesosphere/mesos-hydra <- hydra version
>>
>
> Many codes floating around. Much excitement on new compiler features. Lots
> of hard work and testing going on. That said, the point I was try to make
> is "Vectoring in" resources, with a variety of parameters as a companion to
> your idea, is warranted for these aforementioned use cases
> and other opportunities.
>
>
>>
>>> Great idea. I do like it very much.
>>>
>>> hth,
>>> James
>>>
>>>
>>> On 06/06/2016 05:06 AM, Stephen Gran wrote:
>>>
>>>> Hi,
>>>>
>>>> This looks poten

Re: Apply for assignment of jira MESOS-5425

[Joris Van Remoortere]

Hello!
Thanks for sharing your feedback regarding the range performance. Indeed as you 
partition things like agent ports heavily this can be detrimental.

We have looked at this in the past, and the right thing to do is indeed to use 
the interval set in Stout.

That being said, the reason we haven't done so yet is that the Resources object 
currently exposes the protobuf used to store the data internally.

The approach we would like to take is to introduce a c++ based resource and 
resources object that is unversion (used only internally).

Primarily Mpark, BenM, XuJYan, and I have been thinking about how to tackle 
this larger task.

Please bring this up at the next community sync so we can see how we might cut 
this into manageable chunks.

Thanks,
Joris



Sent from my iPhone
> On Jun 12, 2016, at 4:33 AM, Yan Yan YY Hu  wrote:
> 
> Thanks, Guangya. I will send another mail to apply for it.
> 
> 
> Best regards!
> **
> Yanyan Hu(胡彦彦) Ph.D.
> Cloud Infrastructure & Technology Team 
> Building 19 Zhongguancun Software Park, 8 Dongbeiwang WestRoad, Haidian
> District, Beijing,P.R.C.100094
> E-mail: yanya...@cn.ibm.com
> Tel: 8610-58748025
> ***
> 
> Guangya Liu ---2016-06-12 上午 11:09:34---You may want to post an email to 
> dev@mesos.apache.org to request add you as a contributor first.
> 
> From: Guangya Liu 
> To: dev 
> Cc: Joseph Wu , Seetharami Seelam 
> Date: 2016-06-12 上午 11:09
> Subject: Re: Apply for assignment of jira MESOS-5425
> 
> 
> 
> 
> You may want to post an email to dev@mesos.apache.org to request add you as
> a contributor first.
> 
> Please take a look at
> https://github.com/apache/mesos/blob/master/docs/newbie-guide.md#getting-started-guidance
> 
> Thanks,
> 
> Guangya
> 
> On Sun, Jun 12, 2016 at 10:44 AM, Yan Yan YY Hu  wrote:
> 
> >
> > Dear Mesos team,
> >
> > We are now trying to use Mesos to manage container cluster in large scale.
> > During our test, we found some performance issue about Ranges value
> > subtraction. The current implementation is low-efficient and causes serious
> > overhead to Mesos allocator. Joseph Wu has helped to file a jira to track
> > this issue:https://issues.apache.org/jira/browse/MESOS-5425. We have
> > prepared a fix by re-implementing the Ranges subtraction using IntervalSet
> > data type and we hope to get assignment of this jira to contribute our fix
> > back to Mesos. Thank you so much!
> >
> >
> > Best regards!
> > **
> > Yanyan Hu(胡彦彦) Ph.D.
> > Cloud Infrastructure & Technology Team
> > Building 19 Zhongguancun Software Park, 8 Dongbeiwang WestRoad, Haidian
> > District, Beijing,P.R.C.100094
> > E-mail: yanya...@cn.ibm.com
> > Tel: 8610-58748025
> > ***
> >
> 
> 
> 

Re: Rack awareness support for Mesos

[Joris Van Remoortere]

>
> #1. Stick with attributes for rack awareness

I don't think this is the right approach; however, there seem to be 2
components to this discussion:

1. How the values are presented (Attributes vs. a new type-aware structure)
2. How the values are determined (scripts vs. automation vs. modules)

It seems you are more interested in working on #2. If that's the case,
please make sure that you don't assume anything about #1, as we not
everyone agrees that we will use the existing attributes in the future.

For #2, you should focus on an API (module or script results) that will
support all the different methods the community wants to use to generate
this data.

As you mentioned, updating the values for a running agent is not
straightforward. A lot of design work will need to go into how these values
are propagated to frameworks that have made assumptions about them, and
which values are allowed to change vs. not.

—
*Joris Van Remoortere*
Mesosphere

On Tue, Jun 14, 2016 at 10:04 AM, Aaron Carey  wrote:

> #3 would be very helpful for us. Also related:
>
> https://issues.apache.org/jira/browse/MESOS-3059
>
> --
>
> Aaron Carey
> Production Engineer - Cloud Pipeline
> Industrial Light & Magic
> London
> 020 3751 9150
>
> 
> From: Du, Fan [fan...@intel.com]
> Sent: 14 June 2016 07:24
> To: u...@mesos.apache.org; dev@mesos.apache.org
> Cc: Joris Van Remoortere; vinodk...@apache.org
> Subject: Re: Rack awareness support for Mesos
>
> Hi everyone
>
> Let me summarize the discussion about Rack awareness in the community so
> far. First thanks for all the comments, advices or challenges! :)
>
> #1. Stick with attributes for rack awareness
>
> For compatibility with existing framework, I tend to be ok with using
> attributes to convey the rack information, but with the goal to do it
> automatically, easy to maintain and with good attributes schema. This
> will bring up below question where the controversy starts.
>
> #2. Scripts vs programmatic way
>
> Both can be used to set attributes, I've made my arguments in the Jira
> and the Design doc, I'm not gonna to argue more here. But please take a
> look discussion at MESOS-3366 before, which allow resources/attributes
> discovery.
>
> A module to implement *slaveAttributesDecorator* hook will works like
> a charm here in a static way. And need to justify attributes updating.
>
> #3. Allow updating attributes
> Several cases need to be covered here:
>
> a). Mesos runs inside VMs or container, where live migration happens, so
> rack information need to be updated.
>
> b). LLDP packets are broadcasted by the interval 10s~30s, a vendor
> specific implementation, and rack information are usually stored in LLDP
> daemon to be queried. Worst cases(nodes fresh reboot, or daemon restart)
> would be: Mesos slave have to wait 10s~30s for a valid rack information
> before register to master. Allow updating attributes will mitigate this
> problem.
>
> c). Framework affinity
>
> Framework X prefers to run on the same nodes with another framwork Y.
> For example, it's desirable for Shark or Spark-SQL to reside on the
> *worker* node where Alluxio(former Tachyon) to gain more performance
> boosting as SPARK-6707 ticket message {tachyon=true;us-east-1=false}
>
> If framework could advertise agent attributes in the ResourcesOffer
> process, awesome!
>
>
> #4. Rearrange agents in a more scalable manner, like per rack basis
>
> Randomly offering agents resource to framework does not improve data
> locality, imagine the likelihood of a framework getting resources
> underneath the same rack, at the scale of +3 nodes. Moreover time to
> randomly shuffle the agents also grows.
>
> How about rearranging the agent in a per rack basis, and a minor change
> to the way how resources are allocated will fix this.
>
>
> I might not see the whole picture here, so comments are welcomed!
>
>
> On 2016/6/6 17:17, Du, Fan wrote:
> > Hi, Mesos folks
> >
> > I’ve been thinking about Mesos rack awareness support for a while,
> >
> > it’s a common interest for lots of data center applications to provide
> > data locality,
> >
> > fault tolerance and better task placement. Create MESOS-5545 to track
> > the story,
> >
> > and here is the initial design doc [1] to support rack awareness in
> Mesos.
> >
> > Looking forward to hear any comments from end user and other developers,
> >
> > Thanks!
> >
> > [1]:
> >
> https://docs.google.com/document/d/1rql_LZSwtQzBPALnk0qCLsmxcT3-zB7X7aJp-H3xxyE/edit?usp=sharing
> >
>

Re: Rack awareness support for Mesos

[Joris Van Remoortere]

> On the condition of compatible with existing framework which already rely
on parsing attributes for rack information.
There is currently nothing in Mesos that specifies the format or structure
for rack information in attributes.
The fact that operators / frameworks have decided to add this information
out of band is their problem to solve.
We don't need to be backwards compatible with something we never published
to begin with. This is why it's ok for us to consider adding a typed form
of failure domain information that is separate from the typeless string
attributes.

Since your interest is in the determination of the values, as opposed to
their propagation, I would just urge that you keep in mind that we may (as
a project) not want to support this information as the current string
attributes.



—
*Joris Van Remoortere*
Mesosphere

On Tue, Jun 14, 2016 at 3:02 PM, Du, Fan  wrote:

>
>
> On 2016/6/14 20:32, Joris Van Remoortere wrote:
>
>> #1. Stick with attributes for rack awareness
>>
>> I don't think this is the right approach; however, there seem to be 2
>> components to this discussion:
>>
>> 1. How the values are presented (Attributes vs. a new type-aware
>> structure)
>> 2. How the values are determined (scripts vs. automation vs. modules)
>>
>> It seems you are more interested in working on #2. If that's the case,
>> please make sure that you don't assume anything about #1, as we not
>> everyone agrees that we will use the existing attributes in the future.
>>
>
> On the condition of compatible with existing framework which already rely
> on parsing attributes for rack information.
>
> Quotes from my original statements:
> > For compatibility with existing framework, I tend to be ok with using
> > attributes to convey the rack information
>
> By all means, no matter what internal structures to use, current behavior
> should be honored. btw, I'm also thinking about #1, it's too earlier to
> bring up the details so far before the ticket got ACCEPTED.
>
> Any way, I'm always open to all kind of discussion, thanks for your
> comments! Joris.
>
> For #2, you should focus on an API (module or script results) that will
>> support all the different methods the community wants to use to generate
>> this data.
>>
>> As you mentioned, updating the values for a running agent is not
>> straightforward. A lot of design work will need to go into how these
>> values are propagated to frameworks that have made assumptions about
>> them, and which values are allowed to change vs. not.
>>
>> —
>> *Joris Van Remoortere*
>> Mesosphere
>>
>> On Tue, Jun 14, 2016 at 10:04 AM, Aaron Carey > <mailto:aca...@ilm.com>> wrote:
>>
>> #3 would be very helpful for us. Also related:
>>
>> https://issues.apache.org/jira/browse/MESOS-3059
>>
>> --
>>
>> Aaron Carey
>> Production Engineer - Cloud Pipeline
>> Industrial Light & Magic
>> London
>> 020 3751 9150
>>
>> 
>> From: Du, Fan [fan...@intel.com <mailto:fan...@intel.com>]
>> Sent: 14 June 2016 07:24
>> To: u...@mesos.apache.org <mailto:u...@mesos.apache.org>;
>> dev@mesos.apache.org <mailto:dev@mesos.apache.org>
>> Cc: Joris Van Remoortere; vinodk...@apache.org
>> <mailto:vinodk...@apache.org>
>>
>> Subject: Re: Rack awareness support for Mesos
>>
>> Hi everyone
>>
>> Let me summarize the discussion about Rack awareness in the community
>> so
>> far. First thanks for all the comments, advices or challenges! :)
>>
>> #1. Stick with attributes for rack awareness
>>
>> For compatibility with existing framework, I tend to be ok with using
>> attributes to convey the rack information, but with the goal to do it
>> automatically, easy to maintain and with good attributes schema. This
>> will bring up below question where the controversy starts.
>>
>> #2. Scripts vs programmatic way
>>
>> Both can be used to set attributes, I've made my arguments in the Jira
>> and the Design doc, I'm not gonna to argue more here. But please take
>> a
>> look discussion at MESOS-3366 before, which allow resources/attributes
>> discovery.
>>
>> A module to implement *slaveAttributesDecorator* hook will works like
>> a charm here in a static way. And need to justify attributes updating.
>>
>> #3. Allow updating attributes
>>

Re: Multiple framework DRF tuning

[Joris Van Remoortere]

Your e-mail seems truncated or mangled in some way (at least for me). Could
you please resend it?

—
*Joris Van Remoortere*
Mesosphere

On Tue, Jun 14, 2016 at 10:00 PM, meghdoot bhattacharya <
meghdoo...@yahoo.com.invalid> wrote:

>
>
> I wanted to follow up on 2 issues that we discussed few years back on the
> below blog described in this section Mesos delayed offers to frameworks
>
>
> http://www.ebaytechblog.com/2014/04/04/delivering-ebays-ci-solution-with-apache-mesos-part-i/
>
>
>
> We had sample commits back then for private build. and I am sure code has
> changed drastically since.
>
>
> 1. Reset allocation counter across all frameworks when new frameworks join
> (some sort of normalization)* ben's fix for allocations
> https://issues.apache.org/jira/browse/MES… · asathye/mesos@b7ad40d
>
>
> 2. Precision math not resulting in 0 share value.
> * Set double precision at 0.01 · asathye/mesos@56c144f
>
>
>
> Has these been addressed in some form? Does Mesos-4687 address point 2
> above?
>
>
>
> Thx
>
>
> |
> |
> |
> |   ||
>
>|
>
>   |
> |
> ||
> * Set double precision at 0.01 · asathye/mesos@56c144f
>  mesos - Mirror of Apache Mesos  |   |
>
>   |
>
>   |
>
>
>
>
>
>
> |
> |
> |
> |   ||
>
>|
>
>   |
> |
> ||
> * ben's fix for allocations https://issues.apache.org/jira/browse/MES… ...
>  …OS-1086 * reset allocations on new framework addition  |   |
>
>   |
>
>   |
>
>
>

Re: [GPU] [Allocation] "Scarce" Resource Allocation

[Joris Van Remoortere]

With this 4th sorter approach, how does quota work for scarce resources?

—
*Joris Van Remoortere*
Mesosphere

On Thu, Jun 16, 2016 at 11:26 AM, Guangya Liu  wrote:

> Hi Ben,
>
> The pre-condition for four stage allocation is that we need to put
> different resources to different sorters:
>
> 1) roleSorter only include non scarce resources.
> 2) quotaRoleSorter only include non revocable & non scarce resources.
> 3) revocableSorter only include revocable & non scarce resources. This will
> be handled in MESOS-4923 <https://issues.apache.org/jira/browse/MESOS-4923
> >
> 4) scarceSorter only include scarce resources.
>
> Take your case above:
> 999 agents with (cpus:4,mem:1024,disk:1024)
> 1 agent with (gpus:1,cpus:4,mem:1024,disk:1024)
>
> The four sorters would be:
> 1) roleSorter include 1000 agents with (cpus:4,mem:1024,disk:1024)
> 2) quotaRoleSorter include 1000 agents with (cpus:4,mem:1024,disk:1024)
> 3) revocableSorter include nothing as I have no revocable resources here.
> 4) scarceSorter include 1 agent with (gpus:1)
>
> When allocate resources, even if a role got the agent with gpu resources,
> its share will only be counter by scarceSorter but not other sorters, and
> will not impact other sorters.
>
> The above solution is actually kind of enhancement to "exclude scarce
> resources" as the scarce resources also obey the DRF algorithm with this.
>
> This solution can be also treated as diving the whole resources pool
> logically to scarce and non scarce resource pool. 1), 2) and 3) will handle
> non scarce resources while 4) focus on scarce resources.
>
> Thanks,
>
> Guangya
>
> On Thu, Jun 16, 2016 at 2:10 AM, Benjamin Mahler 
> wrote:
>
> > Hm.. can you expand on how adding another allocation stage for only
> scarce
> > resources would behave well? It seems to have a number of problems when I
> > think through it.
> >
> > On Sat, Jun 11, 2016 at 7:59 AM, Guangya Liu  wrote:
> >
> >> Hi Ben,
> >>
> >> For long term goal, instead of creating sub-pool, what about adding a
> new
> >> sorter to handle **scare** resources? The current logic in allocator was
> >> divided to two stages: allocation for quota, allocation for non quota
> >> resources.
> >>
> >> I think that the future logic in allocator would be divided to four
> >> stages:
> >> 1) allocation for quota
> >> 2) allocation for reserved resources
> >> 3) allocation for revocable resources
> >> 4) allocation for scare resources
> >>
> >> Thanks,
> >>
> >> Guangy
> >>
> >> On Sat, Jun 11, 2016 at 10:50 AM, Benjamin Mahler 
> >> wrote:
> >>
> >>> I wanted to start a discussion about the allocation of "scarce"
> >>> resources. "Scarce" in this context means resources that are not
> present on
> >>> every machine. GPUs are the first example of a scarce resource that we
> >>> support as a known resource type.
> >>>
> >>> Consider the behavior when there are the following agents in a cluster:
> >>>
> >>> 999 agents with (cpus:4,mem:1024,disk:1024)
> >>> 1 agent with (gpus:1,cpus:4,mem:1024,disk:1024)
> >>>
> >>> Here there are 1000 machines but only 1 has GPUs. We call GPUs a
> >>> "scarce" resource here because they are only present on a small
> percentage
> >>> of the machines.
> >>>
> >>> We end up with some problematic behavior here with our current
> >>> allocation model:
> >>>
> >>> (1) If a role wishes to use both GPU and non-GPU resources for
> >>> tasks, consuming 1 GPU will lead DRF to consider the role to have a
> 100%
> >>> share of the cluster, since it consumes 100% of the GPUs in the
> cluster.
> >>> This framework will then not receive any other offers.
> >>>
> >>> (2) Because we do not have revocation yet, if a framework decides
> to
> >>> consume the non-GPU resources on a GPU machine, it will prevent the GPU
> >>> workloads from running!
> >>>
> >>> 
> >>>
> >>> I filed an epic [1] to track this. The plan for the short-term is to
> >>> introduce two mechanisms to mitigate these issues:
> >>>
> >>> -Introduce a resource fairness exclusion list. This allows the
> >>> shares of resources like "gpus" to be excluded from the dominant share.
> >>>
> >>> -Introduce a GPU

Re: Multiple framework DRF tuning

[Joris Van Remoortere]

Hey Meghdoot,

Mesos-4687 does indeed fix the precision math problem. From the JIRA you
can see it is fixed in 0.28.0, 0.27.2, 0.26.1, 0.25.1, 0.24.2.

For #1 it doesn't look like an allocation count reset was ever merged.

I've CCd BenM so we can discuss whether there are any bad consequences from
doing this.

—
*Joris Van Remoortere*
Mesosphere

On Wed, Jun 15, 2016 at 8:33 PM, meghdoot bhattacharya <
meghdoo...@yahoo.com.invalid> wrote:

> Resending with fixed links.
>
> I wanted to follow up on 2 issues that we discussed few years back on the
> below blog described in this section Mesos delayed offers to frameworks
>
>
> http://www.ebaytechblog.com/2014/04/04/delivering-ebays-ci-solution-with-apache-mesos-part-i/
>
>
>
> We had sample commits back then for private build. and I am sure code has
> changed drastically since.
>
>
> 1. Reset allocation counter across all frameworks when new frameworks join
> (some sort of normalization)
> https://github.com/asathye/mesos/commit/b7ad40dc254390751069f55a6756efcf7d1bb9aa
>
>
>
>
> 2. Precision math not resulting in 0 share value.
>
> https://github.com/asathye/mesos/commit/56c144f8cf48bb14c3f0fb639e9791170c1b6267
>
>
>
>
>
> Has these been addressed in some form? Does Mesos-4687 address point 2
> above?
>
>
> Thx
>
>   From: Joris Van Remoortere 
>  To: "dev@mesos.apache.org" ; meghdoot bhattacharya
> 
>  Sent: Wednesday, June 15, 2016 12:47 AM
>  Subject: Re: Multiple framework DRF tuning
>
> Your e-mail seems truncated or mangled in some way (at least for me). Could
> you please resend it?
>
> —
> *Joris Van Remoortere*
> Mesosphere
>
> On Tue, Jun 14, 2016 at 10:00 PM, meghdoot bhattacharya <
> meghdoo...@yahoo.com.invalid> wrote:
>
> >
> >
> > I wanted to follow up on 2 issues that we discussed few years back on the
> > below blog described in this section Mesos delayed offers to frameworks
> >
> >
> >
> http://www.ebaytechblog.com/2014/04/04/delivering-ebays-ci-solution-with-apache-mesos-part-i/
> >
> >
> >
> > We had sample commits back then for private build. and I am sure code has
> > changed drastically since.
> >
> >
> > 1. Reset allocation counter across all frameworks when new frameworks
> join
> > (some sort of normalization)* ben's fix for allocations
> > https://issues.apache.org/jira/browse/MES… · asathye/mesos@b7ad40d
> >
> >
> > 2. Precision math not resulting in 0 share value.
> > * Set double precision at 0.01 · asathye/mesos@56c144f
> >
> >
> >
> > Has these been addressed in some form? Does Mesos-4687 address point 2
> > above?
> >
> >
> >
> > Thx
> >
> >
> > |
> > |
> > |
> > |  ||
> >
> >|
> >
> >  |
> > |
> > ||
> > * Set double precision at 0.01 · asathye/mesos@56c144f
> >  mesos - Mirror of Apache Mesos  |  |
> >
> >  |
> >
> >  |
> >
> >
> >
> >
> >
> >
> > |
> > |
> > |
> > |  ||
> >
> >|
> >
> >  |
> > |
> > ||
> > * ben's fix for allocations https://issues.apache.org/jira/browse/MES…
> ...
> >  …OS-1086 * reset allocations on new framework addition  |  |
> >
> >  |
> >
> >  |
> >
> >
> >
>
>
>

Re: [GPU] [Allocation] "Scarce" Resource Allocation

[Joris Van Remoortere]

@Fan,

In the community meeting a question was raised around which frameworks
might be ready to use this.
Can you provide some more context for immediate use cases on the framework
side?


—
*Joris Van Remoortere*
Mesosphere

On Fri, Jun 17, 2016 at 12:51 AM, Du, Fan  wrote:

> A couple of rough thoughts in the early morning:
>
> a. Is there any quantitative way to decide a resource is kind of scare? I
> mean how to aid operator to make this decision to use/not use this
> functionality when deploying mesos.
>
> b. Scare resource extend from GPU to, name a few, Xeon Phi, FPGA, what
> about make the proposal more generic and future proof?
>
>
>
> On 2016/6/11 10:50, Benjamin Mahler wrote:
>
>> I wanted to start a discussion about the allocation of "scarce" resources.
>> "Scarce" in this context means resources that are not present on every
>> machine. GPUs are the first example of a scarce resource that we support
>> as
>> a known resource type.
>>
>> Consider the behavior when there are the following agents in a cluster:
>>
>> 999 agents with (cpus:4,mem:1024,disk:1024)
>> 1 agent with (gpus:1,cpus:4,mem:1024,disk:1024)
>>
>> Here there are 1000 machines but only 1 has GPUs. We call GPUs a "scarce"
>> resource here because they are only present on a small percentage of the
>> machines.
>>
>> We end up with some problematic behavior here with our current allocation
>> model:
>>
>>  (1) If a role wishes to use both GPU and non-GPU resources for tasks,
>> consuming 1 GPU will lead DRF to consider the role to have a 100% share of
>> the cluster, since it consumes 100% of the GPUs in the cluster. This
>> framework will then not receive any other offers.
>>
>>  (2) Because we do not have revocation yet, if a framework decides to
>> consume the non-GPU resources on a GPU machine, it will prevent the GPU
>> workloads from running!
>>
>> 
>>
>> I filed an epic [1] to track this. The plan for the short-term is to
>> introduce two mechanisms to mitigate these issues:
>>
>>  -Introduce a resource fairness exclusion list. This allows the shares
>> of resources like "gpus" to be excluded from the dominant share.
>>
>>  -Introduce a GPU_AWARE framework capability. This indicates that the
>> scheduler is aware of GPUs and will schedule tasks accordingly. Old
>> schedulers will not have the capability and will not receive any offers
>> for
>> GPU machines. If a scheduler has the capability, we'll advise that they
>> avoid placing their additional non-GPU workloads on the GPU machines.
>>
>> 
>>
>> Longer term, we'll want a more robust way to manage scarce resources. The
>> first thought we had was to have sub-pools of resources based on machine
>> profile and perform fair sharing / quota within each pool. This addresses
>> (1) cleanly, and for (2) the operator needs to explicitly disallow non-GPU
>> frameworks from participating in the GPU pool.
>>
>> Unfortunately, by excluding non-GPU frameworks from the GPU pool we may
>> have a lower level of utilization. In the even longer term, as we add
>> revocation it will be possible to allow a scheduler desiring GPUs to
>> revoke
>> the resources allocated to the non-GPU workloads running on the GPU
>> machines. There are a number of things we need to put in place to support
>> revocation ([2], [3], [4], etc), so I'm glossing over the details here.
>>
>> If anyone has any thoughts or insight in this area, please share!
>>
>> Ben
>>
>> [1] https://issues.apache.org/jira/browse/MESOS-5377
>> [2] https://issues.apache.org/jira/browse/MESOS-5524
>> [3] https://issues.apache.org/jira/browse/MESOS-5527
>> [4] https://issues.apache.org/jira/browse/MESOS-4392
>>
>>

Re: Failed to shutdown socket with fd xxx

[Joris Van Remoortere]

Can you provide:
1. The version that you are upgrading from.
2. Whether you made any OS / init system changes alongside this upgrade
(just to narrow the scope).

It is possible that you are upgrading from a version that did not have
systemd support to one that does. If so, the upgrade may require restarting
the tasks (either by themselves, or just starting a fresh agent). Please
check out some of the work in MESOS-3007 to get a better understanding of
what the issue I am referring to is.

If you can verify that you are making one of these transitions from a bad
world to a good world, then you can devise a plan for your upgrade.

Joris

—
*Joris Van Remoortere*
Mesosphere

On Fri, Jun 17, 2016 at 8:28 AM, Qiang Chen  wrote:

> Hi all,
>
> I met an issue when upgrading mesos-slave to 0.28.2.
>
> At the process of recovering mesos-slave / framework container stage, it
> produced the following errors.
>
>
> ```
> Log file created at: 2016/06/15 15:01:43
> Running on machine: mesos-slave-online005-xxx.cloud.xxx.domain
> Log line format: [IWEF]mmdd hh:mm:ss.uu threadid file:line] msg
> W0615 15:01:43.285518  4182 linux_launcher.cpp:197] Couldn't find pid
> '42322' in 'mesos_executors.slice'. This can lead to lack of proper
> resource isolation
> W0615 15:01:43.286182  4182 linux_launcher.cpp:197] Couldn't find pid
> '42312' in 'mesos_executors.slice'. This can lead to lack of proper
> resource isolation
> W0615 15:01:43.286669  4182 linux_launcher.cpp:197] Couldn't find pid
> '42309' in 'mesos_executors.slice'. This can lead to lack of proper
> resource isolation
> W0615 15:01:43.287144  4182 linux_launcher.cpp:197] Couldn't find pid
> '42304' in 'mesos_executors.slice'. This can lead to lack of proper
> resource isolation
> W0615 15:01:43.287636  4182 linux_launcher.cpp:197] Couldn't find pid
> '42300' in 'mesos_executors.slice'. This can lead to lack of proper
> resource isolation
> W0615 15:01:43.288120  4182 linux_launcher.cpp:197] Couldn't find pid
> '42317' in 'mesos_executors.slice'. This can lead to lack of proper
> resource isolation
> E0615 15:01:43.471676  4201 process.cpp:1958] Failed to shutdown socket
> with fd 24: Transport endpoint is not connected
> E0615 15:01:43.476007  4201 process.cpp:1958] Failed to shutdown socket
> with fd 24: Transport endpoint is not connected
> E0615 15:01:43.476143  4201 process.cpp:1958] Failed to shutdown socket
> with fd 24: Transport endpoint is not connected
> E0615 15:01:43.476272  4201 process.cpp:1958] Failed to shutdown socket
> with fd 24: Transport endpoint is not connected
> E0615 15:01:43.476483  4201 process.cpp:1958] Failed to shutdown socket
> with fd 24: Transport endpoint is not connected
> E0615 15:01:43.476618  4201 process.cpp:1958] Failed to shutdown socket
> with fd 24: Transport endpoint is not connected
>
> ```
>
> And it will also cause the OOM errors, such as:
>
> ```
> I0615 15:01:43.324935  4172 mem.cpp:602] Started listening for OOM events
> for container f50b4c7a-d1d2-4fc8-abb9-5ab549f168dc
> I0615 15:01:43.325469 4172 mem.cpp:722] Started listening on low memory
> pressure events for container f50b4c7a-d1d2-4fc8-abb9-5ab549f168dc
> I0615 15:01:43.326004  4172 mem.cpp:722] Started listening on medium
> memory pressure events for container f50b4c7a-d1d2-4fc8-abb9-5ab549f168dc
> I0615 15:01:43.326539  4172 mem.cpp:722] Started listening on critical
> memory pressure events for container f50b4c7a-d1d2-4fc8-abb9-5ab549f168dc
>
> ```
>
> Did someone suffer this? thanks.
>
> --
> Best Regards,
> Chen, Qiang
>
>

Re: Failed to shutdown socket with fd xxx

[Joris Van Remoortere]

The shutdown errors are not the issue.
The concerning part is this warning:

> W0615 15:01:43.285518  4182 linux_launcher.cpp:197] Couldn't find pid
> '42322' in 'mesos_executors.slice'. This can lead to lack of proper
> resource isolation

That indicates a transition from the old systemd lack of support to the new
support.

—
*Joris Van Remoortere*
Mesosphere

On Fri, Jun 17, 2016 at 2:35 PM, haosdent  wrote:

> Hi, @Qiang.
>
> @Joseph have a nice explain about at Shutdown failed on fd
>
> http://search-hadoop.com/m/0Vlr6pe7qb2MJX8B1&subj=Re+Benign+Shutdown+failed+on+fd+error+messages
> Those errors could be ignored.
>
> For
> ```
> I0615 15:01:43.324935  4172 mem.cpp:602] Started listening for OOM events
> for container f50b4c7a-d1d2-4fc8-abb9-5ab549f168dc
> ```
>
> These are normal info log, it happen when Mesos CgroupMemIsolator register
> oom hooks for your containers.
>
> On Fri, Jun 17, 2016 at 8:22 PM, Joris Van Remoortere  >
> wrote:
>
> > Can you provide:
> > 1. The version that you are upgrading from.
> > 2. Whether you made any OS / init system changes alongside this upgrade
> > (just to narrow the scope).
> >
> > It is possible that you are upgrading from a version that did not have
> > systemd support to one that does. If so, the upgrade may require
> restarting
> > the tasks (either by themselves, or just starting a fresh agent). Please
> > check out some of the work in MESOS-3007 to get a better understanding of
> > what the issue I am referring to is.
> >
> > If you can verify that you are making one of these transitions from a bad
> > world to a good world, then you can devise a plan for your upgrade.
> >
> > Joris
> >
> > —
> > *Joris Van Remoortere*
> > Mesosphere
> >
> > On Fri, Jun 17, 2016 at 8:28 AM, Qiang Chen  wrote:
> >
> > > Hi all,
> > >
> > > I met an issue when upgrading mesos-slave to 0.28.2.
> > >
> > > At the process of recovering mesos-slave / framework container stage,
> it
> > > produced the following errors.
> > >
> > >
> > > ```
> > > Log file created at: 2016/06/15 15:01:43
> > > Running on machine: mesos-slave-online005-xxx.cloud.xxx.domain
> > > Log line format: [IWEF]mmdd hh:mm:ss.uu threadid file:line] msg
> > > W0615 15:01:43.285518  4182 linux_launcher.cpp:197] Couldn't find pid
> > > '42322' in 'mesos_executors.slice'. This can lead to lack of proper
> > > resource isolation
> > > W0615 15:01:43.286182  4182 linux_launcher.cpp:197] Couldn't find pid
> > > '42312' in 'mesos_executors.slice'. This can lead to lack of proper
> > > resource isolation
> > > W0615 15:01:43.286669  4182 linux_launcher.cpp:197] Couldn't find pid
> > > '42309' in 'mesos_executors.slice'. This can lead to lack of proper
> > > resource isolation
> > > W0615 15:01:43.287144  4182 linux_launcher.cpp:197] Couldn't find pid
> > > '42304' in 'mesos_executors.slice'. This can lead to lack of proper
> > > resource isolation
> > > W0615 15:01:43.287636  4182 linux_launcher.cpp:197] Couldn't find pid
> > > '42300' in 'mesos_executors.slice'. This can lead to lack of proper
> > > resource isolation
> > > W0615 15:01:43.288120  4182 linux_launcher.cpp:197] Couldn't find pid
> > > '42317' in 'mesos_executors.slice'. This can lead to lack of proper
> > > resource isolation
> > > E0615 15:01:43.471676  4201 process.cpp:1958] Failed to shutdown socket
> > > with fd 24: Transport endpoint is not connected
> > > E0615 15:01:43.476007  4201 process.cpp:1958] Failed to shutdown socket
> > > with fd 24: Transport endpoint is not connected
> > > E0615 15:01:43.476143  4201 process.cpp:1958] Failed to shutdown socket
> > > with fd 24: Transport endpoint is not connected
> > > E0615 15:01:43.476272  4201 process.cpp:1958] Failed to shutdown socket
> > > with fd 24: Transport endpoint is not connected
> > > E0615 15:01:43.476483  4201 process.cpp:1958] Failed to shutdown socket
> > > with fd 24: Transport endpoint is not connected
> > > E0615 15:01:43.476618  4201 process.cpp:1958] Failed to shutdown socket
> > > with fd 24: Transport endpoint is not connected
> > >
> > > ```
> > >
> > > And it will also cause the OOM errors, such as:
> > >
> > > ```
> > > I0615 15:01:43.324935  4172 mem.cpp:602] Started listening for OOM
> events
> > > for container f50b4c7a-d1d2-4fc8-abb9-5ab549f168dc
> > > I0615 15:01:43.325469 4172 mem.cpp:722] Started listening on low memory
> > > pressure events for container f50b4c7a-d1d2-4fc8-abb9-5ab549f168dc
> > > I0615 15:01:43.326004  4172 mem.cpp:722] Started listening on medium
> > > memory pressure events for container
> f50b4c7a-d1d2-4fc8-abb9-5ab549f168dc
> > > I0615 15:01:43.326539  4172 mem.cpp:722] Started listening on critical
> > > memory pressure events for container
> f50b4c7a-d1d2-4fc8-abb9-5ab549f168dc
> > >
> > > ```
> > >
> > > Did someone suffer this? thanks.
> > >
> > > --
> > > Best Regards,
> > > Chen, Qiang
> > >
> > >
> >
>
>
>
> --
> Best Regards,
> Haosdent Huang
>

Re: Slack as the canonical chat channel

[Joris Van Remoortere]

+1 Slack

—
*Joris Van Remoortere*
Mesosphere

On Fri, Jun 17, 2016 at 10:04 PM, Vinod Kone  wrote:

> Looks like people have jumped the gun here before I sent the email :)
>
> Here is the context. During the community sync we discussed about using
> *Slack* or *HipChat* as our official chat channel instead of our current
> #mesos IRC channel on freenode.
>
> The main reasons for using Slack/Hipchat are
>
>- In-client chat history
>- Discoverability of work group specific channels
>- Email notifications when offline
>- Modern UX and clients
>
> During the sync most people preferred the move to *Slack*. I wanted to get
> a sense from other community members as well through this email. Please let
> us know what you think.
>
> Note that even if we move to Slack, we will make sure people can still
> connect using IRC clients and that the chat history is publicly available
> (per ASF guidelines). During the transition period, we might mirror
> messages from Slack channel to IRC and vice-versa.
>
> Thoughts?
>
> On Fri, Jun 17, 2016 at 8:52 AM, Vinit Mahedia 
> wrote:
>
> > +1 Slack.
> >
> > On Fri, Jun 17, 2016 at 12:59 AM, Jay JN Guo 
> > wrote:
> >
> > > +1 Slack!
> > >
> > > /J
> > >
> > > Vaibhav Khanduja  wrote on 06/16/2016
> > 22:26:27:
> > >
> > > > From: Vaibhav Khanduja 
> > > > To: dev@mesos.apache.org
> > > > Date: 06/16/2016 22:26
> > > > Subject: Re: Notification: Community Meeting @ Thu Jun 16, 2016 3pm
> > > > - 4pm (Apache Mesos)
> > > >
> > > > + 1 slack
> > > >
> > > > Sent from my iPhone. Please excuse for typos and brevity of this
> > message.
> > > >
> > > > > On Jun 16, 2016, at 6:46 PM, haosdent  wrote:
> > > > >
> > > > > +1 For Slack.
> > > > >
> > > > >> On Fri, Jun 17, 2016 at 7:33 AM, Greg Mann 
> > > wrote:
> > > > >>
> > > > >> Hello all,
> > > > >> Here are the notes from our community sync meeting this afternoon:
> > > > >>
> > > > >> Attendees:
> > > > >>
> > > > >> Mesosphere: Joris, Greg, Haris, Artem, Joseph, Kapil, Anand,
> > Gilbert,
> > > > >> Harpreet, Kevin, Vinod, Jie, Joerg, MPark
> > > > >>
> > > > >> Uber: Zhitao Li
> > > > >>
> > > > >> Agenda/Note:
> > > > >>
> > > > >>   -
> > > > >>
> > > > >>   Reviewing the list of maintainers on
> > > > >>   http://mesos.apache.org/documentation/latest/committers/
> > > > >>   -
> > > > >>
> > > > >>  Add components for
> > > > >>  -
> > > > >>
> > > > >> Documentation (docs/*)
> > > > >> -
> > > > >>
> > > > >> Windows (*windows*)
> > > > >> -
> > > > >>
> > > > >> C++ standards (docs/c++-style-guide.md)
> > > > >> -
> > > > >>
> > > > >> HTTP API (http.*)
> > > > >> -
> > > > >>
> > > > >> Persistence
> > > > >> -
> > > > >>
> > > > >> Test infrastructure (src/tests/*)
> > > > >> -
> > > > >>
> > > > >> Build-related
> > > > >> -
> > > > >>
> > > > >>Autotools, CMake
> > > > >>-
> > > > >>
> > > > >> Subdivide Stout
> > > > >> -
> > > > >>
> > > > >> Subdivide Libprocess (3rdparty/libprocess/*)
> > > > >> -
> > > > >>
> > > > >> Subdivide Container-related things
> > (src/slave/containerizer/*)
> > > > >> -
> > > > >>
> > > > >>Networking
> > > > >>-
> > > > >>
> > > > >>Storage
> > > > >>-
> > > > >>
> > > > >> Resource allocation/Scheduler
> > > > >> -

Re: Failed to shutdown socket with fd xxx

[Joris Van Remoortere]

>
> For "That indicates a transition from the old systemd lack of support to
> the new support. "
> >> lack of what support ? would explain more details, and how to fix this?
> or may have other cause ?


There were a few versions of Mesos where we were not yet aware of some of
the issues with running under systemd. There was a fix for the
LinuxLauncher in 0.25 (https://issues.apache.org/jira/browse/MESOS-3425)
and further fixes for the posix launcher and docker containerizer in 0.28
and some backports. See the systemd documentation at the bottom of this
page: http://mesos.apache.org/documentation/latest/agent-recovery/

It's possible that you have tasks left over from before we had this
support, which means they are not running under the executor slice. These
technically could lose their isolation (as mentioned in the warning). If
you care about the isolation (you likely do in production), then the only
remedy is to restart them.

—
*Joris Van Remoortere*
Mesosphere

On Mon, Jun 20, 2016 at 4:45 AM, Qiang Chen  wrote:

> Thanks @Haosdent for the link to explain the shutdown errors. so I can
> ignore this...
>
> @Joris,
>
> 1. I upgraded form 0.25.0 to 0.28.2 in centos 7 which  has systemd support.
> 2. I didn't make any OS / init system changes
>
> For "That indicates a transition from the old systemd lack of support to
> the new support. "
> >> lack of what support ? would explain more details, and how to fix this?
> or may have other cause ?
>
> Thanks great again!
>
>
> On 2016年06月17日 21:31, Joris Van Remoortere wrote:
>
> [image: Boxbe] <https://www.boxbe.com/overview> This message is eligible
> for Automatic Cleanup! (jo...@mesosphere.io) Add cleanup rule
> <https://www.boxbe.com/popup?url=https%3A%2F%2Fwww.boxbe.com%2Fcleanup%3Fkey%3DINo0V0shoF5SDDeFNLmOQcDrkM6vuyhBbTAdJ5Ek4fI%253D%26token%3D5pye7msFkBYF5q0SSLYtlGWaWu8a6Imv%252F0E2lgbtu%252BgVEFau%252BV9i3BQYfTGspspkIaoukz1oy8IOSGPyscO1GfcEZlPEs2k3hUGSvAHO6cSuBmHqxd7TnZwBy5RkAx7yt2on45nEbm4%253D&tc_serial=25796382411&tc_rand=1671551284&utm_source=stf&utm_medium=email&utm_campaign=ANNO_CLEANUP_ADD&utm_content=001>
> | More info
> <http://blog.boxbe.com/general/boxbe-automatic-cleanup?tc_serial=25796382411&tc_rand=1671551284&utm_source=stf&utm_medium=email&utm_campaign=ANNO_CLEANUP_ADD&utm_content=001>
>
>
> The shutdown errors are not the issue.
> The concerning part is this warning:
>
>> W0615 15:01:43.285518  4182 linux_launcher.cpp:197] Couldn't find pid
>> '42322' in 'mesos_executors.slice'. This can lead to lack of proper
>> resource isolation
>
> That indicates a transition from the old systemd lack of support to the
> new support.
>
> —
> *Joris Van Remoortere*
> Mesosphere
>
> On Fri, Jun 17, 2016 at 2:35 PM, haosdent  wrote:
>
>> Hi, @Qiang.
>>
>> @Joseph have a nice explain about at Shutdown failed on fd
>>
>> http://search-hadoop.com/m/0Vlr6pe7qb2MJX8B1&subj=Re+Benign+Shutdown+failed+on+fd+error+messages
>> Those errors could be ignored.
>>
>> For
>> ```
>> I0615 15:01:43.324935  4172 mem.cpp:602] Started listening for OOM events
>> for container f50b4c7a-d1d2-4fc8-abb9-5ab549f168dc
>> ```
>>
>> These are normal info log, it happen when Mesos CgroupMemIsolator register
>> oom hooks for your containers.
>>
>> On Fri, Jun 17, 2016 at 8:22 PM, Joris Van Remoortere <
>> jo...@mesosphere.io>
>> wrote:
>>
>> > Can you provide:
>> > 1. The version that you are upgrading from.
>> > 2. Whether you made any OS / init system changes alongside this upgrade
>> > (just to narrow the scope).
>> >
>> > It is possible that you are upgrading from a version that did not have
>> > systemd support to one that does. If so, the upgrade may require
>> restarting
>> > the tasks (either by themselves, or just starting a fresh agent). Please
>> > check out some of the work in MESOS-3007 to get a better understanding
>> of
>> > what the issue I am referring to is.
>> >
>> > If you can verify that you are making one of these transitions from a
>> bad
>> > world to a good world, then you can devise a plan for your upgrade.
>> >
>> > Joris
>> >
>> > —
>> > *Joris Van Remoortere*
>> > Mesosphere
>> >
>> > On Fri, Jun 17, 2016 at 8:28 AM, Qiang Chen < 
>> qzsc...@gmail.com> wrote:
>> >
>> > > Hi all,
>> > >
>> > > I met an issue when upgrading mesos-slave to 0.28.2.
>> > >
>> > > At the pr

Re: Persistent volume ownership issue

[Joris Van Remoortere]

For the case where a container drops down in privileges and still wants to
create a new file, this will result in an error if it is at the root of the
persistent volume right?

Is the recommended pattern then to always create a stub directory at the
root of the persistent volume, and then launch any lower privileged apps
underneath that? For example:

/ <- Root of persistent volume (Owned by framework user / root)
/Database/ <- Stub directory (Owned by lower privileged user)

All new files by the lower privileged app must be created under /Database/*
?
It would result in an error if the App tried to create /Database-backups/ ?
Only the framework as its original user would be able to do that?

—
*Joris Van Remoortere*
Mesosphere

On Tue, Jun 21, 2016 at 8:25 AM, Jie Yu  wrote:

> Hi folks,
>
> Currently, the ownership of the persistent volumes are set to be the same
> as the sandbox. In the implementation, we call `chown -R` on the persistent
> volume to match that of the sandbox each time before we mount it into the
> container.
>
> Recently, we realized that this behavior is not ideal. Especially, if a
> task created some files in the persistent volume, and the owner of those
> file might be different than the task's user. For instance, a task is
> running under root and it creates some database files under user 'database'
> and launch the database process under user 'database'. When the database
> process is restarted by the scheduler, the current behavior is that the
> we'll do a 'chown -R root.root' on the persistent volume, causes database
> files to be chown to 'root'.
>
> The true fix of this problem is to allow frameworks to explicit specify
> owner of persistent volumes during creation. THis is captured in this
> ticket:
> https://issues.apache.org/jira/browse/MESOS-4893
>
> In the short-term (for 1.0), I propose that, instead of doing a recursive
> chown, we do a non-recursive chown. That'll allow the new task to at least
> create new files under the persistent volume, but do not change ownership
> of files created by previous tasks. It should be a very simple fix which we
> can ship in 1.0. We'll ship MESOS-4893 after 1.0. What do you guys think?
>
> Thanks,
> - Jie
>

Re: Mesos CLI

[Joris Van Remoortere]

+1 for maintaining in repo.
+1 for C++. Are the tools in libprocess not sufficient to make this easy to
write? What is missing that would make it easier to write in C++?

—
*Joris Van Remoortere*
Mesosphere

On Wed, Jun 22, 2016 at 2:24 AM, Zhou Z Xing  wrote:

> +1 for keeping it in Mesos repo and rewrite it in C++.
>
> Also, do we need a CLI work group on this?
>
> Thanks & Best Wishes,
>
> Tom Xing(邢舟)
> Emerging Technology Institute, IBM China Software Development Lab
> --
> IBM China Software Development Laboratory (CSDL)
> Notes ID:Zhou Z Xing/China/IBM
> Phone :86-10-82450442
> e-Mail :xingz...@cn.ibm.com
> Address :Building No.28, ZhongGuanCun Software Park, No.8 Dong Bei Wang
> West Road, Haidian District, Beijing, P.R.China 100193
> 地址 :中国北京市海淀区东北旺西路8号 中关村软件园28号楼 100193
>
>
> [image: Inactive hide details for Benjamin Mahler ---2016-06-22 上午
> 06:12:40---+1 for keeping it in the repo. We can establish maint]Benjamin
> Mahler ---2016-06-22 上午 06:12:40---+1 for keeping it in the repo. We can
> establish maintainers for the CLI to ensure that it can mainta
>
> From: Benjamin Mahler 
> To: dev , jfarr...@apache.org
> Date: 2016-06-22 上午 06:12
> Subject: Re: Mesos CLI
> --
>
>
>
> +1 for keeping it in the repo.
>
> We can establish maintainers for the CLI to ensure that it can maintain a
> reasonable update cadence. Note that we haven't done this well for the
> webui and CLI, so we need to make sure we do it better this time around.
>
> If the architecture allows for easy integration of custom commands (written
> in any language) then it should enable users to create their own helpful
> CLI commands that we can eventually pull in and support in a first class
> way.
>
> On Tue, Jun 21, 2016 at 1:26 PM, Jake Farrell  wrote:
>
> > +1 to in repo
> >
> > C++ would be nice to maintain language parity, GO would be a great choice
> > also
> >
> > -Jake
> >
> > On Tue, Jun 21, 2016 at 3:15 PM, Vinod Kone  wrote:
> >
> > > +1 for keeping it in repo.
> > >
> > > Would be nice if the CLI can be written entirely in C++ though, to
> avoid
> > > supporting more languages.
> > >
> > > On Tue, Jun 21, 2016 at 12:12 PM, Jie Yu  wrote:
> > >
> > > > I personally prefer it being part of the Mesos repo so that when
> people
> > > > install our package, they'll get the command line tools as well. That
> > > also
> > > > avoids the potential version mismatch between Mesos and CLI (as you
> > > > mentioned).
> > > >
> > > > What does others think?
> > > >
> > > > - Jie
> > > >
> > > > On Tue, Jun 21, 2016 at 10:20 AM, Kevin Klues 
> > wrote:
> > > >
> > > > > I've created an Epic to track this:
> > > > > https://issues.apache.org/jira/browse/MESOS-5676
> > > > >
> > > > > There have been efforts on this that have failed in the past (e.g.
> > > > > https://github.com/mesosphere/mesos-cli)
> > > > >
> > > > > I'm curious what people's thoughts are in terms of keeping the CLI
> > > > > integrated into mesos itself vs. maintaining it outside in a
> separate
> > > > > repo. There are advantages / disadvantages to both.  The primary
> > > > > advantage of keeping it in is (in theory) it can keep better pace
> > with
> > > > > Mesos itself and will be fixed if any new / changed features break
> > its
> > > > > unit tests.  The advantage of keeping it out is that it evolve more
> > > > > easily and is not subject to the limitations of the Mesos build
> > > > > system.
> > > > >
> > > > > On Mon, Jun 20, 2016 at 11:05 AM, Haris Choudhary
> > > > >  wrote:
> > > > > > Hey All,
> > > > > >
> > > > > > We are finalizing a Design Doc for the redesign and hope to send
> it
> > > out
> > > > > in
> > > > > > the next few days.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Jun 20, 2016 at 9:47 AM, Zhitao Li <
> zhitaoli...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > >> +1
> > > > > >>
> > > > > >> Very interested in participating on design/feature discussions
> and
> > > > > making
> > > >

Re: Mesos CLI

[Joris Van Remoortere]

>
> Is there strong opposition to python for any
> reason other than code parity?

For me the history of a shortage of python maintainers / committers is the
concern.
As this tool gains popularity I would worry that new feature requests would
be slow to get merged based on when the python experts have time.

This is a trade-off we should consider. The best option may still be for it
to be in Python, this is why I'm asking if there are particular things that
our helper libraries don't provide which you are leveraging in python.

—
*Joris Van Remoortere*
Mesosphere

On Thu, Jun 23, 2016 at 12:12 AM, Kevin Klues  wrote:

> Here is a link to the design doc:
>
> https://docs.google.com/document/d/1r6Iv4Efu8v8IBrcUTjgYkvZ32WVscgYqrD07OyIglsA/edit?ts=57573bba#
>
> It sounds like people would like to see the CLI written in C++.
> However, the reference implementation we have is written completely in
> python. Please see the section on "Reference Implementation" in the
> design doc for a discussion of why we chose this.
>
> Do people consider a C++ implementation of the CLI a requirement or
> simply a desirable. Is there strong opposition to python for any
> reason other than code parity?
>
> On Wed, Jun 22, 2016 at 1:01 PM, Joris Van Remoortere
>  wrote:
> > +1 for maintaining in repo.
> > +1 for C++. Are the tools in libprocess not sufficient to make this easy
> to
> > write? What is missing that would make it easier to write in C++?
> >
> > —
> > *Joris Van Remoortere*
> > Mesosphere
> >
> > On Wed, Jun 22, 2016 at 2:24 AM, Zhou Z Xing 
> wrote:
> >
> >> +1 for keeping it in Mesos repo and rewrite it in C++.
> >>
> >> Also, do we need a CLI work group on this?
> >>
> >> Thanks & Best Wishes,
> >>
> >> Tom Xing(邢舟)
> >> Emerging Technology Institute, IBM China Software Development Lab
> >> --
> >> IBM China Software Development Laboratory (CSDL)
> >> Notes ID:Zhou Z Xing/China/IBM
> >> Phone :86-10-82450442
> >> e-Mail :xingz...@cn.ibm.com
> >> Address :Building No.28, ZhongGuanCun Software Park, No.8 Dong Bei Wang
> >> West Road, Haidian District, Beijing, P.R.China 100193
> >> 地址 :中国北京市海淀区东北旺西路8号 中关村软件园28号楼 100193
> >>
> >>
> >> [image: Inactive hide details for Benjamin Mahler ---2016-06-22 上午
> >> 06:12:40---+1 for keeping it in the repo. We can establish
> maint]Benjamin
> >> Mahler ---2016-06-22 上午 06:12:40---+1 for keeping it in the repo. We can
> >> establish maintainers for the CLI to ensure that it can mainta
> >>
> >> From: Benjamin Mahler 
> >> To: dev , jfarr...@apache.org
> >> Date: 2016-06-22 上午 06:12
> >> Subject: Re: Mesos CLI
> >> --
> >>
> >>
> >>
> >> +1 for keeping it in the repo.
> >>
> >> We can establish maintainers for the CLI to ensure that it can maintain
> a
> >> reasonable update cadence. Note that we haven't done this well for the
> >> webui and CLI, so we need to make sure we do it better this time around.
> >>
> >> If the architecture allows for easy integration of custom commands
> (written
> >> in any language) then it should enable users to create their own helpful
> >> CLI commands that we can eventually pull in and support in a first class
> >> way.
> >>
> >> On Tue, Jun 21, 2016 at 1:26 PM, Jake Farrell 
> wrote:
> >>
> >> > +1 to in repo
> >> >
> >> > C++ would be nice to maintain language parity, GO would be a great
> choice
> >> > also
> >> >
> >> > -Jake
> >> >
> >> > On Tue, Jun 21, 2016 at 3:15 PM, Vinod Kone 
> wrote:
> >> >
> >> > > +1 for keeping it in repo.
> >> > >
> >> > > Would be nice if the CLI can be written entirely in C++ though, to
> >> avoid
> >> > > supporting more languages.
> >> > >
> >> > > On Tue, Jun 21, 2016 at 12:12 PM, Jie Yu 
> wrote:
> >> > >
> >> > > > I personally prefer it being part of the Mesos repo so that when
> >> people
> >> > > > install our package, they'll get the command line tools as well.
> That
> >> > > also
> >> > > > avoids the potential version mismatch between Mesos and CLI (as
> you
> >> > > > mentioned).
> >> > > >
> >> > > > What does others think?
> >&g