Re: Discussion about upgrading 3rdparty libraries

[zhiwei]

If a user upgrades a dependency, he only need to update the
3rdparty/libray_name directory with his updates, no need to re-run the
bootstrap script.

After the user upgrade a dependency, he need to re-run `make` command, and
make command can catch up this new change.


On Tue, Mar 8, 2016 at 1:58 PM, Alex Clemmer 
wrote:

> It's not clear to me what happens when a user upgrades a dependency.
> It seems like they'd need to re-run the `bootstrap` script, is that
> correct?
>
> On Mon, Mar 7, 2016 at 5:23 PM, zhiwei  wrote:
> > I don't think store tarballs in Mesos git repository is a good idea, even
> > in another 3rdparty repo.
> >
> > *So my opinion is:*
> >
> > Add a 3rdparty configure file(/support/3rdparty.config), the file format
> > can be:
> >
> > library_name git_repo_url git_tag/branch/commit_id .patch_file
> >
> > zookeeper https://github.com/apache/zookeeper.git release-3.4.8
> > zookeeper.patch
> > leveldb https://github.com/google/leveldb.git v1.4 leveldb.patch
> > ... ... ... ...
> >
> > In bootstrap file:
> >
> > 1. Traverse the support/3rdparty.config
> >
> > 2. If there is already a 3rdparty/library_name directory, skip and
> continue
> > another item.
> >
> > 3. Clone the library code, switch to the defined tag/branch/commit_id,
> > apply the .patch file.
> >
> > 4. Do the rest steps.
> >
> > So if users want to use their own 3rdparty libraries, he can checkout
> code
> > to the /3rdparty/library_name and the bootstrap script will skip this
> > library.
> >
> > And in Mesos repository, we only need to maintain the
> > support/3rdparty.config file and the .patch files.
> >
> >
> > On Tue, Mar 8, 2016 at 2:05 AM, Alex Clemmer <
> clemmer.alexan...@gmail.com>
> > wrote:
> >
> >> So, at this point we have a bunch of different reviews open for
> >> this[1], and I'd like to use this as an opportunity to start nudging
> >> people towards thinking about possibly transitioning to a scheme where
> >> the tarballs that are currently held in the `3rdparty/` directory are
> >> moved to some external place, and retrieved for users out-of-band, as
> >> necessary to build Mesos.
> >>
> >> In particular, doing this (as you all likely know) is very expensive
> >> because git stores a complete copy of the entire tarball, for each
> >> different revision in history, so if you have updated a tarball twice,
> >> you have two complete copies rolling around in the `.git/` folder. It
> >> seems like there are not many benefits for keeping this scheme, other
> >> than the fact that it's pretty easy to implement.
> >>
> >> I'm not sure what it would take to transition the autotools build
> >> system, but just recapping earlier what I said about the CMake build
> >> system: The easiest thing to do (which we've already mostly done) is
> >> to allow people to rope in tarballs from some mirror of the `3rdparty`
> >> github repository[2]. Right now we have facilities that let you host
> >> it either on your local FS or on a remote URL, and we'll download (if
> >> necessary) and untar into the familliar place in the `build/` folder.
> >> Easy! We could even have `bootstrap` clone the repository and make
> >> CMake automatically pull in that repository if it's out of date.
> >>
> >> Thoughts? I recognize that this might be overcomplicating the problem
> >> a bit, but I figured I'd throw the hat in the ring because this has
> >> always kind of bothered me. :)
> >>
> >>
> >> [1] They are:
> >> https://reviews.apache.org/r/44252/
> >> https://reviews.apache.org/r/44382/
> >> https://reviews.apache.org/r/44372/
> >> https://reviews.apache.org/r/44378/
> >> https://reviews.apache.org/r/44376/
> >> https://reviews.apache.org/r/44257/
> >>
> >> [2] https://github.com/3rdparty/mesos-3rdparty
> >>
> >> On Tue, Mar 1, 2016 at 10:48 AM, Alex Clemmer
> >>  wrote:
> >> > It doesn't seem to be the case that these things are mutually
> >> > exclusive -- it is well within our purview to accept only a specific
> >> > range of versions for any particular dependency, and error out if
> >> > someone tries to select a version outside that range. The only thing
> >> > these commits add is more fine-grained control over which of the
> >> > supported versions you are allowed to select.
> >> >
> >> > At this point, there are no such guards, but that is certainly
> >> > something that can be added.
> >> >
> >> > On Tue, Mar 1, 2016 at 10:17 AM, Neil Conway 
> >> wrote:
> >> >> The prospect of downloading dependencies from "rando" locations is
> >> >> concerning to me :)
> >> >>
> >> >> Mesos can easily come to depend on implementation details of a
> >> >> dependency that might change in a minor release. For example, a
> recent
> >> >> change [1] depends on the connection retry logic in the Zk client
> >> >> library in a fairly delicate way. I also wouldn't want users to
> >> >> randomly upgrade to, say, protobuf 2.6.1 without it being thoroughly
> 

Re: [VOTE] Release Apache Mesos 0.27.2 (rc1)

[Kevin Klues]

Sure, that's fine with me. Since the fix for the flaky test was a flaw
in the test itself there shouldn't be any issues.

Thanks for double checking. Especially since my vote was non-binding.

Kevin

On Mon, Mar 7, 2016 at 7:33 PM, Michael Park  wrote:
> Kevin,
>
> Sorry for missing your flaky test patch. It seems like we added the 0.27.2
> to the backports list after you had indicated that 0.24.2, 0.25.1, and
> 0.26.1
> needs to include the patch. We should have asked whether this needs to
> be included in 0.27.2 as well. I think we missed it because there were many
> patches that needed to be included in 0.24.2, 0.25.1 and 0.26.1 but not
> 0.27.2
> since they had made it into 0.27.0 or 0.27.1.
>
> Having said that, I'm inclined to agree with Joris and proceed since it
> doesn't
> have much of an impact in terms of the resulting binary, for example.
>
> Could you confirm or deny whether you're ok with this?
>
> Thanks,
>
> MPark
>
> On 4 March 2016 at 15:51, Joris Van Remoortere
>  wrote:
>>
>> +1 (binding)
>> Greg's upgrade scripts & CI results
>>
>> The missing commit is for a flaky test which doesn't influence the
>> production binaries.
>> Unless we need to cut another RC for a bug, I suggest we move ahead.
>>
>> On Wed, Mar 2, 2016 at 10:36 AM, Jörg Schad  wrote:
>>
>> > Except the missing fix for Mesos-4518, if we consider cutting a rc2
>> > for that, maybe we could include the fix for MESOS-4677 as well (see
>> > failing ROOT_CGROUPS_Pids_and_Tids test below).
>> > +1 (non-binding)
>> >
>> > All the failing tests I encountered seem to be known.
>> >
>> > Centos 7
>> > * LimitedCpuIsolatorTest.ROOT_CGROUPS_Pids_and_Tids  (fixed with
>> > MESOS-4677 for 0.28 )
>> > * LinuxFilesystemIsolatorTest.ROOT_MultipleContainers (open ticket
>> > MESOS-4423)
>> >
>> > Centos 7 - SSL
>> > All green
>> >
>> > Centos 6 (+/- SSL)
>> > * MemoryPressureMesosTest.CGROUPS_ROOT_SlaveRecovery (reopened
>> > MESOS-4047)
>> >
>> > Debian 8 (+/- SSL)
>> > * DockerContainerizerTest.ROOT_DOCKER_Kill (seems the same issue as
>> > MESOS-3937)
>> >
>> > Ubuntu 15 (+/- SSL)
>> > Green
>> >
>> > Ubuntu 14 (+/- SSL)
>> > Green
>> >
>> > Ubuntu 12 (+/- SSL)
>> > Green
>> >
>> > On Tue, Mar 1, 2016 at 10:18 PM, Kevin Klues  wrote:
>> > > -1 (non-binding)
>> > >
>> > > This release
>> > > candidate
>> > > should have included the backport to re
>> > > s
>> > > olv
>> > > e
>> > > MESOS-4518 .
>> > > All of the other release candidates that came out as backports
>> > > recently
>> > > have included this, but somehow this one was overlooked.
>> > >
>> > >
>> > >
>> > >
>> > > On Tue, Mar 1, 2016 at 4:35 PM, Greg Mann  wrote:
>> > >
>> > >> I was able to successfully test a simple upgrade scenario between
>> > >> 0.26.1-rc1 and 0.27.2-rc1 using Niklas's upgrade testing script,
>> > >> which
>> > I've
>> > >> modified slightly and reposted here:
>> > https://reviews.apache.org/r/44229/
>> > >>
>> > >> On Tue, Mar 1, 2016 at 2:22 PM, Kevin Klues 
>> > >> wrote:
>> > >>
>> > >> > The others all seem to have them though:
>> > >> >
>> > >> >
>> > >> >
>> > >>
>> >
>> > https://github.com/apache/mesos/commits/0.26.1-rc1/src/tests/master_tests.cpp
>> > >> >
>> > >> >
>> > >>
>> >
>> > https://github.com/apache/mesos/commits/0.25.1-rc1/src/tests/master_tests.cpp
>> > >> >
>> > >> >
>> > >>
>> >
>> > https://github.com/apache/mesos/commits/0.24.2-rc1/src/tests/master_tests.cpp
>> > >> >
>> > >> > Just not:
>> > >> >
>> > >> >
>> > >>
>> >
>> > https://github.com/apache/mesos/commits/0.27.2-rc1/src/tests/master_tests.cpp
>> > >> >
>> > >> > On Tue, Mar 1, 2016 at 2:17 PM, Kevin Klues 
>> > wrote:
>> > >> > > Looks like this rc is missing this commit:
>> > >> > >
>> > >> > >
>> > >> >
>> > >>
>> >
>> > https://github.com/apache/mesos/commit/d3108d776b6f7121e37176eda686ecc7245be4cd
>> > >> > >
>> > >> > > On Tue, Mar 1, 2016 at 2:08 PM, Joris Van Remoortere
>> > >> > >  wrote:
>> > >> > >> @Michael Browning:
>> > >> > >>>
>> > >> > >>> MasterTest.MaxCompletedTasksPerFrameworkFlag [flaky, tracked in
>> > >> > >>> MESOS-4518]
>> > >> > >>
>> > >> > >> This is supposed to be fixed in this release. It is concerning
>> > >> > >> that
>> > >> this
>> > >> > >> came up.
>> > >> > >> Can you verify this and provide logs to Kevin Klues?
>> > >> > >>
>> > >> > >>
>> > >> > >> —
>> > >> > >> Joris Van Remoortere
>> > >> > >> Mesosphere
>> > >> > >>
>> > >> > >> On Tue, Mar 1, 2016 at 2:00 PM, Michael Browning <
>> > >> > invitapri...@gmail.com>
>> > >> > >> wrote:
>> > >> > >>>
>> > >> > >>> +1 (non-binding)
>> > >> > >>>
>> > >> > >>> Fedora 23: `make check` non-root OK
>> > >> > >>> OS X: `make check` non-root OK
>> > >> > >>> Ubuntu 14.04: `make check` non-root, three failures:
>> > >> > >>> ContainerLoggerTest.DefaultToSandbox [flaky, 

Re: Extend PerfEventIsolator to support all possible Perf Events.

[Benjamin Mahler]

Why the dynamic map rather than following the existing convention in
PerfStatistics? Seems inconsistent.

On Monday, February 22, 2016, Bartłomiej Płotka  wrote:

> Hi,
>
> We want to introduce the support in PerfEventIsolator for all possible*
> perf events*.
> I'd love to have some feedback here! (:
>
> Please see JIRA issue for details:
> https://issues.apache.org/jira/browse/MESOS-4595
> Patch: https://reviews.apache.org/r/43358/
>
> Kind Regards,
> Bartlomiej Plotka (bplotka)
>

Re: Enable compiler optimization by default?

[Benjamin Mahler]

I won't -1 but please see below before proceeding.

This was the original behavior and this will bring it full circle :)
See the thread here, and the additional links:

http://markmail.org/thread/ylqbxezgz6rtb7v2
https://reviews.apache.org/r/26426/#comment_rc96919-56878
https://issues.apache.org/jira/browse/MESOS-1985

It would be great to include an understanding of this context along with
the proposal.

On Thu, Feb 18, 2016 at 10:59 AM, Joris Van Remoortere <
joris.van.remoort...@gmail.com> wrote:

> +1 (binding)
> with the condition that we fix the flag mixing problem between CXXFLAGS and
> --enable-optimize --enable-debug, even if it is to disallow it for now.
> I want to avoid surprising behavior with implicit flags such as optimize:
> passing irrelevant CXXFLAGS magically turns it into a debug build.
>
> On Thu, Feb 18, 2016 at 1:27 PM, Neil Conway 
> wrote:
>
> > Great! I created https://issues.apache.org/jira/browse/MESOS-4709 to
> > track this issue.
> >
> > Neil
> >
> > On Thu, Feb 18, 2016 at 12:43 AM, Jan Schlicht 
> wrote:
> > > +1
> > >
> > > On Thu, Feb 18, 2016 at 2:34 AM, Klaus Ma 
> > wrote:
> > >
> > >> +1;
> > >>
> > >> So our CI will also update to use optimisation flags, right?  We need
> to
> > >> highlight this in upgrade document to our user; I used to meet so
> > strange
> > >> behaviour after changing -O level.
> > >>
> > >> On Thu, Feb 18, 2016 at 8:51 AM James DeFelice <
> > james.defel...@gmail.com>
> > >> wrote:
> > >>
> > >> > +1
> > >> > On Feb 17, 2016 7:24 PM, "Neil Conway" 
> wrote:
> > >> >
> > >> > > Hi folks,
> > >> > >
> > >> > > At present, Mesos defaults to compiling with "-O0"; to enable
> > compiler
> > >> > > optimizations, the user needs to specify "--enable-optimize".
> > >> > >
> > >> > > I'd like to propose we change the default, for a few reasons:
> > >> > >
> > >> > > (1) The autoconf default for CFLAGS/CXXFLAGS is "-O2 -g".
> > Anecdotally,
> > >> > > I think most software packages compile with a reasonable level of
> > >> > > optimizations enabled by default.
> > >> > >
> > >> > > (2) I think we should make the default configure flags appropriate
> > for
> > >> > > end-users (rather than Mesos developers): developers will be
> > familiar
> > >> > > enough with Mesos to tune the configure flags according to their
> own
> > >> > > preferences.
> > >> > >
> > >> > > (3) The performance consequences of not enabling compiler
> > >> > > optimizations can be pretty severe: 5x in a benchmark I just ran,
> > and
> > >> > > we've seen between 2x and 30x (!) performance differences for some
> > >> > > real-world workloads.
> > >> > >
> > >> > > Neil
> > >> > >
> > >> >
> > >> --
> > >>
> > >> Regards,
> > >> 
> > >> Da (Klaus), Ma (马达), PMP® | Advisory Software Engineer
> > >> IBM Platform Development & Support, STG, IBM GCG
> > >> +86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me
> > >>
> > >
> > >
> > >
> > > --
> > > *Jan Schlicht*
> > > Distributed Systems Engineer, Mesosphere
> >
>

Re: State of registrar

[Benjamin Mahler]

Apologies for the long delay.

I wouldn't call it experimental (that comment is stale), you should feel
free to turn on strictness. Strictness enforces that agents that were
removed by an old master cannot re-join with a new master. This preserves
the steady state behavior: if the master removes an agent, it does not
allow it to return. Ideally, the flag is removed and strictness is the
default, but we didn't feel comfortable removing it until we had state
backup support in the master. Turning off strictness allows for an escape
hatch if state is lost. Now that we are persisting more information than
just the list of agents, this escape hatch doesn't restore the other state
(like maintenance schedules, quota information, etc).

As for why it's not on by default today, we found that many frameworks,
like Aurora and Marathon, are capable of handling a removed agent
re-surfacing in the cluster and so it wasn't critical to turn this on.
Also, we also realized that we need to re-work the partition handling in
Mesos in order to give frameworks the control over how to react to an
unreachable agent.

Does that clarify things?

On Mon, Feb 1, 2016 at 11:04 AM, Zhitao Li  wrote:

> Hi,
>
> I've been reading related documentation on Mesos website and trying to
> understand the current status of registrar.
>
> I noticed that we still consider "--registrar_strict" as experimental, but
> I can't find the back story of what's needed to finish the project or the
> JIRA so tracking that.
>
> Also, does anyone have recommendations on whether we should turn this flag
> on, and what benefits cluster operator would get?
>
> Thanks.

Re: Mesos Maintenance Enhancement (post-MVP)

[Klaus Ma]

Done.


Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform OpenSource Technology, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

On Tue, Mar 8, 2016 at 10:17 AM, Benjamin Mahler  wrote:

> Can you send an invite to bmah...@apache.org?
>
> On Sun, Mar 6, 2016 at 6:40 PM, Klaus Ma  wrote:
>
> > Hi team,
> >
> > I started the following working group to enhance Mesos Maintenance for
> some
> > cases. Please join the working group if you're interesting in it :).
> >
> > *Working Group*:
> > https://groups.google.com/d/forum/mesos-maintenance-working-group
> >
> > If any comments, please let me know.
> >
> > 
> > Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
> > Platform OpenSource Technology, STG, IBM GCG
> > +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
> >
>

Re: Mesos Maintenance Enhancement (post-MVP)

[Benjamin Mahler]

Can you send an invite to bmah...@apache.org?

On Sun, Mar 6, 2016 at 6:40 PM, Klaus Ma  wrote:

> Hi team,
>
> I started the following working group to enhance Mesos Maintenance for some
> cases. Please join the working group if you're interesting in it :).
>
> *Working Group*:
> https://groups.google.com/d/forum/mesos-maintenance-working-group
>
> If any comments, please let me know.
>
> 
> Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
> Platform OpenSource Technology, STG, IBM GCG
> +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
>

Re: Looking for shepherd for MESOS-4705

[Benjamin Mahler]

Thanks for looking into this bug, I left some comments.

On Sun, Mar 6, 2016 at 11:28 PM, Du, Fan  wrote:

> Hi
>
> Could I have a shepherd for MESOS-4705<
> https://issues.apache.org/jira/browse/MESOS-4705> ?
> A trivial bug fix with RR https://reviews.apache.org/r/44379/
>
> Thanks a lot.
>
>
>

Re: Discussion about upgrading 3rdparty libraries

[zhiwei]

I don't think store tarballs in Mesos git repository is a good idea, even
in another 3rdparty repo.

*So my opinion is:*

Add a 3rdparty configure file(/support/3rdparty.config), the file format
can be:

library_name git_repo_url git_tag/branch/commit_id .patch_file

zookeeper https://github.com/apache/zookeeper.git release-3.4.8
zookeeper.patch
leveldb https://github.com/google/leveldb.git v1.4 leveldb.patch
... ... ... ...

In bootstrap file:

1. Traverse the support/3rdparty.config

2. If there is already a 3rdparty/library_name directory, skip and continue
another item.

3. Clone the library code, switch to the defined tag/branch/commit_id,
apply the .patch file.

4. Do the rest steps.

So if users want to use their own 3rdparty libraries, he can checkout code
to the /3rdparty/library_name and the bootstrap script will skip this
library.

And in Mesos repository, we only need to maintain the
support/3rdparty.config file and the .patch files.


On Tue, Mar 8, 2016 at 2:05 AM, Alex Clemmer 
wrote:

> So, at this point we have a bunch of different reviews open for
> this[1], and I'd like to use this as an opportunity to start nudging
> people towards thinking about possibly transitioning to a scheme where
> the tarballs that are currently held in the `3rdparty/` directory are
> moved to some external place, and retrieved for users out-of-band, as
> necessary to build Mesos.
>
> In particular, doing this (as you all likely know) is very expensive
> because git stores a complete copy of the entire tarball, for each
> different revision in history, so if you have updated a tarball twice,
> you have two complete copies rolling around in the `.git/` folder. It
> seems like there are not many benefits for keeping this scheme, other
> than the fact that it's pretty easy to implement.
>
> I'm not sure what it would take to transition the autotools build
> system, but just recapping earlier what I said about the CMake build
> system: The easiest thing to do (which we've already mostly done) is
> to allow people to rope in tarballs from some mirror of the `3rdparty`
> github repository[2]. Right now we have facilities that let you host
> it either on your local FS or on a remote URL, and we'll download (if
> necessary) and untar into the familliar place in the `build/` folder.
> Easy! We could even have `bootstrap` clone the repository and make
> CMake automatically pull in that repository if it's out of date.
>
> Thoughts? I recognize that this might be overcomplicating the problem
> a bit, but I figured I'd throw the hat in the ring because this has
> always kind of bothered me. :)
>
>
> [1] They are:
> https://reviews.apache.org/r/44252/
> https://reviews.apache.org/r/44382/
> https://reviews.apache.org/r/44372/
> https://reviews.apache.org/r/44378/
> https://reviews.apache.org/r/44376/
> https://reviews.apache.org/r/44257/
>
> [2] https://github.com/3rdparty/mesos-3rdparty
>
> On Tue, Mar 1, 2016 at 10:48 AM, Alex Clemmer
>  wrote:
> > It doesn't seem to be the case that these things are mutually
> > exclusive -- it is well within our purview to accept only a specific
> > range of versions for any particular dependency, and error out if
> > someone tries to select a version outside that range. The only thing
> > these commits add is more fine-grained control over which of the
> > supported versions you are allowed to select.
> >
> > At this point, there are no such guards, but that is certainly
> > something that can be added.
> >
> > On Tue, Mar 1, 2016 at 10:17 AM, Neil Conway 
> wrote:
> >> The prospect of downloading dependencies from "rando" locations is
> >> concerning to me :)
> >>
> >> Mesos can easily come to depend on implementation details of a
> >> dependency that might change in a minor release. For example, a recent
> >> change [1] depends on the connection retry logic in the Zk client
> >> library in a fairly delicate way. I also wouldn't want users to
> >> randomly upgrade to, say, protobuf 2.6.1 without it being thoroughly
> >> tested. Increasing the support matrix of different users on different
> >> platforms running arbitrarily different versions of third-party
> >> dependencies doesn't seem like a net improvement to me.
> >>
> >> My two cents: if Windows requires additional dependencies that we
> >> aren't currently vendoring, I would personally opt for (a) vendoring
> >> those additional dependencies (b) ensuring that the vendored versions
> >> we ship are modern enough to support all the platforms we care about.
> >> Are there important use-cases that aren't supported by this scheme?
> >>
> >> Neil
> >>
> >> [1]
> https://github.com/apache/mesos/commit/c2d496ed430eaf7daee3e57edefa39c25af2aa43
> >>
> >> On Tue, Mar 1, 2016 at 10:00 AM, Alex Clemmer
> >>  wrote:
> >>> I guess a tl;dr might be in order.
> >>>
> >>> Basically: the CMake build system already supports roping in 

Re: [VOTE] Release Apache Mesos 0.28.0 (rc1)

[Jie Yu]

I'd like to fix https://issues.apache.org/jira/browse/MESOS-4888 as well if
you guys plan to cut another RC

On Mon, Mar 7, 2016 at 10:16 AM, Daniel Osborne <
daniel.osbo...@metaswitch.com> wrote:

> -1
>
> If it doesn’t cause too much pain, I'm hoping we can squeeze a relatively
> small patch which restores Mesos' ability to extract Docker assigned IPs.
> This has been broken with Docker 1.10's release over  a month ago, and
> prevents service discovery and DNS from working.
>
> Mesos-4370: https://issues.apache.org/jira/browse/MESOS-4370
> RB# 43093: https://reviews.apache.org/r/43093/
>
> I've built 0.28.0-rc1 with this patch and can confirm that it fixes it as
> expected.
>
> Apologies for not bringing this to attention earlier.
>
> Thanks all,
> Dan
>
> -Original Message-
> From: Vinod Kone [mailto:vinodk...@apache.org]
> Sent: Thursday, March 3, 2016 5:44 PM
> To: dev ; user 
> Subject: [VOTE] Release Apache Mesos 0.28.0 (rc1)
>
> Hi all,
>
>
> Please vote on releasing the following candidate as Apache Mesos 0.28.0.
>
>
> 0.28.0 includes the following:
>
>
> 
>
>   * [MESOS-4343] - A new cgroups isolator for enabling the net_cls
> subsystem in
>
> Linux. The cgroups/net_cls isolator allows operators to provide network
>
>
> performance isolation and network segmentation for containers within a
> Mesos
>
> cluster. To enable the cgroups/net_cls isolator, append
> `cgroups/net_cls` to
>
> the `--isolation` flag when starting the slave. Please refer to
>
>
> docs/mesos-containerizer.md for more details.
>
>
>
>
>
>   * [MESOS-4687] - The implementation of scalar resource values (e.g., "2.5
>
>
> CPUs") has changed. Mesos now reliably supports resources with up to
> three
>
> decimal digits of precision (e.g., "2.501 CPUs"); resources with more
> than
>
> three decimal digits of precision will be rounded. Internally,
> resource math
>
> is now done using a fixed-point format that supports three decimal
> digits of
>
> precision, and then converted to/from floating point for input and
> output,
>
> respectively. Frameworks that do their own resource math and manipulate
>
>
> fractional resources may observe differences in roundoff error and
> numerical
>
> precision.
>
>
>
>
>
>   * [MESOS-4479] - Reserved resources can now optionally include "labels".
>
>
> Labels are a set of key-value pairs that can be used to associate
> metadata
>
> with a reserved resource. For example, frameworks can use this feature
> to
>
> distinguish between two reservations for the same role at the same
> agent
>
> that are intended for different purposes.
>
>
>
>
>
>   * [MESOS-2840] - **Experimental** support for container images in Mesos
>
>
> containerizer (a.k.a. Unified Containerizer). This allows frameworks to
>
>
> launch Docker/Appc containers using Mesos containerizer without
> relying on
>
> docker daemon (engine) or rkt. The isolation of the containers is done
> using
>
> isolators. Please refer to docs/container-image.md for currently
> supported
>
> features and limitations.
>
>
>
>
>
>   * [MESOS-4793] - **Experimental** support for v1 Executor HTTP API. This
>
>
> allows executors to send HTTP requests to the /api/v1/executor agent
>
>
> endpoint without the need for an executor driver. Please refer to
>
>
> docs/executor-http-api.md for more details.
>
>
>
>
>
> Additional API Changes:
>
>
>   * [MESOS-4066] - Agent should not return partial state when a request is
> made to /state endpoint during recovery.
>
>   * [MESOS-4547] - Introduce TASK_KILLING state.
>
>
>   * [MESOS-4712] - Remove 'force' field from the Subscribe Call in v1
> Scheduler API.
>
>   * [MESOS-4591] - Change the object of ReserveResources and CreateVolume
> ACLs to `roles`.
>
>   * [MESOS-4712] - Remove 'force' field from the Subscribe Call in v1
> Scheduler API.
>
>   * [MESOS-4591] - Change the object of ReserveResources and CreateVolume
> ACLs to `roles`.
>
>   * [MESOS-3583] - Add stream IDs for HTTP schedulers.
>
>
> The CHANGELOG for the release is available at:
>
>
> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=0.28.0-rc1
>
>
> 
>
>
> The candidate for Mesos 0.28.0 release is available at:
>
> https://dist.apache.org/repos/dist/dev/mesos/0.28.0-rc1/mesos-0.28.0.tar.gz
>
>
> The tag to be voted on is 0.28.0-rc1:
>
> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=0.28.0-rc1
>
>
> The MD5 checksum of the tarball can be found at:
>
>
> https://dist.apache.org/repos/dist/dev/mesos/0.28.0-rc1/mesos-0.28.0.tar.gz.md5
>
>
> The signature of the tarball can be found at:
>
>
> https://dist.apache.org/repos/dist/dev/mesos/0.28.0-rc1/mesos-0.28.0.tar.gz.asc
>
>
> The PGP key used to 

Re: [VOTE] Release Apache Mesos 0.28.0 (rc1)

[Greg Mann]

+1 (non-binding)

`make check` and `sudo make check` on OSX 10.10.5 and CentOS 7.1, with
libevent and SSL enabled.

All tests pass on CentOS 7.1

When running the sudo tests on OSX, three failures were observed:
ExamplesTest.JavaFramework is tracked by
https://issues.apache.org/jira/browse/MESOS-3582
FetcherCacheTest.LocalUncachedExtract and FetcherCacheHttpTest.HttpMixed:
these seem to be failing due to a lack of filesystem permissions. Since
this has only been observed on OSX, it doesn't look like a blocker to me. I
didn't find a ticket covering this specific failure, so I created one here:
https://issues.apache.org/jira/browse/MESOS-4890

On Mon, Mar 7, 2016 at 12:57 PM, Vinod Kone  wrote:

> Looks like there is a bunch of discussion around MESOS-4370 and RB# 43093.
> Kapil, as the shepherd, what is your take on the priority and readiness of
> this?
>
> On Mon, Mar 7, 2016 at 1:16 PM, Daniel Osborne <
> daniel.osbo...@metaswitch.com> wrote:
>
> > -1
> >
> > If it doesn’t cause too much pain, I'm hoping we can squeeze a relatively
> > small patch which restores Mesos' ability to extract Docker assigned IPs.
> > This has been broken with Docker 1.10's release over  a month ago, and
> > prevents service discovery and DNS from working.
> >
> > Mesos-4370: https://issues.apache.org/jira/browse/MESOS-4370
> > RB# 43093: https://reviews.apache.org/r/43093/
> >
> > I've built 0.28.0-rc1 with this patch and can confirm that it fixes it as
> > expected.
> >
> > Apologies for not bringing this to attention earlier.
> >
> > Thanks all,
> > Dan
> >
> > -Original Message-
> > From: Vinod Kone [mailto:vinodk...@apache.org]
> > Sent: Thursday, March 3, 2016 5:44 PM
> > To: dev ; user 
> > Subject: [VOTE] Release Apache Mesos 0.28.0 (rc1)
> >
> > Hi all,
> >
> >
> > Please vote on releasing the following candidate as Apache Mesos 0.28.0.
> >
> >
> > 0.28.0 includes the following:
> >
> >
> >
> 
> >
> >   * [MESOS-4343] - A new cgroups isolator for enabling the net_cls
> > subsystem in
> >
> > Linux. The cgroups/net_cls isolator allows operators to provide
> network
> >
> >
> > performance isolation and network segmentation for containers within
> a
> > Mesos
> >
> > cluster. To enable the cgroups/net_cls isolator, append
> > `cgroups/net_cls` to
> >
> > the `--isolation` flag when starting the slave. Please refer to
> >
> >
> > docs/mesos-containerizer.md for more details.
> >
> >
> >
> >
> >
> >   * [MESOS-4687] - The implementation of scalar resource values (e.g.,
> "2.5
> >
> >
> > CPUs") has changed. Mesos now reliably supports resources with up to
> > three
> >
> > decimal digits of precision (e.g., "2.501 CPUs"); resources with more
> > than
> >
> > three decimal digits of precision will be rounded. Internally,
> > resource math
> >
> > is now done using a fixed-point format that supports three decimal
> > digits of
> >
> > precision, and then converted to/from floating point for input and
> > output,
> >
> > respectively. Frameworks that do their own resource math and
> manipulate
> >
> >
> > fractional resources may observe differences in roundoff error and
> > numerical
> >
> > precision.
> >
> >
> >
> >
> >
> >   * [MESOS-4479] - Reserved resources can now optionally include
> "labels".
> >
> >
> > Labels are a set of key-value pairs that can be used to associate
> > metadata
> >
> > with a reserved resource. For example, frameworks can use this
> feature
> > to
> >
> > distinguish between two reservations for the same role at the same
> > agent
> >
> > that are intended for different purposes.
> >
> >
> >
> >
> >
> >   * [MESOS-2840] - **Experimental** support for container images in Mesos
> >
> >
> > containerizer (a.k.a. Unified Containerizer). This allows frameworks
> to
> >
> >
> > launch Docker/Appc containers using Mesos containerizer without
> > relying on
> >
> > docker daemon (engine) or rkt. The isolation of the containers is
> done
> > using
> >
> > isolators. Please refer to docs/container-image.md for currently
> > supported
> >
> > features and limitations.
> >
> >
> >
> >
> >
> >   * [MESOS-4793] - **Experimental** support for v1 Executor HTTP API.
> This
> >
> >
> > allows executors to send HTTP requests to the /api/v1/executor agent
> >
> >
> > endpoint without the need for an executor driver. Please refer to
> >
> >
> > docs/executor-http-api.md for more details.
> >
> >
> >
> >
> >
> > Additional API Changes:
> >
> >
> >   * [MESOS-4066] - Agent should not return partial state when a request
> is
> > made to /state endpoint during recovery.
> >
> >   * [MESOS-4547] - Introduce TASK_KILLING state.
> >
> >
> >   * [MESOS-4712] - Remove 'force' field from the Subscribe Call in v1
> > Scheduler API.
> >
> >   * [MESOS-4591] - Change the object 

Re: [VOTE] Release Apache Mesos 0.28.0 (rc1)

[Vinod Kone]

Looks like there is a bunch of discussion around MESOS-4370 and RB# 43093.
Kapil, as the shepherd, what is your take on the priority and readiness of
this?

On Mon, Mar 7, 2016 at 1:16 PM, Daniel Osborne <
daniel.osbo...@metaswitch.com> wrote:

> -1
>
> If it doesn’t cause too much pain, I'm hoping we can squeeze a relatively
> small patch which restores Mesos' ability to extract Docker assigned IPs.
> This has been broken with Docker 1.10's release over  a month ago, and
> prevents service discovery and DNS from working.
>
> Mesos-4370: https://issues.apache.org/jira/browse/MESOS-4370
> RB# 43093: https://reviews.apache.org/r/43093/
>
> I've built 0.28.0-rc1 with this patch and can confirm that it fixes it as
> expected.
>
> Apologies for not bringing this to attention earlier.
>
> Thanks all,
> Dan
>
> -Original Message-
> From: Vinod Kone [mailto:vinodk...@apache.org]
> Sent: Thursday, March 3, 2016 5:44 PM
> To: dev ; user 
> Subject: [VOTE] Release Apache Mesos 0.28.0 (rc1)
>
> Hi all,
>
>
> Please vote on releasing the following candidate as Apache Mesos 0.28.0.
>
>
> 0.28.0 includes the following:
>
>
> 
>
>   * [MESOS-4343] - A new cgroups isolator for enabling the net_cls
> subsystem in
>
> Linux. The cgroups/net_cls isolator allows operators to provide network
>
>
> performance isolation and network segmentation for containers within a
> Mesos
>
> cluster. To enable the cgroups/net_cls isolator, append
> `cgroups/net_cls` to
>
> the `--isolation` flag when starting the slave. Please refer to
>
>
> docs/mesos-containerizer.md for more details.
>
>
>
>
>
>   * [MESOS-4687] - The implementation of scalar resource values (e.g., "2.5
>
>
> CPUs") has changed. Mesos now reliably supports resources with up to
> three
>
> decimal digits of precision (e.g., "2.501 CPUs"); resources with more
> than
>
> three decimal digits of precision will be rounded. Internally,
> resource math
>
> is now done using a fixed-point format that supports three decimal
> digits of
>
> precision, and then converted to/from floating point for input and
> output,
>
> respectively. Frameworks that do their own resource math and manipulate
>
>
> fractional resources may observe differences in roundoff error and
> numerical
>
> precision.
>
>
>
>
>
>   * [MESOS-4479] - Reserved resources can now optionally include "labels".
>
>
> Labels are a set of key-value pairs that can be used to associate
> metadata
>
> with a reserved resource. For example, frameworks can use this feature
> to
>
> distinguish between two reservations for the same role at the same
> agent
>
> that are intended for different purposes.
>
>
>
>
>
>   * [MESOS-2840] - **Experimental** support for container images in Mesos
>
>
> containerizer (a.k.a. Unified Containerizer). This allows frameworks to
>
>
> launch Docker/Appc containers using Mesos containerizer without
> relying on
>
> docker daemon (engine) or rkt. The isolation of the containers is done
> using
>
> isolators. Please refer to docs/container-image.md for currently
> supported
>
> features and limitations.
>
>
>
>
>
>   * [MESOS-4793] - **Experimental** support for v1 Executor HTTP API. This
>
>
> allows executors to send HTTP requests to the /api/v1/executor agent
>
>
> endpoint without the need for an executor driver. Please refer to
>
>
> docs/executor-http-api.md for more details.
>
>
>
>
>
> Additional API Changes:
>
>
>   * [MESOS-4066] - Agent should not return partial state when a request is
> made to /state endpoint during recovery.
>
>   * [MESOS-4547] - Introduce TASK_KILLING state.
>
>
>   * [MESOS-4712] - Remove 'force' field from the Subscribe Call in v1
> Scheduler API.
>
>   * [MESOS-4591] - Change the object of ReserveResources and CreateVolume
> ACLs to `roles`.
>
>   * [MESOS-4712] - Remove 'force' field from the Subscribe Call in v1
> Scheduler API.
>
>   * [MESOS-4591] - Change the object of ReserveResources and CreateVolume
> ACLs to `roles`.
>
>   * [MESOS-3583] - Add stream IDs for HTTP schedulers.
>
>
> The CHANGELOG for the release is available at:
>
>
> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=0.28.0-rc1
>
>
> 
>
>
> The candidate for Mesos 0.28.0 release is available at:
>
> https://dist.apache.org/repos/dist/dev/mesos/0.28.0-rc1/mesos-0.28.0.tar.gz
>
>
> The tag to be voted on is 0.28.0-rc1:
>
> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=0.28.0-rc1
>
>
> The MD5 checksum of the tarball can be found at:
>
>
> https://dist.apache.org/repos/dist/dev/mesos/0.28.0-rc1/mesos-0.28.0.tar.gz.md5
>
>
> The signature of the tarball can be found at:
>
>
> 

RE: [VOTE] Release Apache Mesos 0.28.0 (rc1)

[Daniel Osborne]

-1

If it doesn’t cause too much pain, I'm hoping we can squeeze a relatively small 
patch which restores Mesos' ability to extract Docker assigned IPs. This has 
been broken with Docker 1.10's release over  a month ago, and prevents service 
discovery and DNS from working.

Mesos-4370: https://issues.apache.org/jira/browse/MESOS-4370
RB# 43093: https://reviews.apache.org/r/43093/

I've built 0.28.0-rc1 with this patch and can confirm that it fixes it as 
expected.

Apologies for not bringing this to attention earlier.

Thanks all,
Dan

-Original Message-
From: Vinod Kone [mailto:vinodk...@apache.org] 
Sent: Thursday, March 3, 2016 5:44 PM
To: dev ; user 
Subject: [VOTE] Release Apache Mesos 0.28.0 (rc1)

Hi all,


Please vote on releasing the following candidate as Apache Mesos 0.28.0.


0.28.0 includes the following:



  * [MESOS-4343] - A new cgroups isolator for enabling the net_cls subsystem in

Linux. The cgroups/net_cls isolator allows operators to provide network


performance isolation and network segmentation for containers within a Mesos

cluster. To enable the cgroups/net_cls isolator, append `cgroups/net_cls` to

the `--isolation` flag when starting the slave. Please refer to


docs/mesos-containerizer.md for more details.





  * [MESOS-4687] - The implementation of scalar resource values (e.g., "2.5


CPUs") has changed. Mesos now reliably supports resources with up to three

decimal digits of precision (e.g., "2.501 CPUs"); resources with more than

three decimal digits of precision will be rounded. Internally, resource math

is now done using a fixed-point format that supports three decimal digits of

precision, and then converted to/from floating point for input and output,

respectively. Frameworks that do their own resource math and manipulate


fractional resources may observe differences in roundoff error and numerical

precision.





  * [MESOS-4479] - Reserved resources can now optionally include "labels".


Labels are a set of key-value pairs that can be used to associate metadata

with a reserved resource. For example, frameworks can use this feature to

distinguish between two reservations for the same role at the same agent

that are intended for different purposes.





  * [MESOS-2840] - **Experimental** support for container images in Mesos


containerizer (a.k.a. Unified Containerizer). This allows frameworks to


launch Docker/Appc containers using Mesos containerizer without relying on

docker daemon (engine) or rkt. The isolation of the containers is done using

isolators. Please refer to docs/container-image.md for currently supported

features and limitations.





  * [MESOS-4793] - **Experimental** support for v1 Executor HTTP API. This


allows executors to send HTTP requests to the /api/v1/executor agent


endpoint without the need for an executor driver. Please refer to


docs/executor-http-api.md for more details.





Additional API Changes:


  * [MESOS-4066] - Agent should not return partial state when a request is made 
to /state endpoint during recovery.

  * [MESOS-4547] - Introduce TASK_KILLING state.


  * [MESOS-4712] - Remove 'force' field from the Subscribe Call in v1 Scheduler 
API.

  * [MESOS-4591] - Change the object of ReserveResources and CreateVolume ACLs 
to `roles`.

  * [MESOS-4712] - Remove 'force' field from the Subscribe Call in v1 Scheduler 
API.

  * [MESOS-4591] - Change the object of ReserveResources and CreateVolume ACLs 
to `roles`.

  * [MESOS-3583] - Add stream IDs for HTTP schedulers.


The CHANGELOG for the release is available at:

https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=0.28.0-rc1




The candidate for Mesos 0.28.0 release is available at:

https://dist.apache.org/repos/dist/dev/mesos/0.28.0-rc1/mesos-0.28.0.tar.gz


The tag to be voted on is 0.28.0-rc1:

https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=0.28.0-rc1


The MD5 checksum of the tarball can be found at:

https://dist.apache.org/repos/dist/dev/mesos/0.28.0-rc1/mesos-0.28.0.tar.gz.md5


The signature of the tarball can be found at:

https://dist.apache.org/repos/dist/dev/mesos/0.28.0-rc1/mesos-0.28.0.tar.gz.asc


The PGP key used to sign the release is here:

https://dist.apache.org/repos/dist/release/mesos/KEYS


The JAR is up in Maven in a staging repository here:

https://repository.apache.org/content/repositories/orgapachemesos-1112


Please vote on releasing this package as Apache Mesos 0.28.0!


The vote is open until Tue Mar  10 17:00:00 PST 2016 and passes if a majority 
of at least 3 +1 PMC votes are cast.


[ ] +1 Release this package as Apache Mesos 0.28.0

[ ] -1 Do not release this package 

Re: Discussion about upgrading 3rdparty libraries

[Alex Clemmer]

So, at this point we have a bunch of different reviews open for
this[1], and I'd like to use this as an opportunity to start nudging
people towards thinking about possibly transitioning to a scheme where
the tarballs that are currently held in the `3rdparty/` directory are
moved to some external place, and retrieved for users out-of-band, as
necessary to build Mesos.

In particular, doing this (as you all likely know) is very expensive
because git stores a complete copy of the entire tarball, for each
different revision in history, so if you have updated a tarball twice,
you have two complete copies rolling around in the `.git/` folder. It
seems like there are not many benefits for keeping this scheme, other
than the fact that it's pretty easy to implement.

I'm not sure what it would take to transition the autotools build
system, but just recapping earlier what I said about the CMake build
system: The easiest thing to do (which we've already mostly done) is
to allow people to rope in tarballs from some mirror of the `3rdparty`
github repository[2]. Right now we have facilities that let you host
it either on your local FS or on a remote URL, and we'll download (if
necessary) and untar into the familliar place in the `build/` folder.
Easy! We could even have `bootstrap` clone the repository and make
CMake automatically pull in that repository if it's out of date.

Thoughts? I recognize that this might be overcomplicating the problem
a bit, but I figured I'd throw the hat in the ring because this has
always kind of bothered me. :)


[1] They are:
https://reviews.apache.org/r/44252/
https://reviews.apache.org/r/44382/
https://reviews.apache.org/r/44372/
https://reviews.apache.org/r/44378/
https://reviews.apache.org/r/44376/
https://reviews.apache.org/r/44257/

[2] https://github.com/3rdparty/mesos-3rdparty

On Tue, Mar 1, 2016 at 10:48 AM, Alex Clemmer
 wrote:
> It doesn't seem to be the case that these things are mutually
> exclusive -- it is well within our purview to accept only a specific
> range of versions for any particular dependency, and error out if
> someone tries to select a version outside that range. The only thing
> these commits add is more fine-grained control over which of the
> supported versions you are allowed to select.
>
> At this point, there are no such guards, but that is certainly
> something that can be added.
>
> On Tue, Mar 1, 2016 at 10:17 AM, Neil Conway  wrote:
>> The prospect of downloading dependencies from "rando" locations is
>> concerning to me :)
>>
>> Mesos can easily come to depend on implementation details of a
>> dependency that might change in a minor release. For example, a recent
>> change [1] depends on the connection retry logic in the Zk client
>> library in a fairly delicate way. I also wouldn't want users to
>> randomly upgrade to, say, protobuf 2.6.1 without it being thoroughly
>> tested. Increasing the support matrix of different users on different
>> platforms running arbitrarily different versions of third-party
>> dependencies doesn't seem like a net improvement to me.
>>
>> My two cents: if Windows requires additional dependencies that we
>> aren't currently vendoring, I would personally opt for (a) vendoring
>> those additional dependencies (b) ensuring that the vendored versions
>> we ship are modern enough to support all the platforms we care about.
>> Are there important use-cases that aren't supported by this scheme?
>>
>> Neil
>>
>> [1] 
>> https://github.com/apache/mesos/commit/c2d496ed430eaf7daee3e57edefa39c25af2aa43
>>
>> On Tue, Mar 1, 2016 at 10:00 AM, Alex Clemmer
>>  wrote:
>>> I guess a tl;dr might be in order.
>>>
>>> Basically: the CMake build system already supports roping in tarballs
>>> from rando places on the filesystem or Internet, so I think it makes
>>> sense to rope them in at configure time, and so I'm proposing we
>>> re-appropriate the sophisticated tools we already have to do this for
>>> WIndows, into a more general solution that is useful to other exotic
>>> platforms, rather than just Windows.
>>>
>>> As always, super interested to hear feedback, I'd love to know if I
>>> missed something.
>>>
>>> On Tue, Mar 1, 2016 at 9:58 AM, Alex Clemmer
>>>  wrote:
 This is a great time to discuss the Mesos dependency channel story in
 general, because it has had to evolve a bit to fit the requirements of
 Windows, and some of the issues you describe are issues we had to
 resolve (at least partially) to support the Windows integration work.

 More particularly, our problems are: first, Windows frequently
 requires newer versions of dependencies (due to poor support of MSVC
 1900), so we have had to develop reasonably robust version-selection
 mechanisms, so that Windows can get specific versions of different
 packages. This means that the Mesos project does not have to evolve
 the 

Re: Compile failure

[Joris Van Remoortere]

Hello Walter,

Did you see this section on the Getting Started
 page?

# 'Mesos > 0.21.0' requires 'subversion > 1.8' devel package,
# which is not available in the default repositories.
# Create a WANdisco SVN repo file to install the correct version:
$ sudo cat > /etc/yum.repos.d/wandisco-svn.repo  wrote:

> Hi,
>
> Trying to build the mesos apache-mesos-0.27.1 on Oracle Linux 7
>
> configure: error: cannot find libsvn_subr-1 headers
> ---
> libsubversion-1 is required for mesos to build.
>
> I search for packages to install , and found the subversion-devel one, but
> doesn't seems to exist when I yum
>
> sh-4.2# yum install subversion-devel
> Loaded plugins: ulninfo
> No package subversion-devel available.
> Error: Nothing to do
>
> Can somebody advise what to install?
>
> Walter
>
>
>
> This e-mail may contain confidential information. If you are not an
> addressee or otherwise authorised to receive this message, you should not
> use, copy, disclose or take any action based on this e-mail. If you have
> received this e-mail in error, please inform the sender promptly and delete
> this message and any attachments immediately.
>