[mesos-site] branch asf-site updated: rename asf file

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/mesos-site.git


The following commit(s) were added to refs/heads/asf-site by this push:
 new 1eb6636  rename asf file
1eb6636 is described below

commit 1eb6636d3f957fa3fe9942ad2d27d409c8682297
Author: Andreas Peters 
AuthorDate: Tue Jun 29 08:38:12 2021 +0200

rename asf file
---
 .asf.yml => .asf.yaml | 0
 1 file changed, 0 insertions(+), 0 deletions(-)

diff --git a/.asf.yml b/.asf.yaml
similarity index 100%
rename from .asf.yml
rename to .asf.yaml

[mesos-site] branch asf-site updated: add asf file to enable the website also after the 1.7.2021

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/mesos-site.git


The following commit(s) were added to refs/heads/asf-site by this push:
 new 15163d7  add asf file to enable the website also after the 1.7.2021
15163d7 is described below

commit 15163d78c65c4be833ea01dd43b253702569175d
Author: Andreas Peters 
AuthorDate: Thu Jun 24 11:09:12 2021 +0200

add asf file to enable the website also after the 1.7.2021
---
 .asf.yml | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/.asf.yml b/.asf.yml
new file mode 100644
index 000..170e1e3
--- /dev/null
+++ b/.asf.yml
@@ -0,0 +1,3 @@
+publish:
+  whoami: asf-site
+

[mesos] 01/02: Add recipient address in Host header field

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git

commit de8d9e8dc419445b2f9b22868d4ce03a11fc3505
Author: Abdul Qadeer 
AuthorDate: Mon Nov 23 14:30:55 2020 -0800

Add recipient address in Host header field
---
 3rdparty/libprocess/src/encoder.hpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rdparty/libprocess/src/encoder.hpp 
b/3rdparty/libprocess/src/encoder.hpp
index 629b94f..23c8097 100644
--- a/3rdparty/libprocess/src/encoder.hpp
+++ b/3rdparty/libprocess/src/encoder.hpp
@@ -122,7 +122,7 @@ public:
 << "User-Agent: libprocess/" << message.from << "\r\n"
 << "Libprocess-From: " << message.from << "\r\n"
 << "Connection: Keep-Alive\r\n"
-<< "Host: \r\n";
+<< "Host: " << message.to.address << "\r\n";
 
 if (message.body.size() > 0) {
   out << "Transfer-Encoding: chunked\r\n\r\n"

[mesos] 02/02: Update ProcessTest.Remote

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git

commit 0a6cf7639bf71e6333ef541f001f5885ac00b2d5
Author: Abdul Qadeer 
AuthorDate: Thu Dec 3 16:43:06 2020 -0800

Update ProcessTest.Remote
---
 3rdparty/libprocess/src/tests/process_tests.cpp | 13 ++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/3rdparty/libprocess/src/tests/process_tests.cpp 
b/3rdparty/libprocess/src/tests/process_tests.cpp
index 42295a6..eabd70e 100644
--- a/3rdparty/libprocess/src/tests/process_tests.cpp
+++ b/3rdparty/libprocess/src/tests/process_tests.cpp
@@ -1445,9 +1445,11 @@ TEST_F(ProcessTest, Remote)
   RemoteProcess process;
   spawn(process);
 
-  Future handler;
+  Future pid;
+  Future body;
   EXPECT_CALL(process, handler(_, _))
-.WillOnce(FutureSatisfy());
+.WillOnce(DoAll(FutureArg<0>(),
+FutureArg<1>()));
 
   Try create = Socket::create();
   ASSERT_SOME(create);
@@ -1463,12 +1465,17 @@ TEST_F(ProcessTest, Remote)
   message.name = "handler";
   message.from = UPID("sender", sender.get());
   message.to = process.self();
+  message.body = "hello world";
 
   const string data = MessageEncoder::encode(message);
 
   AWAIT_READY(socket.send(data));
 
-  AWAIT_READY(handler);
+  AWAIT_READY(body);
+  ASSERT_EQ("hello world", body.get());
+
+  AWAIT_READY(pid);
+  ASSERT_EQ(message.from, pid.get());
 
   terminate(process);
   wait(process);

[mesos] branch master updated (206da61 -> 0a6cf76)

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git.


from 206da61  Updated `building.md` for the 1.11.0 release.
 new de8d9e8  Add recipient address in Host header field
 new 0a6cf76  Update ProcessTest.Remote

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 3rdparty/libprocess/src/encoder.hpp |  2 +-
 3rdparty/libprocess/src/tests/process_tests.cpp | 13 ++---
 2 files changed, 11 insertions(+), 4 deletions(-)

[mesos] branch master updated (5d68474 -> f86cb65)

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git.


from 5d68474  Added MESOS-10134 to the 1.7.4 CHANGELOG.
 new a5b7765  Fixed Website bot to work with docker user namespaces.
 new af01d50  Merged build.sh and entrypoint.sh.
 new 0a9e90a  Reduced build parallelism to avoid aborted builds in CI.
 new 28cabfc  Added `--no-same-owner` option to tar command.
 new 311c47e  Added `--no-same-owner` option to tar command.
 new d335288  Added `--no-same-owner` option to tar command.
 new f86cb65  Added cleanup of build dir to website bot.

The 7 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 3rdparty/Makefile.am |  4 +--
 3rdparty/libprocess/3rdparty/Makefile.am |  2 +-
 3rdparty/stout/3rdparty/Makefile.am  |  2 +-
 support/mesos-website.sh | 10 +++
 support/mesos-website/Dockerfile |  6 +++--
 support/mesos-website/build.sh   | 44 ---
 support/mesos-website/entrypoint.sh  | 45 +---
 7 files changed, 47 insertions(+), 66 deletions(-)
 delete mode 100755 support/mesos-website/build.sh

[mesos] 05/07: Added `--no-same-owner` option to tar command.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git

commit 311c47e4a9169bdc6cdc962383c711a38c7167d2
Author: Vinod Kone 
AuthorDate: Thu Sep 10 15:20:41 2020 -0500

Added `--no-same-owner` option to tar command.

This ensures extracted tarballs are owned by root when running the
build as root.

Review: https://reviews.apache.org/r/72860
---
 3rdparty/libprocess/3rdparty/Makefile.am | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rdparty/libprocess/3rdparty/Makefile.am 
b/3rdparty/libprocess/3rdparty/Makefile.am
index e148f2d..77e9073 100644
--- a/3rdparty/libprocess/3rdparty/Makefile.am
+++ b/3rdparty/libprocess/3rdparty/Makefile.am
@@ -97,7 +97,7 @@ CLEAN_EXTRACTED = \
 # which cause the packages to get extracted as necessary. We also
 # apply any patches as appropriate.
 %-stamp: $(BUNDLED_DIR)/%.tar.gz
-   gzip -d -c $^ | tar xf -
+   gzip -d -c $^ | tar xf - --no-same-owner
test ! -e $(top_srcdir)/../$*.patch || patch -d $* -p1 
<$(top_srcdir)/../$*.patch
touch $@
 

[mesos] 07/07: Added cleanup of build dir to website bot.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git

commit f86cb659fc38a198261b8d62bd463f8d3eb313f0
Author: Vinod Kone 
AuthorDate: Thu Sep 10 16:30:12 2020 -0500

Added cleanup of build dir to website bot.

Review: https://reviews.apache.org/r/72862
---
 support/mesos-website/entrypoint.sh | 5 -
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/support/mesos-website/entrypoint.sh 
b/support/mesos-website/entrypoint.sh
index 2cc4ab2..c8af020 100755
--- a/support/mesos-website/entrypoint.sh
+++ b/support/mesos-website/entrypoint.sh
@@ -21,7 +21,10 @@ set -e
 set -o pipefail
 
 function exit_hook {
-  # Remove generated documents when exit.
+  # Remove mesos build directory on exit.
+  rm -rf /mesos/build
+
+  # Remove generated documents on exit.
   cd /mesos/site && bundle exec rake clean_docs
 }
 

[mesos] 03/07: Reduced build parallelism to avoid aborted builds in CI.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git

commit 0a9e90af54edc4a1c48a406b54404246a86d1499
Author: Vinod Kone 
AuthorDate: Thu Sep 10 07:09:14 2020 -0500

Reduced build parallelism to avoid aborted builds in CI.

Review: https://reviews.apache.org/r/72858
---
 support/mesos-website/entrypoint.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/support/mesos-website/entrypoint.sh 
b/support/mesos-website/entrypoint.sh
index 6ecfbe0..2cc4ab2 100755
--- a/support/mesos-website/entrypoint.sh
+++ b/support/mesos-website/entrypoint.sh
@@ -44,7 +44,7 @@ fi
 mkdir -p build
 pushd build
 ../configure --disable-python
-make -j6
+make -j3 # Higher parallelism sometimes causes CI to get stuck.
 popd # build
 
 # Generate the endpoint docs from the latest mesos and agent binaries.

[mesos] 02/07: Merged build.sh and entrypoint.sh.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git

commit af01d50663e3c1ec81177dd8baeecae3e5ba8819
Author: Vinod Kone 
AuthorDate: Wed Sep 9 20:17:50 2020 -0500

Merged build.sh and entrypoint.sh.

Review: https://reviews.apache.org/r/72857
---
 support/mesos-website/build.sh  | 57 -
 support/mesos-website/entrypoint.sh | 38 +
 2 files changed, 33 insertions(+), 62 deletions(-)

diff --git a/support/mesos-website/build.sh b/support/mesos-website/build.sh
deleted file mode 100755
index 45ab892..000
--- a/support/mesos-website/build.sh
+++ /dev/null
@@ -1,57 +0,0 @@
-#!/usr/bin/env bash
-
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements.  See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership.  The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License.  You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# This is a script for building Mesos website.
-set -e
-set -o pipefail
-
-function exit_hook {
-  # Remove generated documents when exit.
-  cd /mesos/site && bundle exec rake clean_docs
-}
-
-trap exit_hook EXIT
-
-file_owner_uid=`stat . --format=%u`
-current_user_uid=`id -u`
-if [ $file_owner_uid -ne $current_user_uid ];
-then
-  echo "
-The mounted mesos sources are owned by UID $file_owner_uid
-which is different from the current user UID $current_user_uid
-inside the container. Please check that dockerd has
-user namespace remapping configured properly.
-  "
-  exit 1
-fi
-
-# Build mesos to get the latest master and agent binaries.
-./bootstrap
-mkdir -p build
-pushd build
-../configure --disable-python
-make -j6
-popd # build
-
-# Generate the endpoint docs from the latest mesos and agent binaries.
-./support/generate-endpoint-help.py
-
-# Build the website.
-pushd site
-bundle exec rake
-popd # site
diff --git a/support/mesos-website/entrypoint.sh 
b/support/mesos-website/entrypoint.sh
index 2029949..6ecfbe0 100755
--- a/support/mesos-website/entrypoint.sh
+++ b/support/mesos-website/entrypoint.sh
@@ -16,14 +16,42 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-# This is a wrapper script for building Mesos website.
-# TODO(vinod): Merge this with `build.sh` now that we dont need to switch
-# users in ASF CI as user namespacing has been implemented.
+# This is a script for building Mesos website.
 set -e
 set -o pipefail
 
+function exit_hook {
+  # Remove generated documents when exit.
+  cd /mesos/site && bundle exec rake clean_docs
+}
+
+trap exit_hook EXIT
+
+file_owner_uid=`stat . --format=%u`
+current_user_uid=`id -u`
+if [ $file_owner_uid -ne $current_user_uid ]; then
+  echo "
+The mounted mesos sources are owned by UID $file_owner_uid
+which is different from the current user UID $current_user_uid
+inside the container. Please check that dockerd has
+user namespace remapping configured properly.
+  "
+  exit 1
+fi
+
+# Build mesos to get the latest master and agent binaries.
+./bootstrap
+mkdir -p build
+pushd build
+../configure --disable-python
+make -j6
+popd # build
+
+# Generate the endpoint docs from the latest mesos and agent binaries.
+./support/generate-endpoint-help.py
+
+# Build the website.
 pushd site
 bundle install
+bundle exec rake
 popd # site
-
-./support/mesos-website/build.sh

[mesos] 04/07: Added `--no-same-owner` option to tar command.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git

commit 28cabfc61248908c42c91d4287730cad42dd0400
Author: Vinod Kone 
AuthorDate: Thu Sep 10 15:18:44 2020 -0500

Added `--no-same-owner` option to tar command.

This ensures extracted tarballs are owned by root when running the
build as root.

Review: https://reviews.apache.org/r/72859
---
 3rdparty/stout/3rdparty/Makefile.am | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rdparty/stout/3rdparty/Makefile.am 
b/3rdparty/stout/3rdparty/Makefile.am
index 5a9..7edb7d2 100644
--- a/3rdparty/stout/3rdparty/Makefile.am
+++ b/3rdparty/stout/3rdparty/Makefile.am
@@ -71,7 +71,7 @@ CLEAN_EXTRACTED = \
 # which cause the packages to get extracted as necessary. We also
 # apply any patches as appropriate.
 %-stamp: $(BUNDLED_DIR)/%.tar.gz
-   gzip -d -c $^ | tar xf -
+   gzip -d -c $^ | tar xf - --no-same-owner
test ! -e $(top_srcdir)/../$*.patch || patch -d $* -p1 
<$(top_srcdir)/../$*.patch
touch $@
 

[mesos] 01/07: Fixed Website bot to work with docker user namespaces.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git

commit a5b7765d3ef2905d11ffadbb877377b07624a99c
Author: Vinod Kone 
AuthorDate: Wed Sep 9 17:20:58 2020 -0500

Fixed Website bot to work with docker user namespaces.

ASF CI has enabled user namespaces for docker daemon on the build
machines that are used to build the website. This commit makes the
requisite changes to work with user namespacing.

Review: https://reviews.apache.org/r/72856
---
 support/mesos-website.sh| 10 +++---
 support/mesos-website/Dockerfile|  6 --
 support/mesos-website/build.sh  | 13 +
 support/mesos-website/entrypoint.sh | 12 
 4 files changed, 24 insertions(+), 17 deletions(-)

diff --git a/support/mesos-website.sh b/support/mesos-website.sh
index 04a6646..af98f36 100755
--- a/support/mesos-website.sh
+++ b/support/mesos-website.sh
@@ -32,15 +32,11 @@ docker build --no-cache=true -t $TAG support/mesos-website
 
 trap 'docker rmi $TAG' EXIT
 
-# NOTE: We set `LOCAL_USER_ID` environment variable to enable running the
-# container process with the same UID as the user running this script. This
-# ensures that any writes to the mounted volumes will have the same permissions
-# as the user running the script, making it easy to do cleanups; otherwise
-# any writes will have permissions of UID 0 by default on Linux.
-
+# NOTE: ASF CI remaps the host `jenkins` user to UID 0 inside the container
+# so we don't need to do any user switching inside the container to be able
+# to write to the mounted host volumes with the correct permissions.
 docker run \
   --rm \
-  -e LOCAL_USER_ID="$(id -u "$USER")" \
   -v "$MESOS_DIR":/mesos:Z \
   -v "$MESOS_SITE_DIR/content":/mesos/site/publish:Z \
   $TAG
diff --git a/support/mesos-website/Dockerfile b/support/mesos-website/Dockerfile
index 611c496..bcd62fe 100644
--- a/support/mesos-website/Dockerfile
+++ b/support/mesos-website/Dockerfile
@@ -3,8 +3,10 @@ MAINTAINER "d...@mesos.apache.org"
 
 LABEL Description="This image is used for generating Mesos web site."
 
-# The mesos build image drops down to user `mesos`, but
-# we need priviledged access to install packages below.
+# Set the root user explicitly because the base image
+# `mesos/mesos-build` uses `mesos` user by default. Also, note that
+# `root` inside the container is mapped to `jenkins` user on the build
+# machine host in ASF CI.
 USER root
 
 # Install dependencies.
diff --git a/support/mesos-website/build.sh b/support/mesos-website/build.sh
index afbec48..45ab892 100755
--- a/support/mesos-website/build.sh
+++ b/support/mesos-website/build.sh
@@ -27,6 +27,19 @@ function exit_hook {
 
 trap exit_hook EXIT
 
+file_owner_uid=`stat . --format=%u`
+current_user_uid=`id -u`
+if [ $file_owner_uid -ne $current_user_uid ];
+then
+  echo "
+The mounted mesos sources are owned by UID $file_owner_uid
+which is different from the current user UID $current_user_uid
+inside the container. Please check that dockerd has
+user namespace remapping configured properly.
+  "
+  exit 1
+fi
+
 # Build mesos to get the latest master and agent binaries.
 ./bootstrap
 mkdir -p build
diff --git a/support/mesos-website/entrypoint.sh 
b/support/mesos-website/entrypoint.sh
index 72fd723..2029949 100755
--- a/support/mesos-website/entrypoint.sh
+++ b/support/mesos-website/entrypoint.sh
@@ -16,18 +16,14 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-# This is a wrapper script for building Mesos website as a non-root user.
+# This is a wrapper script for building Mesos website.
+# TODO(vinod): Merge this with `build.sh` now that we dont need to switch
+# users in ASF CI as user namespacing has been implemented.
 set -e
 set -o pipefail
 
-# This needs to be run under `root` user for `bundle exec rake` to
-# work properly. See MESOS-7859.
 pushd site
 bundle install
 popd # site
 
-# Create a local user account.
-useradd -u $LOCAL_USER_ID -s /bin/bash -m tempuser
-
-# Build mesos and the website as the new user.
-su -c support/mesos-website/build.sh tempuser
+./support/mesos-website/build.sh

[mesos] 06/07: Added `--no-same-owner` option to tar command.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git

commit d3352889d075bb84ed86c69e8acd9b52697d0a1b
Author: Vinod Kone 
AuthorDate: Thu Sep 10 15:21:37 2020 -0500

Added `--no-same-owner` option to tar command.

This ensures extracted tarballs are owned by root when running the
build as root.

Review: https://reviews.apache.org/r/72861
---
 3rdparty/Makefile.am | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/3rdparty/Makefile.am b/3rdparty/Makefile.am
index 23f49ec..26099e2 100644
--- a/3rdparty/Makefile.am
+++ b/3rdparty/Makefile.am
@@ -178,7 +178,7 @@ CLEAN_EXTRACTED =   \
 # which cause the packages to get extracted as necessary. We also
 # apply any patches as appropriate.
 %-stamp: %.tar.gz
-   gzip -d -c $^ | tar xf -
+   gzip -d -c $^ | tar xf - --no-same-owner
test ! -e $(srcdir)/$*.patch || patch -d $* -p1 <$(srcdir)/$*.patch
touch $@
 
@@ -191,7 +191,7 @@ CLEAN_EXTRACTED =   \
 CSI_STAMPS = $(CSI_V0)-stamp $(CSI_V1)-stamp
 $(CSI_STAMPS): csi-%-stamp: csi-%.tar.gz
$(MKDIR_P) csi-$*
-   gzip -d -c $^ | (cd csi-$* && tar xf -)
+   gzip -d -c $^ | (cd csi-$* && tar xf - --no-same-owner)
test ! -e $(srcdir)/csi-$*.patch || \
  patch -d csi-$*/spec-$* -p1 <$(srcdir)/csi-$*.patch
touch $@

[mesos] branch master updated: Removed stale MPI framework.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git


The following commit(s) were added to refs/heads/master by this push:
 new 0ef073c  Removed stale MPI framework.
0ef073c is described below

commit 0ef073c1719216415b63a2ead3f9e33d28aa0212
Author: Vinod Kone 
AuthorDate: Mon Mar 2 14:34:32 2020 -0600

Removed stale MPI framework.

The framework code hasn't been touched in years and likely
doesn't work anymore.

Review: https://reviews.apache.org/r/72196
---
 Makefile.am  |   4 -
 configure.ac |   2 -
 mpi/README   |  63 --
 mpi/mpiexec-mesos.in |  39 -
 mpi/mpiexec-mesos.py | 234 ---
 5 files changed, 342 deletions(-)

diff --git a/Makefile.am b/Makefile.am
index 11192ac..b227461 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -56,10 +56,6 @@ AM_DISTCHECK_CONFIGURE_FLAGS = 
--disable-python-dependency-install
 AM_DISTCHECK_CONFIGURE_FLAGS += --disable-dependency-tracking
 
 
-# MPI framework.
-EXTRA_DIST += mpi/README mpi/mpiexec-mesos.in mpi/mpiexec-mesos.py
-
-
 if HAS_JAVA
 maven-install:
@cd src && $(MAKE) $(AM_MAKEFLAGS) maven-install
diff --git a/configure.ac b/configure.ac
index 4aa4f76..46ca1c7 100644
--- a/configure.ac
+++ b/configure.ac
@@ -200,8 +200,6 @@ AC_CONFIG_FILES([include/mesos/version.hpp])
 
 AC_CONFIG_FILES([src/java/generated/org/apache/mesos/MesosNativeLibrary.java])
 
-AC_CONFIG_FILES([mpi/mpiexec-mesos], [chmod +x mpi/mpiexec-mesos])
-
 
 ###
 # Optional features.
diff --git a/mpi/README b/mpi/README
deleted file mode 100644
index 691ae78..000
--- a/mpi/README
+++ /dev/null
@@ -1,63 +0,0 @@
-Mesos MPICH2 framework readme
-
-
-Table of Contents:
-1) Installing MPICH2
-2) Running the Mesos MPICH2 framework
-
-=
-1) INSTALLING MPICH2:
-=
-- This framework was developed for MPICH2 1.2 (mpd was deprecated
-  starting 1.3) on Linux(Ubuntu 11.10) and OS X Lion.
-  WARNING: MPICH2 later than 1.2 may not run on Mesos.
-  (You can configure newer version to include mpd,
-   but then mpdtrace may still be missing and
-   the executor script used for Mesos relies on it.)
-
-- You can install MPICH2 from scratch. You can get MPICH2 as well as
-  installation directions here:
-  http://www.mcs.anl.gov/research/projects/mpich2/. This tutorial
-  follows the latter. Unpack the tar.gz and...
-
-- To use MPI with Mesos, make sure to have MPICH2 installed on every
-  machine in your cluster.
-
-Setting up:
--> Install and configure:
-mac:  ./configure --prefix=/Users/_your_username_/mpich2-install
-ubuntu :  ./configure --prefix=/home/_your_username_/mpich2-install
- Then...
- sudo make
- sudo make install
-
-
--> Optional: add mpich binaries to PATH. You can specify the path to
-   installed MPICH2 binaries using mpiexec-meso's '--path' option
-mac:  sudo vim ~/.bash_profile
-  export PATH=/Users/_your_username_/mpich2-install/bin:$PATH
-ubuntu :  sudo vim ~/.bashrc
-  export PATH=/home/_your_username_/mpich2-install/bin:$PATH
-
--> Create .mpd conf file in home directory:
-  echo "secretword=nil" > ~/.mpd.conf
- chmod 600 .mpd.conf
-
--> Check installation - these should all return the PATH's set above
-  which mpd
-  which mpiexec
-  which mpirun
-
-
-=
-2) RUNNING THE MESOS MPICH2 FRAMEWORK
-=
-
-Using/testing mpiexec-mesos:
--> Start a Mesos master and slaves
-
--> How to run a Hello, World! program (pass the -h flag to see help options):
-   mpicc helloworld.c -helloworld
-  ./mpiexec-mesos 127.0.0.1:5050 ./helloworld
-   Paths to mesos, protobuf, and distribute eggs can be specified by setting
-   respective environment variables in mpiexec-mesos.
diff --git a/mpi/mpiexec-mesos.in b/mpi/mpiexec-mesos.in
deleted file mode 100644
index dc06953..000
--- a/mpi/mpiexec-mesos.in
+++ /dev/null
@@ -1,39 +0,0 @@
-#!/bin/sh
-#
-# This is a wrapper script for the MPI launcher framework.
-#
-# This script uses MESOS_SOURCE_DIR and MESOS_BUILD_DIR which come
-# from configuration substitutions.
-MESOS_SOURCE_DIR=@abs_top_srcdir@
-MESOS_BUILD_DIR=@abs_top_builddir@
-
-# Use colors for errors.
-. ${MESOS_SOURCE_DIR}/support/colors.sh
-
-# Force the use of the Python interpreter configured during building.
-test ! -z "${PYTHON}" && \
-  echo "${RED}Ignoring PYTHON environment variable (using @PYTHON@)${NORMAL}"
-
-PYTHON=@PYTHON@
-
-SETUPTOOLS=`echo ${MESOS_BUILD_DIR}/3rdparty/setuptools-*/`
-
-# Just warn in the case when build with --disable-bundled.
-test ! -e ${S

[mesos] branch vinodkone-patch-1 deleted (was b8a8554)

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a change to branch vinodkone-patch-1
in repository https://gitbox.apache.org/repos/asf/mesos.git.


 was b8a8554  Removed the link to mesos.slackarchive.io

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.

[mesos] branch master updated: Removed the link to mesos.slackarchive.io

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git


The following commit(s) were added to refs/heads/master by this push:
 new 5987264  Removed the link to mesos.slackarchive.io
5987264 is described below

commit 598726423731f65d4b17a6543970f74fa876038b
Author: Vinod Kone 
AuthorDate: Fri Jan 3 17:26:51 2020 -0600

Removed the link to mesos.slackarchive.io

slackarchive.io is no longer active. Also, since we upgraded to
Slack's standard plan, archiving is built-in to Slack.
---
 site/source/community.html.md | 2 --
 1 file changed, 2 deletions(-)

diff --git a/site/source/community.html.md b/site/source/community.html.md
index 67ce8b0..19083bb 100644
--- a/site/source/community.html.md
+++ b/site/source/community.html.md
@@ -41,8 +41,6 @@ layout: community_section
 
 To request an invite for slack please click https://mesos-slackin.herokuapp.com;>here.
 
-All slack communication is publicly archived http://mesos.slackarchive.io;>here.
-
 Thank you https://www.criteo.com;>Criteo for sponsoring 
Slack standard plan.
 
 

[mesos] branch vinodkone-patch-1 updated (701fe22 -> b8a8554)

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a change to branch vinodkone-patch-1
in repository https://gitbox.apache.org/repos/asf/mesos.git.


 discard 701fe22  Removed the link the slackarchive.io
 new b8a8554  Removed the link to mesos.slackarchive.io

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (701fe22)
\
 N -- N -- N   refs/heads/vinodkone-patch-1 (b8a8554)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:

[mesos] 01/01: Removed the link to mesos.slackarchive.io

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch vinodkone-patch-1
in repository https://gitbox.apache.org/repos/asf/mesos.git

commit b8a85545bdc94766d4bc8a09c01c9648d942eda3
Author: Vinod Kone 
AuthorDate: Fri Jan 3 17:26:51 2020 -0600

Removed the link to mesos.slackarchive.io

slackarchive.io is no longer active. Also, since we upgraded to
Slack's standard plan, archiving is built-in to Slack.
---
 site/source/community.html.md | 2 --
 1 file changed, 2 deletions(-)

diff --git a/site/source/community.html.md b/site/source/community.html.md
index 67ce8b0..19083bb 100644
--- a/site/source/community.html.md
+++ b/site/source/community.html.md
@@ -41,8 +41,6 @@ layout: community_section
 
 To request an invite for slack please click https://mesos-slackin.herokuapp.com;>here.
 
-All slack communication is publicly archived http://mesos.slackarchive.io;>here.
-
 Thank you https://www.criteo.com;>Criteo for sponsoring 
Slack standard plan.
 
 

[mesos] 01/01: Removed the link the slackarchive.io

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch vinodkone-patch-1
in repository https://gitbox.apache.org/repos/asf/mesos.git

commit 701fe22d9b32731a8cede5301983e27e816922a5
Author: Vinod Kone 
AuthorDate: Fri Jan 3 17:26:51 2020 -0600

Removed the link the slackarchive.io

slackarchive.io is no longer active. Also, since we upgraded to Slack's 
standard plan, archiving is built-in.
---
 site/source/community.html.md | 2 --
 1 file changed, 2 deletions(-)

diff --git a/site/source/community.html.md b/site/source/community.html.md
index 67ce8b0..19083bb 100644
--- a/site/source/community.html.md
+++ b/site/source/community.html.md
@@ -41,8 +41,6 @@ layout: community_section
 
 To request an invite for slack please click https://mesos-slackin.herokuapp.com;>here.
 
-All slack communication is publicly archived http://mesos.slackarchive.io;>here.
-
 Thank you https://www.criteo.com;>Criteo for sponsoring 
Slack standard plan.
 
 

[mesos] branch vinodkone-patch-1 created (now 701fe22)

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a change to branch vinodkone-patch-1
in repository https://gitbox.apache.org/repos/asf/mesos.git.


  at 701fe22  Removed the link the slackarchive.io

This branch includes the following new commits:

 new 701fe22  Removed the link the slackarchive.io

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

[mesos] branch master updated: Fixed bad link

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git


The following commit(s) were added to refs/heads/master by this push:
 new 427b927  Fixed bad link
427b927 is described below

commit 427b927a7eae048cf554297ead97b57e5488c13c
Author: Jonathan Robson 
AuthorDate: Wed Oct 23 08:34:21 2019 -0500

Fixed bad link

The bintray link was incorrect. Now working correctly
---
 docs/binary-packages.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/binary-packages.md b/docs/binary-packages.md
index 02c95eb..07e0933 100644
--- a/docs/binary-packages.md
+++ b/docs/binary-packages.md
@@ -13,7 +13,7 @@ Download and install the latest stable RPM binary from the 
[Bintray Repository](
 #bintray-mesos-el - packages by mesos from Bintray
 [bintray-mesos-el]
 name=bintray-mesos-el
-baseurl=https://dl.bintray.com/apache/mesos/el/7/x86_64
+baseurl=https://dl.bintray.com/apache/mesos/el7/x86_64
 gpgcheck=0
 repo_gpgcheck=0
 enabled=1

[mesos] branch master updated: Fixed the link to RAO staging repo in the release guide.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git


The following commit(s) were added to refs/heads/master by this push:
 new 9297e2d  Fixed the link to RAO staging repo in the release guide.
9297e2d is described below

commit 9297e2d3b0d44b553fc89bcf5f6109c76cc53668
Author: Vinod Kone 
AuthorDate: Fri Aug 30 10:10:05 2019 -0500

Fixed the link to RAO staging repo in the release guide.
---
 docs/release-guide.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/docs/release-guide.md b/docs/release-guide.md
index ce9c7c1..bae7414 100644
--- a/docs/release-guide.md
+++ b/docs/release-guide.md
@@ -173,7 +173,7 @@ This guide describes the process of doing an official 
release of Mesos.
scheme. `R` is release candidate version that starts with 1.
 
 4. Tag the release externally and deploy the corresponding JAR to the
-   [Apache maven repository](https://repository.apache.org). It is recommended
+   [Apache Maven repository](https://repository.apache.org). It is recommended
to use the `support/vote.sh` script to accomplish this.
 
 $ ./support/vote.sh X.Y.Z R
@@ -195,7 +195,7 @@ This guide describes the process of doing an official 
release of Mesos.
 
 1. If the vote does not pass (any -1s or showstopper bugs)
1. Send a reply to the original VOTE email with subject "[RESULT][VOTE] 
Release Apache Mesos X.Y.Z (rcN)" and mention that the vote is canceled.
-   2. Go to https://repository.apache.org/#stagingRepositories and "Drop" the 
staging repository containing the JAR (you can find the link to the staging 
repository in the original VOTE email).
+   2. Go to [Apache Maven staging 
repositories](https://repository.apache.org/#stagingRepositories) and "Drop" 
the staging repository containing the JAR (you can find the exact link to the 
staging repository in the original VOTE email).
3. Track the issues as new JIRAs for the release.
 
 2. When all known issues are resolved, update the `CHANGELOG` with the newly

[mesos] branch master updated: Updated release guide.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git


The following commit(s) were added to refs/heads/master by this push:
 new 00f7f48  Updated release guide.
00f7f48 is described below

commit 00f7f48e4f4ef599a41c5f63e0afe6c51cc6894f
Author: Vinod Kone 
AuthorDate: Thu Aug 29 11:29:30 2019 -0500

Updated release guide.

Added steps for dropping the staging repository on failed vote for an ARC.
---
 docs/release-guide.md | 6 --
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/docs/release-guide.md b/docs/release-guide.md
index 9ea54cc..ce9c7c1 100644
--- a/docs/release-guide.md
+++ b/docs/release-guide.md
@@ -193,8 +193,10 @@ This guide describes the process of doing an official 
release of Mesos.
 
 ## Preparing a New Release Candidate
 
-1. If the vote does not pass (any -1s or showstopper bugs), track the issues
-   as new JIRAs for the release.
+1. If the vote does not pass (any -1s or showstopper bugs)
+   1. Send a reply to the original VOTE email with subject "[RESULT][VOTE] 
Release Apache Mesos X.Y.Z (rcN)" and mention that the vote is canceled.
+   2. Go to https://repository.apache.org/#stagingRepositories and "Drop" the 
staging repository containing the JAR (you can find the link to the staging 
repository in the original VOTE email).
+   3. Track the issues as new JIRAs for the release.
 
 2. When all known issues are resolved, update the `CHANGELOG` with the newly
fixed JIRAs.

[mesos] branch master updated: Updated configure.ac to correct openssl/libevent setup.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git


The following commit(s) were added to refs/heads/master by this push:
 new dfcd31f  Updated configure.ac to correct openssl/libevent setup.
dfcd31f is described below

commit dfcd31fab6a99940fe1e33c3c6ba780f6b88b29d
Author: Till Toenshoff 
AuthorDate: Mon Jul 22 18:00:10 2019 -0500

Updated configure.ac to correct openssl/libevent setup.

Fixes a problem introduced by configure.ac reordering in commit 4df2b62.

Review: https://reviews.apache.org/r/71138/
---
 configure.ac | 18 ++
 1 file changed, 10 insertions(+), 8 deletions(-)

diff --git a/configure.ac b/configure.ac
index 0e3058c..ae40d56 100644
--- a/configure.ac
+++ b/configure.ac
@@ -901,14 +901,6 @@ libcrypto is required for an SSL-enabled build.
 
   LIBS="-lssl -lcrypto $LIBS"
 
-  if test "x$enable_libevent" = "xyes"; then
-if test "x$with_bundled_libevent" != "xyes"; then
-  MESOS_HAVE_LIBEVENT()
-  MESOS_HAVE_LIBEVENT_SSL()
-fi
-  else
-AC_MSG_ERROR([SSL is currently only supported with libevent])
-  fi
   AC_DEFINE([USE_SSL_SOCKET], [1])
 fi
 
@@ -1605,6 +1597,16 @@ if test "x$enable_libevent" = "xyes"; then
   fi
 fi
 
+if test "x$enable_ssl" = "xyes"; then
+  if test "x$enable_libevent" = "xyes"; then
+if test "x$with_bundled_libevent" != "xyes"; then
+  MESOS_HAVE_LIBEVENT_SSL()
+fi
+  else
+AC_MSG_ERROR([SSL is currently only supported with libevent])
+  fi
+fi
+
 AM_CONDITIONAL([ENABLE_LIBEVENT],
[test "x$enable_libevent" = "xyes"])
 AM_CONDITIONAL([WITH_BUNDLED_LIBEVENT],

[mesos] branch master updated: Documented EOF in ATTACH_CONTAINER_INPUT

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git


The following commit(s) were added to refs/heads/master by this push:
 new 11bdb5f  Documented EOF in ATTACH_CONTAINER_INPUT
11bdb5f is described below

commit 11bdb5f996c8cfc94d325001cd0a861b654a08a4
Author: Bilal Amarni 
AuthorDate: Mon Jul 22 16:57:18 2019 +0200

Documented EOF in ATTACH_CONTAINER_INPUT

This documents how to indicate when EOF is reached on STDIN.

See 
https://github.com/dcos/dcos-core-cli/blob/00e1d8f4eddca5d961152f32669e02ed81d5/python/lib/dcos/dcos/mesos.py#L1657-L1658
---
 docs/operator-http-api.md | 17 -
 1 file changed, 16 insertions(+), 1 deletion(-)

diff --git a/docs/operator-http-api.md b/docs/operator-http-api.md
index 2d4a9b6..1167838 100644
--- a/docs/operator-http-api.md
+++ b/docs/operator-http-api.md
@@ -3816,7 +3816,9 @@ they may contain subtypes of either DATA or CONTROL. DATA 
messages
 must be of type STDIN and contain the actual data to stream to the
 STDIN of the container being attached to. Currently, the only valid
 CONTROL message sends a heartbeat to keep the connection alive. We may
-add more CONTROL messages in the future.
+add more CONTROL messages in the future. An empty DATA message of type
+STDIN indicates EOF. If the container was launched with TTYInfo in their
+ContainerInfo, an EOT DATA message is expected instead.
 
 ```
 ATTACH_CONTAINER_INPUT HTTP Request (JSON):
@@ -3868,6 +3870,19 @@ Accept: application/json
   }
 }
   }
+}215
+{
+  "type": "ATTACH_CONTAINER_INPUT",
+  "attach_container_input": {
+"type": "PROCESS_IO",
+"process_io": {
+  "type": "DATA",
+  "data": {
+"type": "STDIN",
+"data": ""
+  }
+}
+  }
 }
 ...
 

[mesos] branch master updated: Updated ReviewBot to tail 100 KB instead of 1 MB.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git


The following commit(s) were added to refs/heads/master by this push:
 new 97fded5  Updated ReviewBot to tail 100 KB instead of 1 MB.
97fded5 is described below

commit 97fded51512ef3c5a4755a861a811e5329d9857a
Author: Vinod Kone 
AuthorDate: Mon Jul 8 13:55:22 2019 -0500

Updated ReviewBot to tail 100 KB instead of 1 MB.

1 MB worth of trailing data does not seem to be needed from
experience. It just causes comment bloat.
---
 support/verify-reviews.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/support/verify-reviews.py b/support/verify-reviews.py
index c9c85f9..045a9ef 100755
--- a/support/verify-reviews.py
+++ b/support/verify-reviews.py
@@ -47,7 +47,7 @@ from datetime import datetime
 
 from common import REVIEWBOARD_URL
 
-REVIEW_SIZE = 100  # 1 MB in bytes.
+REVIEW_SIZE = 10  # 100 KB in bytes.
 
 # Parse arguments.
 parser = argparse.ArgumentParser(

[mesos] branch master updated: Added thanks to Criteo for Slack plan in community.html.md.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git


The following commit(s) were added to refs/heads/master by this push:
 new 5a624bc  Added thanks to Criteo for Slack plan in community.html.md.
5a624bc is described below

commit 5a624bc61836dae0126c934e1cc97711deb43c3e
Author: Vinod Kone 
AuthorDate: Wed Apr 24 09:54:12 2019 -0500

Added thanks to Criteo for Slack plan in community.html.md.
---
 site/source/community.html.md | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/site/source/community.html.md b/site/source/community.html.md
index 09a829b..67ce8b0 100644
--- a/site/source/community.html.md
+++ b/site/source/community.html.md
@@ -43,6 +43,9 @@ layout: community_section
 
 All slack communication is publicly archived http://mesos.slackarchive.io;>here.
 
+Thank you https://www.criteo.com;>Criteo for sponsoring 
Slack standard plan.
+
+
 IRC 
  If you want to connect to our slack team through an IRC client please 
click https://my.slack.com/gateways;>here after signing up.
 

[mesos] branch master updated: Updated default executor semantics in the doc.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git


The following commit(s) were added to refs/heads/master by this push:
 new 26630d0  Updated default executor semantics in the doc.
26630d0 is described below

commit 26630d02c6b40a5e631dc5dccb50c61c26f25145
Author: Vinod Kone 
AuthorDate: Thu Apr 11 13:16:20 2019 -0500

Updated default executor semantics in the doc.

Noted that scheduler killing a task will kill all tasks in a task group.
---
 docs/app-framework-development-guide.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/app-framework-development-guide.md 
b/docs/app-framework-development-guide.md
index e64cd92..8f30045 100644
--- a/docs/app-framework-development-guide.md
+++ b/docs/app-framework-development-guide.md
@@ -144,7 +144,7 @@ The current semantics of the default executor are as folows:
 -- There is no resource isolation between different tasks or task groups 
within an executor.
Tasks' resources are added to the executor container.
 
--- If any of the tasks exits with a non-zero exit code, all the tasks in the 
task group
+-- If any of the tasks exits with a non-zero exit code or killed by the 
scheduler, all the tasks in the task group
are killed automatically. The default executor commits suicide if there are 
no active task groups.

 

[mesos] branch master updated: Updated documentation of default executor.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git


The following commit(s) were added to refs/heads/master by this push:
 new 277d38d  Updated documentation of default executor.
277d38d is described below

commit 277d38d7277f70bf3e90052b41eaa4740562a4ee
Author: Vinod Kone 
AuthorDate: Wed Apr 10 18:27:50 2019 -0500

Updated documentation of default executor.

Documented that it supports multiple task groups. Made other minor 
adjustments.
---
 docs/app-framework-development-guide.md | 16 ++--
 1 file changed, 10 insertions(+), 6 deletions(-)

diff --git a/docs/app-framework-development-guide.md 
b/docs/app-framework-development-guide.md
index 3648ab6..e64cd92 100644
--- a/docs/app-framework-development-guide.md
+++ b/docs/app-framework-development-guide.md
@@ -130,19 +130,23 @@ commands or Docker containers.
 
 The current semantics of the default executor are as folows:
 
--- Tasks are launched as nested containers underneath the executor container.
+-- Task group is an atomic unit of deployment of a scheduler onto the default 
executor.
+
+-- The default executor can run one or more task groups (since Mesos 1.2) and 
each task group can be launched by the scheduler at different points in time.
+
+-- All task groups' tasks are launched as nested containers underneath the 
executor container.
 
 -- Task containers and executor container share resources like cpu, memory,
network and volumes.
+   
+-- Each task can have its own separate root file system (e.g., Docker image).
 
--- There is no resource isolation between different tasks within an executor.
+-- There is no resource isolation between different tasks or task groups 
within an executor.
Tasks' resources are added to the executor container.
 
 -- If any of the tasks exits with a non-zero exit code, all the tasks in the 
task group
-   are killed and the executor shuts down.
-
--- Multiple task groups are not supported.
-
+   are killed automatically. The default executor commits suicide if there are 
no active task groups.
+   
 
 Once the default executor is considered **stable**, the command executor will 
be deprecated in favor of it.
 

[mesos] branch master updated: Adding Mesoscon 2018 resources

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git


The following commit(s) were added to refs/heads/master by this push:
 new 623fd7c  Adding Mesoscon 2018 resources
623fd7c is described below

commit 623fd7c6c15124b0d77af878129f3e70abaa14f7
Author: Dominik Dary 
AuthorDate: Tue Mar 26 13:51:49 2019 +0100

Adding Mesoscon 2018 resources
---
 docs/presentations.md | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/docs/presentations.md b/docs/presentations.md
index 8fb664c..c2775ea 100644
--- a/docs/presentations.md
+++ b/docs/presentations.md
@@ -6,6 +6,9 @@ layout: documentation
 # Video and Slides of Mesos Presentations
 _(Listed in reverse chronological order)_
 
+## MesosCon North America 2018
+[Video 
playlist](https://www.youtube.com/playlist?list=PL-cRvJ6sAbfjvQCLT3ktrpnVwrJKTodfm)
 + [Slides](https://mesoscon18.sched.com/)
+
 ## Jolt: Running Distributed, Fault-Tolerant Tests at Scale using Mesos
 [Video](https://www.youtube.com/watch?v=2uGwlVs8Cpw)
 Sunil Shah, Kyle Kelly, and Timmy Zhu Presented November 1, 2017 at [Bay Area 
Mesos User Group 
Meetup](https://www.meetup.com/Bay-Area-Mesos-User-Group/events/244469969/)

[mesos] branch master updated: Updated ReviewBot to verify reviews by checking for updates recursively.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git


The following commit(s) were added to refs/heads/master by this push:
 new 2c65877  Updated ReviewBot to verify reviews by checking for updates 
recursively.
2c65877 is described below

commit 2c658770e7e273672d64fe6c7deaed05828e9a15
Author: Vinod Kone 
AuthorDate: Thu Feb 28 12:21:09 2019 -0600

Updated ReviewBot to verify reviews by checking for updates recursively.

If any of the dependent reviews has an updated diff or dependency, it
now triggers the ReviewBot. Previously only updates to the tail
review in the chain triggered the ReviewBot.

Review: https://reviews.apache.org/r/70060/
---
 support/verify-reviews.py | 86 ++-
 1 file changed, 56 insertions(+), 30 deletions(-)

diff --git a/support/verify-reviews.py b/support/verify-reviews.py
index f03869a..377792d 100755
--- a/support/verify-reviews.py
+++ b/support/verify-reviews.py
@@ -115,6 +115,11 @@ class ReviewError(Exception):
 pass
 
 
+def parse_time(timestamp):
+"""Parse time in ReviewBoard date format."""
+return datetime.strptime(timestamp, "%Y-%m-%dT%H:%M:%SZ")
+
+
 def shell(command, working_dir=None):
 """Run a shell command."""
 out = subprocess.check_output(
@@ -303,8 +308,51 @@ def verify_review(review_request):
 cleanup()
 
 
+def review_updated(review_request, review_time):
+"""Returns whether this review request chain was updated after review
+   time."""
+# If the latest diff on this review request was uploaded after the last
+# review from `USER`, we need to verify it.
+diffs_url = review_request["links"]["diffs"]["href"]
+diffs = api(diffs_url)
+diff_time = None
+if "diffs" in diffs:
+# Get the timestamp of the latest diff.
+timestamp = diffs["diffs"][-1]["timestamp"]
+diff_time = parse_time(timestamp)
+print("Latest diff timestamp: %s" % diff_time)
+
+if diff_time and review_time < diff_time:
+return True
+
+# If the latest dependency change on this review request happened after the
+# last review from `USER`, we need to verify it.
+changes_url = review_request["links"]["changes"]["href"]
+changes = api(changes_url)
+dependency_time = None
+for change in changes["changes"]:
+if "depends_on" in change["fields_changed"]:
+timestamp = change["timestamp"]
+dependency_time = parse_time(timestamp)
+print("Latest dependency change timestamp: %s" % dependency_time)
+break
+
+if dependency_time and review_time < dependency_time:
+return True
+
+# Recursively check if any of the dependent review requests need
+# verification.
+for review in review_request["depends_on"]:
+review_url = review["href"]
+print("Dependent review: %s " % review_url)
+if review_updated(api(review_url)["review_request"], review_time):
+return True
+
+return False
+
+
 def needs_verification(review_request):
-"""Return True if this review request needs to be verified."""
+"""Returns whether this review request chain needs to be verified."""
 print("Checking if review: %s needs verification" % review_request["id"])
 
 # Skip if the review blocks another review.
@@ -312,45 +360,23 @@ def needs_verification(review_request):
 print("Skipping blocking review %s" % review_request["id"])
 return False
 
-diffs_url = review_request["links"]["diffs"]["href"]
-diffs = api(diffs_url)
-
-if "diffs" not in diffs:  # No diffs attached!
-print("Skipping review %s as it has no diffs" % review_request["id"])
-return False
-
-# Get the timestamp of the latest diff.
-timestamp = diffs["diffs"][-1]["timestamp"]
-rb_date_format = "%Y-%m-%dT%H:%M:%SZ"
-diff_time = datetime.strptime(timestamp, rb_date_format)
-print("Latest diff timestamp: %s" % diff_time)
-
-# Get the timestamp of the latest review from this script.
+# Get the timestamp of the latest review from `USER`.
 reviews_url = review_request["links"]["reviews"]["href"]
 reviews = api(reviews_url + "?max-results=200")
 review_time = None
 for review in reversed(reviews["reviews"]):
 if review[&q

[mesos] branch master updated: Output Review Request URL in Reviewbot output.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git


The following commit(s) were added to refs/heads/master by this push:
 new 4442c79  Output Review Request URL in Reviewbot output.
4442c79 is described below

commit 4442c793708a8d7d3dfe2b17f56265d101bc8d9e
Author: Vinod Kone 
AuthorDate: Mon Feb 25 17:24:19 2019 -0600

Output Review Request URL in Reviewbot output.

The URL makes it easy to get to the Review from log messages.

Review: https://reviews.apache.org/r/70055
---
 support/verify-reviews.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/support/verify-reviews.py b/support/verify-reviews.py
index ab4888a..f03869a 100755
--- a/support/verify-reviews.py
+++ b/support/verify-reviews.py
@@ -197,7 +197,8 @@ def apply_reviews(review_request, reviews):
 
 def post_review(review_request, message):
 """Post a review on the review board."""
-print("Posting review: %s" % message)
+print("Posting review on %s :\n%s" \
+  % (review_request["absolute_url"], message))
 
 review_url = review_request["links"]["reviews"]["href"]
 data = urllib.parse.urlencode({"body_top": message, "public": "true"})

[mesos] branch master updated: Increased the number of review requests queried by ReviewBot to 200.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git


The following commit(s) were added to refs/heads/master by this push:
 new 2527ddc  Increased the number of review requests queried by ReviewBot 
to 200.
2527ddc is described below

commit 2527ddccdefa7688459116146ae666da57019d83
Author: Vinod Kone 
AuthorDate: Wed Feb 20 15:19:24 2019 -0600

Increased the number of review requests queried by ReviewBot to 200.

Without the `max-results` query parameter, ReviewBoard API returns 25
results by default.

Review: https://reviews.apache.org/r/70026
---
 support/verify-reviews.py | 8 ++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/support/verify-reviews.py b/support/verify-reviews.py
index 84f..ab4888a 100755
--- a/support/verify-reviews.py
+++ b/support/verify-reviews.py
@@ -70,10 +70,14 @@ parser.add_argument(
 " that will need verification")
 
 # Unless otherwise specified consider pending review requests to Mesos updated
-# since 03/01/2014.
+# since 03/01/2014. Note that we need to explicitly set `max-results` query
+# parameter because the default is 25 (max allowed is 200).
+# TODO(vinod): Paginate through the API to get all pending review requests
+# instead of just getting the first 200.
 group = "mesos"
 last_updated = "2014-03-01T00:00:00"
-query_parameters = "?to-groups=%s=pending=%s" \
+query_parameters = \
+"?to-groups=%s=pending=%s=200" \
 % (group, last_updated)
 parser.add_argument(
 "-q",

[mesos] branch master updated: FaultDomain, conventions for additional hierarchy.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git


The following commit(s) were added to refs/heads/master by this push:
 new 437968d  FaultDomain, conventions for additional hierarchy.
437968d is described below

commit 437968d7ba1115ee4713c6e04f74d731b4b55ee0
Author: James DeFelice 
AuthorDate: Wed May 30 17:43:27 2018 -0400

FaultDomain, conventions for additional hierarchy.
---
 include/mesos/mesos.proto| 12 
 include/mesos/v1/mesos.proto | 12 
 2 files changed, 24 insertions(+)

diff --git a/include/mesos/mesos.proto b/include/mesos/mesos.proto
index c96ed31..67db603 100644
--- a/include/mesos/mesos.proto
+++ b/include/mesos/mesos.proto
@@ -834,6 +834,18 @@ message ExecutorInfo {
  * in the same zone, in different zones in the same region, or in
  * different regions. Note that all masters in a given Mesos cluster
  * must be in the same region.
+ *
+ * Complex deployments may have additional levels of hierarchy: for example,
+ * multiple racks might be grouped together into "halls" and multiple DCs in
+ * the same geographical vicinity might be grouped together. As a convention,
+ * the recommended way to represent additional levels of hierarchy is via dot-
+ * separated labels in the existing zone and region fields. For example, the
+ * fact that racks "abc" and "def" are in the same hall might be represented
+ * using the zone names "rack-abc.hall-1" and "rack-def.hall-1", for example.
+ * Software that is not aware of this additional structure will compare the
+ * zone names for equality- hence, the two zones will be treated as different
+ * (unrelated) zones. Software that is "hall-aware" can inspect the zone names
+ * and make use of the additional hierarchy.
  */
 message DomainInfo {
   message FaultDomain {
diff --git a/include/mesos/v1/mesos.proto b/include/mesos/v1/mesos.proto
index bbd343c..50b4ff3 100644
--- a/include/mesos/v1/mesos.proto
+++ b/include/mesos/v1/mesos.proto
@@ -832,6 +832,18 @@ message ExecutorInfo {
  * in the same zone, in different zones in the same region, or in
  * different regions. Note that all masters in a given Mesos cluster
  * must be in the same region.
+ *
+ * Complex deployments may have additional levels of hierarchy: for example,
+ * multiple racks might be grouped together into "halls" and multiple DCs in
+ * the same geographical vicinity might be grouped together. As a convention,
+ * the recommended way to represent additional levels of hierarchy is via dot-
+ * separated labels in the existing zone and region fields. For example, the
+ * fact that racks "abc" and "def" are in the same hall might be represented
+ * using the zone names "rack-abc.hall-1" and "rack-def.hall-1", for example.
+ * Software that is not aware of this additional structure will compare the
+ * zone names for equality- hence, the two zones will be treated as different
+ * (unrelated) zones. Software that is "hall-aware" can inspect the zone names
+ * and make use of the additional hierarchy.
  */
 message DomainInfo {
   message FaultDomain {

[mesos] branch master updated: Added MESOS-8887 to the 1.5.3 CHANGELOG.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git


The following commit(s) were added to refs/heads/master by this push:
 new f1dc505  Added MESOS-8887 to the 1.5.3 CHANGELOG.
f1dc505 is described below

commit f1dc50568dcc90cec7158205dca86a2398a42dcd
Author: Vinod Kone 
AuthorDate: Fri Feb 15 14:51:22 2019 -0600

Added MESOS-8887 to the 1.5.3 CHANGELOG.
---
 CHANGELOG | 1 +
 1 file changed, 1 insertion(+)

diff --git a/CHANGELOG b/CHANGELOG
index 26b5ef9..becb170 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -892,6 +892,7 @@ Release Notes - Mesos - Version 1.5.3 (WIP)
 * This is a bug fix release.
 
 ** Bug
+  * [MESOS-8887] - Unreachable tasks are not GC'ed when unreachable agent is 
GC'ed.
   * [MESOS-9210] - Mesos v1 scheduler library does not properly handle 
SUBSCRIBE retries.
   * [MESOS-9532] - ResourceOffersTest.ResourceOfferWithMultipleSlaves is flaky.
   * [MESOS-9533] - CniIsolatorTest.ROOT_CleanupAfterReboot is flaky.

[mesos] 01/04: Fixed variable names in `Master::_doRegistryGC()`.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch 1.5.x
in repository https://gitbox.apache.org/repos/asf/mesos.git

commit 4d853875c12ee56e206d6f82b51b78dc37234696
Author: Vinod Kone 
AuthorDate: Sat Feb 2 09:51:09 2019 -0600

Fixed variable names in `Master::_doRegistryGC()`.

Substituted `slave` with `slaveId` to be consistent with the code base.
No functional changes.

Review: https://reviews.apache.org/r/69907
---
 src/master/master.cpp | 22 +++---
 1 file changed, 11 insertions(+), 11 deletions(-)

diff --git a/src/master/master.cpp b/src/master/master.cpp
index 36b167f..c22caf3 100644
--- a/src/master/master.cpp
+++ b/src/master/master.cpp
@@ -1879,7 +1879,7 @@ void Master::doRegistryGc()
 TimeInfo currentTime = protobuf::getCurrentTime();
 hashset toRemove;
 
-foreachpair (const SlaveID& slave,
+foreachpair (const SlaveID& slaveId,
  const TimeInfo& removalTime,
  slaves) {
   // Count-based GC.
@@ -1887,7 +1887,7 @@ void Master::doRegistryGc()
 
   size_t liveCount = count - toRemove.size();
   if (liveCount > flags.registry_max_agent_count) {
-toRemove.insert(slave);
+toRemove.insert(slaveId);
 continue;
   }
 
@@ -1896,7 +1896,7 @@ void Master::doRegistryGc()
   currentTime.nanoseconds() - removalTime.nanoseconds());
 
   if (age > flags.registry_max_agent_age) {
-toRemove.insert(slave);
+toRemove.insert(slaveId);
   }
 }
 
@@ -1948,26 +1948,26 @@ void Master::_doRegistryGc(
   // operation, but there isn't an easy way to do that.
 
   size_t numRemovedUnreachable = 0;
-  foreach (const SlaveID& slave, toRemoveUnreachable) {
-if (!slaves.unreachable.contains(slave)) {
-  LOG(WARNING) << "Failed to garbage collect " << slave
+  foreach (const SlaveID& slaveId, toRemoveUnreachable) {
+if (!slaves.unreachable.contains(slaveId)) {
+  LOG(WARNING) << "Failed to garbage collect " << slaveId
<< " from the unreachable list";
   continue;
 }
 
-slaves.unreachable.erase(slave);
+slaves.unreachable.erase(slaveId);
 numRemovedUnreachable++;
   }
 
   size_t numRemovedGone = 0;
-  foreach (const SlaveID& slave, toRemoveGone) {
-if (!slaves.gone.contains(slave)) {
-  LOG(WARNING) << "Failed to garbage collect " << slave
+  foreach (const SlaveID& slaveId, toRemoveGone) {
+if (!slaves.gone.contains(slaveId)) {
+  LOG(WARNING) << "Failed to garbage collect " << slaveId
<< " from the gone list";
   continue;
 }
 
-slaves.gone.erase(slave);
+slaves.gone.erase(slaveId);
 numRemovedGone++;
   }
 

[mesos] 04/04: Added MESOS-8887 to the 1.5.3 CHANGELOG.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch 1.5.x
in repository https://gitbox.apache.org/repos/asf/mesos.git

commit 47667c2d634f7a9f80d99bfe0410516b69885599
Author: Vinod Kone 
AuthorDate: Fri Feb 15 14:51:22 2019 -0600

Added MESOS-8887 to the 1.5.3 CHANGELOG.
---
 CHANGELOG | 1 +
 1 file changed, 1 insertion(+)

diff --git a/CHANGELOG b/CHANGELOG
index bd471bb..2e42c98 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -3,6 +3,7 @@ Release Notes - Mesos - Version 1.5.3 (WIP)
 * This is a bug fix release.
 
 ** Bug
+  * [MESOS-8887] - Unreachable tasks are not GC'ed when unreachable agent is 
GC'ed.
   * [MESOS-9210] - Mesos v1 scheduler library does not properly handle 
SUBSCRIBE retries.
   * [MESOS-9532] - ResourceOffersTest.ResourceOfferWithMultipleSlaves is flaky.
   * [MESOS-9533] - CniIsolatorTest.ROOT_CleanupAfterReboot is flaky.

[mesos] 02/04: Removed unreachable tasks from `Master::Framework` on agent GC.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch 1.5.x
in repository https://gitbox.apache.org/repos/asf/mesos.git

commit f049981470791c59e48732930fd24110af5e2c75
Author: Vinod Kone 
AuthorDate: Sat Feb 2 10:01:56 2019 -0600

Removed unreachable tasks from `Master::Framework` on agent GC.

NOTE: Modified code during cherry-picking.

Unreachable tasks are stored in `Framework` struct of the master, but
they were not removed when an unreachable agent is GCed from the
registry. This patch fixes it so that the unreachable tasks are also
cleaned up.

Review: https://reviews.apache.org/r/69908
---
 src/master/master.cpp | 15 +++
 1 file changed, 15 insertions(+)

diff --git a/src/master/master.cpp b/src/master/master.cpp
index c22caf3..995ff55 100644
--- a/src/master/master.cpp
+++ b/src/master/master.cpp
@@ -1956,6 +1956,21 @@ void Master::_doRegistryGc(
 }
 
 slaves.unreachable.erase(slaveId);
+
+// TODO(vinod): Consider moving these tasks into `completedTasks` by
+// transitioning them to a terminal state and sending status updates.
+// But it's not clear what this state should be. If a framework
+// reconciles these tasks after this point it would get `TASK_UNKNOWN`
+// which seems appropriate but we don't keep tasks in this state in-memory.
+foreachvalue (Framework* framework, frameworks.registered) {
+  foreach (const TaskID& taskId, framework->unreachableTasks.keys()) {
+const Owned& task = framework->unreachableTasks.at(taskId);
+if (task->slave_id() ==  slaveId) {
+  framework->unreachableTasks.erase(taskId);
+}
+  }
+}
+
 numRemovedUnreachable++;
   }
 

[mesos] branch 1.5.x updated (467282e -> 47667c2)

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a change to branch 1.5.x
in repository https://gitbox.apache.org/repos/asf/mesos.git.


from 467282e  Added MESOS-9555 to the 1.5.3 CHANGELOG.
 new 4d85387  Fixed variable names in `Master::_doRegistryGC()`.
 new f049981  Removed unreachable tasks from `Master::Framework` on agent 
GC.
 new a91b087  Tested unreachable task behavior on agent GC.
 new 47667c2  Added MESOS-8887 to the 1.5.3 CHANGELOG.

The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 CHANGELOG |  1 +
 src/master/master.cpp | 37 ++---
 src/tests/partition_tests.cpp | 40 
 3 files changed, 59 insertions(+), 19 deletions(-)

[mesos] 03/04: Tested unreachable task behavior on agent GC.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch 1.5.x
in repository https://gitbox.apache.org/repos/asf/mesos.git

commit a91b08778b96a0d6b47d96ece4c817eb6daaa0c8
Author: Vinod Kone 
AuthorDate: Tue Feb 5 16:55:19 2019 -0600

Tested unreachable task behavior on agent GC.

Updated `PartitionTest, RegistryGcByCount` test. This test fails
without the previous patch.

Review: https://reviews.apache.org/r/69909
---
 src/tests/partition_tests.cpp | 40 
 1 file changed, 32 insertions(+), 8 deletions(-)

diff --git a/src/tests/partition_tests.cpp b/src/tests/partition_tests.cpp
index e7f4ebe..2f121ef 100644
--- a/src/tests/partition_tests.cpp
+++ b/src/tests/partition_tests.cpp
@@ -2603,16 +2603,30 @@ TEST_F_TEMP_DISABLED_ON_WINDOWS(PartitionTest, 
RegistryGcByCount)
 
   EXPECT_CALL(sched, registered(, _, _));
 
-  Future resourceOffers;
+  Future> offers;
   EXPECT_CALL(sched, resourceOffers(, _))
-.WillOnce(FutureSatisfy())
+.WillOnce(FutureArg<1>())
 .WillRepeatedly(Return()); // Ignore subsequent offers.
 
   driver.start();
 
   // Need to make sure the framework AND slave have registered with
   // master. Waiting for resource offers should accomplish both.
-  AWAIT_READY(resourceOffers);
+  AWAIT_READY(offers);
+
+  TaskInfo task = createTask(offers->at(0), SLEEP_COMMAND(60));
+
+  Future unreachableStatus;
+  EXPECT_CALL(sched, statusUpdate(, _))
+.WillOnce(FutureArg<1>());
+
+  // Drop RunTaskMessage so that the task stays in TASK_STAGING.
+  Future runTaskMessage =
+DROP_PROTOBUF(RunTaskMessage(), _, _);
+
+  driver.launchTasks(offers->at(0).id(), {task});
+
+  AWAIT_READY(runTaskMessage);
 
   EXPECT_CALL(sched, offerRescinded(, _))
 .WillRepeatedly(Return());
@@ -2644,6 +2658,8 @@ TEST_F_TEMP_DISABLED_ON_WINDOWS(PartitionTest, 
RegistryGcByCount)
 
   Clock::advance(Milliseconds(100));
 
+  AWAIT_READY(unreachableStatus);
+
   AWAIT_READY(slaveLost1);
 
   // Shutdown the first slave. This is necessary because we only drop
@@ -2701,13 +2717,13 @@ TEST_F_TEMP_DISABLED_ON_WINDOWS(PartitionTest, 
RegistryGcByCount)
   slave2.get()->terminate();
   slave2->reset();
 
-  // Do explicit reconciliation for a random task ID on `slave1`. GC
+  // Do explicit reconciliation for the task launched on `slave1`. GC
   // has not occurred yet (since `registry_gc_interval` has not
   // elapsed since the master was started), so the slave should be in
   // the unreachable list; hence `unreachable_time` should be set on
   // the result of the reconciliation request.
   TaskStatus status1;
-  status1.mutable_task_id()->set_value(id::UUID::random().toString());
+  status1.mutable_task_id()->CopyFrom(task.task_id());
   status1.mutable_slave_id()->CopyFrom(slaveId1);
   status1.set_state(TASK_STAGING); // Dummy value.
 
@@ -2722,15 +2738,23 @@ TEST_F_TEMP_DISABLED_ON_WINDOWS(PartitionTest, 
RegistryGcByCount)
   EXPECT_EQ(TaskStatus::REASON_RECONCILIATION, reconcileUpdate1->reason());
   EXPECT_EQ(partitionTime1, reconcileUpdate1->unreachable_time());
 
+  JSON::Object stats = Metrics();
+  EXPECT_EQ(1, stats.values["master/tasks_unreachable"]);
+
   // Advance the clock to cause GC to be performed.
   Clock::advance(masterFlags.registry_gc_interval);
   Clock::settle();
 
-  // Do explicit reconciliation for a random task ID on `slave1`.
+  // Ensure task has been removed from unreachable list once the
+  // agent is GC'ed.
+  stats = Metrics();
+  EXPECT_EQ(0, stats.values["master/tasks_unreachable"]);
+
+  // Do explicit reconciliation for the task launched on `slave1`.
   // Because the agent has been removed from the unreachable list in
   // the registry, `unreachable_time` should NOT be set.
   TaskStatus status2;
-  status2.mutable_task_id()->set_value(id::UUID::random().toString());
+  status2.mutable_task_id()->CopyFrom(task.task_id());
   status2.mutable_slave_id()->CopyFrom(slaveId1);
   status2.set_state(TASK_STAGING); // Dummy value.
 
@@ -2764,7 +2788,7 @@ TEST_F_TEMP_DISABLED_ON_WINDOWS(PartitionTest, 
RegistryGcByCount)
   EXPECT_EQ(TaskStatus::REASON_RECONCILIATION, reconcileUpdate3->reason());
   EXPECT_EQ(partitionTime2, reconcileUpdate3->unreachable_time());
 
-  JSON::Object stats = Metrics();
+  stats = Metrics();
   EXPECT_EQ(2, stats.values["master/slave_unreachable_scheduled"]);
   EXPECT_EQ(2, stats.values["master/slave_unreachable_completed"]);
   EXPECT_EQ(2, stats.values["master/slave_removals"]);

[mesos] 03/04: Tested unreachable task behavior on agent GC.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch 1.6.x
in repository https://gitbox.apache.org/repos/asf/mesos.git

commit aee50e6c7fbb3a2564eb20eefad599a60b6ab742
Author: Vinod Kone 
AuthorDate: Tue Feb 5 16:55:19 2019 -0600

Tested unreachable task behavior on agent GC.

Updated `PartitionTest, RegistryGcByCount` test. This test fails
without the previous patch.

Review: https://reviews.apache.org/r/69909
---
 src/tests/partition_tests.cpp | 40 
 1 file changed, 32 insertions(+), 8 deletions(-)

diff --git a/src/tests/partition_tests.cpp b/src/tests/partition_tests.cpp
index 390b7c8..0746fa0 100644
--- a/src/tests/partition_tests.cpp
+++ b/src/tests/partition_tests.cpp
@@ -2928,16 +2928,30 @@ TEST_F_TEMP_DISABLED_ON_WINDOWS(PartitionTest, 
RegistryGcByCount)
 
   EXPECT_CALL(sched, registered(, _, _));
 
-  Future resourceOffers;
+  Future> offers;
   EXPECT_CALL(sched, resourceOffers(, _))
-.WillOnce(FutureSatisfy())
+.WillOnce(FutureArg<1>())
 .WillRepeatedly(Return()); // Ignore subsequent offers.
 
   driver.start();
 
   // Need to make sure the framework AND slave have registered with
   // master. Waiting for resource offers should accomplish both.
-  AWAIT_READY(resourceOffers);
+  AWAIT_READY(offers);
+
+  TaskInfo task = createTask(offers->at(0), SLEEP_COMMAND(60));
+
+  Future unreachableStatus;
+  EXPECT_CALL(sched, statusUpdate(, _))
+.WillOnce(FutureArg<1>());
+
+  // Drop RunTaskMessage so that the task stays in TASK_STAGING.
+  Future runTaskMessage =
+DROP_PROTOBUF(RunTaskMessage(), _, _);
+
+  driver.launchTasks(offers->at(0).id(), {task});
+
+  AWAIT_READY(runTaskMessage);
 
   EXPECT_CALL(sched, offerRescinded(, _))
 .WillRepeatedly(Return());
@@ -2969,6 +2983,8 @@ TEST_F_TEMP_DISABLED_ON_WINDOWS(PartitionTest, 
RegistryGcByCount)
 
   Clock::advance(Milliseconds(100));
 
+  AWAIT_READY(unreachableStatus);
+
   AWAIT_READY(slaveLost1);
 
   // Shutdown the first slave. This is necessary because we only drop
@@ -3026,13 +3042,13 @@ TEST_F_TEMP_DISABLED_ON_WINDOWS(PartitionTest, 
RegistryGcByCount)
   slave2.get()->terminate();
   slave2->reset();
 
-  // Do explicit reconciliation for a random task ID on `slave1`. GC
+  // Do explicit reconciliation for the task launched on `slave1`. GC
   // has not occurred yet (since `registry_gc_interval` has not
   // elapsed since the master was started), so the slave should be in
   // the unreachable list; hence `unreachable_time` should be set on
   // the result of the reconciliation request.
   TaskStatus status1;
-  status1.mutable_task_id()->set_value(id::UUID::random().toString());
+  status1.mutable_task_id()->CopyFrom(task.task_id());
   status1.mutable_slave_id()->CopyFrom(slaveId1);
   status1.set_state(TASK_STAGING); // Dummy value.
 
@@ -3047,15 +3063,23 @@ TEST_F_TEMP_DISABLED_ON_WINDOWS(PartitionTest, 
RegistryGcByCount)
   EXPECT_EQ(TaskStatus::REASON_RECONCILIATION, reconcileUpdate1->reason());
   EXPECT_EQ(partitionTime1, reconcileUpdate1->unreachable_time());
 
+  JSON::Object stats = Metrics();
+  EXPECT_EQ(1, stats.values["master/tasks_unreachable"]);
+
   // Advance the clock to cause GC to be performed.
   Clock::advance(masterFlags.registry_gc_interval);
   Clock::settle();
 
-  // Do explicit reconciliation for a random task ID on `slave1`.
+  // Ensure task has been removed from unreachable list once the
+  // agent is GC'ed.
+  stats = Metrics();
+  EXPECT_EQ(0, stats.values["master/tasks_unreachable"]);
+
+  // Do explicit reconciliation for the task launched on `slave1`.
   // Because the agent has been removed from the unreachable list in
   // the registry, `unreachable_time` should NOT be set.
   TaskStatus status2;
-  status2.mutable_task_id()->set_value(id::UUID::random().toString());
+  status2.mutable_task_id()->CopyFrom(task.task_id());
   status2.mutable_slave_id()->CopyFrom(slaveId1);
   status2.set_state(TASK_STAGING); // Dummy value.
 
@@ -3089,7 +3113,7 @@ TEST_F_TEMP_DISABLED_ON_WINDOWS(PartitionTest, 
RegistryGcByCount)
   EXPECT_EQ(TaskStatus::REASON_RECONCILIATION, reconcileUpdate3->reason());
   EXPECT_EQ(partitionTime2, reconcileUpdate3->unreachable_time());
 
-  JSON::Object stats = Metrics();
+  stats = Metrics();
   EXPECT_EQ(2, stats.values["master/slave_unreachable_scheduled"]);
   EXPECT_EQ(2, stats.values["master/slave_unreachable_completed"]);
   EXPECT_EQ(2, stats.values["master/slave_removals"]);

[mesos] 01/04: Fixed variable names in `Master::_doRegistryGC()`.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch 1.6.x
in repository https://gitbox.apache.org/repos/asf/mesos.git

commit 5ec28f92096a22cfbf87594cf9dd1ac1d58d1047
Author: Vinod Kone 
AuthorDate: Sat Feb 2 09:51:09 2019 -0600

Fixed variable names in `Master::_doRegistryGC()`.

Substituted `slave` with `slaveId` to be consistent with the code base.
No functional changes.

Review: https://reviews.apache.org/r/69907
---
 src/master/master.cpp | 24 
 1 file changed, 12 insertions(+), 12 deletions(-)

diff --git a/src/master/master.cpp b/src/master/master.cpp
index 8776663..e32d157 100644
--- a/src/master/master.cpp
+++ b/src/master/master.cpp
@@ -1869,7 +1869,7 @@ void Master::doRegistryGc()
 TimeInfo currentTime = protobuf::getCurrentTime();
 hashset toRemove;
 
-foreachpair (const SlaveID& slave,
+foreachpair (const SlaveID& slaveId,
  const TimeInfo& removalTime,
  slaves) {
   // Count-based GC.
@@ -1877,7 +1877,7 @@ void Master::doRegistryGc()
 
   size_t liveCount = count - toRemove.size();
   if (liveCount > flags.registry_max_agent_count) {
-toRemove.insert(slave);
+toRemove.insert(slaveId);
 continue;
   }
 
@@ -1886,7 +1886,7 @@ void Master::doRegistryGc()
   currentTime.nanoseconds() - removalTime.nanoseconds());
 
   if (age > flags.registry_max_agent_age) {
-toRemove.insert(slave);
+toRemove.insert(slaveId);
   }
 }
 
@@ -1939,29 +1939,29 @@ void Master::_doRegistryGc(
   // operation, but there isn't an easy way to do that.
 
   size_t numRemovedUnreachable = 0;
-  foreach (const SlaveID& slave, toRemoveUnreachable) {
-if (!slaves.unreachable.contains(slave)) {
-  LOG(WARNING) << "Failed to garbage collect " << slave
+  foreach (const SlaveID& slaveId, toRemoveUnreachable) {
+if (!slaves.unreachable.contains(slaveId)) {
+  LOG(WARNING) << "Failed to garbage collect " << slaveId
<< " from the unreachable list";
 
   continue;
 }
 
-slaves.unreachable.erase(slave);
-slaves.unreachableTasks.erase(slave);
+slaves.unreachable.erase(slaveId);
+slaves.unreachableTasks.erase(slaveId);
 numRemovedUnreachable++;
   }
 
   size_t numRemovedGone = 0;
-  foreach (const SlaveID& slave, toRemoveGone) {
-if (!slaves.gone.contains(slave)) {
-  LOG(WARNING) << "Failed to garbage collect " << slave
+  foreach (const SlaveID& slaveId, toRemoveGone) {
+if (!slaves.gone.contains(slaveId)) {
+  LOG(WARNING) << "Failed to garbage collect " << slaveId
<< " from the gone list";
 
   continue;
 }
 
-slaves.gone.erase(slave);
+slaves.gone.erase(slaveId);
 numRemovedGone++;
   }
 

[mesos] 02/04: Removed unreachable tasks from `Master::Framework` on agent GC.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch 1.6.x
in repository https://gitbox.apache.org/repos/asf/mesos.git

commit e7cff7afe99dbfc20a45d0435bed557b2595f772
Author: Vinod Kone 
AuthorDate: Sat Feb 2 10:01:56 2019 -0600

Removed unreachable tasks from `Master::Framework` on agent GC.

Unreachable tasks are stored in `Slaves` and `Framework` structs of
the master, but they were only being removed from the former when
an unreachable agent is GCed from the registry. This patch fixes it
so that the latter is also cleaned up.

Review: https://reviews.apache.org/r/69908
---
 src/master/master.cpp | 20 
 1 file changed, 20 insertions(+)

diff --git a/src/master/master.cpp b/src/master/master.cpp
index e32d157..77629e4 100644
--- a/src/master/master.cpp
+++ b/src/master/master.cpp
@@ -1948,7 +1948,27 @@ void Master::_doRegistryGc(
 }
 
 slaves.unreachable.erase(slaveId);
+
+// TODO(vinod): Consider moving these tasks into `completedTasks` by
+// transitioning them to a terminal state and sending status updates.
+// But it's not clear what this state should be. If a framework
+// reconciles these tasks after this point it would get `TASK_UNKNOWN`
+// which seems appropriate but we don't keep tasks in this state in-memory.
+if (slaves.unreachableTasks.contains(slaveId)) {
+  foreachkey (const FrameworkID& frameworkId,
+  slaves.unreachableTasks.at(slaveId)) {
+Framework* framework = getFramework(frameworkId);
+if (framework != nullptr) {
+  foreach (const TaskID& taskId,
+   slaves.unreachableTasks.at(slaveId).get(frameworkId)) {
+framework->unreachableTasks.erase(taskId);
+  }
+}
+  }
+}
+
 slaves.unreachableTasks.erase(slaveId);
+
 numRemovedUnreachable++;
   }
 

[mesos] branch 1.6.x updated (01b4a07 -> 577008c)

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a change to branch 1.6.x
in repository https://gitbox.apache.org/repos/asf/mesos.git.


from 01b4a07  Added MESOS-9555 to the 1.6.2 CHANGELOG.
 new 5ec28f9  Fixed variable names in `Master::_doRegistryGC()`.
 new e7cff7a  Removed unreachable tasks from `Master::Framework` on agent 
GC.
 new aee50e6  Tested unreachable task behavior on agent GC.
 new 577008c  Added MESOS-8887 to the 1.6.2 CHANGELOG.

The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 CHANGELOG |  1 +
 src/master/master.cpp | 44 +++
 src/tests/partition_tests.cpp | 40 +++
 3 files changed, 65 insertions(+), 20 deletions(-)

[mesos] 04/04: Added MESOS-8887 to the 1.6.2 CHANGELOG.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch 1.6.x
in repository https://gitbox.apache.org/repos/asf/mesos.git

commit 577008c2b5fcdc8a9c1be6e1e35dd8588083488d
Author: Vinod Kone 
AuthorDate: Fri Feb 15 14:38:15 2019 -0600

Added MESOS-8887 to the 1.6.2 CHANGELOG.
---
 CHANGELOG | 1 +
 1 file changed, 1 insertion(+)

diff --git a/CHANGELOG b/CHANGELOG
index 0f4db4a..8adafbb 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -8,6 +8,7 @@ Release Notes - Mesos - Version 1.6.2 (WIP)
   * [MESOS-8418] - mesos-agent high cpu usage because of numerous /proc/mounts 
reads.
   * [MESOS-8545] - AgentAPIStreamingTest.AttachInputToNestedContainerSession 
is flaky.
   * [MESOS-8568] - Command checks should always call `WAIT_NESTED_CONTAINER` 
before `REMOVE_NESTED_CONTAINER`
+  * [MESOS-8887] - Unreachable tasks are not GC'ed when unreachable agent is 
GC'ed.
   * [MESOS-8907] - Docker image fetcher fails with HTTP/2.
   * [MESOS-8917] - Agent leaking file descriptors into forked processes
   * [MESOS-8921] - Autotools don't work with newer OpenJDK versions

[mesos] 01/04: Fixed variable names in `Master::_doRegistryGC()`.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch 1.7.x
in repository https://gitbox.apache.org/repos/asf/mesos.git

commit f0cd3b7b62807fe377b1b47bf1bf364b18c4a373
Author: Vinod Kone 
AuthorDate: Sat Feb 2 09:51:09 2019 -0600

Fixed variable names in `Master::_doRegistryGC()`.

Substituted `slave` with `slaveId` to be consistent with the code base.
No functional changes.

Review: https://reviews.apache.org/r/69907
---
 src/master/master.cpp | 24 
 1 file changed, 12 insertions(+), 12 deletions(-)

diff --git a/src/master/master.cpp b/src/master/master.cpp
index 873d685..354a512 100644
--- a/src/master/master.cpp
+++ b/src/master/master.cpp
@@ -1820,7 +1820,7 @@ void Master::doRegistryGc()
 TimeInfo currentTime = protobuf::getCurrentTime();
 hashset toRemove;
 
-foreachpair (const SlaveID& slave,
+foreachpair (const SlaveID& slaveId,
  const TimeInfo& removalTime,
  slaves) {
   // Count-based GC.
@@ -1828,7 +1828,7 @@ void Master::doRegistryGc()
 
   size_t liveCount = count - toRemove.size();
   if (liveCount > flags.registry_max_agent_count) {
-toRemove.insert(slave);
+toRemove.insert(slaveId);
 continue;
   }
 
@@ -1837,7 +1837,7 @@ void Master::doRegistryGc()
   currentTime.nanoseconds() - removalTime.nanoseconds());
 
   if (age > flags.registry_max_agent_age) {
-toRemove.insert(slave);
+toRemove.insert(slaveId);
   }
 }
 
@@ -1890,29 +1890,29 @@ void Master::_doRegistryGc(
   // operation, but there isn't an easy way to do that.
 
   size_t numRemovedUnreachable = 0;
-  foreach (const SlaveID& slave, toRemoveUnreachable) {
-if (!slaves.unreachable.contains(slave)) {
-  LOG(WARNING) << "Failed to garbage collect " << slave
+  foreach (const SlaveID& slaveId, toRemoveUnreachable) {
+if (!slaves.unreachable.contains(slaveId)) {
+  LOG(WARNING) << "Failed to garbage collect " << slaveId
<< " from the unreachable list";
 
   continue;
 }
 
-slaves.unreachable.erase(slave);
-slaves.unreachableTasks.erase(slave);
+slaves.unreachable.erase(slaveId);
+slaves.unreachableTasks.erase(slaveId);
 numRemovedUnreachable++;
   }
 
   size_t numRemovedGone = 0;
-  foreach (const SlaveID& slave, toRemoveGone) {
-if (!slaves.gone.contains(slave)) {
-  LOG(WARNING) << "Failed to garbage collect " << slave
+  foreach (const SlaveID& slaveId, toRemoveGone) {
+if (!slaves.gone.contains(slaveId)) {
+  LOG(WARNING) << "Failed to garbage collect " << slaveId
<< " from the gone list";
 
   continue;
 }
 
-slaves.gone.erase(slave);
+slaves.gone.erase(slaveId);
 numRemovedGone++;
   }
 

[mesos] branch 1.7.x updated (d018bab -> 6fcf701)

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a change to branch 1.7.x
in repository https://gitbox.apache.org/repos/asf/mesos.git.


from d018bab  Added MESOS-9555 to the 1.7.2 CHANGELOG.
 new f0cd3b7  Fixed variable names in `Master::_doRegistryGC()`.
 new c72a4f9  Removed unreachable tasks from `Master::Framework` on agent 
GC.
 new 1a506a4  Tested unreachable task behavior on agent GC.
 new 6fcf701  Added MESOS-8887 to the 1.7.2 CHANGELOG.

The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 CHANGELOG |  1 +
 src/master/master.cpp | 44 +++
 src/tests/partition_tests.cpp | 40 +++
 3 files changed, 65 insertions(+), 20 deletions(-)

[mesos] 02/04: Removed unreachable tasks from `Master::Framework` on agent GC.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch 1.7.x
in repository https://gitbox.apache.org/repos/asf/mesos.git

commit c72a4f909054e5efa75d9e5d8dde71b0083402c1
Author: Vinod Kone 
AuthorDate: Sat Feb 2 10:01:56 2019 -0600

Removed unreachable tasks from `Master::Framework` on agent GC.

Unreachable tasks are stored in `Slaves` and `Framework` structs of
the master, but they were only being removed from the former when
an unreachable agent is GCed from the registry. This patch fixes it
so that the latter is also cleaned up.

Review: https://reviews.apache.org/r/69908
---
 src/master/master.cpp | 20 
 1 file changed, 20 insertions(+)

diff --git a/src/master/master.cpp b/src/master/master.cpp
index 354a512..a6bc23f 100644
--- a/src/master/master.cpp
+++ b/src/master/master.cpp
@@ -1899,7 +1899,27 @@ void Master::_doRegistryGc(
 }
 
 slaves.unreachable.erase(slaveId);
+
+// TODO(vinod): Consider moving these tasks into `completedTasks` by
+// transitioning them to a terminal state and sending status updates.
+// But it's not clear what this state should be. If a framework
+// reconciles these tasks after this point it would get `TASK_UNKNOWN`
+// which seems appropriate but we don't keep tasks in this state in-memory.
+if (slaves.unreachableTasks.contains(slaveId)) {
+  foreachkey (const FrameworkID& frameworkId,
+  slaves.unreachableTasks.at(slaveId)) {
+Framework* framework = getFramework(frameworkId);
+if (framework != nullptr) {
+  foreach (const TaskID& taskId,
+   slaves.unreachableTasks.at(slaveId).get(frameworkId)) {
+framework->unreachableTasks.erase(taskId);
+  }
+}
+  }
+}
+
 slaves.unreachableTasks.erase(slaveId);
+
 numRemovedUnreachable++;
   }
 

[mesos] 04/04: Added MESOS-8887 to the 1.7.2 CHANGELOG.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch 1.7.x
in repository https://gitbox.apache.org/repos/asf/mesos.git

commit 6fcf70167076bbe6fb10ca04876939fe0e3379d9
Author: Vinod Kone 
AuthorDate: Fri Feb 15 14:33:00 2019 -0600

Added MESOS-8887 to the 1.7.2 CHANGELOG.
---
 CHANGELOG | 1 +
 1 file changed, 1 insertion(+)

diff --git a/CHANGELOG b/CHANGELOG
index 7cd5bf6..922e750 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -3,6 +3,7 @@ Release Notes - Mesos - Version 1.7.2 (WIP)
 * This is a bug fix release.
 
 ** Bug
+  * [MESOS-8887] - Unreachable tasks are not GC'ed when unreachable agent is 
GC'ed.
   * [MESOS-9210] - Mesos v1 scheduler library does not properly handle 
SUBSCRIBE retries.
   * [MESOS-9517] - SLRP should treat gRPC timeouts as non-terminal errors, 
instead of reporting OPERATION_FAILED.
   * [MESOS-9531] - chown error handling is incorrect in createSandboxDirectory.

[mesos] branch master updated: Added MESOS-8887 to the 1.6.2 CHANGELOG.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git


The following commit(s) were added to refs/heads/master by this push:
 new dbdc088  Added MESOS-8887 to the 1.6.2 CHANGELOG.
dbdc088 is described below

commit dbdc088f6b8d9e95c5b5622f986e5a417cc3d3f6
Author: Vinod Kone 
AuthorDate: Fri Feb 15 14:38:15 2019 -0600

Added MESOS-8887 to the 1.6.2 CHANGELOG.
---
 CHANGELOG | 1 +
 1 file changed, 1 insertion(+)

diff --git a/CHANGELOG b/CHANGELOG
index 51b0a97..44267f8 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -452,6 +452,7 @@ Release Notes - Mesos - Version 1.6.2 (WIP)
   * [MESOS-8418] - mesos-agent high cpu usage because of numerous /proc/mounts 
reads.
   * [MESOS-8545] - AgentAPIStreamingTest.AttachInputToNestedContainerSession 
is flaky.
   * [MESOS-8568] - Command checks should always call `WAIT_NESTED_CONTAINER` 
before `REMOVE_NESTED_CONTAINER`
+  * [MESOS-8887] - Unreachable tasks are not GC'ed when unreachable agent is 
GC'ed.
   * [MESOS-8907] - Docker image fetcher fails with HTTP/2.
   * [MESOS-8917] - Agent leaking file descriptors into forked processes
   * [MESOS-8921] - Autotools don't work with newer OpenJDK versions

[mesos] branch master updated: Added MESOS-8887 to the 1.7.2 CHANGELOG.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git


The following commit(s) were added to refs/heads/master by this push:
 new 35dd1b8  Added MESOS-8887 to the 1.7.2 CHANGELOG.
35dd1b8 is described below

commit 35dd1b8515ca52d0f3c9ce14cee15396a74a521d
Author: Vinod Kone 
AuthorDate: Fri Feb 15 14:33:00 2019 -0600

Added MESOS-8887 to the 1.7.2 CHANGELOG.
---
 CHANGELOG | 1 +
 1 file changed, 1 insertion(+)

diff --git a/CHANGELOG b/CHANGELOG
index 5b858a3..51b0a97 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -3,6 +3,7 @@ Release Notes - Mesos - Version 1.7.2 (WIP)
 * This is a bug fix release.
 
 ** Bug
+  * [MESOS-8887] - Unreachable tasks are not GC'ed when unreachable agent is 
GC'ed.
   * [MESOS-9210] - Mesos v1 scheduler library does not properly handle 
SUBSCRIBE retries.
   * [MESOS-9517] - SLRP should treat gRPC timeouts as non-terminal errors, 
instead of reporting OPERATION_FAILED.
   * [MESOS-9531] - chown error handling is incorrect in createSandboxDirectory.

[mesos] branch master updated (7a52eef -> 38e41c5)

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git.


from 7a52eef  Added MESOS-9555 to the 1.5.3 CHANGELOG.
 new e10034f  Fixed variable names in `Master::_doRegistryGC()`.
 new d794ab7  Removed unreachable tasks from `Master::Framework` on agent 
GC.
 new 38e41c5  Tested unreachable task behavior on agent GC.

The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 src/master/master.cpp | 44 +++
 src/tests/partition_tests.cpp | 40 +++
 2 files changed, 64 insertions(+), 20 deletions(-)

[mesos] 02/03: Removed unreachable tasks from `Master::Framework` on agent GC.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git

commit d794ab792f102aa6642b66b7b5dd9520ef283d94
Author: Vinod Kone 
AuthorDate: Sat Feb 2 10:01:56 2019 -0600

Removed unreachable tasks from `Master::Framework` on agent GC.

Unreachable tasks are stored in `Slaves` and `Framework` structs of
the master, but they were only being removed from the former when
an unreachable agent is GCed from the registry. This patch fixes it
so that the latter is also cleaned up.

Review: https://reviews.apache.org/r/69908
---
 src/master/master.cpp | 20 
 1 file changed, 20 insertions(+)

diff --git a/src/master/master.cpp b/src/master/master.cpp
index 4870227..106d924 100644
--- a/src/master/master.cpp
+++ b/src/master/master.cpp
@@ -1875,7 +1875,27 @@ void Master::_doRegistryGc(
 }
 
 slaves.unreachable.erase(slaveId);
+
+// TODO(vinod): Consider moving these tasks into `completedTasks` by
+// transitioning them to a terminal state and sending status updates.
+// But it's not clear what this state should be. If a framework
+// reconciles these tasks after this point it would get `TASK_UNKNOWN`
+// which seems appropriate but we don't keep tasks in this state in-memory.
+if (slaves.unreachableTasks.contains(slaveId)) {
+  foreachkey (const FrameworkID& frameworkId,
+  slaves.unreachableTasks.at(slaveId)) {
+Framework* framework = getFramework(frameworkId);
+if (framework != nullptr) {
+  foreach (const TaskID& taskId,
+   slaves.unreachableTasks.at(slaveId).get(frameworkId)) {
+framework->unreachableTasks.erase(taskId);
+  }
+}
+  }
+}
+
 slaves.unreachableTasks.erase(slaveId);
+
 numRemovedUnreachable++;
   }
 

[mesos] 01/03: Fixed variable names in `Master::_doRegistryGC()`.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git

commit e10034fb4fede1aaf58b25c7e9fe3e677826350b
Author: Vinod Kone 
AuthorDate: Sat Feb 2 09:51:09 2019 -0600

Fixed variable names in `Master::_doRegistryGC()`.

Substituted `slave` with `slaveId` to be consistent with the code base.
No functional changes.

Review: https://reviews.apache.org/r/69907
---
 src/master/master.cpp | 24 
 1 file changed, 12 insertions(+), 12 deletions(-)

diff --git a/src/master/master.cpp b/src/master/master.cpp
index 8e23785..4870227 100644
--- a/src/master/master.cpp
+++ b/src/master/master.cpp
@@ -1796,7 +1796,7 @@ void Master::doRegistryGc()
 TimeInfo currentTime = protobuf::getCurrentTime();
 hashset toRemove;
 
-foreachpair (const SlaveID& slave,
+foreachpair (const SlaveID& slaveId,
  const TimeInfo& removalTime,
  slaves) {
   // Count-based GC.
@@ -1804,7 +1804,7 @@ void Master::doRegistryGc()
 
   size_t liveCount = count - toRemove.size();
   if (liveCount > flags.registry_max_agent_count) {
-toRemove.insert(slave);
+toRemove.insert(slaveId);
 continue;
   }
 
@@ -1813,7 +1813,7 @@ void Master::doRegistryGc()
   currentTime.nanoseconds() - removalTime.nanoseconds());
 
   if (age > flags.registry_max_agent_age) {
-toRemove.insert(slave);
+toRemove.insert(slaveId);
   }
 }
 
@@ -1866,29 +1866,29 @@ void Master::_doRegistryGc(
   // operation, but there isn't an easy way to do that.
 
   size_t numRemovedUnreachable = 0;
-  foreach (const SlaveID& slave, toRemoveUnreachable) {
-if (!slaves.unreachable.contains(slave)) {
-  LOG(WARNING) << "Failed to garbage collect " << slave
+  foreach (const SlaveID& slaveId, toRemoveUnreachable) {
+if (!slaves.unreachable.contains(slaveId)) {
+  LOG(WARNING) << "Failed to garbage collect " << slaveId
<< " from the unreachable list";
 
   continue;
 }
 
-slaves.unreachable.erase(slave);
-slaves.unreachableTasks.erase(slave);
+slaves.unreachable.erase(slaveId);
+slaves.unreachableTasks.erase(slaveId);
 numRemovedUnreachable++;
   }
 
   size_t numRemovedGone = 0;
-  foreach (const SlaveID& slave, toRemoveGone) {
-if (!slaves.gone.contains(slave)) {
-  LOG(WARNING) << "Failed to garbage collect " << slave
+  foreach (const SlaveID& slaveId, toRemoveGone) {
+if (!slaves.gone.contains(slaveId)) {
+  LOG(WARNING) << "Failed to garbage collect " << slaveId
<< " from the gone list";
 
   continue;
 }
 
-slaves.gone.erase(slave);
+slaves.gone.erase(slaveId);
 numRemovedGone++;
   }
 

[mesos] 03/03: Tested unreachable task behavior on agent GC.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git

commit 38e41c5fe4a3cd41b89961b8e3d4e03f2f35a6c3
Author: Vinod Kone 
AuthorDate: Tue Feb 5 16:55:19 2019 -0600

Tested unreachable task behavior on agent GC.

Updated `PartitionTest, RegistryGcByCount` test. This test fails
without the previous patch.

Review: https://reviews.apache.org/r/69909
---
 src/tests/partition_tests.cpp | 40 
 1 file changed, 32 insertions(+), 8 deletions(-)

diff --git a/src/tests/partition_tests.cpp b/src/tests/partition_tests.cpp
index 15af47b..6cdfe973 100644
--- a/src/tests/partition_tests.cpp
+++ b/src/tests/partition_tests.cpp
@@ -2932,16 +2932,30 @@ TEST_F_TEMP_DISABLED_ON_WINDOWS(PartitionTest, 
RegistryGcByCount)
 
   EXPECT_CALL(sched, registered(, _, _));
 
-  Future resourceOffers;
+  Future> offers;
   EXPECT_CALL(sched, resourceOffers(, _))
-.WillOnce(FutureSatisfy())
+.WillOnce(FutureArg<1>())
 .WillRepeatedly(Return()); // Ignore subsequent offers.
 
   driver.start();
 
   // Need to make sure the framework AND slave have registered with
   // master. Waiting for resource offers should accomplish both.
-  AWAIT_READY(resourceOffers);
+  AWAIT_READY(offers);
+
+  TaskInfo task = createTask(offers->at(0), SLEEP_COMMAND(60));
+
+  Future unreachableStatus;
+  EXPECT_CALL(sched, statusUpdate(, _))
+.WillOnce(FutureArg<1>());
+
+  // Drop RunTaskMessage so that the task stays in TASK_STAGING.
+  Future runTaskMessage =
+DROP_PROTOBUF(RunTaskMessage(), _, _);
+
+  driver.launchTasks(offers->at(0).id(), {task});
+
+  AWAIT_READY(runTaskMessage);
 
   EXPECT_CALL(sched, offerRescinded(, _))
 .WillRepeatedly(Return());
@@ -2973,6 +2987,8 @@ TEST_F_TEMP_DISABLED_ON_WINDOWS(PartitionTest, 
RegistryGcByCount)
 
   Clock::advance(Milliseconds(100));
 
+  AWAIT_READY(unreachableStatus);
+
   AWAIT_READY(slaveLost1);
 
   // Set the expectation before resetting `slave1`.
@@ -3031,13 +3047,13 @@ TEST_F_TEMP_DISABLED_ON_WINDOWS(PartitionTest, 
RegistryGcByCount)
   slave2.get()->terminate();
   slave2->reset();
 
-  // Do explicit reconciliation for a random task ID on `slave1`. GC
+  // Do explicit reconciliation for the task launched on `slave1`. GC
   // has not occurred yet (since `registry_gc_interval` has not
   // elapsed since the master was started), so the slave should be in
   // the unreachable list; hence `unreachable_time` should be set on
   // the result of the reconciliation request.
   TaskStatus status1;
-  status1.mutable_task_id()->set_value(id::UUID::random().toString());
+  status1.mutable_task_id()->CopyFrom(task.task_id());
   status1.mutable_slave_id()->CopyFrom(slaveId1);
   status1.set_state(TASK_STAGING); // Dummy value.
 
@@ -3052,15 +3068,23 @@ TEST_F_TEMP_DISABLED_ON_WINDOWS(PartitionTest, 
RegistryGcByCount)
   EXPECT_EQ(TaskStatus::REASON_RECONCILIATION, reconcileUpdate1->reason());
   EXPECT_EQ(partitionTime1, reconcileUpdate1->unreachable_time());
 
+  JSON::Object stats = Metrics();
+  EXPECT_EQ(1, stats.values["master/tasks_unreachable"]);
+
   // Advance the clock to cause GC to be performed.
   Clock::advance(masterFlags.registry_gc_interval);
   Clock::settle();
 
-  // Do explicit reconciliation for a random task ID on `slave1`.
+  // Ensure task has been removed from unreachable list once the
+  // agent is GC'ed.
+  stats = Metrics();
+  EXPECT_EQ(0, stats.values["master/tasks_unreachable"]);
+
+  // Do explicit reconciliation for the task launched on `slave1`.
   // Because the agent has been removed from the unreachable list in
   // the registry, `unreachable_time` should NOT be set.
   TaskStatus status2;
-  status2.mutable_task_id()->set_value(id::UUID::random().toString());
+  status2.mutable_task_id()->CopyFrom(task.task_id());
   status2.mutable_slave_id()->CopyFrom(slaveId1);
   status2.set_state(TASK_STAGING); // Dummy value.
 
@@ -3094,7 +3118,7 @@ TEST_F_TEMP_DISABLED_ON_WINDOWS(PartitionTest, 
RegistryGcByCount)
   EXPECT_EQ(TaskStatus::REASON_RECONCILIATION, reconcileUpdate3->reason());
   EXPECT_EQ(partitionTime2, reconcileUpdate3->unreachable_time());
 
-  JSON::Object stats = Metrics();
+  stats = Metrics();
   EXPECT_EQ(2, stats.values["master/slave_unreachable_scheduled"]);
   EXPECT_EQ(2, stats.values["master/slave_unreachable_completed"]);
   EXPECT_EQ(2, stats.values["master/slave_removals"]);

[mesos] branch master updated: Minor typo fix in downloads page.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git


The following commit(s) were added to refs/heads/master by this push:
 new 3b9bf30  Minor typo fix in downloads page.
3b9bf30 is described below

commit 3b9bf3035ebd2b6f378394c60f0ba18dbc9c8b11
Author: Vinod Kone 
AuthorDate: Mon Jan 28 13:47:24 2019 -0800

Minor typo fix in downloads page.
---
 site/source/downloads.html.erb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/source/downloads.html.erb b/site/source/downloads.html.erb
index 85de7e0..259ca43 100644
--- a/site/source/downloads.html.erb
+++ b/site/source/downloads.html.erb
@@ -36,7 +36,7 @@ breadcrumb: Downloads
   The codebase can also be downloaded from the Apache git repository at:
 https://gitbox.apache.org/repos/asf/mesos.git.
 This repository is also mirrored on
-https://github.com/apache/mesos;>on Github.
+https://github.com/apache/mesos;>GitHub.
   
 
   Once you have downloaded Mesos, follow the

[mesos] branch master updated: Updated text in website downloads page.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git


The following commit(s) were added to refs/heads/master by this push:
 new 54a35ad  Updated text in website downloads page.
54a35ad is described below

commit 54a35ad61a53b88ac5bffbdb49f3d09066c81a5f
Author: Vinod Kone 
AuthorDate: Mon Jan 28 13:44:27 2019 -0800

Updated text in website downloads page.
---
 site/source/downloads.html.erb | 6 ++
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/site/source/downloads.html.erb b/site/source/downloads.html.erb
index 514546a..85de7e0 100644
--- a/site/source/downloads.html.erb
+++ b/site/source/downloads.html.erb
@@ -35,10 +35,8 @@ breadcrumb: Downloads
   Getting the code via source control
   The codebase can also be downloaded from the Apache git repository at:
 https://gitbox.apache.org/repos/asf/mesos.git.
-A read-only git repository is also mirrored on
-https://github.com/apache/mesos/tree/master;>on Github.
-(Note: we use http://reviews.apache.org;>Review Board for
-patches, not pull requests on Github)
+This repository is also mirrored on
+https://github.com/apache/mesos;>on Github.
   
 
   Once you have downloaded Mesos, follow the

[mesos] branch master updated: Manually copy test reports to host fs.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git


The following commit(s) were added to refs/heads/master by this push:
 new aee2e7e  Manually copy test reports to host fs.
aee2e7e is described below

commit aee2e7e761b2bf564bce0dc1971aadd2b68e3224
Author: Vinod Kone 
AuthorDate: Fri Nov 16 13:26:12 2018 -0600

Manually copy test reports to host fs.

Review: https://reviews.apache.org/r/69513
---
 support/mesos-build/entrypoint.sh | 13 -
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/support/mesos-build/entrypoint.sh 
b/support/mesos-build/entrypoint.sh
index ec98cc8..865c44a 100755
--- a/support/mesos-build/entrypoint.sh
+++ b/support/mesos-build/entrypoint.sh
@@ -26,7 +26,18 @@ git clone --depth 1 file:///SRC "${SRCDIR}"
 
 cd "${SRCDIR}"
 
-export GTEST_OUTPUT=xml:report.xml
+# NOTE: We have googletest write test reports into container fs and manually
+# copy out to host fs because we have seen some strange behavior in CI where
+# gooletest writing directly into mounted host directory sometimes fails (with
+# no proper error message).
+function copy_out_test_reports {
+ find ${SRCDIR} -name \*-tests.xml -exec cp {} /SRC \;
+}
+
+trap copy_out_test_reports EXIT
+
+export GTEST_OUTPUT=xml:${SRCDIR}/
+
 export DISTCHECK_CONFIGURE_FLAGS=${CONFIGURATION}
 export ${ENVIRONMENT}
 

[mesos] branch master updated: Fixed xml test report path.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git


The following commit(s) were added to refs/heads/master by this push:
 new 6f3bfb4  Fixed xml test report path.
6f3bfb4 is described below

commit 6f3bfb4bd57d806538a2006949f569c21173017e
Author: Vinod Kone 
AuthorDate: Fri Nov 16 13:54:17 2018 -0600

Fixed xml test report path.
---
 support/mesos-build/entrypoint.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/support/mesos-build/entrypoint.sh 
b/support/mesos-build/entrypoint.sh
index dd97f6d..ec98cc8 100755
--- a/support/mesos-build/entrypoint.sh
+++ b/support/mesos-build/entrypoint.sh
@@ -26,7 +26,7 @@ git clone --depth 1 file:///SRC "${SRCDIR}"
 
 cd "${SRCDIR}"
 
-export GTEST_OUTPUT=xml
+export GTEST_OUTPUT=xml:report.xml
 export DISTCHECK_CONFIGURE_FLAGS=${CONFIGURATION}
 export ${ENVIRONMENT}
 

[mesos] 01/02: Reverted xml output file location.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git

commit 029e11c5f49e9f7d84c940781a3d508963b385a4
Author: Vinod Kone 
AuthorDate: Thu Nov 15 17:57:21 2018 -0600

Reverted xml output file location.

Having google test directly write XML reports into the mounted /SRC
directory is giving intermittent "Unable to open file" error.

This change will have google test write xml output into build directory
inside the container for now. Note that since build directory is cleaned
up as part of the docker image, the output reports wont be accessible
to Jenkins.
---
 support/mesos-build/entrypoint.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/support/mesos-build/entrypoint.sh 
b/support/mesos-build/entrypoint.sh
index b75b4df..dd97f6d 100755
--- a/support/mesos-build/entrypoint.sh
+++ b/support/mesos-build/entrypoint.sh
@@ -26,7 +26,7 @@ git clone --depth 1 file:///SRC "${SRCDIR}"
 
 cd "${SRCDIR}"
 
-export GTEST_OUTPUT=xml:/SRC/
+export GTEST_OUTPUT=xml
 export DISTCHECK_CONFIGURE_FLAGS=${CONFIGURATION}
 export ${ENVIRONMENT}
 

[mesos] branch master updated (e524652 -> f3db5ad)

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git.


from e524652  Added MESOS-7574 to 1.5.3, 1.6.2, and 1.7.1 CHANGELOGs.
 new 029e11c  Reverted xml output file location.
 new f3db5ad  Improved log messages in master when adding/removing 
tasks/executors.

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 src/master/master.cpp | 39 ++-
 support/mesos-build/entrypoint.sh |  2 +-
 2 files changed, 31 insertions(+), 10 deletions(-)

[mesos] 02/02: Improved log messages in master when adding/removing tasks/executors.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git

commit f3db5ad2ca4b01f000e9d204ea970434d585f38b
Author: Vinod Kone 
AuthorDate: Thu Nov 15 15:01:18 2018 -0600

Improved log messages in master when adding/removing tasks/executors.

Made the log messages and the calling sites consistent and also added
one for adding an executor.

Review: https://reviews.apache.org/r/61128/
---
 src/master/master.cpp | 39 ++-
 1 file changed, 30 insertions(+), 9 deletions(-)

diff --git a/src/master/master.cpp b/src/master/master.cpp
index 1e326ec..9458ff1 100644
--- a/src/master/master.cpp
+++ b/src/master/master.cpp
@@ -4053,6 +4053,17 @@ void Master::addExecutor(
   CHECK(slave->connected) << "Adding executor " << executorInfo.executor_id()
   << " to disconnected agent " << *slave;
 
+  // Note that we explicitly convert from protobuf to `Resources` here
+  // and then use the result below to avoid performance penalty for multiple
+  // conversions and validations implied by conversion.
+  // Conversion is safe, as resources have already passed validation.
+  const Resources resources = executorInfo.resources();
+
+  LOG(INFO) << "Adding executor '" << executorInfo.executor_id()
+<< "' with resources " << resources
+<< " of framework " << *framework
+<< " on agent " << *slave;
+
   slave->addExecutor(framework->id(), executorInfo);
   framework->addExecutor(slave->id, executorInfo);
 }
@@ -4068,6 +4079,17 @@ void Master::addTask(
   CHECK(slave->connected) << "Adding task " << task.task_id()
   << " to disconnected agent " << *slave;
 
+  // Note that we explicitly convert from protobuf to `Resources` here
+  // and then use the result below to avoid performance penalty for multiple
+  // conversions and validations implied by conversion.
+  // Conversion is safe, as resources have already passed validation.
+  const Resources resources = task.resources();
+
+  LOG(INFO) << "Adding task " << task.task_id()
+<< " with resources " << resources
+<< " of framework " << *framework
+<< " on agent " << *slave;
+
   // Add the task to the framework and slave.
   Task* t = new Task(protobuf::createTask(task, TASK_STAGING, 
framework->id()));
 
@@ -11082,12 +11104,17 @@ void Master::removeExecutor(
 
   ExecutorInfo executor = slave->executors[frameworkId][executorId];
 
+  // Note that we explicitly convert from protobuf to `Resources` here
+  // and then use the result below to avoid performance penalty for multiple
+  // conversions and validations implied by conversion.
+  // Conversion is safe, as resources have already passed validation.
+  const Resources resources = executor.resources();
+
   LOG(INFO) << "Removing executor '" << executorId
-<< "' with resources " << executor.resources()
+<< "' with resources " << resources
 << " of framework " << frameworkId << " on agent " << *slave;
 
-  allocator->recoverResources(
-  frameworkId, slave->id, executor.resources(), None());
+  allocator->recoverResources(frameworkId, slave->id, resources, None());
 
   Framework* framework = getFramework(frameworkId);
   if (framework != nullptr) { // The framework might not be reregistered yet.
@@ -12227,12 +12254,6 @@ void Slave::addTask(Task* task)
   if (!protobuf::isTerminalState(task->state())) {
 usedResources[frameworkId] += resources;
   }
-
-  // Note that we use `Resources` for output as it's faster than
-  // logging raw protobuf data.
-  LOG(INFO) << "Adding task " << taskId
-<< " with resources " << resources
-<< " on agent " << *this;
 }
 
 

[mesos] branch master updated: Simplified writing out test report xml files.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git


The following commit(s) were added to refs/heads/master by this push:
 new 4650527  Simplified writing out test report xml files.
4650527 is described below

commit 46505272dda99373fd5075d848c86f7f96b176cc
Author: Vinod Kone 
AuthorDate: Wed Nov 7 14:34:05 2018 -0800

Simplified writing out test report xml files.

Review: https://reviews.apache.org/r/69291
---
 support/mesos-build/entrypoint.sh | 8 +---
 1 file changed, 1 insertion(+), 7 deletions(-)

diff --git a/support/mesos-build/entrypoint.sh 
b/support/mesos-build/entrypoint.sh
index 012f003..b75b4df 100755
--- a/support/mesos-build/entrypoint.sh
+++ b/support/mesos-build/entrypoint.sh
@@ -26,7 +26,7 @@ git clone --depth 1 file:///SRC "${SRCDIR}"
 
 cd "${SRCDIR}"
 
-export GTEST_OUTPUT=xml:report.xml
+export GTEST_OUTPUT=xml:/SRC/
 export DISTCHECK_CONFIGURE_FLAGS=${CONFIGURATION}
 export ${ENVIRONMENT}
 
@@ -81,9 +81,3 @@ case ${BUILDTOOL} in
 exit 1
 ;;
 esac
-
-
-# Copy out the test report files.
-cp 3rdparty/stout/tests/report.xml /SRC/stout_report.xml
-cp 3rdparty/libprocess/src/tests/report.xml /SRC/libprocess_report.xml
-cp src/tests/report.xml /SRC/mesos_report.xml

[mesos] branch master updated: Updated Build bot to write out test report xml files.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git


The following commit(s) were added to refs/heads/master by this push:
 new 08bd06e  Updated Build bot to write out test report xml files.
08bd06e is described below

commit 08bd06e3275e2d44cca234bfd6cdf3d0f84acbf1
Author: Vinod Kone 
AuthorDate: Wed Nov 7 13:35:21 2018 -0800

Updated Build bot to write out test report xml files.

This can be used by Jenkins to display test trends.

Review: https://reviews.apache.org/r/69279/
---
 support/mesos-build.sh| 4 
 support/mesos-build/entrypoint.sh | 6 ++
 2 files changed, 10 insertions(+)

diff --git a/support/mesos-build.sh b/support/mesos-build.sh
index d146cc1..c2d121d 100755
--- a/support/mesos-build.sh
+++ b/support/mesos-build.sh
@@ -34,6 +34,10 @@ if ! $(git diff-index --quiet HEAD --); then
   exit 1
 fi
 
+# NOTE: We chmod the directory here so that the docker containter can
+# copy out the test report xml files from the container file system.
+chmod 777 ${MESOS_DIR}
+
 docker run \
   --rm \
   -v "${MESOS_DIR}":/SRC:Z \
diff --git a/support/mesos-build/entrypoint.sh 
b/support/mesos-build/entrypoint.sh
index ec98cc8..012f003 100755
--- a/support/mesos-build/entrypoint.sh
+++ b/support/mesos-build/entrypoint.sh
@@ -81,3 +81,9 @@ case ${BUILDTOOL} in
 exit 1
 ;;
 esac
+
+
+# Copy out the test report files.
+cp 3rdparty/stout/tests/report.xml /SRC/stout_report.xml
+cp 3rdparty/libprocess/src/tests/report.xml /SRC/libprocess_report.xml
+cp src/tests/report.xml /SRC/mesos_report.xml

[mesos] branch master updated: Add info about Rhythm framework

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git


The following commit(s) were added to refs/heads/master by this push:
 new 2fa6598  Add info about Rhythm framework
2fa6598 is described below

commit 2fa65986de3ad707ee3afb6c59c744d1c3b93400
Author: Michał Łowicki 
AuthorDate: Sat Nov 3 23:39:28 2018 +0100

Add info about Rhythm framework
---
 docs/frameworks.md | 1 +
 1 file changed, 1 insertion(+)

diff --git a/docs/frameworks.md b/docs/frameworks.md
index d98675d..368ecad 100644
--- a/docs/frameworks.md
+++ b/docs/frameworks.md
@@ -35,6 +35,7 @@ layout: documentation
 * [GoDocker](https://bitbucket.org/osallou/go-docker) is a batch computing job 
scheduler like SGE, Torque, etc. It schedules batch computing tasks via webui, 
API or CLI for system or LDAP users, mounting their home directory or other 
shared resources in a Docker container. It targets scientists, not developers, 
and provides plugin mechanisms to extend or modify the default behavior.
 * [Jenkins](https://github.com/jenkinsci/mesos-plugin) is a continuous 
integration server. The mesos-jenkins plugin allows it to dynamically launch 
workers on a Mesos cluster depending on the workload.
 * [JobServer](http://www.grandlogic.com/content/html_docs/jobserver.html) is a 
distributed job scheduler and processor  which allows developers to build 
custom batch processing Tasklets using point and click web UI.
+* [Rhythm](https://github.com/mlowicki/rhythm) is a cron-like job scheduler 
supporting ACL and Vault for secrets management.
 
 ## Data Storage
 

[mesos] branch master updated: Updated verify-reviews.py to use current interpreter in subprocesses.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git


The following commit(s) were added to refs/heads/master by this push:
 new 568fcdf  Updated verify-reviews.py to use current interpreter in 
subprocesses.
568fcdf is described below

commit 568fcdfd29788d9df89a51ffae7969c2bf0ea173
Author: Armand Grillet 
AuthorDate: Mon Oct 8 15:01:28 2018 -0500

Updated verify-reviews.py to use current interpreter in subprocesses.

This changes the command used in `support/verify-reviews.py` when
running `support/apply-reviews.py` as a subprocess. It was previously
`"python"`, which is generally Python 2, and is now `sys.executable`.

That way, if verify-reviews.py is run with Python 3 (as it should),
apply-reviews.py will be run with the same Python 3 interpreter. This
should fix the `ImportError` issues we have recently seen in our CI.

Review: https://reviews.apache.org/r/68951/
---
 support/verify-reviews.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/support/verify-reviews.py b/support/verify-reviews.py
index 56321ae..552dc36 100755
--- a/support/verify-reviews.py
+++ b/support/verify-reviews.py
@@ -94,7 +94,7 @@ def shell(command):
 def apply_review(review_id):
 """Apply a review using the script apply-reviews.py."""
 print("Applying review %s" % review_id)
-shell("python support/apply-reviews.py -n -r %s" % review_id)
+shell("%s support/apply-reviews.py -n -r %s" % (sys.executable, review_id))
 
 
 def apply_reviews(review_request, reviews, handler):

[mesos] branch master updated: Fixed bug in `verify-reviews` due to mismatched types.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git


The following commit(s) were added to refs/heads/master by this push:
 new 5b348b6  Fixed bug in `verify-reviews` due to mismatched types.
5b348b6 is described below

commit 5b348b6070f0d0403cb69b6a7fa638fc46b7ff49
Author: Andrew Schwartzmeyer 
AuthorDate: Fri Oct 5 15:39:49 2018 -0500

Fixed bug in `verify-reviews` due to mismatched types.

Because Python is not type-safe, we encountered a bug in the code
executed on non-Windows platforms that was expecting `output` to be a
normal Python string instead of a Python byte string (with encoded
content). To fix this, we now always decode the bytes into a string,
so that the logic afterwards only has one type to deal with.

Review: https://reviews.apache.org/r/68826/
---
 support/verify-reviews.py | 8 +---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/support/verify-reviews.py b/support/verify-reviews.py
index 72b7eb5..56321ae 100755
--- a/support/verify-reviews.py
+++ b/support/verify-reviews.py
@@ -195,10 +195,12 @@ def verify_review(review_request, handler):
 # If we are here because the docker build command failed, read the
 # output from `build_output` file. For all other command failures read
 # the output from `e.output`.
+#
+# Decode the RHS so that `output` is always a string.
 if os.path.exists(build_output):
-output = open(build_output).read()
+output = open(build_output).read().decode(sys.stdout.encoding)
 else:
-output = err.output
+output = err.output.decode(sys.stdout.encoding)
 
 if platform.system() == 'Windows':
 # We didn't output anything during the build (because `tee`
@@ -208,7 +210,7 @@ def verify_review(review_request, handler):
 # fixed in pylint 1.7.
 # TODO(ArmandGrillet): Remove this once pylint updated to >= 1.7.
 # pylint: disable=no-member
-sys.stdout.buffer.write(output)
+sys.stdout.buffer.write(output.encode())
 
 # Truncate the output when posting the review as it can be very large.
 if len(output) > REVIEW_SIZE:

[mesos] branch 1.5.x updated: Added a log line to `MesosContainerizer::kill()`.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch 1.5.x
in repository https://gitbox.apache.org/repos/asf/mesos.git


The following commit(s) were added to refs/heads/1.5.x by this push:
 new ba960ed  Added a log line to `MesosContainerizer::kill()`.
ba960ed is described below

commit ba960ed45e80119eadf398abd72609538fbc983e
Author: Vinod Kone 
AuthorDate: Wed Oct 3 09:51:53 2018 -0500

Added a log line to `MesosContainerizer::kill()`.

Review: https://reviews.apache.org/r/68912
---
 src/slave/containerizer/mesos/containerizer.cpp | 4 
 1 file changed, 4 insertions(+)

diff --git a/src/slave/containerizer/mesos/containerizer.cpp 
b/src/slave/containerizer/mesos/containerizer.cpp
index 2e718fe..9f3df1f 100644
--- a/src/slave/containerizer/mesos/containerizer.cpp
+++ b/src/slave/containerizer/mesos/containerizer.cpp
@@ -2696,6 +2696,10 @@ Future MesosContainerizerProcess::kill(
 
   const Owned& container = containers_.at(containerId);
 
+  LOG_BASED_ON_CLASS(container->containerClass())
+<< "Sending " << strsignal(signal) << " to container "
+<< containerId << " in " << container->state << " state";
+
   // This can happen when we try to signal the container before it
   // is launched. We destroy the container forcefully in this case.
   //

[mesos] branch 1.6.x updated: Added a log line to `MesosContainerizer::kill()`.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch 1.6.x
in repository https://gitbox.apache.org/repos/asf/mesos.git


The following commit(s) were added to refs/heads/1.6.x by this push:
 new 6739eab  Added a log line to `MesosContainerizer::kill()`.
6739eab is described below

commit 6739eab1616987cca34dd71474849376ba644161
Author: Vinod Kone 
AuthorDate: Wed Oct 3 09:51:53 2018 -0500

Added a log line to `MesosContainerizer::kill()`.

Review: https://reviews.apache.org/r/68912
---
 src/slave/containerizer/mesos/containerizer.cpp | 4 
 1 file changed, 4 insertions(+)

diff --git a/src/slave/containerizer/mesos/containerizer.cpp 
b/src/slave/containerizer/mesos/containerizer.cpp
index 4ff26c3..d9550c0 100644
--- a/src/slave/containerizer/mesos/containerizer.cpp
+++ b/src/slave/containerizer/mesos/containerizer.cpp
@@ -2724,6 +2724,10 @@ Future MesosContainerizerProcess::kill(
 
   const Owned& container = containers_.at(containerId);
 
+  LOG_BASED_ON_CLASS(container->containerClass())
+<< "Sending " << strsignal(signal) << " to container "
+<< containerId << " in " << container->state << " state";
+
   // This can happen when we try to signal the container before it
   // is launched. We destroy the container forcefully in this case.
   //

[mesos] branch 1.7.x updated: Added a log line to `MesosContainerizer::kill()`.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch 1.7.x
in repository https://gitbox.apache.org/repos/asf/mesos.git


The following commit(s) were added to refs/heads/1.7.x by this push:
 new 3d48604  Added a log line to `MesosContainerizer::kill()`.
3d48604 is described below

commit 3d48604b7582d0063f31fb581b3f938b03f001d2
Author: Vinod Kone 
AuthorDate: Wed Oct 3 09:51:53 2018 -0500

Added a log line to `MesosContainerizer::kill()`.

Review: https://reviews.apache.org/r/68912
---
 src/slave/containerizer/mesos/containerizer.cpp | 4 
 1 file changed, 4 insertions(+)

diff --git a/src/slave/containerizer/mesos/containerizer.cpp 
b/src/slave/containerizer/mesos/containerizer.cpp
index a89474b..6c27000 100644
--- a/src/slave/containerizer/mesos/containerizer.cpp
+++ b/src/slave/containerizer/mesos/containerizer.cpp
@@ -2826,6 +2826,10 @@ Future MesosContainerizerProcess::kill(
 
   const Owned& container = containers_.at(containerId);
 
+  LOG_BASED_ON_CLASS(container->containerClass())
+<< "Sending " << strsignal(signal) << " to container "
+<< containerId << " in " << container->state << " state";
+
   // This can happen when we try to signal the container before it
   // is launched. We destroy the container forcefully in this case.
   //

[mesos] branch master updated: Clarify existence of FrameworkID in SUBSCRIBE calls.

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git


The following commit(s) were added to refs/heads/master by this push:
 new 4f8092c  Clarify existence of FrameworkID in SUBSCRIBE calls.
4f8092c is described below

commit 4f8092c50ddff640518d2879a7faeaa3e45900e9
Author: Aaron Wood 
AuthorDate: Thu Aug 30 15:42:38 2018 -0500

Clarify existence of FrameworkID in SUBSCRIBE calls.

I wanted to clarify how `FrameworkID` should or shouldn't be set in
`SUBSCRIBE` calls.

This closes #205
---
 docs/scheduler-http-api.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/docs/scheduler-http-api.md b/docs/scheduler-http-api.md
index 79370ec..84f6187 100644
--- a/docs/scheduler-http-api.md
+++ b/docs/scheduler-http-api.md
@@ -55,7 +55,7 @@ The response returned from the `SUBSCRIBE` call (see 
[below](#subscribe)) is enc
 
 This is the first step in the communication process between the scheduler and 
the master. This is also to be considered as subscription to the "/scheduler" 
event stream.
 
-To subscribe with the master, the scheduler sends an HTTP POST with a 
`SUBSCRIBE` message including the required FrameworkInfo. Note that if 
"subscribe.framework_info.id" is not set, master considers the scheduler as a 
new one and subscribes it by assigning it a FrameworkID. The HTTP response is a 
stream in RecordIO format; the event stream begins with a `SUBSCRIBED` event 
(see details in **Events** section). The response also includes the 
`Mesos-Stream-Id` header, which is used by the m [...]
+To subscribe with the master, the scheduler sends an HTTP POST with a 
`SUBSCRIBE` message including the required FrameworkInfo. Note that if 
"subscribe.framework_info.id" and "FrameworkID" are not set, the master 
considers the scheduler as a new one and subscribes it by assigning it a 
FrameworkID. The HTTP response is a stream in RecordIO format; the event stream 
begins with a `SUBSCRIBED` event (see details in **Events** section). The 
response also includes the `Mesos-Stream-Id` header, [...]
 
 ```
 SUBSCRIBE Request (JSON):
@@ -97,7 +97,7 @@ Mesos-Stream-Id: 130ae4e3-6b13-4ef4-baa9-9f2e85c3e9af
 
 ```
 
-Alternatively, if "subscribe.framework_info.id" is set, master considers this 
a request from an already subscribed scheduler reconnecting after a 
disconnection (e.g., due to master/scheduler failover or network disconnection) 
and responds
+Alternatively, if "subscribe.framework_info.id" and "FrameworkID" are set, the 
master considers this a request from an already subscribed scheduler 
reconnecting after a disconnection (e.g., due to master/scheduler failover or 
network disconnection) and responds
 with a `SUBSCRIBED` event. For further details, see the **Disconnections** 
section below.
 
 NOTE: In the old version of the API, (re-)registered callbacks also included 
MasterInfo, which contained information about the master the driver currently 
connected to. With the new API, since schedulers explicitly subscribe with the 
leading master (see details below in **Master Detection** section), it's not 
relevant anymore.

[mesos] branch master updated: Update nested-container-and-task-group.md

[vinodkone]

This is an automated email from the ASF dual-hosted git repository.

vinodkone pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git


The following commit(s) were added to refs/heads/master by this push:
 new fc41758  Update nested-container-and-task-group.md
fc41758 is described below

commit fc41758f15bb0d2b7c4294e36f6aa5e5d248f147
Author: Tarun Gupta Akirala 
AuthorDate: Wed Jun 20 18:49:00 2018 -0700

Update nested-container-and-task-group.md
---
 docs/nested-container-and-task-group.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/docs/nested-container-and-task-group.md 
b/docs/nested-container-and-task-group.md
index 0026366..3325ff8 100644
--- a/docs/nested-container-and-task-group.md
+++ b/docs/nested-container-and-task-group.md
@@ -14,11 +14,11 @@ some resources (e.g., network namespace, volumes) but not 
others
 (e.g., container image, resource limits). Here are the use cases for
 pod:
 
-* Run a side-car container (e.g., logger, backup) next the main
+* Run a side-car container (e.g., logger, backup) next to the main
   application controller.
 * Run an adapter container (e.g., metrics endpoint, queue consumer)
   next to the main container.
-* Run transient tasks inside a pod for a operations which are
+* Run transient tasks inside a pod for operations which are
   short-lived and whose exit does not imply that a pod should
   exit (e.g., a task which backs up data in a persistent volume).
 * Provide performance isolation between latency-critical application

mesos git commit: Optimized ranges subtraction operation.

[vinodkone]

Repository: mesos
Updated Branches:
  refs/heads/master 04d8d1d3a -> 077f122d5


Optimized ranges subtraction operation.

The current ranges subtraction uses boost IntervalSet.
Based on the profiling result of MESOS-8989, the ranges
subtraction operation is about 2~3 times more expensive
than that of addition. The conversion cost from Ranges
to IntervalSet may be the culprit.

This patch optimizes the ranges subtraction operation by
converting Ranges to a vector of internal sub-ranges, sorting
the vector based on sub-range start and then applying a
one-pass algorithm similar to that of addition.

Review: https://reviews.apache.org/r/67965/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/077f122d
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/077f122d
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/077f122d

Branch: refs/heads/master
Commit: 077f122d52671412a2ab5d992d535712cc154002
Parents: 04d8d1d
Author: Meng Zhu 
Authored: Fri Jul 27 10:26:22 2018 -0700
Committer: Vinod Kone 
Committed: Fri Jul 27 10:26:22 2018 -0700

--
 src/common/values.cpp  | 115 
 src/tests/values_tests.cpp |  16 +-
 src/v1/values.cpp  | 115 
 3 files changed, 224 insertions(+), 22 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/077f122d/src/common/values.cpp
--
diff --git a/src/common/values.cpp b/src/common/values.cpp
index afe4137..f04d115 100644
--- a/src/common/values.cpp
+++ b/src/common/values.cpp
@@ -229,6 +229,105 @@ void coalesce(Value::Ranges* result, vector ranges)
   CHECK_EQ(result->range_size(), count);
 }
 
+
+// Subtract `right_` from `left_`, and return the result as Value::Ranges.
+Value::Ranges subtract(const Value::Ranges& left_, const Value::Ranges& right_)
+{
+  if (left_.range_size() == 0 || right_.range_size() == 0) {
+return left_;
+  }
+
+  // Convert the input `Ranges` to `vector` and
+  // sort the vector based on the start of a range.
+  auto sortRanges = [](const Value::Ranges& ranges) {
+vector result;
+result.reserve(ranges.range_size());
+
+foreach (const Value::Range& range, ranges.range()) {
+  result.push_back({range.begin(), range.end()});
+}
+
+std::sort(
+result.begin(),
+result.end(),
+[](const internal::Range& left, const internal::Range& right) {
+  return left.start < right.start;
+});
+
+return result;
+  };
+
+  Value::Ranges result;
+
+  vector left = sortRanges(left_);
+  vector right = sortRanges(right_);
+
+  vector::iterator itLeft = left.begin();
+  for (vector::const_iterator itRight = right.cbegin();
+   itLeft != left.end() && itRight != right.cend();) {
+// Non-overlap:
+// L: |___|
+// R:   |___|
+if (itLeft->end < itRight->start) {
+  Value::Range* newRange = result.add_range();
+  newRange->set_begin(itLeft->start);
+  newRange->set_end(itLeft->end);
+
+  itLeft++;
+  continue;
+}
+
+// Non-overlap:
+// L:   |___|
+// R: |___|
+if (itLeft->start > itRight->end) {
+  itRight++;
+  continue;
+}
+
+if (itLeft->start < itRight->start) {
+  Value::Range* newRange = result.add_range();
+  newRange->set_begin(itLeft->start);
+  newRange->set_end(itRight->start - 1);
+
+  if (itLeft->end <= itRight->end) {
+// L: |_|
+// R:||
+itLeft++;
+  } else {
+// L: ||
+// R:|___|
+itLeft->start = itRight->end + 1;
+itRight++;
+  }
+} else { // itLeft->start >= itRight->start
+  if (itLeft->end <= itRight->end) {
+// L:   ||
+// R: ||
+itLeft++;
+  } else {
+// L:   |_|
+// R: ||
+itLeft->start = itRight->end + 1;
+itRight++;
+  }
+}
+  }
+
+  // Traverse what's left in the `left`, if any.
+  while (itLeft != left.end()) {
+// TODO(mzhu): Consider reserving the exact size.
+Value::Range* newRange = result.add_range();
+newRange->set_begin(itLeft->start);
+newRange->set_end(itLeft->end);
+
+itLeft++;
+  }
+
+  return result;
+}
+
+
 } // namespace internal {
 
 
@@ -352,6 +451,7 @@ bool operator<=(const Value::Ranges& _left, const 
Value::Ranges& _right)
 }
 
 
+// TODO(mzhu): Make this consistent with how we do subtraction.
 Value::Ranges operator+(const Value::Ranges& left, const Value::Ranges& right)
 {
   Value::Ranges result;
@@ -362,12 +462,11 @@ Value::Ranges operator+(const Value::Ranges& left, const 
Value::Ranges& right)
 
 Value::Ranges operator-(const Value::Ranges& left, const Value::Ranges& right)
 {
-  

mesos git commit: Document SUPPRESS HTTP call [MESOS-7211].

[vinodkone]

Repository: mesos
Updated Branches:
  refs/heads/master 9af920c75 -> a7638a833


Document SUPPRESS HTTP call [MESOS-7211].

This closes #301


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/a7638a83
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/a7638a83
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/a7638a83

Branch: refs/heads/master
Commit: a7638a833acd82d5cc4f1d374b1867510c76
Parents: 9af920c
Author: Thodoris Zois 
Authored: Mon Jul 23 10:19:01 2018 -0700
Committer: Vinod Kone 
Committed: Mon Jul 23 10:19:01 2018 -0700

--
 docs/scheduler-http-api.md | 26 ++
 1 file changed, 26 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/a7638a83/docs/scheduler-http-api.md
--
diff --git a/docs/scheduler-http-api.md b/docs/scheduler-http-api.md
index 3d7d789..79370ec 100644
--- a/docs/scheduler-http-api.md
+++ b/docs/scheduler-http-api.md
@@ -484,6 +484,32 @@ HTTP/1.1 202 Accepted
 
 ```
 
+### SUPPRESS
+Sent by the scheduler when it doesn't need offers for a given set of its 
roles. When Mesos master receives this request, it will stop sending offers for 
the given set of roles to the framework. As a special case, if roles are not 
specified, all subscribed roles of this framework are suppressed.
+
+Note that master continues to send offers to other subscribed roles of this 
framework that are not suppressed. Also, status updates about tasks, executors 
and agents are not affected by this call. 
+
+If the scheduler wishes to receive offers for the suppressed roles again 
(e.g., it needs to schedule new workloads), it can send `REVIVE` call.
+
+```
+SUPPRESS Request (JSON):
+POST /api/v1/scheduler  HTTP/1.1
+
+Host: masterhost:5050
+Content-Type: application/json
+Mesos-Stream-Id: 130ae4e3-6b13-4ef4-baa9-9f2e85c3e9af
+
+{
+  "framework_id" : {"value" : "12220-3440-12532-2345"},
+  "type" : "SUPPRESS",
+  "suppress" : {"roles": }
+}
+
+SUPPRESS Response:
+HTTP/1.1 202 Accepted
+
+```
+
 ## Events
 
 Schedulers are expected to keep a **persistent** connection to the 
"/scheduler" endpoint (even after getting a `SUBSCRIBED` HTTP Response event). 
This is indicated by the "Connection: keep-alive" and "Transfer-Encoding: 
chunked" headers with *no* "Content-Length" header set. All subsequent events 
that are relevant to this framework generated by Mesos are streamed on this 
connection. The master encodes each Event in RecordIO format, i.e., string 
representation of the length of the event in bytes followed by JSON or binary 
Protobuf (possibly compressed) encoded event. The length of an event is a 
64-bit unsigned integer (encoded as a textual value) and will never be "0". 
Also, note that the RecordIO encoding should be decoded by the scheduler 
whereas the underlying HTTP chunked encoding is typically invisible at the 
application (scheduler) layer. The type of content encoding used for the events 
will be determined by the accept header of the POST request (e.g., Accept: 
application/json)
 .

mesos git commit: Added Dutch and Belgium Mesos Users Group.

[vinodkone]

Repository: mesos
Updated Branches:
  refs/heads/master 9b67ea040 -> 4e7e3670e


Added Dutch and Belgium Mesos Users Group.

This closes #298


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/4e7e3670
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/4e7e3670
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/4e7e3670

Branch: refs/heads/master
Commit: 4e7e3670e12e95a3e5aa060a0afd957fa0e80919
Parents: 9b67ea0
Author: janisz 
Authored: Thu Jun 21 22:39:56 2018 -0500
Committer: Vinod Kone 
Committed: Thu Jun 21 22:39:56 2018 -0500

--
 site/source/community/user-groups.html.md | 4 
 1 file changed, 4 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/4e7e3670/site/source/community/user-groups.html.md
--
diff --git a/site/source/community/user-groups.html.md 
b/site/source/community/user-groups.html.md
index 2e94752..af3f9a7 100644
--- a/site/source/community/user-groups.html.md
+++ b/site/source/community/user-groups.html.md
@@ -9,6 +9,9 @@ Mesos User Groups (MUGs) are responsible for organizing local 
meetups, hackathon
 
 ## Local Groups
 
+### Belgium
+* [Belgian Mesos User Group](https://www.meetup.com/Belgian-Mesos-User-Group/)
+
 ### China
 * [Shanghai](http://www.weibo.com/u/3068800961)
 * [Shenzhen](http://www.meetup.com/Shenzhen-Mesos-User-Group/)
@@ -30,6 +33,7 @@ Mesos User Groups (MUGs) are responsible for organizing local 
meetups, hackathon
 * [Tokyo](https://mesos.connpass.com/)
 
 ### Netherlands
+* [Dutch Mesos User Group](https://www.meetup.com/Dutch-Mesos-User-Group/)
 * [Amsterdam](http://www.meetup.com/Mesos-Amsterdam/)
 
 ### Poland

mesos git commit: Reformat email address in blog to avoid un-necessary updates to website.

[vinodkone]

Repository: mesos
Updated Branches:
  refs/heads/master b863f6c78 -> 77279f025


Reformat email address in blog to avoid un-necessary updates to website.

Email in "" format in a markdown file causes random
characters to be generated in the resulting html file causing the
website bot generating a new commit everytime even though nothing
changed.

The fix here is to remove the angular brackets.


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/77279f02
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/77279f02
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/77279f02

Branch: refs/heads/master
Commit: 77279f0256d12edc8208dfd96a88fea205bde8a4
Parents: b863f6c
Author: Vinod Kone 
Authored: Thu May 10 13:07:11 2018 -0700
Committer: Vinod Kone 
Committed: Thu May 10 13:10:08 2018 -0700

--
 .../blog/2017-12-07-performance-working-group-progress-report.md   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/77279f02/site/source/blog/2017-12-07-performance-working-group-progress-report.md
--
diff --git 
a/site/source/blog/2017-12-07-performance-working-group-progress-report.md 
b/site/source/blog/2017-12-07-performance-working-group-progress-report.md
index 0f59515..e641d25 100644
--- a/site/source/blog/2017-12-07-performance-working-group-progress-report.md
+++ b/site/source/blog/2017-12-07-performance-working-group-progress-report.md
@@ -87,4 +87,4 @@ We're currently targeting the following areas for 
improvements:
   * 
**[Libprocess](https://github.com/apache/mesos/tree/master/3rdparty/libprocess) 
HTTP performance**: This will be undertaken as part of improving the v1 API 
performance, since it is HTTP-based.
 * **Master state API performance**: Currently, API queries of the master's 
state are serviced by the same master actor that processes all of the messages 
from schedulers and agents. Since the query processing can block the master 
from processing other events, users need to be careful not to query the master 
excessively. In practice, the master gets queried quite heavily due to the 
presence of several tools that rely on the master's state (e.g. DNS tooling, 
UIs, CLIs, etc) and so this is a critical problem for users. This effort will 
leverage the state streaming API to stream the state to a different actor that 
can serve the state API requests. This will ensure that expensive state queries 
do not affect the master's ability to process events.
 
-If you are a user and would like to suggest some areas for performance 
improvement, please let us know by emailing .
+If you are a user and would like to suggest some areas for performance 
improvement, please let us know by emailing `d...@apache.mesos.org`.

[2/2] mesos git commit: Added MESOS-8552 to the 1.5.1 CHANGELOG.

[vinodkone]

Added MESOS-8552 to the 1.5.1 CHANGELOG.


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/ae4f94a8
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/ae4f94a8
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/ae4f94a8

Branch: refs/heads/1.5.x
Commit: ae4f94a8e429ca0628bb5accff37e7757706e5b4
Parents: 01bb8e7
Author: Vinod Kone 
Authored: Thu Feb 8 17:23:09 2018 -0800
Committer: Vinod Kone 
Committed: Thu Feb 8 17:23:58 2018 -0800

--
 CHANGELOG | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/ae4f94a8/CHANGELOG
--
diff --git a/CHANGELOG b/CHANGELOG
index e126e86..7de3911 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -5,6 +5,7 @@ Release Notes - Mesos - Version 1.5.1 (WIP)
 ** Bug
  * [MESOS-7742] - Race conditions in IOSwitchboard: listening on unix socket 
and premature closing of the connection.
  * [MESOS-8411] - Killing a queued task can lead to the command executor never 
terminating.
+ * [MESOS-8552] - Failing `CGROUPS_ROOT_PidNamespaceForward` and 
`CGROUPS_ROOT_PidNamespaceBackward`
 
 
 Release Notes - Mesos - Version 1.5.0

mesos git commit: Added MESOS-8552 to the 1.5.1 CHANGELOG.

[vinodkone]

Repository: mesos
Updated Branches:
  refs/heads/master 97d505e5c -> 0d1d00bd1


Added MESOS-8552 to the 1.5.1 CHANGELOG.


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/0d1d00bd
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/0d1d00bd
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/0d1d00bd

Branch: refs/heads/master
Commit: 0d1d00bd16474042c19a62c555ddcbf1a1717a3e
Parents: 97d505e
Author: Vinod Kone 
Authored: Thu Feb 8 17:23:09 2018 -0800
Committer: Vinod Kone 
Committed: Thu Feb 8 17:23:09 2018 -0800

--
 CHANGELOG | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/0d1d00bd/CHANGELOG
--
diff --git a/CHANGELOG b/CHANGELOG
index 785c102..0706fff 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -5,6 +5,7 @@ Release Notes - Mesos - Version 1.5.1 (WIP)
 ** Bug
  * [MESOS-7742] - Race conditions in IOSwitchboard: listening on unix socket 
and premature closing of the connection.
  * [MESOS-8411] - Killing a queued task can lead to the command executor never 
terminating.
+ * [MESOS-8552] - Failing `CGROUPS_ROOT_PidNamespaceForward` and 
`CGROUPS_ROOT_PidNamespaceBackward`
 
 
 Release Notes - Mesos - Version 1.5.0

mesos git commit: Fixed two slave recovery tests.

[vinodkone]

Repository: mesos
Updated Branches:
  refs/heads/master a018cf33d -> 97d505e5c


Fixed two slave recovery tests.

These two tests are affected by the lingering executor fix
(#65109). We should wait to make sure tasks are actually launched
on the executor before trigger the agent failover. Otherwise, the
container will be shutdown upon re-registration because the executor
has never received any task.

Review: https://reviews.apache.org/r/65572/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/97d505e5
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/97d505e5
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/97d505e5

Branch: refs/heads/master
Commit: 97d505e5cc27a11b7538cc81fa57d9e4fd10aaf9
Parents: a018cf3
Author: Meng Zhu 
Authored: Thu Feb 8 17:19:48 2018 -0800
Committer: Vinod Kone 
Committed: Thu Feb 8 17:19:48 2018 -0800

--
 src/tests/slave_recovery_tests.cpp | 22 ++
 1 file changed, 22 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/97d505e5/src/tests/slave_recovery_tests.cpp
--
diff --git a/src/tests/slave_recovery_tests.cpp 
b/src/tests/slave_recovery_tests.cpp
index 77aa60c..10f8dd3 100644
--- a/src/tests/slave_recovery_tests.cpp
+++ b/src/tests/slave_recovery_tests.cpp
@@ -5156,7 +5156,11 @@ TEST_F(MesosContainerizerSlaveRecoveryTest, 
CGROUPS_ROOT_PidNamespaceForward)
   // Scheduler expectations.
   EXPECT_CALL(sched, registered(_, _, _));
 
+  Future statusStarting;
+  Future statusRunning;
   EXPECT_CALL(sched, statusUpdate(_, _))
+.WillOnce(FutureArg<1>())
+.WillOnce(FutureArg<1>())
 .WillRepeatedly(Return());
 
   Future offers1;
@@ -5188,6 +5192,13 @@ TEST_F(MesosContainerizerSlaveRecoveryTest, 
CGROUPS_ROOT_PidNamespaceForward)
 
   ContainerID containerId = *(containers->begin());
 
+  // Wait until task is running.
+  AWAIT_READY(statusStarting);
+  EXPECT_EQ(TASK_STARTING, statusStarting->state());
+
+  AWAIT_READY(statusRunning);
+  EXPECT_EQ(TASK_RUNNING, statusRunning->state());
+
   // Stop the slave.
   slave.get()->terminate();
 
@@ -5260,7 +5271,11 @@ TEST_F(MesosContainerizerSlaveRecoveryTest, 
CGROUPS_ROOT_PidNamespaceBackward)
   // Scheduler expectations.
   EXPECT_CALL(sched, registered(_, _, _));
 
+  Future statusStarting;
+  Future statusRunning;
   EXPECT_CALL(sched, statusUpdate(_, _))
+.WillOnce(FutureArg<1>())
+.WillOnce(FutureArg<1>())
 .WillRepeatedly(Return());
 
   Future offers1;
@@ -5292,6 +5307,13 @@ TEST_F(MesosContainerizerSlaveRecoveryTest, 
CGROUPS_ROOT_PidNamespaceBackward)
 
   ContainerID containerId = *(containers->begin());
 
+  // Wait until task is running.
+  AWAIT_READY(statusStarting);
+  EXPECT_EQ(TASK_STARTING, statusStarting->state());
+
+  AWAIT_READY(statusRunning);
+  EXPECT_EQ(TASK_RUNNING, statusRunning->state());
+
   // Stop the slave.
   slave.get()->terminate();
 

mesos git commit: Added documentation for fault domains.

[vinodkone]

Repository: mesos
Updated Branches:
  refs/heads/master 25feb869e -> 5a1433576


Added documentation for fault domains.

Fault domains are a new feature in 1.5 which did not yet have
a corresponding description in the documentation.

Review: https://reviews.apache.org/r/65437/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/5a143357
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/5a143357
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/5a143357

Branch: refs/heads/master
Commit: 5a1433576eca20f15f1ea309fc202f4bbaf3b6c7
Parents: 25feb86
Author: Benno Evers 
Authored: Fri Feb 2 14:12:21 2018 -0800
Committer: Vinod Kone 
Committed: Fri Feb 2 14:14:04 2018 -0800

--
 docs/configuration/master-and-agent.md |   3 +
 docs/fault-domains.md  | 106 
 docs/home.md   |   1 +
 3 files changed, 110 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/5a143357/docs/configuration/master-and-agent.md
--
diff --git a/docs/configuration/master-and-agent.md 
b/docs/configuration/master-and-agent.md
index f247498..df18681 100644
--- a/docs/configuration/master-and-agent.md
+++ b/docs/configuration/master-and-agent.md
@@ -95,6 +95,9 @@ different zones). Agents configured to use a different region 
than the
 master's region will not appear in resource offers to frameworks that have
 not enabled the REGION_AWARE capability. This value can be
 specified as either a JSON-formatted string or a file path containing JSON.
+
+See the [documentation](../fault-domains.md) for further details.
+
 
 Example:
 {

http://git-wip-us.apache.org/repos/asf/mesos/blob/5a143357/docs/fault-domains.md
--
diff --git a/docs/fault-domains.md b/docs/fault-domains.md
new file mode 100644
index 000..08f13b5
--- /dev/null
+++ b/docs/fault-domains.md
@@ -0,0 +1,106 @@
+---
+title: Apache Mesos - Domains and Regions
+layout: documentation
+---
+
+# Regions and Fault Domains
+
+Starting with Mesos 1.5, it is possible to place Mesos masters and agents into
+*domains*, which are logical groups of machines that share some 
characteristics.
+
+Currently, fault domains are the only supported type of domains, which are
+groups of machines with similar failure characteristics.
+
+A fault domain is a 2 level hierarchy of regions and zones. The mapping from
+fault domains to physical infrastructure is up to the operator to configure,
+although it is recommended that machines in the same zones have low latency to
+each other.
+
+In cloud environments, regions and zones can be mapped to the "region" and
+"availability zone" concepts exposed by most cloud providers, respectively.
+In on-premise deployments, regions and zones can be mapped to data centers and
+racks, respectively.
+
+Schedulers may prefer to place network-intensive workloads in the same domain,
+as this may improve performance. Conversely, a single failure that affects a
+host in a domain may be more likely to affect other hosts in the same domain;
+hence, schedulers may prefer to place workloads that require high availability
+in multiple domains. For example, all the hosts in a single rack might lose
+power or network connectivity simultaneously.
+
+The `--domain` flag can be used to specify the fault domain of a master or
+agent node. The value of this flag must be a file path or a JSON dictionary
+with the key `fault_domain` and subkeys `region` and `zone` mapping to
+arbitrary strings:
+
+mesos-master --domain='{"fault_domain": {"region": "eu", "zone": "rack1"}}'
+
+mesos-agent  --domain='{"fault_domain": {"region": "eu", "zone": "rack2"}}'
+
+Frameworks can learn about the domain of an agent by inspecting the `domain`
+field in the received offer, which contains a `DomainInfo` that has the
+same structure as the JSON dictionary above.
+
+
+# Constraints
+
+When configuring fault domains for the masters and agents, the following
+constraints must be obeyed:
+
+ * If a mesos master is not configured with a domain, it will reject connection
+   attempts from agents with a domain.
+
+   This is done because the master is not able to determine whether or not the
+   agent would be remote in this case.
+
+ * Agents with no configured domain are assumed to be in the same domain as the
+   master.
+
+   If this behaviour isn't desired, the `--require_agent_domain` flag on the
+   master can be used to enforce that domains are configured on all agents by
+   having the master reject all registration attempts by agents without a
+   configured domain.
+
+ * If one master is configured with a domain, all other masters 

mesos git commit: Updated api-client-libraries.md.

[vinodkone]

Repository: mesos
Updated Branches:
  refs/heads/master a9f5ef3a4 -> a24be7195


Updated api-client-libraries.md.

* Adding Go client for the Mesos Operator API

This closes #258


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/a24be719
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/a24be719
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/a24be719

Branch: refs/heads/master
Commit: a24be7195768d3e160a45b0d8d99bd1008518e0d
Parents: a9f5ef3
Author: Demitri Swan 
Authored: Thu Jan 18 10:48:45 2018 -0800
Committer: Vinod Kone 
Committed: Thu Jan 18 10:48:45 2018 -0800

--
 docs/api-client-libraries.md | 8 
 1 file changed, 8 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/a24be719/docs/api-client-libraries.md
--
diff --git a/docs/api-client-libraries.md b/docs/api-client-libraries.md
index 9a9972b..4d623bd 100644
--- a/docs/api-client-libraries.md
+++ b/docs/api-client-libraries.md
@@ -136,4 +136,12 @@ run into any issues, file them with the library 
maintainers.*
   mesos-operator-api-client
   JavaScript
 
+
+  
+https://github.com/miroswan/mesops;>mesops
+  
+  
+Go
+  
+
 

mesos git commit: Fixed comparison logic for additive reconfiguration policy.

[vinodkone]

Repository: mesos
Updated Branches:
  refs/heads/1.5.x cc4f022ba -> db71e2f62


Fixed comparison logic for additive reconfiguration policy.

The case where multiple resources have the same name
was not handled correctly, and could result in false
negatives.

Review: https://reviews.apache.org/r/65074/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/db71e2f6
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/db71e2f6
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/db71e2f6

Branch: refs/heads/1.5.x
Commit: db71e2f6247c8c6e1290eab663d120d9cfd45a6c
Parents: cc4f022
Author: Benno Evers 
Authored: Fri Jan 12 11:33:57 2018 -0800
Committer: Vinod Kone 
Committed: Fri Jan 12 12:10:14 2018 -0800

--
 include/mesos/resources.hpp |  5 ++
 src/common/resources.cpp| 22 +++-
 src/slave/compatibility.cpp | 73 -
 src/slave/compatibility.hpp |  4 +-
 src/tests/slave_compatibility_tests.cpp | 80 
 5 files changed, 142 insertions(+), 42 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/db71e2f6/include/mesos/resources.hpp
--
diff --git a/include/mesos/resources.hpp b/include/mesos/resources.hpp
index 69bf823..7afe0d8 100644
--- a/include/mesos/resources.hpp
+++ b/include/mesos/resources.hpp
@@ -492,6 +492,11 @@ public:
   // Error if the conversion cannot be applied.
   Try apply(const ResourceConversion& conversion) const;
 
+  // Finds a resource object with the same metadata (i.e. AllocationInfo,
+  // ReservationInfo, etc.) as the given one, ignoring the actual value.
+  // If multiple matching resources exist, the first match is returned.
+  Option match(const Resource& resource) const;
+
   // Obtains the conversion from the given operation and applies the
   // conversion. This method serves a syntax sugar for applying a
   // resource conversion.

http://git-wip-us.apache.org/repos/asf/mesos/blob/db71e2f6/src/common/resources.cpp
--
diff --git a/src/common/resources.cpp b/src/common/resources.cpp
index 69e8e34..262dcfb 100644
--- a/src/common/resources.cpp
+++ b/src/common/resources.cpp
@@ -260,7 +260,7 @@ bool operator!=(const Resource::DiskInfo& left, const 
Resource::DiskInfo& right)
 }
 
 
-bool operator==(const Resource& left, const Resource& right)
+static bool compareResourceMetadata(const Resource& left, const Resource& 
right)
 {
   if (left.name() != right.name() || left.type() != right.type()) {
 return false;
@@ -315,6 +315,15 @@ bool operator==(const Resource& left, const Resource& 
right)
 return false;
   }
 
+  return true;
+}
+
+
+bool operator==(const Resource& left, const Resource& right) {
+  if (!compareResourceMetadata(left, right)) {
+return false;
+  }
+
   if (left.type() == Value::SCALAR) {
 return left.scalar() == right.scalar();
   } else if (left.type() == Value::RANGES) {
@@ -1852,6 +1861,17 @@ Option Resources::ephemeral_ports() const
 }
 
 
+Option Resources::match(const Resource& resource) const
+{
+  foreach (const Resource_& resource_, resources) {
+if (compareResourceMetadata(resource_.resource, resource)) {
+  return resource_.resource;
+}
+  }
+
+  return None();
+}
+
 /
 // Private member functions.
 /

http://git-wip-us.apache.org/repos/asf/mesos/blob/db71e2f6/src/slave/compatibility.cpp
--
diff --git a/src/slave/compatibility.cpp b/src/slave/compatibility.cpp
index 4ead4a5..5551d93 100644
--- a/src/slave/compatibility.cpp
+++ b/src/slave/compatibility.cpp
@@ -19,6 +19,8 @@
 #include 
 #include 
 
+#include 
+#include 
 #include 
 
 #include "mesos/type_utils.hpp"
@@ -49,33 +51,6 @@ Try equal(
 }
 
 
-// T is instantiated below as either `Resource` or `Attribute`.
-template
-Try getMatchingValue(
-  const T& previous,
-  const google::protobuf::RepeatedPtrField& values)
-{
-  auto match = std::find_if(
-  values.begin(),
-  values.end(),
-  [](const T& value) {
-return previous.name() == value.name();
-  });
-
-  if (match == values.end()) {
-return Error("Couldn't find '" + previous.name() + "'");
-  }
-
-  if (match->type() != previous.type()) {
-return Error(
-"Type of '" + previous.name() + "' changed from " +
-stringify(previous.type()) + " to " + stringify(match->type()));
-  }
-
-  return *match;
-}
-
-
 Try additive(
 const SlaveInfo& previous,
 const SlaveInfo& current)
@@ -101,16 +76,20 @@ Try additive(
   

mesos git commit: Fixed comparison logic for additive reconfiguration policy.

[vinodkone]

Repository: mesos
Updated Branches:
  refs/heads/master e39809bd6 -> 49f4b2aff


Fixed comparison logic for additive reconfiguration policy.

The case where multiple resources have the same name
was not handled correctly, and could result in false
negatives.

Review: https://reviews.apache.org/r/65074/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/49f4b2af
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/49f4b2af
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/49f4b2af

Branch: refs/heads/master
Commit: 49f4b2aff4d9f0603947b04071203a55d008a61c
Parents: e39809b
Author: Benno Evers 
Authored: Fri Jan 12 11:33:57 2018 -0800
Committer: Vinod Kone 
Committed: Fri Jan 12 11:35:00 2018 -0800

--
 include/mesos/resources.hpp |  5 ++
 src/common/resources.cpp| 22 +++-
 src/slave/compatibility.cpp | 73 -
 src/slave/compatibility.hpp |  4 +-
 src/tests/slave_compatibility_tests.cpp | 80 
 5 files changed, 142 insertions(+), 42 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/49f4b2af/include/mesos/resources.hpp
--
diff --git a/include/mesos/resources.hpp b/include/mesos/resources.hpp
index 69bf823..7afe0d8 100644
--- a/include/mesos/resources.hpp
+++ b/include/mesos/resources.hpp
@@ -492,6 +492,11 @@ public:
   // Error if the conversion cannot be applied.
   Try apply(const ResourceConversion& conversion) const;
 
+  // Finds a resource object with the same metadata (i.e. AllocationInfo,
+  // ReservationInfo, etc.) as the given one, ignoring the actual value.
+  // If multiple matching resources exist, the first match is returned.
+  Option match(const Resource& resource) const;
+
   // Obtains the conversion from the given operation and applies the
   // conversion. This method serves a syntax sugar for applying a
   // resource conversion.

http://git-wip-us.apache.org/repos/asf/mesos/blob/49f4b2af/src/common/resources.cpp
--
diff --git a/src/common/resources.cpp b/src/common/resources.cpp
index 69e8e34..262dcfb 100644
--- a/src/common/resources.cpp
+++ b/src/common/resources.cpp
@@ -260,7 +260,7 @@ bool operator!=(const Resource::DiskInfo& left, const 
Resource::DiskInfo& right)
 }
 
 
-bool operator==(const Resource& left, const Resource& right)
+static bool compareResourceMetadata(const Resource& left, const Resource& 
right)
 {
   if (left.name() != right.name() || left.type() != right.type()) {
 return false;
@@ -315,6 +315,15 @@ bool operator==(const Resource& left, const Resource& 
right)
 return false;
   }
 
+  return true;
+}
+
+
+bool operator==(const Resource& left, const Resource& right) {
+  if (!compareResourceMetadata(left, right)) {
+return false;
+  }
+
   if (left.type() == Value::SCALAR) {
 return left.scalar() == right.scalar();
   } else if (left.type() == Value::RANGES) {
@@ -1852,6 +1861,17 @@ Option Resources::ephemeral_ports() const
 }
 
 
+Option Resources::match(const Resource& resource) const
+{
+  foreach (const Resource_& resource_, resources) {
+if (compareResourceMetadata(resource_.resource, resource)) {
+  return resource_.resource;
+}
+  }
+
+  return None();
+}
+
 /
 // Private member functions.
 /

http://git-wip-us.apache.org/repos/asf/mesos/blob/49f4b2af/src/slave/compatibility.cpp
--
diff --git a/src/slave/compatibility.cpp b/src/slave/compatibility.cpp
index 4ead4a5..5551d93 100644
--- a/src/slave/compatibility.cpp
+++ b/src/slave/compatibility.cpp
@@ -19,6 +19,8 @@
 #include 
 #include 
 
+#include 
+#include 
 #include 
 
 #include "mesos/type_utils.hpp"
@@ -49,33 +51,6 @@ Try equal(
 }
 
 
-// T is instantiated below as either `Resource` or `Attribute`.
-template
-Try getMatchingValue(
-  const T& previous,
-  const google::protobuf::RepeatedPtrField& values)
-{
-  auto match = std::find_if(
-  values.begin(),
-  values.end(),
-  [](const T& value) {
-return previous.name() == value.name();
-  });
-
-  if (match == values.end()) {
-return Error("Couldn't find '" + previous.name() + "'");
-  }
-
-  if (match->type() != previous.type()) {
-return Error(
-"Type of '" + previous.name() + "' changed from " +
-stringify(previous.type()) + " to " + stringify(match->type()));
-  }
-
-  return *match;
-}
-
-
 Try additive(
 const SlaveInfo& previous,
 const SlaveInfo& current)
@@ -101,16 +76,20 @@ Try additive(
 

mesos git commit: Fixes WG table formatting and add Community WG.

[vinodkone]

Repository: mesos
Updated Branches:
  refs/heads/master 07430e701 -> e35e6808b


Fixes WG table formatting and add Community WG.

Edits to http://mesos.apache.org/documentation/latest/working-groups/:

- Fix table formatting
- Add the Community WG (with @vinodkone as the owner since he originally
  proposed the group)

This closes #252


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/e35e6808
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/e35e6808
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/e35e6808

Branch: refs/heads/master
Commit: e35e6808b518ab3f9c1754f64f38d2c539844a8e
Parents: 07430e7
Author: Judith Malnick <judith.patud...@gmail.com>
Authored: Mon Jan 8 10:55:05 2018 -0800
Committer: Vinod Kone <vinodk...@gmail.com>
Committed: Mon Jan 8 10:55:05 2018 -0800

--
 docs/working-groups.md | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/e35e6808/docs/working-groups.md
--
diff --git a/docs/working-groups.md b/docs/working-groups.md
index df6f050..ab7fb21 100644
--- a/docs/working-groups.md
+++ b/docs/working-groups.md
@@ -31,4 +31,5 @@ In order to distinguish emails for a specific working group, 
please tag the emai
 | Name | Leads | Slack Channel | Meetings |
 |--|---|---|--|
 | Containerization | [Jie Yu](mailto:yujie@gmail.com) | #containerizer | 
[Every other Thursday at 9:00 AM 
PST](https://docs.google.com/document/d/1z55a7tLZFoRWVuUxz1FZwgxkHeugtc2nHR89skFXSpU/edit?usp=sharing)
 |
-| Performance | [Benjamin Mahler](mailto:bmah...@apache.org) | #performance | 
[Monthly at 10 AM PST on Wednesday]() 
|https://docs.google.com/document/d/12hWGuzbqyNWc2l1ysbPcXwc0pzHEy4bodagrlNGCuQU/edit?usp=sharing)
 |
\ No newline at end of file
+| Performance | [Benjamin Mahler](mailto:bmah...@apache.org) | #performance | 
[Monthly at 10:00 AM PST on 
Wednesday](https://docs.google.com/document/d/12hWGuzbqyNWc2l1ysbPcXwc0pzHEy4bodagrlNGCuQU/edit?usp=sharing)
 |
+| Community | [Vinod Kone](mailto:vinodk...@apache.org) | #community | [Every 
other Monday at 10:30 AM 
PST](https://docs.google.com/document/d/1vgi434dYkkZHs49EK4F4eMmM-3JG4f3qg-N5En-4ubg/edit?usp=sharing)
 |

mesos git commit: Added new reconfiguration_policy flag to docs/updgrades.md.

[vinodkone]

Repository: mesos
Updated Branches:
  refs/heads/1.5.x 944d74ad3 -> d6a73f6ab


Added new reconfiguration_policy flag to docs/updgrades.md.

Review: https://reviews.apache.org/r/65019/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/d6a73f6a
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/d6a73f6a
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/d6a73f6a

Branch: refs/heads/1.5.x
Commit: d6a73f6ab550b6798d644099274fcd7bde8d15f0
Parents: 944d74a
Author: Benno Evers 
Authored: Mon Jan 8 10:14:32 2018 -0800
Committer: Vinod Kone 
Committed: Mon Jan 8 10:15:05 2018 -0800

--
 docs/upgrades.md | 7 +++
 1 file changed, 7 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/d6a73f6a/docs/upgrades.md
--
diff --git a/docs/upgrades.md b/docs/upgrades.md
index f2eb5a9..29739cb 100644
--- a/docs/upgrades.md
+++ b/docs/upgrades.md
@@ -51,6 +51,7 @@ We categorize the changes as follows:
   C Built-in executors send a 
TASK_STARTING update
   A Network ports 
isolator
   C Relative source 
root paths for disk resources
+  A Agent state recovery after 
resource changes
 
   
 
@@ -59,6 +60,7 @@ We categorize the changes as follows:
   A container_ports_watch_interval
   A check_agent_port_range_only
   D executor_secret_key
+  A reconfiguration_policy
 
   
 
@@ -395,6 +397,11 @@ We categorize the changes as follows:
   * `show_nested`: Whether to show nested containers.
   * `show_standalone`: Whether to show standalone containers.
 
+
+* A new agent flag `--reconfiguration_policy` has been added. By setting the 
value of this flag to `additive`,
+  operators can allow the agent to be restarted with increased resources 
without requiring the agent ID to be
+  changed. Note that if this feature is used, the master version is required 
to be >= 1.5 as well.
+
 ## Upgrading from 1.3.x to 1.4.x ##
 
 

mesos git commit: Added new reconfiguration_policy flag to docs/updgrades.md.

[vinodkone]

Repository: mesos
Updated Branches:
  refs/heads/master 8be2b0f58 -> 07430e701


Added new reconfiguration_policy flag to docs/updgrades.md.

Review: https://reviews.apache.org/r/65019/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/07430e70
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/07430e70
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/07430e70

Branch: refs/heads/master
Commit: 07430e7011f0cf7df589e5022c7ed6fb70776b37
Parents: 8be2b0f
Author: Benno Evers 
Authored: Mon Jan 8 10:14:32 2018 -0800
Committer: Vinod Kone 
Committed: Mon Jan 8 10:14:32 2018 -0800

--
 docs/upgrades.md | 7 +++
 1 file changed, 7 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/07430e70/docs/upgrades.md
--
diff --git a/docs/upgrades.md b/docs/upgrades.md
index f2eb5a9..29739cb 100644
--- a/docs/upgrades.md
+++ b/docs/upgrades.md
@@ -51,6 +51,7 @@ We categorize the changes as follows:
   C Built-in executors send a 
TASK_STARTING update
   A Network ports 
isolator
   C Relative source 
root paths for disk resources
+  A Agent state recovery after 
resource changes
 
   
 
@@ -59,6 +60,7 @@ We categorize the changes as follows:
   A container_ports_watch_interval
   A check_agent_port_range_only
   D executor_secret_key
+  A reconfiguration_policy
 
   
 
@@ -395,6 +397,11 @@ We categorize the changes as follows:
   * `show_nested`: Whether to show nested containers.
   * `show_standalone`: Whether to show standalone containers.
 
+
+* A new agent flag `--reconfiguration_policy` has been added. By setting the 
value of this flag to `additive`,
+  operators can allow the agent to be restarted with increased resources 
without requiring the agent ID to be
+  changed. Note that if this feature is used, the master version is required 
to be >= 1.5 as well.
+
 ## Upgrading from 1.3.x to 1.4.x ##
 
 

[1/2] mesos git commit: Correctly reset slave status when aborting a registration.

[vinodkone]

Repository: mesos
Updated Branches:
  refs/heads/1.5.x 5453f3fa4 -> 79d189260


Correctly reset slave status when aborting a registration.

Previously, the slave was not erased from the `registering`
and `reregistering` sets in the master for some code paths
that would result in a failed (re-)registration attempt.

This could lead to a state where the reason of the unsuccessful
(re-)registration attempt is fixed on the agent, but the master
ignores subsequent attempts because it assumes the previous
operation is still in progress.

Review: https://reviews.apache.org/r/64506/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/4bc5f034
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/4bc5f034
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/4bc5f034

Branch: refs/heads/1.5.x
Commit: 4bc5f03497b64ae39d30418e803d9ded88acb74a
Parents: 5453f3f
Author: Benno Evers 
Authored: Tue Jan 2 10:58:23 2018 -0800
Committer: Vinod Kone 
Committed: Tue Jan 2 11:45:27 2018 -0800

--
 src/master/master.cpp  | 27 +++
 src/tests/master_tests.cpp | 16 
 2 files changed, 43 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/4bc5f034/src/master/master.cpp
--
diff --git a/src/master/master.cpp b/src/master/master.cpp
index 04378a8..4528687 100644
--- a/src/master/master.cpp
+++ b/src/master/master.cpp
@@ -6178,12 +6178,16 @@ void Master::_registerSlave(
 LOG(WARNING) << "Failed to parse version '" << version << "'"
  << " of agent at " << pid << ": " << parsedVersion.error()
  << "; ignoring agent registration attempt";
+
+slaves.registering.erase(pid);
 return;
   } else if (parsedVersion.get() < MINIMUM_AGENT_VERSION) {
 LOG(WARNING) << "Ignoring registration attempt from old agent at "
  << pid << ": agent version is " << parsedVersion.get()
  << ", minimum supported agent version is "
  << MINIMUM_AGENT_VERSION;
+
+slaves.registering.erase(pid);
 return;
   }
 
@@ -6198,6 +6202,8 @@ void Master::_registerSlave(
  << "domain " << slaveInfo.domain() << " "
  << "but the master has no configured domain. "
  << "Ignoring agent registration attempt";
+
+slaves.registering.erase(pid);
 return;
   }
 
@@ -6418,12 +6424,15 @@ void Master::reregisterSlave(
 << slaveInfo.id() << " at " << from << " ("
 << slaveInfo.hostname() << ")";
 
+  // TODO(bevers): Create a guard object calling `insert()` in its constructor
+  // and `erase()` in its destructor, to avoid the manual bookkeeping.
   slaves.reregistering.insert(slaveInfo.id());
 
   // Update all resources passed by the agent to `POST_RESERVATION_REFINEMENT`
   // format. We do this as early as possible so that we only use a single
   // format inside master, and downgrade again if necessary when they leave the
   // master (e.g. when writing to the registry).
+  //
   // TODO(bevers): Also convert the resources in `ExecutorInfos` and `Tasks`
   // here for consistency.
   convertResourceFormat(
@@ -6490,6 +6499,8 @@ void Master::_reregisterSlave(
   << "Ignoring re-register agent message from agent "
   << slaveInfo.id() << " at " << pid << " ("
   << slaveInfo.hostname() << ") as a gone operation is already in 
progress";
+
+slaves.reregistering.erase(slaveInfo.id());
 return;
   }
 
@@ -6500,6 +6511,8 @@ void Master::_reregisterSlave(
 ShutdownMessage message;
 message.set_message("Agent has been marked gone");
 send(pid, message);
+
+slaves.reregistering.erase(slaveInfo.id());
 return;
   }
 
@@ -6536,12 +6549,16 @@ void Master::_reregisterSlave(
 LOG(WARNING) << "Failed to parse version '" << version << "'"
  << " of agent at " << pid << ": " << parsedVersion.error()
  << "; ignoring agent re-registration attempt";
+
+slaves.reregistering.erase(slaveInfo.id());
 return;
   } else if (parsedVersion.get() < MINIMUM_AGENT_VERSION) {
 LOG(WARNING) << "Ignoring re-registration attempt from old agent at "
  << pid << ": agent version is " << parsedVersion.get()
  << ", minimum supported agent version is "
  << MINIMUM_AGENT_VERSION;
+
+slaves.reregistering.erase(slaveInfo.id());
 return;
   }
 
@@ -6556,6 +6573,8 @@ void Master::_reregisterSlave(
  << "domain " << slaveInfo.domain() << " "
  << "but the master has no configured domain."
  << "Ignoring agent re-registration attempt";
+
+

[2/2] mesos git commit: Added a master flag to disallow agents without domain.

[vinodkone]

Added a master flag to disallow agents without domain.

Added a new `--require_agent_domain` flag and implementation. When
set to true, it will cause the master to refuse (re-)registration
attempts for agents with no configured domain.

This is intended as a safety net for operators, who could forget to
configure the fault domain of a remote agent and let it join the
cluster. If this happens, an agent in a remote region will be
considered a local agent by the master and frameworks (because agent's
fault domain is not configured), causing tasks to potentially land in a
remote agent which is undesirable.

Review: https://reviews.apache.org/r/64507/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/79d18926
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/79d18926
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/79d18926

Branch: refs/heads/1.5.x
Commit: 79d189260eb705c73fd7fa356cef6e28351cb490
Parents: 4bc5f03
Author: Benno Evers 
Authored: Tue Jan 2 10:58:35 2018 -0800
Committer: Vinod Kone 
Committed: Tue Jan 2 11:45:33 2018 -0800

--
 docs/configuration/master.md |  8 
 src/master/flags.cpp |  5 +++
 src/master/flags.hpp |  1 +
 src/master/master.cpp| 24 ++
 src/tests/master_tests.cpp   | 93 +++
 5 files changed, 131 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/79d18926/docs/configuration/master.md
--
diff --git a/docs/configuration/master.md b/docs/configuration/master.md
index 6af16a0..3ff97ea 100644
--- a/docs/configuration/master.md
+++ b/docs/configuration/master.md
@@ -487,6 +487,14 @@ after which the operation is considered a failure. 
(default: 20secs)
 
 
   
+--[no-]require_agent_domain
+  
+  
+If true, only agents with a configured domain can register. (default: false)
+  
+
+
+  
 --roles=VALUE
   
   

http://git-wip-us.apache.org/repos/asf/mesos/blob/79d18926/src/master/flags.cpp
--
diff --git a/src/master/flags.cpp b/src/master/flags.cpp
index 18f405b..b49cb63 100644
--- a/src/master/flags.cpp
+++ b/src/master/flags.cpp
@@ -642,6 +642,11 @@ mesos::internal::master::Flags::Flags()
   "the IP address which the master will try to bind to.\n"
   "Cannot be used in conjunction with `--ip`.");
 
+  add(::require_agent_domain,
+  "require_agent_domain",
+  "If true, only agents with a configured domain can register.\n",
+  false);
+
   add(::domain,
   "domain",
   "Domain that the master belongs to. Mesos currently only supports\n"

http://git-wip-us.apache.org/repos/asf/mesos/blob/79d18926/src/master/flags.hpp
--
diff --git a/src/master/flags.hpp b/src/master/flags.hpp
index edda71a..dabb414 100644
--- a/src/master/flags.hpp
+++ b/src/master/flags.hpp
@@ -96,6 +96,7 @@ public:
   Duration registry_gc_interval;
   Duration registry_max_agent_age;
   size_t registry_max_agent_count;
+  bool require_agent_domain;
   Option domain;
 
   // The following flags are executable specific (e.g., since we only

http://git-wip-us.apache.org/repos/asf/mesos/blob/79d18926/src/master/master.cpp
--
diff --git a/src/master/master.cpp b/src/master/master.cpp
index 4528687..03eb178 100644
--- a/src/master/master.cpp
+++ b/src/master/master.cpp
@@ -6207,6 +6207,18 @@ void Master::_registerSlave(
 return;
   }
 
+  // Don't allow agents without domain if domains are required.
+  // We don't shutdown the agent to allow it to restart itself with
+  // the correct domain and without losing tasks.
+  if (flags.require_agent_domain && !slaveInfo.has_domain()) {
+LOG(WARNING) << "Agent at " << pid << " attempted to register without "
+ << "a domain, but this master is configured to require agent "
+ << "domains. Ignoring agent registration attempt";
+
+slaves.registering.erase(pid);
+return;
+  }
+
   // Check if this slave is already registered (because it retries).
   if (Slave* slave = slaves.registered.get(pid)) {
 if (!slave->connected) {
@@ -6578,6 +6590,18 @@ void Master::_reregisterSlave(
 return;
   }
 
+  // Don't allow agents without domain if domains are required.
+  // We don't shutdown the agent to allow it to restart itself with
+  // the correct domain and without losing tasks.
+  if (flags.require_agent_domain && !slaveInfo.has_domain()) {
+LOG(WARNING) << "Agent at " << pid << " attempted to register without "
+ << "a domain, but this master is configured to require agent 

[2/2] mesos git commit: Added a master flag to disallow agents without domain.

[vinodkone]

Added a master flag to disallow agents without domain.

Added a new `--require_agent_domain` flag and implementation. When
set to true, it will cause the master to refuse (re-)registration
attempts for agents with no configured domain.

This is intended as a safety net for operators, who could forget to
configure the fault domain of a remote agent and let it join the
cluster. If this happens, an agent in a remote region will be
considered a local agent by the master and frameworks (because agent's
fault domain is not configured), causing tasks to potentially land in a
remote agent which is undesirable.

Review: https://reviews.apache.org/r/64507/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/d02d6604
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/d02d6604
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/d02d6604

Branch: refs/heads/master
Commit: d02d66043e2beedbb7cd1c1bbf78ce8d506d82b7
Parents: 3eb57ca
Author: Benno Evers 
Authored: Tue Jan 2 10:58:35 2018 -0800
Committer: Vinod Kone 
Committed: Tue Jan 2 10:58:35 2018 -0800

--
 docs/configuration/master.md |  8 
 src/master/flags.cpp |  5 +++
 src/master/flags.hpp |  1 +
 src/master/master.cpp| 24 ++
 src/tests/master_tests.cpp   | 93 +++
 5 files changed, 131 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/d02d6604/docs/configuration/master.md
--
diff --git a/docs/configuration/master.md b/docs/configuration/master.md
index 6af16a0..3ff97ea 100644
--- a/docs/configuration/master.md
+++ b/docs/configuration/master.md
@@ -487,6 +487,14 @@ after which the operation is considered a failure. 
(default: 20secs)
 
 
   
+--[no-]require_agent_domain
+  
+  
+If true, only agents with a configured domain can register. (default: false)
+  
+
+
+  
 --roles=VALUE
   
   

http://git-wip-us.apache.org/repos/asf/mesos/blob/d02d6604/src/master/flags.cpp
--
diff --git a/src/master/flags.cpp b/src/master/flags.cpp
index 18f405b..b49cb63 100644
--- a/src/master/flags.cpp
+++ b/src/master/flags.cpp
@@ -642,6 +642,11 @@ mesos::internal::master::Flags::Flags()
   "the IP address which the master will try to bind to.\n"
   "Cannot be used in conjunction with `--ip`.");
 
+  add(::require_agent_domain,
+  "require_agent_domain",
+  "If true, only agents with a configured domain can register.\n",
+  false);
+
   add(::domain,
   "domain",
   "Domain that the master belongs to. Mesos currently only supports\n"

http://git-wip-us.apache.org/repos/asf/mesos/blob/d02d6604/src/master/flags.hpp
--
diff --git a/src/master/flags.hpp b/src/master/flags.hpp
index edda71a..dabb414 100644
--- a/src/master/flags.hpp
+++ b/src/master/flags.hpp
@@ -96,6 +96,7 @@ public:
   Duration registry_gc_interval;
   Duration registry_max_agent_age;
   size_t registry_max_agent_count;
+  bool require_agent_domain;
   Option domain;
 
   // The following flags are executable specific (e.g., since we only

http://git-wip-us.apache.org/repos/asf/mesos/blob/d02d6604/src/master/master.cpp
--
diff --git a/src/master/master.cpp b/src/master/master.cpp
index 4528687..03eb178 100644
--- a/src/master/master.cpp
+++ b/src/master/master.cpp
@@ -6207,6 +6207,18 @@ void Master::_registerSlave(
 return;
   }
 
+  // Don't allow agents without domain if domains are required.
+  // We don't shutdown the agent to allow it to restart itself with
+  // the correct domain and without losing tasks.
+  if (flags.require_agent_domain && !slaveInfo.has_domain()) {
+LOG(WARNING) << "Agent at " << pid << " attempted to register without "
+ << "a domain, but this master is configured to require agent "
+ << "domains. Ignoring agent registration attempt";
+
+slaves.registering.erase(pid);
+return;
+  }
+
   // Check if this slave is already registered (because it retries).
   if (Slave* slave = slaves.registered.get(pid)) {
 if (!slave->connected) {
@@ -6578,6 +6590,18 @@ void Master::_reregisterSlave(
 return;
   }
 
+  // Don't allow agents without domain if domains are required.
+  // We don't shutdown the agent to allow it to restart itself with
+  // the correct domain and without losing tasks.
+  if (flags.require_agent_domain && !slaveInfo.has_domain()) {
+LOG(WARNING) << "Agent at " << pid << " attempted to register without "
+ << "a domain, but this master is configured to require 

[1/2] mesos git commit: Correctly reset slave status when aborting a registration.

[vinodkone]

Repository: mesos
Updated Branches:
  refs/heads/master 5e88bc076 -> d02d66043


Correctly reset slave status when aborting a registration.

Previously, the slave was not erased from the `registering`
and `reregistering` sets in the master for some code paths
that would result in a failed (re-)registration attempt.

This could lead to a state where the reason of the unsuccessful
(re-)registration attempt is fixed on the agent, but the master
ignores subsequent attempts because it assumes the previous
operation is still in progress.

Review: https://reviews.apache.org/r/64506/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/3eb57cae
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/3eb57cae
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/3eb57cae

Branch: refs/heads/master
Commit: 3eb57cae3674fc835c784cac9eaa63e1aab7ba1c
Parents: 5e88bc0
Author: Benno Evers 
Authored: Tue Jan 2 10:58:23 2018 -0800
Committer: Vinod Kone 
Committed: Tue Jan 2 10:58:23 2018 -0800

--
 src/master/master.cpp  | 27 +++
 src/tests/master_tests.cpp | 16 
 2 files changed, 43 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/3eb57cae/src/master/master.cpp
--
diff --git a/src/master/master.cpp b/src/master/master.cpp
index 04378a8..4528687 100644
--- a/src/master/master.cpp
+++ b/src/master/master.cpp
@@ -6178,12 +6178,16 @@ void Master::_registerSlave(
 LOG(WARNING) << "Failed to parse version '" << version << "'"
  << " of agent at " << pid << ": " << parsedVersion.error()
  << "; ignoring agent registration attempt";
+
+slaves.registering.erase(pid);
 return;
   } else if (parsedVersion.get() < MINIMUM_AGENT_VERSION) {
 LOG(WARNING) << "Ignoring registration attempt from old agent at "
  << pid << ": agent version is " << parsedVersion.get()
  << ", minimum supported agent version is "
  << MINIMUM_AGENT_VERSION;
+
+slaves.registering.erase(pid);
 return;
   }
 
@@ -6198,6 +6202,8 @@ void Master::_registerSlave(
  << "domain " << slaveInfo.domain() << " "
  << "but the master has no configured domain. "
  << "Ignoring agent registration attempt";
+
+slaves.registering.erase(pid);
 return;
   }
 
@@ -6418,12 +6424,15 @@ void Master::reregisterSlave(
 << slaveInfo.id() << " at " << from << " ("
 << slaveInfo.hostname() << ")";
 
+  // TODO(bevers): Create a guard object calling `insert()` in its constructor
+  // and `erase()` in its destructor, to avoid the manual bookkeeping.
   slaves.reregistering.insert(slaveInfo.id());
 
   // Update all resources passed by the agent to `POST_RESERVATION_REFINEMENT`
   // format. We do this as early as possible so that we only use a single
   // format inside master, and downgrade again if necessary when they leave the
   // master (e.g. when writing to the registry).
+  //
   // TODO(bevers): Also convert the resources in `ExecutorInfos` and `Tasks`
   // here for consistency.
   convertResourceFormat(
@@ -6490,6 +6499,8 @@ void Master::_reregisterSlave(
   << "Ignoring re-register agent message from agent "
   << slaveInfo.id() << " at " << pid << " ("
   << slaveInfo.hostname() << ") as a gone operation is already in 
progress";
+
+slaves.reregistering.erase(slaveInfo.id());
 return;
   }
 
@@ -6500,6 +6511,8 @@ void Master::_reregisterSlave(
 ShutdownMessage message;
 message.set_message("Agent has been marked gone");
 send(pid, message);
+
+slaves.reregistering.erase(slaveInfo.id());
 return;
   }
 
@@ -6536,12 +6549,16 @@ void Master::_reregisterSlave(
 LOG(WARNING) << "Failed to parse version '" << version << "'"
  << " of agent at " << pid << ": " << parsedVersion.error()
  << "; ignoring agent re-registration attempt";
+
+slaves.reregistering.erase(slaveInfo.id());
 return;
   } else if (parsedVersion.get() < MINIMUM_AGENT_VERSION) {
 LOG(WARNING) << "Ignoring re-registration attempt from old agent at "
  << pid << ": agent version is " << parsedVersion.get()
  << ", minimum supported agent version is "
  << MINIMUM_AGENT_VERSION;
+
+slaves.reregistering.erase(slaveInfo.id());
 return;
   }
 
@@ -6556,6 +6573,8 @@ void Master::_reregisterSlave(
  << "domain " << slaveInfo.domain() << " "
  << "but the master has no configured domain."
  << "Ignoring agent re-registration attempt";
+
+

[8/8] mesos git commit: Skip registry update when nothing changed.

[vinodkone]

Skip registry update when nothing changed.

Most of the time, agents will not change their configuration
when they restart, so we can skip updating the registry in this
case.

Review: https://reviews.apache.org/r/64101/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/7b0812e9
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/7b0812e9
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/7b0812e9

Branch: refs/heads/master
Commit: 7b0812e9bf851b8baadadf1c1a95f3790a43d788
Parents: b612e9d
Author: Benno Evers 
Authored: Tue Dec 5 13:56:20 2017 -0800
Committer: Vinod Kone 
Committed: Tue Dec 5 13:58:52 2017 -0800

--
 src/master/master.cpp | 54 ++
 1 file changed, 45 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/7b0812e9/src/master/master.cpp
--
diff --git a/src/master/master.cpp b/src/master/master.cpp
index f9740e0..16cdde7 100644
--- a/src/master/master.cpp
+++ b/src/master/master.cpp
@@ -6582,9 +6582,11 @@ void Master::_reregisterSlave(
 
 // TODO(bevers): Verify that the checkpointed resources sent by the
 // slave match the ones stored in `slave`.
-registrar->apply(Owned(new UpdateSlave(slaveInfo)))
-  .onAny(defer(self(),
-  ::___reregisterSlave,
+
+// Skip updating the registry if `slaveInfo` did not change from its
+// previously known state.
+if (slaveInfo == slave->info) {
+  ___reregisterSlave(
   slaveInfo,
   pid,
   executorInfos,
@@ -6593,17 +6595,35 @@ void Master::_reregisterSlave(
   version,
   agentCapabilities,
   resourceVersions,
-  lambda::_1));
-
+  true);
+} else {
+  registrar->apply(Owned(new UpdateSlave(slaveInfo)))
+.onAny(defer(self(),
+::___reregisterSlave,
+slaveInfo,
+pid,
+executorInfos,
+tasks,
+frameworks,
+version,
+agentCapabilities,
+resourceVersions,
+lambda::_1));
+}
   } else if (slaves.recovered.contains(slaveInfo.id())) {
 // The agent likely is re-registering after a master failover as it
 // is in the list recovered from the registry.
 VLOG(1) << "Re-admitting recovered agent " << slaveInfo.id()
 << " at " << pid << "(" << slaveInfo.hostname() << ")";
 
-registrar->apply(Owned(new UpdateSlave(slaveInfo)))
-  .onAny(defer(self(),
-  ::__reregisterSlave,
+SlaveInfo recoveredInfo = slaves.recovered.at(slaveInfo.id());
+convertResourceFormat(
+recoveredInfo.mutable_resources(), POST_RESERVATION_REFINEMENT);
+
+// Skip updating the registry if `slaveInfo` did not change from its
+// previously known state (see also MESOS-7711).
+if (slaveInfo == recoveredInfo) {
+  __reregisterSlave(
   slaveInfo,
   pid,
   checkpointedResources,
@@ -6614,7 +6634,23 @@ void Master::_reregisterSlave(
   version,
   agentCapabilities,
   resourceVersions,
-  lambda::_1));
+  true);
+} else {
+  registrar->apply(Owned(new UpdateSlave(slaveInfo)))
+.onAny(defer(self(),
+::__reregisterSlave,
+slaveInfo,
+pid,
+checkpointedResources,
+executorInfos,
+tasks,
+frameworks,
+completedFrameworks,
+version,
+agentCapabilities,
+resourceVersions,
+lambda::_1));
+}
   } else {
 // In the common case, the slave has been marked unreachable
 // by the master, so we move the slave to the reachable list and

[2/8] mesos git commit: Added new UpdateSlave registry operation.

[vinodkone]

Added new UpdateSlave registry operation.

This operation can be used to update the stored state
of an existing, admitted slave.

Review: https://reviews.apache.org/r/64009/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/054bd5ab
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/054bd5ab
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/054bd5ab

Branch: refs/heads/master
Commit: 054bd5abc6b9788553cd5beb285bd46bbe2dec0d
Parents: 114519a
Author: Benno Evers 
Authored: Tue Dec 5 13:55:30 2017 -0800
Committer: Vinod Kone 
Committed: Tue Dec 5 13:55:30 2017 -0800

--
 src/master/registry_operations.cpp | 44 
 src/master/registry_operations.hpp | 14 +
 src/tests/registrar_tests.cpp  | 51 +
 3 files changed, 109 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/054bd5ab/src/master/registry_operations.cpp
--
diff --git a/src/master/registry_operations.cpp 
b/src/master/registry_operations.cpp
index f9c2162..149009b 100644
--- a/src/master/registry_operations.cpp
+++ b/src/master/registry_operations.cpp
@@ -51,6 +51,50 @@ Try AdmitSlave::perform(Registry* registry, 
hashset* slaveIDs)
 }
 
 
+UpdateSlave::UpdateSlave(const SlaveInfo& _info) : info(_info)
+{
+  CHECK(info.has_id()) << "SlaveInfo is missing the 'id' field";
+}
+
+
+Try UpdateSlave::perform(Registry* registry, hashset* slaveIDs)
+{
+  if (!slaveIDs->contains(info.id())) {
+return Error("Agent not yet admitted.");
+  }
+
+  for (int i = 0; i < registry->slaves().slaves().size(); i++) {
+Registry::Slave* slave = registry->mutable_slaves()->mutable_slaves(i);
+
+if (slave->info().id() == info.id()) {
+  // The SlaveInfo in the registry is stored in the
+  // `PRE_RESERVATION_REFINEMENT` format, but the equality operator
+  // asserts that resources are in `POST_RESERVATION_REFINEMENT` format,
+  // so we have to upgrade before we can do the comparison.
+  SlaveInfo _previousInfo(slave->info());
+  convertResourceFormat(_previousInfo.mutable_resources(),
+  POST_RESERVATION_REFINEMENT);
+
+  if (info == _previousInfo) {
+return false; // No mutation.
+  }
+
+  // Convert the resource format back to `PRE_RESERVATION_REFINEMENT` so
+  // the data stored in the registry can be read by older master versions.
+  SlaveInfo _info(info);
+  convertResourceFormat(_info.mutable_resources(),
+  PRE_RESERVATION_REFINEMENT);
+
+  slave->mutable_info()->CopyFrom(_info);
+  return true; // Mutation.
+}
+  }
+
+  // Shouldn't happen
+  return Error("Failed to find agent " + stringify(info.id()));
+}
+
+
 // Move a slave from the list of admitted slaves to the list of
 // unreachable slaves.
 MarkSlaveUnreachable::MarkSlaveUnreachable(

http://git-wip-us.apache.org/repos/asf/mesos/blob/054bd5ab/src/master/registry_operations.hpp
--
diff --git a/src/master/registry_operations.hpp 
b/src/master/registry_operations.hpp
index 5b78306..06f68a3 100644
--- a/src/master/registry_operations.hpp
+++ b/src/master/registry_operations.hpp
@@ -41,6 +41,20 @@ private:
 };
 
 
+// Update the SlaveInfo of an existing admitted slave.
+class UpdateSlave : public Operation
+{
+public:
+  explicit UpdateSlave(const SlaveInfo& _info);
+
+protected:
+  virtual Try perform(Registry* registry, hashset* slaveIDs);
+
+private:
+  const SlaveInfo info;
+};
+
+
 // Move a slave from the list of admitted slaves to the list of
 // unreachable slaves.
 class MarkSlaveUnreachable : public Operation

http://git-wip-us.apache.org/repos/asf/mesos/blob/054bd5ab/src/tests/registrar_tests.cpp
--
diff --git a/src/tests/registrar_tests.cpp b/src/tests/registrar_tests.cpp
index 21d78e9..f93129f 100644
--- a/src/tests/registrar_tests.cpp
+++ b/src/tests/registrar_tests.cpp
@@ -265,6 +265,57 @@ TEST_F(RegistrarTest, Admit)
 }
 
 
+TEST_F(RegistrarTest, UpdateSlave)
+{
+  // Add a new slave to the registry.
+  {
+Registrar registrar(flags, state);
+AWAIT_READY(registrar.recover(master));
+
+slave.set_hostname("original");
+AWAIT_TRUE(
+registrar.apply(Owned(new AdmitSlave(slave;
+  }
+
+
+  // Verify that the slave is present, and update its hostname.
+  {
+Registrar registrar(flags, state);
+Future registry = registrar.recover(master);
+AWAIT_READY(registry);
+
+ASSERT_EQ(1, registry->slaves().slaves().size());
+EXPECT_EQ("original", registry->slaves().slaves(0).info().hostname());
+
+slave.set_hostname("changed");
+  

[4/8] mesos git commit: Updated master behaviour to update agent state on reregistration.

[vinodkone]

Updated master behaviour to update agent state on reregistration.

When an agent reregisters, the master will now always update
the agent information it holds in memory, and will write any
changes back to the registry if necessary.

Note that most tests for this are added in a later review, since they
require the capability to actually restart the agent with
a changed state to effectively test the masters response to that.

Review: https://reviews.apache.org/r/64011/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/2e8c7663
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/2e8c7663
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/2e8c7663

Branch: refs/heads/master
Commit: 2e8c76631a8cd0f3a46c9865fa3eb83e2f425de9
Parents: 9015cd3
Author: Benno Evers 
Authored: Tue Dec 5 13:55:42 2017 -0800
Committer: Vinod Kone 
Committed: Tue Dec 5 13:55:42 2017 -0800

--
 src/master/allocator/mesos/hierarchical.cpp |  58 +++-
 src/master/allocator/mesos/hierarchical.hpp |   4 +
 src/master/master.cpp   | 383 +++
 src/master/master.hpp   |  19 +-
 src/tests/hierarchical_allocator_tests.cpp  |  70 +
 5 files changed, 389 insertions(+), 145 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/2e8c7663/src/master/allocator/mesos/hierarchical.cpp
--
diff --git a/src/master/allocator/mesos/hierarchical.cpp 
b/src/master/allocator/mesos/hierarchical.cpp
index b22829b..d348516 100644
--- a/src/master/allocator/mesos/hierarchical.cpp
+++ b/src/master/allocator/mesos/hierarchical.cpp
@@ -22,6 +22,7 @@
 #include 
 #include 
 
+#include 
 #include 
 #include 
 
@@ -622,14 +623,32 @@ void HierarchicalAllocatorProcess::updateSlave(
 
   Slave& slave = slaves.at(slaveId);
 
-  // We unconditionally overwrite the old domain and hostname: Even though
-  // the master places some restrictions on this (i.e. agents are not allowed
-  // to re-register with a different hostname) inside the allocator it
-  // doesn't matter, as the algorithm will work correctly either way.
-  slave.info = info;
-
   bool updated = false;
 
+  // Remove all offer filters for this slave if it was restarted with changed
+  // attributes. We do this because schedulers might have decided that they're
+  // not interested in offers from this slave based on the non-presence of some
+  // required attributes, and right now they have no other way of learning
+  // about this change.
+  // TODO(bennoe): Once the agent lifecycle design is implemented, there is a
+  // better way to notify frameworks about such changes and let them make this
+  // decision. We should think about ways to safely remove this check at that
+  // point in time.
+  if (!(Attributes(info.attributes()) == Attributes(slave.info.attributes( 
{
+updated = true;
+removeFilters(slaveId);
+  }
+
+  if (!(slave.info == info)) {
+updated = true;
+
+// We unconditionally overwrite the old domain and hostname: Even though
+// the master places some restrictions on this (i.e. agents are not allowed
+// to re-register with a different hostname) inside the allocator it
+// doesn't matter, as the algorithm will work correctly either way.
+slave.info = info;
+  }
+
   // Update agent capabilities.
   if (capabilities.isSome()) {
 protobuf::slave::Capabilities newCapabilities(capabilities.get());
@@ -698,6 +717,33 @@ void HierarchicalAllocatorProcess::addResourceProvider(
 }
 
 
+void HierarchicalAllocatorProcess::removeFilters(const SlaveID& slaveId)
+{
+  CHECK(initialized);
+
+  foreachpair (const FrameworkID& id,
+   Framework& framework,
+   frameworks) {
+framework.inverseOfferFilters.erase(slaveId);
+
+// Need a typedef here, otherwise the preprocessor gets confused
+// by the comma in the template argument list.
+typedef hashmap> Filters;
+foreachpair(const string& role,
+Filters& filters,
+framework.offerFilters) {
+  size_t erased = filters.erase(slaveId);
+  if (erased) {
+frameworkSorters.at(role)->activate(id.value());
+framework.suppressedRoles.erase(role);
+  }
+}
+  }
+
+  LOG(INFO) << "Removed all filters for agent " << slaveId;
+}
+
+
 void HierarchicalAllocatorProcess::activateSlave(
 const SlaveID& slaveId)
 {

http://git-wip-us.apache.org/repos/asf/mesos/blob/2e8c7663/src/master/allocator/mesos/hierarchical.hpp
--
diff --git a/src/master/allocator/mesos/hierarchical.hpp 
b/src/master/allocator/mesos/hierarchical.hpp
index 14dcf3d..0622026 

[7/8] mesos git commit: Check if slave is marked gone during reregistration.

[vinodkone]

Check if slave is marked gone during reregistration.

This commit adds checks that test whether a
re-registering agent is being marked gone or has been
marked gone during the re-registration attempt.

Review: https://reviews.apache.org/r/64169/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/b612e9d5
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/b612e9d5
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/b612e9d5

Branch: refs/heads/master
Commit: b612e9d511daa2a5f99ef984c7eb9aeeda79c4db
Parents: 63e3336
Author: Benno Evers 
Authored: Tue Dec 5 13:56:14 2017 -0800
Committer: Vinod Kone 
Committed: Tue Dec 5 13:58:52 2017 -0800

--
 src/master/master.cpp | 54 ++
 1 file changed, 54 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/b612e9d5/src/master/master.cpp
--
diff --git a/src/master/master.cpp b/src/master/master.cpp
index 6a52aad..f9740e0 100644
--- a/src/master/master.cpp
+++ b/src/master/master.cpp
@@ -6472,6 +6472,24 @@ void Master::_reregisterSlave(
 return;
   }
 
+  if (slaves.markingGone.contains(slaveInfo.id())) {
+LOG(INFO)
+  << "Ignoring re-register agent message from agent "
+  << slaveInfo.id() << " at " << pid << " ("
+  << slaveInfo.hostname() << ") as a gone operation is already in 
progress";
+return;
+  }
+
+  if (slaves.gone.contains(slaveInfo.id())) {
+LOG(WARNING) << "Refusing re-registration of agent at " << pid
+ << " because it is already marked gone";
+
+ShutdownMessage message;
+message.set_message("Agent has been marked gone");
+send(pid, message);
+return;
+  }
+
   VLOG(1) << "Authorized re-registration of agent " << slaveInfo.id()
   << " at " << pid << " (" << slaveInfo.hostname() << ")";
 
@@ -6651,6 +6669,24 @@ void Master::__reregisterSlave(
   // should ever fail.
   CHECK(future.get());
 
+  if (slaves.markingGone.contains(slaveInfo.id())) {
+LOG(INFO)
+  << "Ignoring re-register agent message from agent "
+  << slaveInfo.id() << " at " << pid << " ("
+  << slaveInfo.hostname() << ") as a gone operation is already in 
progress";
+return;
+  }
+
+  if (slaves.gone.contains(slaveInfo.id())) {
+LOG(WARNING) << "Refusing re-registration of agent at " << pid
+ << " because it is already marked gone";
+
+ShutdownMessage message;
+message.set_message("Agent has been marked gone");
+send(pid, message);
+return;
+  }
+
   VLOG(1) << "Re-admitted agent " << slaveInfo.id() << " at " << pid
   << " (" << slaveInfo.hostname() << ")";
 
@@ -6851,6 +6887,24 @@ void Master::___reregisterSlave(
   VLOG(1) << "Registry updated for slave " << slaveInfo.id() << " at " << pid
   << "(" << slaveInfo.hostname() << ")";
 
+  if (slaves.markingGone.contains(slaveInfo.id())) {
+LOG(INFO)
+  << "Ignoring re-register agent message from agent "
+  << slaveInfo.id() << " at " << pid << " ("
+  << slaveInfo.hostname() << ") as a gone operation is already in 
progress";
+return;
+  }
+
+  if (slaves.gone.contains(slaveInfo.id())) {
+LOG(WARNING) << "Refusing re-registration of agent at " << pid
+ << " because it is already marked gone";
+
+ShutdownMessage message;
+message.set_message("Agent has been marked gone");
+send(pid, message);
+return;
+  }
+
   if (!slaves.registered.contains(slaveInfo.id())) {
 LOG(WARNING)
   << "Dropping ongoing re-registration attempt of slave " << slaveInfo.id()

[5/8] mesos git commit: Activated AGENT_UPDATE master capability.

[vinodkone]

Activated AGENT_UPDATE master capability.

Review: https://reviews.apache.org/r/64008/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/f4d2c8f5
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/f4d2c8f5
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/f4d2c8f5

Branch: refs/heads/master
Commit: f4d2c8f547e42b65dd0040ddbe35157335577f6a
Parents: 2e8c766
Author: Benno Evers 
Authored: Tue Dec 5 13:55:52 2017 -0800
Committer: Vinod Kone 
Committed: Tue Dec 5 13:55:52 2017 -0800

--
 src/master/constants.cpp   | 13 -
 src/tests/master_tests.cpp | 10 +-
 2 files changed, 21 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/f4d2c8f5/src/master/constants.cpp
--
diff --git a/src/master/constants.cpp b/src/master/constants.cpp
index 55eecfb..1109b48 100644
--- a/src/master/constants.cpp
+++ b/src/master/constants.cpp
@@ -22,7 +22,18 @@ namespace master {
 
 std::vector MASTER_CAPABILITIES()
 {
-  return {}; // Empty for now.
+  MasterInfo::Capability::Type types[] = {
+MasterInfo::Capability::AGENT_UPDATE,
+  };
+
+  std::vector result;
+  foreach (MasterInfo::Capability::Type type, types) {
+MasterInfo::Capability capability;
+capability.set_type(type);
+result.push_back(capability);
+  }
+
+  return result;
 }
 
 } // namespace master {

http://git-wip-us.apache.org/repos/asf/mesos/blob/f4d2c8f5/src/tests/master_tests.cpp
--
diff --git a/src/tests/master_tests.cpp b/src/tests/master_tests.cpp
index 08742ec..64ac2d5 100644
--- a/src/tests/master_tests.cpp
+++ b/src/tests/master_tests.cpp
@@ -4515,7 +4515,15 @@ TEST_F(MasterTest, StateEndpoint)
   
state.values["unregistered_frameworks"].as().values.empty());
 
   ASSERT_TRUE(state.values["capabilities"].is());
-  EXPECT_TRUE(state.values["capabilities"].as().values.empty());
+  EXPECT_FALSE(state.values["capabilities"].as().values.empty());
+
+  JSON::Value masterCapabilities = state.values.at("capabilities");
+
+  // Master should always have the AGENT_UPDATE capability
+  Try expectedCapabilities = JSON::parse("[\"AGENT_UPDATE\"]");
+
+  ASSERT_SOME(expectedCapabilities);
+  EXPECT_TRUE(masterCapabilities.contains(expectedCapabilities.get()));
 }
 
 

[6/8] mesos git commit: Added new `--reconfiguration_policy` slave flag and implementation.

[vinodkone]

Added new `--reconfiguration_policy` slave flag and implementation.

This flag allows operators to select a set of permitted
configuration changes that the slave will tolerate during
recovery while still recovering running tasks and keeping
the same agent id.

The previous behaviour of Mesos is reproduced exactly by setting
this flag to "equal". For now only one additional policy is
provided, "additive".

Review: https://reviews.apache.org/r/64012/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/63e3336b
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/63e3336b
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/63e3336b

Branch: refs/heads/master
Commit: 63e3336b508f5fe3936ed701404cde7d70eee754
Parents: f4d2c8f
Author: Benno Evers 
Authored: Tue Dec 5 13:56:00 2017 -0800
Committer: Vinod Kone 
Committed: Tue Dec 5 13:58:39 2017 -0800

--
 src/CMakeLists.txt  |   1 +
 src/Makefile.am |   2 +
 src/slave/compatibility.cpp | 207 +
 src/slave/compatibility.hpp |  66 +++
 src/slave/flags.cpp |  14 ++
 src/slave/flags.hpp |   1 +
 src/slave/slave.cpp | 119 +
 src/slave/slave.hpp |   9 +
 src/tests/CMakeLists.txt|   1 +
 src/tests/master_tests.cpp  | 257 ++-
 src/tests/slave_compatibility_tests.cpp | 175 ++
 src/tests/slave_recovery_tests.cpp  | 135 ++
 src/tests/slave_tests.cpp   |  98 ++
 13 files changed, 1052 insertions(+), 33 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/63e3336b/src/CMakeLists.txt
--
diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
index 592489d..a76ba1e 100644
--- a/src/CMakeLists.txt
+++ b/src/CMakeLists.txt
@@ -246,6 +246,7 @@ configure_file(
 # SOURCE FILES FOR THE MESOS LIBRARY.
 #
 set(AGENT_SRC
+  slave/compatibility.cpp
   slave/constants.cpp
   slave/container_daemon.cpp
   slave/container_logger.cpp

http://git-wip-us.apache.org/repos/asf/mesos/blob/63e3336b/src/Makefile.am
--
diff --git a/src/Makefile.am b/src/Makefile.am
index d5ca797..05e8b95 100644
--- a/src/Makefile.am
+++ b/src/Makefile.am
@@ -1015,6 +1015,7 @@ libmesos_no_3rdparty_la_SOURCES +=
\
   sched/sched.cpp  \
   scheduler/scheduler.cpp  \
   secret/resolver.cpp  \
+  slave/compatibility.cpp  \
   slave/constants.cpp  \
   slave/container_daemon.cpp   \
   slave/container_logger.cpp   \
@@ -2494,6 +2495,7 @@ mesos_tests_SOURCES = 
\
   tests/scheduler_tests.cpp\
   tests/script.cpp \
   tests/slave_authorization_tests.cpp  \
+  tests/slave_compatibility_tests.cpp  \
   tests/slave_recovery_tests.cpp   \
   tests/slave_validation_tests.cpp \
   tests/slave_tests.cpp\

http://git-wip-us.apache.org/repos/asf/mesos/blob/63e3336b/src/slave/compatibility.cpp
--
diff --git a/src/slave/compatibility.cpp b/src/slave/compatibility.cpp
new file mode 100644
index 000..4ead4a5
--- /dev/null
+++ b/src/slave/compatibility.cpp
@@ -0,0 +1,207 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions 

[1/8] mesos git commit: Convert resource format of messages entering master.

[vinodkone]

Repository: mesos
Updated Branches:
  refs/heads/master 1d24d42ea -> 7b0812e9b


Convert resource format of messages entering master.

Review: https://reviews.apache.org/r/64252/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/114519a4
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/114519a4
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/114519a4

Branch: refs/heads/master
Commit: 114519a46366b0e19f1ae0c64dcd2493c51a068a
Parents: 1d24d42
Author: Benno Evers 
Authored: Tue Dec 5 13:55:22 2017 -0800
Committer: Vinod Kone 
Committed: Tue Dec 5 13:55:22 2017 -0800

--
 src/master/master.cpp  | 48 +++--
 src/master/registry_operations.cpp | 22 ---
 2 files changed, 52 insertions(+), 18 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/114519a4/src/master/master.cpp
--
diff --git a/src/master/master.cpp b/src/master/master.cpp
index e8257e7..f77a1ed 100644
--- a/src/master/master.cpp
+++ b/src/master/master.cpp
@@ -6044,6 +6044,20 @@ void Master::registerSlave(
 
   slaves.registering.insert(from);
 
+  // Update all resources passed by the agent to `POST_RESERVATION_REFINEMENT`
+  // format. We do this as early as possible so that we only use a single
+  // format inside master, and downgrade again if necessary when they leave the
+  // master (e.g. when writing to the registry).
+  // TODO(bevers): Also convert the resources in `ExecutorInfos` and `Tasks`
+  // here for consistency.
+  SlaveInfo _slaveInfo(slaveInfo);
+  convertResourceFormat(
+  _slaveInfo.mutable_resources(), POST_RESERVATION_REFINEMENT);
+
+  std::vector _checkpointedResources(checkpointedResources);
+  convertResourceFormat(
+  &_checkpointedResources, POST_RESERVATION_REFINEMENT);
+
   // Note that the principal may be empty if authentication is not
   // required. Also it is passed along because it may be removed from
   // `authenticated` while the authorization is pending.
@@ -6052,10 +6066,10 @@ void Master::registerSlave(
   authorizeSlave(principal)
 .onAny(defer(self(),
  ::_registerSlave,
- slaveInfo,
+ _slaveInfo,
  from,
  principal,
- checkpointedResources,
+ _checkpointedResources,
  version,
  agentCapabilities,
  resourceVersions,
@@ -6361,6 +6375,20 @@ void Master::reregisterSlave(
   Option error = validation::master::message::reregisterSlave(
   slaveInfo, tasks, checkpointedResources, executorInfos, frameworks);
 
+  // Update all resources passed by the agent to `POST_RESERVATION_REFINEMENT`
+  // format. We do this as early as possible so that we only use a single
+  // format inside master, and downgrade again if necessary when they leave the
+  // master (e.g. when writing to the registry).
+  // TODO(bevers): Also convert the resources in `ExecutorInfos` and `Tasks`
+  // here for consistency.
+  SlaveInfo _slaveInfo(slaveInfo);
+  convertResourceFormat(
+  _slaveInfo.mutable_resources(), POST_RESERVATION_REFINEMENT);
+
+  std::vector _checkpointedResources(checkpointedResources);
+  convertResourceFormat(
+  &_checkpointedResources, POST_RESERVATION_REFINEMENT);
+
   if (error.isSome()) {
 LOG(WARNING) << "Dropping re-registration of agent at " << from
  << " because it sent an invalid re-registration: "
@@ -6382,10 +6410,10 @@ void Master::reregisterSlave(
   authorizeSlave(principal)
 .onAny(defer(self(),
  ::_reregisterSlave,
- slaveInfo,
+ _slaveInfo,
  from,
  principal,
- checkpointedResources,
+ _checkpointedResources,
  executorInfos,
  tasks,
  frameworks,
@@ -11024,11 +11052,7 @@ Slave::Slave(
 const vector& tasks)
   : master(_master),
 id(_info.id()),
-info([&_info]() {
-  convertResourceFormat(
-  _info.mutable_resources(), POST_RESERVATION_REFINEMENT);
-  return _info;
-}()),
+info(_info),
 machineId(_machineId),
 pid(_pid),
 version(_version),
@@ -11036,11 +11060,7 @@ Slave::Slave(
 registeredTime(_registeredTime),
 connected(true),
 active(true),
-checkpointedResources([&_checkpointedResources]() {
-  convertResourceFormat(
-  &_checkpointedResources, POST_RESERVATION_REFINEMENT);
-  return _checkpointedResources;
-}()),
+checkpointedResources(_checkpointedResources),
 observer(nullptr),
 resourceVersions(_resourceVersions)
 {

mesos git commit: Fixed an issue with the scheduler driver subscribe backoff time.

[vinodkone]

Repository: mesos
Updated Branches:
  refs/heads/master 6a98530a5 -> 3caca9716


Fixed an issue with the scheduler driver subscribe backoff time.

When framework failover time is set to zero (which is the
default value), the scheduler driver subscribe backoff time
will also become zero. Ignore failover time if it is zero
when deciding the subscribe backoff time.
Also added a dedicated test.

Review: https://reviews.apache.org/r/63765/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/3caca971
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/3caca971
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/3caca971

Branch: refs/heads/master
Commit: 3caca9716314bba6d87c9e5a8654f4d09fbaf9c2
Parents: 6a98530
Author: Meng Zhu 
Authored: Mon Dec 4 13:12:23 2017 -0800
Committer: Vinod Kone 
Committed: Mon Dec 4 13:12:23 2017 -0800

--
 src/sched/sched.cpp  |  2 +-
 src/tests/scheduler_driver_tests.cpp | 47 +++
 2 files changed, 48 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/3caca971/src/sched/sched.cpp
--
diff --git a/src/sched/sched.cpp b/src/sched/sched.cpp
index 6028499..72bd413 100644
--- a/src/sched/sched.cpp
+++ b/src/sched/sched.cpp
@@ -844,7 +844,7 @@ protected:
 // by 1/10th of the failover timeout.
 if (framework.has_failover_timeout()) {
   Try duration = Duration::create(framework.failover_timeout());
-  if (duration.isSome()) {
+  if (duration.isSome() && duration.get() > Duration::zero()) {
 maxBackoff = std::min(maxBackoff, duration.get() / 10);
   }
 }

http://git-wip-us.apache.org/repos/asf/mesos/blob/3caca971/src/tests/scheduler_driver_tests.cpp
--
diff --git a/src/tests/scheduler_driver_tests.cpp 
b/src/tests/scheduler_driver_tests.cpp
index 14d872b..46af8d2 100644
--- a/src/tests/scheduler_driver_tests.cpp
+++ b/src/tests/scheduler_driver_tests.cpp
@@ -68,6 +68,7 @@ using testing::_;
 using testing::AtMost;
 using testing::DoAll;
 using testing::Return;
+using ::testing::InSequence;
 
 namespace mesos {
 namespace internal {
@@ -76,6 +77,52 @@ namespace tests {
 class MesosSchedulerDriverTest : public MesosTest {};
 
 
+// Ensures that the scheduler driver can backoff correctly when
+// framework failover time is set to zero.
+TEST_F(MesosSchedulerDriverTest, RegistrationWithZeroFailoverTime)
+{
+  Try master = StartMaster();
+  ASSERT_SOME(master);
+
+  FrameworkInfo frameworkInfo = DEFAULT_FRAMEWORK_INFO;
+  frameworkInfo.set_failover_timeout(0);
+
+  MockScheduler sched;
+  MesosSchedulerDriver driver(
+  , frameworkInfo, master.get()->pid, DEFAULT_CREDENTIAL);
+
+  Clock::pause();
+
+  {
+InSequence inSequence;
+
+// Drop the first SUBSCRIBE call so that the driver retries.
+DROP_CALL(
+mesos::scheduler::Call(),
+mesos::scheduler::Call::SUBSCRIBE,
+_,
+master.get()->pid);
+
+// Settling the clock ensures that if a retried SUBSCRIBE
+// call is enqueued it will be processed. Since the backoff
+// is non-zero no new SUBSCRIBE call should be sent when
+// the clock is paused.
+EXPECT_NO_FUTURE_CALLS(
+mesos::scheduler::Call(),
+mesos::scheduler::Call::SUBSCRIBE,
+_,
+master.get()->pid);
+  }
+
+  driver.start();
+
+  Clock::settle();
+
+  driver.stop();
+  driver.join();
+}
+
+
 TEST_F(MesosSchedulerDriverTest, MetricsEndpoint)
 {
   Try master = StartMaster();

[2/2] mesos git commit: Added infrastructure to support master capabilities.

[vinodkone]

Added infrastructure to support master capabilities.

This includes the necessary constants in the protobuf
definition as well as various support methods to get
the current master capabilities.

Note that the actual feature is implemented in a later
commit, so the capability is not yet activated on the
master.

Review: https://reviews.apache.org/r/64007/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/17d186e3
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/17d186e3
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/17d186e3

Branch: refs/heads/master
Commit: 17d186e379d2db3edb04743a061dc05e24c32c9b
Parents: dfb2c62
Author: Benno Evers 
Authored: Wed Nov 29 10:21:57 2017 -0800
Committer: Vinod Kone 
Committed: Wed Nov 29 10:21:57 2017 -0800

--
 include/mesos/mesos.proto | 13 +
 include/mesos/v1/mesos.proto  | 13 +
 src/CMakeLists.txt|  1 +
 src/Makefile.am   |  1 +
 src/common/protobuf_utils.cpp |  6 ++
 src/common/protobuf_utils.hpp | 25 
 src/master/constants.cpp  | 39 ++
 src/master/constants.hpp  |  4 
 src/master/http.cpp   |  9 +
 src/master/master.cpp |  4 
 src/tests/master_tests.cpp|  3 +++
 11 files changed, 118 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/17d186e3/include/mesos/mesos.proto
--
diff --git a/include/mesos/mesos.proto b/include/mesos/mesos.proto
index 8c28b81..b1ebfe2 100644
--- a/include/mesos/mesos.proto
+++ b/include/mesos/mesos.proto
@@ -884,6 +884,19 @@ message MasterInfo {
   // The domain that this master belongs to. All masters in a Mesos
   // cluster should belong to the same region.
   optional DomainInfo domain = 8;
+
+  message Capability {
+enum Type {
+  UNKNOWN = 0;
+
+  // The master can handle slaves whose state
+  // changes after re-registering.
+  AGENT_UPDATE = 1;
+}
+optional Type type = 1;
+  }
+
+  repeated Capability capabilities = 9;
 }
 
 

http://git-wip-us.apache.org/repos/asf/mesos/blob/17d186e3/include/mesos/v1/mesos.proto
--
diff --git a/include/mesos/v1/mesos.proto b/include/mesos/v1/mesos.proto
index 35bc6e1..d535eb4 100644
--- a/include/mesos/v1/mesos.proto
+++ b/include/mesos/v1/mesos.proto
@@ -882,6 +882,19 @@ message MasterInfo {
   // The domain that this master belongs to. All masters in a Mesos
   // cluster should belong to the same region.
   optional DomainInfo domain = 8;
+
+  message Capability {
+enum Type {
+  UNKNOWN = 0;
+
+  // The master can handle slaves whose state
+  // changes after re-registering.
+  AGENT_UPDATE = 1;
+}
+optional Type type = 1;
+  }
+
+  repeated Capability capabilities = 9;
 }
 
 

http://git-wip-us.apache.org/repos/asf/mesos/blob/17d186e3/src/CMakeLists.txt
--
diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
index 6b50062..3f5d946 100644
--- a/src/CMakeLists.txt
+++ b/src/CMakeLists.txt
@@ -427,6 +427,7 @@ set(LOGGING_SRC
   logging/logging.cpp)
 
 set(MASTER_SRC
+  master/constants.cpp
   master/flags.cpp
   master/http.cpp
   master/maintenance.cpp

http://git-wip-us.apache.org/repos/asf/mesos/blob/17d186e3/src/Makefile.am
--
diff --git a/src/Makefile.am b/src/Makefile.am
index ec70b5f..b1ed4e5 100644
--- a/src/Makefile.am
+++ b/src/Makefile.am
@@ -973,6 +973,7 @@ libmesos_no_3rdparty_la_SOURCES +=  
\
   local/local.cpp  \
   logging/flags.cpp\
   logging/logging.cpp  \
+  master/constants.cpp  \
   master/flags.cpp \
   master/http.cpp  \
   master/maintenance.cpp   \

http://git-wip-us.apache.org/repos/asf/mesos/blob/17d186e3/src/common/protobuf_utils.cpp
--
diff --git a/src/common/protobuf_utils.cpp b/src/common/protobuf_utils.cpp
index c0ff306..d557342 100644
--- a/src/common/protobuf_utils.cpp
+++ b/src/common/protobuf_utils.cpp
@@ -51,6 +51,7 @@
 #include "common/resources_utils.hpp"
 
 #include "master/master.hpp"
+#include "master/constants.hpp"
 
 #include "messages/messages.hpp"
 
@@ -534,6 +535,11 @@ MasterInfo 

mesos git commit: Fix bad links.

[vinodkone]

Repository: mesos
Updated Branches:
  refs/heads/master 677305f0b -> a98363fb9


Fix bad links.

Review: https://reviews.apache.org/r/63400/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/a98363fb
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/a98363fb
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/a98363fb

Branch: refs/heads/master
Commit: a98363fb9df4023e3719f70ba734bed54e222bda
Parents: 677305f
Author: Tomasz Janiszewski 
Authored: Wed Nov 8 10:35:33 2017 -0800
Committer: Vinod Kone 
Committed: Wed Nov 8 10:35:33 2017 -0800

--
 docs/app-framework-development-guide.md | 2 +-
 docs/cmake.md   | 5 ++---
 docs/cni.md | 2 +-
 docs/isolators/network-port-mapping.md  | 2 +-
 docs/upgrades.md| 2 +-
 docs/working-groups.md  | 4 ++--
 site/source/blog/2014-03-28-mesos-community-update-1.md | 4 ++--
 site/source/layouts/community_section.erb   | 2 +-
 8 files changed, 11 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/a98363fb/docs/app-framework-development-guide.md
--
diff --git a/docs/app-framework-development-guide.md 
b/docs/app-framework-development-guide.md
index 6e73e4e..e31bffc 100644
--- a/docs/app-framework-development-guide.md
+++ b/docs/app-framework-development-guide.md
@@ -362,7 +362,7 @@ Executor implementation. For example, you may not want a 
1:1 relationship
 between tasks and processes.
 
 If you are writing an executor against Mesos 1.0 or newer, it is recommended
-to use the new [HTTP API](executor-http-api) to talk to Mesos.
+to use the new [HTTP API](executor-http-api.md) to talk to Mesos.
 
 If writing against Mesos 0.28.0 or older, your framework executor must inherit
 from the Executor class. It must override the launchTask() method. You can use

http://git-wip-us.apache.org/repos/asf/mesos/blob/a98363fb/docs/cmake.md
--
diff --git a/docs/cmake.md b/docs/cmake.md
index 84cc319..b8f9e7c 100644
--- a/docs/cmake.md
+++ b/docs/cmake.md
@@ -242,13 +242,12 @@ part of the build (such as protobuf or the JNI headers).
 
 ## Checking `CMAKE_BUILD_TYPE`
 
-See the ["Building debug or release configurations"][config] example for more
-information. In short, not all generators respect the variable
+See the ["Building debug or release 
configurations"](cmake-examples.md#building-debug-or-release-configurations)
+example for more information. In short, not all generators respect the variable
 `CMAKE_BUILD_TYPE` at configuration time, and thus it must not be used in CMake
 logic. A usable alternative (where supported) is a [generator expression][] 
such
 as `$<$:DEBUG_MODE>`.
 
-[config]: cmake-examples.md#building-debug-or-release-configurations
 [generator expression]: 
https://cmake.org/cmake/help/latest/manual/cmake-generator-expressions.7.html#logical-expressions
 
 # Remaining hacks

http://git-wip-us.apache.org/repos/asf/mesos/blob/a98363fb/docs/cni.md
--
diff --git a/docs/cni.md b/docs/cni.md
index 09159cd..c45ce27 100644
--- a/docs/cni.md
+++ b/docs/cni.md
@@ -392,7 +392,7 @@ that services running on the container can be exposed 
outside the host
 on which the container is running.
 
 Unfortunately, there is no CNI plugin available in the
-[containernetworking/cni](https://github.com/containernetwork/cni)
+[containernetworking/cni](https://github.com/containernetworking/cni)
 repository that provides port-mapping functionality.
 Hence, we have developed a port-mapper CNI plugin that resides
 within the Mesos code base called the `mesos-cni-port-mapper`. The

http://git-wip-us.apache.org/repos/asf/mesos/blob/a98363fb/docs/isolators/network-port-mapping.md
--
diff --git a/docs/isolators/network-port-mapping.md 
b/docs/isolators/network-port-mapping.md
index 522f2ed..250aa87 100644
--- a/docs/isolators/network-port-mapping.md
+++ b/docs/isolators/network-port-mapping.md
@@ -182,7 +182,7 @@ allocated by the framework, limiting container transmit 
bandwidth to
 ## Monitoring container network statistics
 
 Mesos exposes statistics from the Linux network stack for each container 
network
-on the [/monitor/statistics](endpoints/slave/monitor/statistics.md) agent 
endpoint.
+on the [/monitor/statistics](../endpoints/slave/monitor/statistics.md) agent 
endpoint.
 
 From the network interface inside the container, we report the following