[3/3] mesos git commit: Removed two std::move in MountInfoTable::read.

2016-10-12 Thread jieyu 
Removed two std::move in MountInfoTable::read.

Review: https://reviews.apache.org/r/51620/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/0ab140ba
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/0ab140ba
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/0ab140ba

Branch: refs/heads/1.0.x
Commit: 0ab140bacc3642deebc1c64af00ddaf8a3393d9c
Parents: 1f04aee
Author: Jie Yu 
Authored: Wed Oct 12 22:49:45 2016 -0700
Committer: Jie Yu 
Committed: Wed Oct 12 22:56:12 2016 -0700

--
 src/linux/fs.cpp | 9 -
 1 file changed, 4 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/0ab140ba/src/linux/fs.cpp
--
diff --git a/src/linux/fs.cpp b/src/linux/fs.cpp
index 1cb329a..978953f 100644
--- a/src/linux/fs.cpp
+++ b/src/linux/fs.cpp
@@ -126,7 +126,7 @@ Try MountInfoTable::read(
   if (entry.target == "/") {
 rootParentId = entry.parent;
   }
-  parentToChildren[entry.parent].push_back(std::move(entry));
+  parentToChildren[entry.parent].push_back(entry);
 }
 
 // Walk the hashmap and construct a list of entries sorted
@@ -144,8 +144,7 @@ Try MountInfoTable::read(
   visitedParents.insert(parentId);
 
   foreach (const MountInfoTable::Entry& entry, parentToChildren[parentId]) 
{
-int newParentId = entry.id;
-sortedEntries.push_back(std::move(entry));
+sortedEntries.push_back(entry);
 
 // It is legal to have a `MountInfoTable` entry whose
 // `entry.id` is the same as its `entry.parent`. This can
@@ -153,8 +152,8 @@ Try MountInfoTable::read(
 // and then keeps the original `/` in RAM. To avoid cycles
 // when walking the mount hierarchy, we only recurse into our
 // children if this case is not satisfied.
-if (parentId != newParentId) {
-  sortFrom(newParentId);
+if (parentId != entry.id) {
+  sortFrom(entry.id);
 }
   }
 };

[2/3] mesos git commit: Added special case when sorting hierarchically in MountInfoTable::read.

2016-10-12 Thread jieyu 
Added special case when sorting hierarchically in MountInfoTable::read.

It is legal to have entries in a `MountInfoTable` whose `entry.id` is
the same as `entry.parent`. This can happen (for example), if a system
boots from the network and then keeps the original `/` in RAM.
However, to avoid cycles when walking the mount hierarchy, we should
not treat these entries as children of their parent so we skip them.

This commit adds functionality to handle this case.

Review: https://reviews.apache.org/r/52596/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/1f04aee1
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/1f04aee1
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/1f04aee1

Branch: refs/heads/1.0.x
Commit: 1f04aee1ed5376d2403818b71f373ca672b7ade1
Parents: ac3d9b8
Author: Kevin Klues 
Authored: Wed Oct 12 22:33:56 2016 -0700
Committer: Jie Yu 
Committed: Wed Oct 12 22:46:36 2016 -0700

--
 src/linux/fs.cpp | 11 ++-
 1 file changed, 10 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/1f04aee1/src/linux/fs.cpp
--
diff --git a/src/linux/fs.cpp b/src/linux/fs.cpp
index d4f2ac2..1cb329a 100644
--- a/src/linux/fs.cpp
+++ b/src/linux/fs.cpp
@@ -146,7 +146,16 @@ Try MountInfoTable::read(
   foreach (const MountInfoTable::Entry& entry, parentToChildren[parentId]) 
{
 int newParentId = entry.id;
 sortedEntries.push_back(std::move(entry));
-sortFrom(newParentId);
+
+// It is legal to have a `MountInfoTable` entry whose
+// `entry.id` is the same as its `entry.parent`. This can
+// happen (for example), if a system boots from the network
+// and then keeps the original `/` in RAM. To avoid cycles
+// when walking the mount hierarchy, we only recurse into our
+// children if this case is not satisfied.
+if (parentId != newParentId) {
+  sortFrom(newParentId);
+}
   }
 };

[1/3] mesos git commit: Added more detailed error message when failing in MountInfoTable::read.

2016-10-12 Thread jieyu 
Repository: mesos
Updated Branches:
  refs/heads/1.0.x 6174d018e -> 0ab140bac


Added more detailed error message when failing in MountInfoTable::read.

Review: https://reviews.apache.org/r/52597/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/ac3d9b8f
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/ac3d9b8f
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/ac3d9b8f

Branch: refs/heads/1.0.x
Commit: ac3d9b8f3011ea4314600eee234c775f9a8252cc
Parents: 6174d01
Author: Kevin Klues 
Authored: Wed Oct 12 22:33:51 2016 -0700
Committer: Jie Yu 
Committed: Wed Oct 12 22:46:35 2016 -0700

--
 src/linux/fs.cpp | 5 -
 1 file changed, 4 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/ac3d9b8f/src/linux/fs.cpp
--
diff --git a/src/linux/fs.cpp b/src/linux/fs.cpp
index 0d025d3..d4f2ac2 100644
--- a/src/linux/fs.cpp
+++ b/src/linux/fs.cpp
@@ -137,7 +137,10 @@ Try MountInfoTable::read(
 vector sortedEntries;
 
 std::function sortFrom = [&](int parentId) {
-  CHECK(!visitedParents.contains(parentId));
+  CHECK(!visitedParents.contains(parentId))
+<< "Cycle found in mount table hierarchy at entry"
+<< " '" << stringify(parentId) << "': " << std::endl << lines.get();
+
   visitedParents.insert(parentId);
 
   foreach (const MountInfoTable::Entry& entry, parentToChildren[parentId]) 
{

[2/7] mesos git commit: Added special case when sorting hierarchically in MountInfoTable::read.

2016-10-12 Thread jieyu 
Added special case when sorting hierarchically in MountInfoTable::read.

It is legal to have entries in a `MountInfoTable` whose `entry.id` is
the same as `entry.parent`. This can happen (for example), if a system
boots from the network and then keeps the original `/` in RAM.
However, to avoid cycles when walking the mount hierarchy, we should
not treat these entries as children of their parent so we skip them.

This commit adds functionality to handle this case.

Review: https://reviews.apache.org/r/52596/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/ccc746a7
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/ccc746a7
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/ccc746a7

Branch: refs/heads/master
Commit: ccc746a7d12cc524120a76aa49a0d69e7303608a
Parents: 70b227f
Author: Kevin Klues 
Authored: Wed Oct 12 22:33:56 2016 -0700
Committer: Jie Yu 
Committed: Wed Oct 12 22:33:56 2016 -0700

--
 src/linux/fs.cpp | 11 ++-
 1 file changed, 10 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/ccc746a7/src/linux/fs.cpp
--
diff --git a/src/linux/fs.cpp b/src/linux/fs.cpp
index d49d1f5..aeec882 100644
--- a/src/linux/fs.cpp
+++ b/src/linux/fs.cpp
@@ -147,7 +147,16 @@ Try MountInfoTable::read(
   foreach (const MountInfoTable::Entry& entry, parentToChildren[parentId]) 
{
 int newParentId = entry.id;
 sortedEntries.push_back(std::move(entry));
-sortFrom(newParentId);
+
+// It is legal to have a `MountInfoTable` entry whose
+// `entry.id` is the same as its `entry.parent`. This can
+// happen (for example), if a system boots from the network
+// and then keeps the original `/` in RAM. To avoid cycles
+// when walking the mount hierarchy, we only recurse into our
+// children if this case is not satisfied.
+if (parentId != newParentId) {
+  sortFrom(newParentId);
+}
   }
 };

[6/7] mesos git commit: Added 'stout/wait.hpp' header.

2016-10-12 Thread jieyu 
Added 'stout/wait.hpp' header.

This was motivated by the need for a default definition of
'W_EXITCODE' (since it is not technically POSIX compliant).

Review: https://reviews.apache.org/r/52705/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/fa56d045
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/fa56d045
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/fa56d045

Branch: refs/heads/master
Commit: fa56d0458860b2b7f035c793a192d7b55180a125
Parents: de69a72
Author: Kevin Klues 
Authored: Wed Oct 12 22:34:21 2016 -0700
Committer: Jie Yu 
Committed: Wed Oct 12 22:34:21 2016 -0700

--
 3rdparty/stout/include/Makefile.am   |  1 +
 3rdparty/stout/include/stout/os/wait.hpp | 37 +++
 2 files changed, 38 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/fa56d045/3rdparty/stout/include/Makefile.am
--
diff --git a/3rdparty/stout/include/Makefile.am 
b/3rdparty/stout/include/Makefile.am
index b0b08d8..1eb9c14 100644
--- a/3rdparty/stout/include/Makefile.am
+++ b/3rdparty/stout/include/Makefile.am
@@ -102,6 +102,7 @@ nobase_include_HEADERS =\
   stout/os/sysctl.hpp  \
   stout/os/touch.hpp   \
   stout/os/utime.hpp   \
+  stout/os/wait.hpp\
   stout/os/write.hpp   \
   stout/os/posix/bootid.hpp\
   stout/os/posix/chown.hpp \

http://git-wip-us.apache.org/repos/asf/mesos/blob/fa56d045/3rdparty/stout/include/stout/os/wait.hpp
--
diff --git a/3rdparty/stout/include/stout/os/wait.hpp 
b/3rdparty/stout/include/stout/os/wait.hpp
new file mode 100644
index 000..06cf10c
--- /dev/null
+++ b/3rdparty/stout/include/stout/os/wait.hpp
@@ -0,0 +1,37 @@
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//  http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#ifndef __STOUT_OS_WAIT_HPP__
+#define __STOUT_OS_WAIT_HPP__
+
+#ifdef __WINDOWS__
+// TODO(klueska): Move all `WAIT` related functions out of
+// `windows.hpp` and into this file.
+#include 
+#else
+#include 
+#endif // __WINDOWS__
+
+
+#ifndef W_EXITCODE
+#define W_EXITCODE(ret, sig)  ((ret) << 8 | (sig))
+#endif
+
+
+namespace os {
+
+// TODO(klueska): Add helper functions for common wait related
+// operations in this header file.
+
+} // namespace os {
+
+#endif // __STOUT_OS_WAIT_HPP__

[4/7] mesos git commit: Added test to test corner cases with sorted 'MountInfoTable::read()'.

2016-10-12 Thread jieyu 
Added test to test corner cases with sorted 'MountInfoTable::read()'.

We allow entries in the MountInfoTable to be out of order, as well as
parent's of themselves. This test makes sure that this functionality
is exercised.

Review: https://reviews.apache.org/r/52703/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/2f005a03
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/2f005a03
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/2f005a03

Branch: refs/heads/master
Commit: 2f005a03d59948c9a41b3fea8f81332e4e74ab3c
Parents: 44c4383
Author: Kevin Klues 
Authored: Wed Oct 12 22:34:15 2016 -0700
Committer: Jie Yu 
Committed: Wed Oct 12 22:34:15 2016 -0700

--
 src/tests/containerizer/fs_tests.cpp | 35 +++
 1 file changed, 35 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/2f005a03/src/tests/containerizer/fs_tests.cpp
--
diff --git a/src/tests/containerizer/fs_tests.cpp 
b/src/tests/containerizer/fs_tests.cpp
index 0dd212f..d8bd1d0 100644
--- a/src/tests/containerizer/fs_tests.cpp
+++ b/src/tests/containerizer/fs_tests.cpp
@@ -22,6 +22,7 @@
 
 #include 
 #include 
+#include 
 #include 
 #include 
 #include 
@@ -184,6 +185,40 @@ TEST_F(FsTest, MountInfoTableReadSorted)
 }
 
 
+TEST_F(FsTest, MountInfoTableReadSortedParentOfSelf)
+{
+  // Construct a mount info table with a few entries out of order as
+  // well as a few having themselves as parents.
+  string lines =
+"1 1 0:00 / / rw shared:6 - sysfs sysfs rw\n"
+"6 5 0:00 / /6 rw shared:6 - sysfs sysfs rw\n"
+"7 6 0:00 / /7 rw shared:6 - sysfs sysfs rw\n"
+"8 8 0:00 / /8 rw shared:6 - sysfs sysfs rw\n"
+"9 8 0:00 / /9 rw shared:6 - sysfs sysfs rw\n"
+"2 1 0:00 / /2 rw shared:6 - sysfs sysfs rw\n"
+"3 2 0:00 / /3 rw shared:6 - sysfs sysfs rw\n"
+"4 3 0:00 / /4 rw shared:6 - sysfs sysfs rw\n"
+"5 4 0:00 / /5 rw shared:6 - sysfs sysfs rw\n";
+
+  // Examine the calling process's mountinfo table.
+  Try table = MountInfoTable::read(lines);
+  ASSERT_SOME(table);
+
+  hashset ids;
+
+  // Verify that all parent entries appear *before* their children.
+  foreach (const MountInfoTable::Entry& entry, table->entries) {
+if (entry.target != "/") {
+  ASSERT_TRUE(ids.contains(entry.parent));
+}
+
+ASSERT_FALSE(ids.contains(entry.id));
+
+ids.insert(entry.id);
+  }
+}
+
+
 TEST_F(FsTest, ROOT_SharedMount)
 {
   string directory = os::getcwd();

[7/7] mesos git commit: Updated mesos containerizer to use new 'stout/wait.hpp' header.

2016-10-12 Thread jieyu 
Updated mesos containerizer to use new 'stout/wait.hpp' header.

Review: https://reviews.apache.org/r/52706/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/78f1ea32
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/78f1ea32
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/78f1ea32

Branch: refs/heads/master
Commit: 78f1ea321fc7fcea0cdb9334e0336d37f6672a1f
Parents: fa56d04
Author: Kevin Klues 
Authored: Wed Oct 12 22:34:24 2016 -0700
Committer: Jie Yu 
Committed: Wed Oct 12 22:34:24 2016 -0700

--
 src/slave/containerizer/mesos/containerizer.cpp | 6 ++
 1 file changed, 2 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/78f1ea32/src/slave/containerizer/mesos/containerizer.cpp
--
diff --git a/src/slave/containerizer/mesos/containerizer.cpp 
b/src/slave/containerizer/mesos/containerizer.cpp
index f5a4858..cc9e2bc 100644
--- a/src/slave/containerizer/mesos/containerizer.cpp
+++ b/src/slave/containerizer/mesos/containerizer.cpp
@@ -14,10 +14,6 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-#ifndef __WINDOWS__
-#include 
-#endif // __WINDOWS__
-
 #include 
 
 #include 
@@ -44,6 +40,8 @@
 #include 
 #include 
 
+#include 
+
 #include "common/protobuf_utils.hpp"
 
 #include "hook/manager.hpp"

[3/7] mesos git commit: Refactored 'MountInfoTable::read()' into two separate functions.

2016-10-12 Thread jieyu 
Refactored 'MountInfoTable::read()' into two separate functions.

The original function now calls a helper which takes a string
representation of a 'MountInfoTable'. In a subsequent commit we will
use this helper to write more meaningful tests to stress the core
logic of the 'MountInfoTable::read()' functionality.

Review: https://reviews.apache.org/r/52704/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/44c43835
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/44c43835
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/44c43835

Branch: refs/heads/master
Commit: 44c438358def99d16024c1365355f01b80e4f66c
Parents: ccc746a
Author: Kevin Klues 
Authored: Wed Oct 12 22:34:12 2016 -0700
Committer: Jie Yu 
Committed: Wed Oct 12 22:34:12 2016 -0700

--
 src/linux/fs.cpp | 35 +--
 src/linux/fs.hpp | 17 +
 2 files changed, 38 insertions(+), 14 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/44c43835/src/linux/fs.cpp
--
diff --git a/src/linux/fs.cpp b/src/linux/fs.cpp
index aeec882..f184d46 100644
--- a/src/linux/fs.cpp
+++ b/src/linux/fs.cpp
@@ -86,24 +86,13 @@ Try supported(const string& fsname)
   return false;
 }
 
-
 Try MountInfoTable::read(
-const Option& pid,
+const string& lines,
 bool hierarchicalSort)
 {
   MountInfoTable table;
 
-  const string path = path::join(
-  "/proc",
-  (pid.isSome() ? stringify(pid.get()) : "self"),
-  "mountinfo");
-
-  Try lines = os::read(path);
-  if (lines.isError()) {
-return Error("Failed to read mountinfo file: " + lines.error());
-  }
-
-  foreach (const string& line, strings::tokenize(lines.get(), "\n")) {
+  foreach (const string& line, strings::tokenize(lines, "\n")) {
 Try parse = MountInfoTable::Entry::parse(line);
 if (parse.isError()) {
   return Error("Failed to parse entry '" + line + "': " + parse.error());
@@ -140,7 +129,7 @@ Try MountInfoTable::read(
 std::function sortFrom = [&](int parentId) {
   CHECK(!visitedParents.contains(parentId))
 << "Cycle found in mount table hierarchy at entry"
-<< " '" << stringify(parentId) << "': " << std::endl << lines.get();
+<< " '" << stringify(parentId) << "': " << std::endl << lines;
 
   visitedParents.insert(parentId);
 
@@ -172,6 +161,24 @@ Try MountInfoTable::read(
 }
 
 
+Try MountInfoTable::read(
+const Option& pid,
+bool hierarchicalSort)
+{
+  const string path = path::join(
+  "/proc",
+  (pid.isSome() ? stringify(pid.get()) : "self"),
+  "mountinfo");
+
+  Try lines = os::read(path);
+  if (lines.isError()) {
+return Error("Failed to read mountinfo file: " + lines.error());
+  }
+
+  return MountInfoTable::read(lines.get(), hierarchicalSort);
+}
+
+
 Try MountInfoTable::Entry::parse(const string& s)
 {
   MountInfoTable::Entry entry;

http://git-wip-us.apache.org/repos/asf/mesos/blob/44c43835/src/linux/fs.hpp
--
diff --git a/src/linux/fs.hpp b/src/linux/fs.hpp
index 090c826..da49c9e 100644
--- a/src/linux/fs.hpp
+++ b/src/linux/fs.hpp
@@ -222,6 +222,23 @@ struct MountInfoTable {
   const Option& pid = None(),
   bool hierarchicalSort = true);
 
+  // Read a mountinfo table from a string.
+  // @param   lines   The contents of a mountinfo table represented as
+  //  a string. Different entries in the string are
+  //  separated by a newline.
+  // @param   hierarchicalSort
+  //  A boolean indicating whether the entries in the
+  //  mountinfo table should be sorted according to
+  //  their parent / child relationship (as opposed to
+  //  the temporal ordering of when they were
+  //  mounted). The two orderings may differ (for
+  //  example) if a filesystem is remounted after some
+  //  of its children have been mounted.
+  // @return  An instance of MountInfoTable if success.
+  static Try read(
+  const std::string& lines,
+  bool hierarchicalSort = true);
+
   // TODO(jieyu): Introduce 'find' methods to find entries that match
   // the given conditions (e.g., target, root, devno, etc.).

[5/7] mesos git commit: Updated 'MountInfoTableReadSorted' test to use a hashset.

2016-10-12 Thread jieyu 
Updated 'MountInfoTableReadSorted' test to use a hashset.

Review: https://reviews.apache.org/r/52811/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/de69a72d
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/de69a72d
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/de69a72d

Branch: refs/heads/master
Commit: de69a72d1b52d8d2dbed215172bda885a4ff5a9e
Parents: 2f005a0
Author: Kevin Klues 
Authored: Wed Oct 12 22:34:17 2016 -0700
Committer: Jie Yu 
Committed: Wed Oct 12 22:34:17 2016 -0700

--
 src/tests/containerizer/fs_tests.cpp | 8 +++-
 1 file changed, 3 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/de69a72d/src/tests/containerizer/fs_tests.cpp
--
diff --git a/src/tests/containerizer/fs_tests.cpp 
b/src/tests/containerizer/fs_tests.cpp
index d8bd1d0..ccdd596 100644
--- a/src/tests/containerizer/fs_tests.cpp
+++ b/src/tests/containerizer/fs_tests.cpp
@@ -18,8 +18,6 @@
 
 #include 
 
-#include 
-
 #include 
 #include 
 #include 
@@ -170,15 +168,15 @@ TEST_F(FsTest, MountInfoTableReadSorted)
   Try table = MountInfoTable::read();
   ASSERT_SOME(table);
 
-  set ids;
+  hashset ids;
 
   // Verify that all parent entries appear *before* their children.
   foreach (const MountInfoTable::Entry& entry, table->entries) {
 if (entry.target != "/") {
-  ASSERT_TRUE(ids.count(entry.parent) == 1);
+  ASSERT_TRUE(ids.contains(entry.parent));
 }
 
-ASSERT_TRUE(ids.count(entry.id) == 0);
+ASSERT_FALSE(ids.contains(entry.id));
 
 ids.insert(entry.id);
   }

[1/7] mesos git commit: Added more detailed error message when failing in MountInfoTable::read.

2016-10-12 Thread jieyu 
Repository: mesos
Updated Branches:
  refs/heads/master 7cbd902bb -> 78f1ea321


Added more detailed error message when failing in MountInfoTable::read.

Review: https://reviews.apache.org/r/52597/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/70b227f7
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/70b227f7
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/70b227f7

Branch: refs/heads/master
Commit: 70b227f7d5662c051d0e978e9e4bfec328854c57
Parents: 7cbd902
Author: Kevin Klues 
Authored: Wed Oct 12 22:33:51 2016 -0700
Committer: Jie Yu 
Committed: Wed Oct 12 22:33:51 2016 -0700

--
 src/linux/fs.cpp | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/70b227f7/src/linux/fs.cpp
--
diff --git a/src/linux/fs.cpp b/src/linux/fs.cpp
index 4b10141..d49d1f5 100644
--- a/src/linux/fs.cpp
+++ b/src/linux/fs.cpp
@@ -138,7 +138,9 @@ Try MountInfoTable::read(
 vector sortedEntries;
 
 std::function sortFrom = [&](int parentId) {
-  CHECK(!visitedParents.contains(parentId)) << lines.get();
+  CHECK(!visitedParents.contains(parentId))
+<< "Cycle found in mount table hierarchy at entry"
+<< " '" << stringify(parentId) << "': " << std::endl << lines.get();
 
   visitedParents.insert(parentId);

mesos git commit: Added Allocator::updateAllocation() change to upgrades.md for 1.1.0.

2016-10-12 Thread yan 
Repository: mesos
Updated Branches:
  refs/heads/master 99cbe69eb -> 8443cba7c


Added Allocator::updateAllocation() change to upgrades.md for 1.1.0.


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/8443cba7
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/8443cba7
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/8443cba7

Branch: refs/heads/master
Commit: 8443cba7cab464ba35fa737fed74221e8a78f143
Parents: 99cbe69
Author: Jiang Yan Xu 
Authored: Wed Oct 12 18:21:09 2016 -0700
Committer: Jiang Yan Xu 
Committed: Wed Oct 12 18:21:26 2016 -0700

--
 docs/upgrades.md | 7 +++
 1 file changed, 7 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/8443cba7/docs/upgrades.md
--
diff --git a/docs/upgrades.md b/docs/upgrades.md
index 7250259..b1e9e46 100644
--- a/docs/upgrades.md
+++ b/docs/upgrades.md
@@ -55,6 +55,9 @@ We categorize the changes as follows:
 
   R Container Logger 
recovery method
 
+
+  C Allocator 
updateAllocation method
+
   
   
   
@@ -208,6 +211,10 @@ We categorize the changes as follows:
 
 * Mesos 1.1 removes the `ContainerLogger`'s `recover()` method.  The 
`ContainerLogger` had an incomplete interface for a stateful implementation.  
This removes the incomplete parts to avoid adding tech debt in the 
containerizer.  Please see 
[MESOS-6371](https://issues.apache.org/jira/browse/MESOS-6371) for more 
information.
 
+
+
+* Mesos 1.1 adds an `offeredResources` argument to the `updateAllocation` 
method. It is used to indicate the resources that the operations passed to 
`updateAllocation` are applied to. 
[MESOS-4431](https://issues.apache.org/jira/browse/MESOS-4431) (paticularly 
[/r/45961/](https://reviews.apache.org/r/45961/)) has more details on the 
motivation.
+
 ## Upgrading from 0.28.x to 1.0.x ##

mesos git commit: Implemented `delegate` method.

2016-10-12 Thread jieyu 
Repository: mesos
Updated Branches:
  refs/heads/master 8b4d83aec -> 99cbe69eb


Implemented `delegate` method.

Review: https://reviews.apache.org/r/51769/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/99cbe69e
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/99cbe69e
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/99cbe69e

Branch: refs/heads/master
Commit: 99cbe69eb4dd7e5bcac7a5df4134d1af9e4a00b8
Parents: 8b4d83a
Author: Avinash sridharan 
Authored: Wed Oct 12 16:04:43 2016 -0700
Committer: Jie Yu 
Committed: Wed Oct 12 17:52:46 2016 -0700

--
 .../cni/plugins/port_mapper/port_mapper.cpp | 154 ++-
 1 file changed, 148 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/99cbe69e/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/port_mapper.cpp
--
diff --git 
a/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/port_mapper.cpp
 
b/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/port_mapper.cpp
index 0473645..2ff8b0e 100644
--- 
a/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/port_mapper.cpp
+++ 
b/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/port_mapper.cpp
@@ -25,11 +25,19 @@
 #include "slave/containerizer/mesos/isolators/network/cni/spec.hpp"
 #include 
"slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/port_mapper.hpp"
 
+namespace io = process::io;
+
+using std::cerr;
+using std::endl;
+using std::map;
 using std::string;
+using std::tuple;
 using std::vector;
 
+using process::Failure;
 using process::Future;
 using process::Owned;
+using process::Subprocess;
 
 using mesos::NetworkInfo;
 
@@ -180,19 +188,23 @@ Try 
PortMapper::create(const string& _cniConfig)
   // 'delegate' as it should be a valid CNI config JSON.
 
   // Make sure the 'delegate' plugin exists.
-  Result delegatePlugin =
+  Result _delegatePlugin =
 _delegateConfig->find("type");
 
-  if (!delegatePlugin.isSome()) {
+  if (!_delegatePlugin.isSome()) {
 return PluginError(
 "Failed to get the delegate plugin 'type'" +
-(delegatePlugin.isError() ? delegatePlugin.error() : "Not found"),
+(_delegatePlugin.isError() ? _delegatePlugin.error() : "Not found"),
 ERROR_BAD_ARGS);
   }
 
-  if (os::which(delegatePlugin->value, cniPath.get()).isNone()) {
+  Option delegatePlugin = os::which(
+  _delegatePlugin->value,
+  cniPath.get());
+
+  if (delegatePlugin.isNone()) {
 return PluginError(
-"Could not find the delegate plugin '" + delegatePlugin->value +
+"Could not find the delegate plugin '" + _delegatePlugin->value +
 "' in '" + cniPath.get() + "'",
 ERROR_BAD_ARGS);
   }
@@ -211,7 +223,7 @@ Try 
PortMapper::create(const string& _cniConfig)
   cniArgs,
   cniPath.get(),
   networkInfo.get(),
-  delegatePlugin->value,
+  delegatePlugin.get(),
   delegateConfig,
   chain->value,
   excludeDevices));
@@ -226,6 +238,136 @@ Try PortMapper::execute()
 
 Result PortMapper::delegate(const string& command)
 {
+  map environment;
+
+  environment["CNI_COMMAND"] = command;
+  environment["CNI_IFNAME"] = cniIfName;
+  environment["CNI_NETNS"] = cniNetNs;
+  environment["CNI_PATH"] = cniPath;
+
+  if (cniContainerId.isSome()) {
+environment["CNI_CONTAINERID"] = cniContainerId.get();
+  }
+
+  if (cniArgs.isSome()) {
+environment["CNI_ARGS"] = cniArgs.get();
+  }
+
+  // Some CNI plugins need to run "iptables" to set up IP Masquerade,
+  // so we need to set the "PATH" environment variable so that the
+  // plugin can locate the "iptables" executable file.
+  Option value = os::getenv("PATH");
+  if (value.isSome()) {
+environment["PATH"] = value.get();
+  } else {
+environment["PATH"] =
+  "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin";
+  }
+
+  Try temp = os::mktemp();
+  if (temp.isError()) {
+return Error("Failed to create the temp file: " + temp.error());
+  }
+
+  Try write = os::write(
+  temp.get(),
+  stringify(delegateConfig));
+
+  if (write.isError()) {
+os::rm(temp.get());
+return Error("Failed to write the temp file: " + write.error());
+  }
+
+  Try s = process::subprocess(
+  delegatePlugin,
+  {delegatePlugin},
+  Subprocess::PATH(temp.get()),
+  Subprocess::PIPE(),
+  Subprocess::PIPE(),
+  nullptr,
+  environment);
+
+  if (s.isError()) {
+return Error(
+"Failed to exec the '" +

[2/2] mesos git commit: Invoke the shutdown executor callback for checkpointed frameworks.

2016-10-12 Thread vinodkone 
Invoke the shutdown executor callback for checkpointed frameworks.

Previously, the executor library used to commit suicide after the
recovery timeout without invoking the executor's shutdown callback.
This behavior was not consistent with the executor driver.

Review: https://reviews.apache.org/r/52756/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/8b4d83ae
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/8b4d83ae
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/8b4d83ae

Branch: refs/heads/master
Commit: 8b4d83aecc937996805e553c96f88486bd84ef35
Parents: a1dc1d3
Author: Anand Mazumdar 
Authored: Wed Oct 12 17:34:40 2016 -0700
Committer: Vinod Kone 
Committed: Wed Oct 12 17:37:02 2016 -0700

--
 src/executor/executor.cpp | 15 ++-
 1 file changed, 10 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/8b4d83ae/src/executor/executor.cpp
--
diff --git a/src/executor/executor.cpp b/src/executor/executor.cpp
index a0e5d83..1d47b52 100644
--- a/src/executor/executor.cpp
+++ b/src/executor/executor.cpp
@@ -478,10 +478,7 @@ protected:
   // Backoff and reconnect only if framework checkpointing is enabled.
   backoff();
 } else {
-  Event event;
-  event.set_type(Event::SHUTDOWN);
-
-  receive(event, true);
+  shutdown();
 }
   }
 
@@ -721,12 +718,20 @@ protected:
 }
 
 if (event.type() == Event::SHUTDOWN) {
-  shutdown();
+  _shutdown();
 }
   }
 
   void shutdown()
   {
+Event event;
+event.set_type(Event::SHUTDOWN);
+
+receive(event, true);
+  }
+
+  void _shutdown()
+  {
 if (!local) {
   spawn(new ShutdownProcess(shutdownGracePeriod), true);
 } else {

[1/2] mesos git commit: Made default executor handle shutdown events while disconnected.

2016-10-12 Thread vinodkone 
Repository: mesos
Updated Branches:
  refs/heads/master 58bfa80c6 -> 8b4d83aec


Made default executor handle shutdown events while disconnected.

Previously, the default executor used to crash with a failed assertion
when the executor library injected a shutdown event when it noticed
a disconnection with the agent for non-checkpointed frameworks and
upon recovery timeout for checkpointed frameworks. This change
modifies it to commit suicide minus the failed assertion.

Review: https://reviews.apache.org/r/52755/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/a1dc1d33
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/a1dc1d33
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/a1dc1d33

Branch: refs/heads/master
Commit: a1dc1d33adce98007a7a88174048f46e4afe1782
Parents: 58bfa80
Author: Anand Mazumdar 
Authored: Wed Oct 12 17:34:36 2016 -0700
Committer: Vinod Kone 
Committed: Wed Oct 12 17:37:00 2016 -0700

--
 src/launcher/default_executor.cpp | 14 --
 1 file changed, 12 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/a1dc1d33/src/launcher/default_executor.cpp
--
diff --git a/src/launcher/default_executor.cpp 
b/src/launcher/default_executor.cpp
index 2454bd7..af4a97f 100644
--- a/src/launcher/default_executor.cpp
+++ b/src/launcher/default_executor.cpp
@@ -683,8 +683,6 @@ protected:
   return;
 }
 
-CHECK_EQ(SUBSCRIBED, state);
-
 LOG(INFO) << "Shutting down";
 
 shuttingDown = true;
@@ -694,6 +692,18 @@ protected:
   return;
 }
 
+// It is possible that the executor library injected the shutdown event
+// upon a disconnection with the agent for non-checkpointed
+// frameworks or after recovery timeout for checkpointed frameworks.
+// This could also happen when the executor is connected but the agent
+// asked it to shutdown because it didn't subscribe in time.
+if (state == CONNECTED || state == DISCONNECTED) {
+  __shutdown();
+  return;
+}
+
+CHECK_EQ(SUBSCRIBED, state);
+
 process::http::connect(agent)
   .onAny(defer(self(), ::_shutdown, lambda::_1));
   }

mesos git commit: Removed a stale TODO in the nested container launch handler.

2016-10-12 Thread bmahler 
Repository: mesos
Updated Branches:
  refs/heads/master d3f560576 -> 58bfa80c6


Removed a stale TODO in the nested container launch handler.

This is a remnant from https://reviews.apache.org/r/52569/.


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/58bfa80c
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/58bfa80c
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/58bfa80c

Branch: refs/heads/master
Commit: 58bfa80c68803ed5a1eeb576df1d021aed229fd3
Parents: d3f5605
Author: Benjamin Mahler 
Authored: Wed Oct 12 17:23:40 2016 -0700
Committer: Benjamin Mahler 
Committed: Wed Oct 12 17:24:23 2016 -0700

--
 src/slave/http.cpp | 2 --
 1 file changed, 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/58bfa80c/src/slave/http.cpp
--
diff --git a/src/slave/http.cpp b/src/slave/http.cpp
index 79061c3..a1fbb7a 100644
--- a/src/slave/http.cpp
+++ b/src/slave/http.cpp
@@ -1961,8 +1961,6 @@ Future Slave::Http::launchNestedContainer(
   }
 #endif
 
-  // TODO(gilbert): The sandbox directory and user are incorrect,
-  // Please update it.
   Future launched = slave->containerizer->launch(
   containerId,
   call.launch_nested_container().command(),

mesos git commit: Updated comments about ExecutorInfo in mesos.proto.

2016-10-12 Thread vinodkone 
Repository: mesos
Updated Branches:
  refs/heads/master 1e514ca02 -> d3f560576


Updated comments about ExecutorInfo in mesos.proto.

Removed the comment that says resources are automatically set
to default values if unset.


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/d3f56057
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/d3f56057
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/d3f56057

Branch: refs/heads/master
Commit: d3f5605762caca6190ea5d419423106f7716f8c0
Parents: 1e514ca
Author: Vinod Kone 
Authored: Wed Oct 12 16:10:57 2016 -0700
Committer: Vinod Kone 
Committed: Wed Oct 12 16:10:57 2016 -0700

--
 include/mesos/mesos.proto| 6 +-
 include/mesos/v1/mesos.proto | 6 +-
 2 files changed, 2 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/d3f56057/include/mesos/mesos.proto
--
diff --git a/include/mesos/mesos.proto b/include/mesos/mesos.proto
index 7d0201e..05988d4 100644
--- a/include/mesos/mesos.proto
+++ b/include/mesos/mesos.proto
@@ -541,11 +541,7 @@ message ExecutorInfo {
 //
 // 1) `command` must not be set when using a default executor.
 //
-// 2) If "cpu" or "mem" resources are not set, default values of 0.1 cpu
-//and 32 MB memory will be used; schedulers should account for these
-//resources when accepting offers.
-//
-// 3) Default executor only accepts a *single* `LAUNCH` or `LAUNCH_GROUP`
+// 2) Default executor only accepts a *single* `LAUNCH` or `LAUNCH_GROUP`
 //offer operation.
 DEFAULT = 1;
 

http://git-wip-us.apache.org/repos/asf/mesos/blob/d3f56057/include/mesos/v1/mesos.proto
--
diff --git a/include/mesos/v1/mesos.proto b/include/mesos/v1/mesos.proto
index 301d1ae..08a536c 100644
--- a/include/mesos/v1/mesos.proto
+++ b/include/mesos/v1/mesos.proto
@@ -541,11 +541,7 @@ message ExecutorInfo {
 //
 // 1) `command` must not be set when using a default executor.
 //
-// 2) If "cpu" or "mem" resources are not set, default values of 0.1 cpu
-//and 32 MB memory will be used; schedulers should account for these
-//resources when accepting offers.
-//
-// 3) Default executor only accepts a *single* `LAUNCH` or `LAUNCH_GROUP`
+// 2) Default executor only accepts a *single* `LAUNCH` or `LAUNCH_GROUP`
 //offer operation.
 DEFAULT = 1;

[3/3] mesos git commit: Supported `ContainerLogger` with nested containers.

2016-10-12 Thread josephwu 
Supported `ContainerLogger` with nested containers.

For nested containers, the sandbox directory still exists.  However,
ExecutorInfo's no longer map directly one-to-one to containers.
That means a nested container does not have an associated ExecutorInfo.

The `ExecutorInfo` parameter provides metadata for the `ContainerLogger`
including the FrameworkID, ExecutorID, environment variables, and
arbitrary Labels.  For nested containers, the top-level parent's
`ExecutorInfo` should be sufficient to provide the same metadata.

Review: https://reviews.apache.org/r/52412/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/1e514ca0
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/1e514ca0
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/1e514ca0

Branch: refs/heads/master
Commit: 1e514ca023526f949e0ed5b91d43636a1b2fd172
Parents: 9588a60
Author: Gilbert Song 
Authored: Wed Oct 12 14:38:43 2016 -0700
Committer: Joseph Wu 
Committed: Wed Oct 12 14:41:30 2016 -0700

--
 src/slave/containerizer/mesos/containerizer.cpp | 21 +++-
 1 file changed, 20 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/1e514ca0/src/slave/containerizer/mesos/containerizer.cpp
--
diff --git a/src/slave/containerizer/mesos/containerizer.cpp 
b/src/slave/containerizer/mesos/containerizer.cpp
index 7ec6f78..f5a4858 100644
--- a/src/slave/containerizer/mesos/containerizer.cpp
+++ b/src/slave/containerizer/mesos/containerizer.cpp
@@ -1333,8 +1333,27 @@ Future MesosContainerizerProcess::_launch(
 environment.values[name] = value;
   }
 
+  // Determine the 'ExecutorInfo' for the logger. If launching a
+  // top level executor container, use the 'ExecutorInfo' from
+  // 'ContainerConfig'. If launching a nested container, use the
+  // 'ExecutorInfo' from its top level parent container.
+  ExecutorInfo executorInfo;
+  if (container->config.has_executor_info()) {
+// The top level executor container case. The 'ExecutorInfo'
+// will always be set in 'ContainerConfig'.
+executorInfo = container->config.executor_info();
+  } else {
+// The nested container case. Use the 'ExecutorInfo' from its root
+// parent container.
+CHECK(containerId.has_parent());
+const ContainerID& rootContainerId = getRootContainerId(containerId);
+CHECK(containers_.contains(rootContainerId));
+CHECK(containers_[rootContainerId]->config.has_executor_info());
+executorInfo = containers_[rootContainerId]->config.executor_info();
+  }
+
   return logger->prepare(
-  container->config.executor_info(),
+  executorInfo,
   container->config.directory())
 .then(defer(
 self(),

[2/3] mesos git commit: Updated CLI bootstrap to search for local virtualenv installations.

2016-10-12 Thread josephwu 
Updated CLI bootstrap to search for local virtualenv installations.

A locally installed virtualenv does not always show up on the `PATH`,
which fails `which virtualenv`.  This adds an extra search location
based on the user's site package install directory.

Review: https://reviews.apache.org/r/52707/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/9588a600
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/9588a600
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/9588a600

Branch: refs/heads/master
Commit: 9588a6009e956ec1629be13df6a41dc98dd0180d
Parents: 8c8ec60
Author: Joseph Wu 
Authored: Wed Oct 12 13:17:56 2016 -0700
Committer: Joseph Wu 
Committed: Wed Oct 12 14:41:30 2016 -0700

--
 src/cli_new/bootstrap | 15 +--
 1 file changed, 13 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/9588a600/src/cli_new/bootstrap
--
diff --git a/src/cli_new/bootstrap b/src/cli_new/bootstrap
index 74a8568..6d62e9a 100755
--- a/src/cli_new/bootstrap
+++ b/src/cli_new/bootstrap
@@ -29,11 +29,22 @@ if [ "${OLD_VIRTUAL_ENV}" != "" ]; then
 fi
 
 # Verify that python and virtualenv are installed.
-if [ "${PYTHON}" = "" ] || [ "${VIRTUALENV}" = "" ]; then
-  echo "You must have python and virtualenv installed in order to continue..."
+if [ "${PYTHON}" = "" ]; then
+  echo "You must have python installed in order to continue..."
   exit 1
 fi
 
+if [ "${VIRTUALENV}" = "" ]; then
+  # Search for a locally installed virtualenv.
+  # See https://docs.python.org/2/library/site.html#site.USER_SITE for details.
+  VIRTUALENV=$(${PYTHON} -c "import site; print site.USER_SITE")/virtualenv.py
+
+  if [ ! -f "${VIRTUALENV}" ]; then
+echo "You must have virtualenv installed in order to continue..."
+exit 1
+  fi
+fi
+
 PYTHON_MAJOR=$(${PYTHON} -c 'import sys; print(sys.version_info[0])')
 PYTHON_MINOR=$(${PYTHON} -c 'import sys; print(sys.version_info[1])')

[1/3] mesos git commit: Windows: Implemented os::execvpe with _spawnvpe.

2016-10-12 Thread josephwu 
Repository: mesos
Updated Branches:
  refs/heads/master 44cc1932c -> 1e514ca02


Windows: Implemented os::execvpe with _spawnvpe.

Review: https://reviews.apache.org/r/52798


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/8c8ec608
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/8c8ec608
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/8c8ec608

Branch: refs/heads/master
Commit: 8c8ec608503394575a4f99fd725010b8920e5efa
Parents: 44cc193
Author: Joseph Wu 
Authored: Wed Oct 12 11:57:18 2016 -0700
Committer: Joseph Wu 
Committed: Wed Oct 12 14:41:29 2016 -0700

--
 3rdparty/stout/include/stout/os/windows/shell.hpp | 14 ++
 3rdparty/stout/include/stout/windows/os.hpp   |  6 --
 2 files changed, 14 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/8c8ec608/3rdparty/stout/include/stout/os/windows/shell.hpp
--
diff --git a/3rdparty/stout/include/stout/os/windows/shell.hpp 
b/3rdparty/stout/include/stout/os/windows/shell.hpp
index 0ababb4..17e3d56 100644
--- a/3rdparty/stout/include/stout/os/windows/shell.hpp
+++ b/3rdparty/stout/include/stout/os/windows/shell.hpp
@@ -149,6 +149,20 @@ inline int execvp(const char* file, char* const argv[])
 }
 
 
+// On Windows, the `_spawnvpe` call creates a new process.
+// In order to emulate the semantics of `execvpe`, we spawn with `_P_WAIT`,
+// which forces the parent process to block on the child. When the child exits,
+// the exit code is propagated back through the parent via `exit()`.
+//
+// The returned value from `_spawnvpe` represents child exit code when
+// `_P_WAIT` is used.
+inline int execvpe(const char* file, char* const argv[], char* const envp[])
+{
+  exit(static_cast(::_spawnvpe(_P_WAIT, file, argv, envp)));
+  return 0;
+}
+
+
 // Concatenates multiple command-line arguments and escapes the values.
 // If `arg` is not specified (or takes the value `0`), the function will
 // scan `argv` until a `nullptr` is encountered.

http://git-wip-us.apache.org/repos/asf/mesos/blob/8c8ec608/3rdparty/stout/include/stout/windows/os.hpp
--
diff --git a/3rdparty/stout/include/stout/windows/os.hpp 
b/3rdparty/stout/include/stout/windows/os.hpp
index 7d6530e..7ca0b5d 100644
--- a/3rdparty/stout/include/stout/windows/os.hpp
+++ b/3rdparty/stout/include/stout/windows/os.hpp
@@ -374,12 +374,6 @@ inline std::string hstrerror(int err)
 }
 
 
-// This function is a portable version of execvpe ('p' means searching
-// executable from PATH and 'e' means setting environments). We add
-// this function because it is not available on all systems.
-inline int execvpe(const char* file, char** argv, char** envp) = delete;
-
-
 inline Try chown(
 uid_t uid,
 gid_t gid,

mesos git commit: Fixed 'SharedPersistentVolumeRescindOnDestroy' test.

2016-10-12 Thread yan 
Repository: mesos
Updated Branches:
  refs/heads/master 713b784a3 -> 44cc1932c


Fixed 'SharedPersistentVolumeRescindOnDestroy' test.


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/44cc1932
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/44cc1932
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/44cc1932

Branch: refs/heads/master
Commit: 44cc1932cb27001f0b73cee011d00c45fbd1764c
Parents: 713b784
Author: Jiang Yan Xu 
Authored: Wed Oct 12 14:11:44 2016 -0700
Committer: Jiang Yan Xu 
Committed: Wed Oct 12 14:11:53 2016 -0700

--
 src/tests/persistent_volume_tests.cpp | 4 
 1 file changed, 4 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/44cc1932/src/tests/persistent_volume_tests.cpp
--
diff --git a/src/tests/persistent_volume_tests.cpp 
b/src/tests/persistent_volume_tests.cpp
index b7d1c1a..6289009 100644
--- a/src/tests/persistent_volume_tests.cpp
+++ b/src/tests/persistent_volume_tests.cpp
@@ -1116,8 +1116,12 @@ TEST_P(PersistentVolumeTest, 
SharedPersistentVolumeRescindOnDestroy)
 
   // Advance the clock until the allocator allocates
   // the recovered resources.
+  Clock::settle();
   Clock::advance(masterFlags.allocation_interval);
 
+  // Resume the clock so the terminating task and executor can be reaped.
+  Clock::resume();
+
   AWAIT_READY(offers1);
 
   offer1 = offers1.get()[0];

mesos git commit: Added a test for verifying nested container environment.

2016-10-12 Thread jieyu 
Repository: mesos
Updated Branches:
  refs/heads/master 6a4296de3 -> 713b784a3


Added a test for verifying nested container environment.

Review: https://reviews.apache.org/r/52774/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/713b784a
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/713b784a
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/713b784a

Branch: refs/heads/master
Commit: 713b784a3a92d655bd341470a9ed6fb7208466a3
Parents: 6a4296d
Author: Jie Yu 
Authored: Wed Oct 12 12:31:59 2016 -0700
Committer: Jie Yu 
Committed: Wed Oct 12 12:31:59 2016 -0700

--
 .../nested_mesos_containerizer_tests.cpp| 93 
 src/tests/mesos.hpp | 13 +++
 2 files changed, 106 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/713b784a/src/tests/containerizer/nested_mesos_containerizer_tests.cpp
--
diff --git a/src/tests/containerizer/nested_mesos_containerizer_tests.cpp 
b/src/tests/containerizer/nested_mesos_containerizer_tests.cpp
index 9b278e5..c690b41 100644
--- a/src/tests/containerizer/nested_mesos_containerizer_tests.cpp
+++ b/src/tests/containerizer/nested_mesos_containerizer_tests.cpp
@@ -1433,6 +1433,99 @@ TEST_F(NestedMesosContainerizerTest, 
ROOT_CGROUPS_WaitAfterDestroy)
   ASSERT_NONE(nestedWait.get());
 }
 
+
+// This test verifies that agent environment variables are not leaked
+// to the nested container, and the environment variables specified in
+// the command for the nested container will be honored.
+TEST_F(NestedMesosContainerizerTest, ROOT_CGROUPS_Environment)
+{
+  slave::Flags flags = CreateSlaveFlags();
+  flags.launcher = "linux";
+  flags.isolation = "cgroups/cpu,filesystem/linux,namespaces/pid";
+
+  Fetcher fetcher;
+
+  Try create = MesosContainerizer::create(
+  flags,
+  true,
+  );
+
+  ASSERT_SOME(create);
+
+  Owned containerizer(create.get());
+
+  SlaveState state;
+  state.id = SlaveID();
+
+  AWAIT_READY(containerizer->recover(state));
+
+  ContainerID containerId;
+  containerId.set_value(UUID::random().toString());
+
+  Try directory = environment->mkdtemp();
+  ASSERT_SOME(directory);
+
+  Future launch = containerizer->launch(
+  containerId,
+  None(),
+  createExecutorInfo("executor", "sleep 1000", "cpus:1"),
+  directory.get(),
+  None(),
+  state.id,
+  map(),
+  true); // TODO(benh): Ever want to test not checkpointing?
+
+  AWAIT_ASSERT_TRUE(launch);
+
+  // Now launch nested container.
+  ContainerID nestedContainerId;
+  nestedContainerId.mutable_parent()->CopyFrom(containerId);
+  nestedContainerId.set_value(UUID::random().toString());
+
+  // Construct a command that verifies that agent environment
+  // variables are not leaked to the nested container.
+  ostringstream script;
+  script << "#!/bin/sh\n";
+
+  foreachkey (const string& key, os::environment()) {
+script << "test -z \"$" << key << "\"\n";
+  }
+
+  mesos::Environment environment = createEnvironment(
+  {{"NESTED_MESOS_CONTAINERIZER_TEST", "ENVIRONMENT"}});
+
+  script << "test $NESTED_MESOS_CONTAINERIZER_TEST = ENVIRONMENT\n";
+
+  CommandInfo command = createCommandInfo(script.str());
+  command.mutable_environment()->CopyFrom(environment);
+
+  launch = containerizer->launch(
+  nestedContainerId,
+  command,
+  None(),
+  None(),
+  state.id);
+
+  AWAIT_ASSERT_TRUE(launch);
+
+  Future

[3/5] mesos git commit: Added fields `chain` and `excludeDevices` to `PortMapper`.

2016-10-12 Thread jieyu 
Added fields `chain` and `excludeDevices` to `PortMapper`.

Review: https://reviews.apache.org/r/51736/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/fba4c1e8
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/fba4c1e8
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/fba4c1e8

Branch: refs/heads/master
Commit: fba4c1e85c4417c79764dd5858192851b2e0eabd
Parents: 06d2e23
Author: Avinash sridharan 
Authored: Wed Oct 12 09:11:30 2016 -0700
Committer: Jie Yu 
Committed: Wed Oct 12 10:59:45 2016 -0700

--
 .../cni/plugins/port_mapper/port_mapper.cpp | 28 +++-
 .../cni/plugins/port_mapper/port_mapper.hpp | 22 +--
 2 files changed, 47 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/fba4c1e8/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/port_mapper.cpp
--
diff --git 
a/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/port_mapper.cpp
 
b/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/port_mapper.cpp
index 0ecf64f..836fed5 100644
--- 
a/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/port_mapper.cpp
+++ 
b/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/port_mapper.cpp
@@ -20,6 +20,7 @@
 #include 
"slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/port_mapper.hpp"
 
 using std::string;
+using std::vector;
 
 using process::Future;
 using process::Owned;
@@ -91,6 +92,29 @@ Try PortMapper::create(const string& 
_cniConfig)
 ERROR_BAD_ARGS));
   }
 
+  vector excludeDevices;
+
+  Result _excludeDevices =
+cniConfig->find("excludeDevices");
+
+  if (_excludeDevices.isError()) {
+return Error(spec::error(
+"Failed to parse field 'excludeDevices': " +
+_excludeDevices.error(),
+ERROR_BAD_ARGS));
+  } else if (_excludeDevices.isSome()) {
+foreach (const JSON::Value& value, _excludeDevices->values) {
+  if (!value.is()) {
+return Error(spec::error(
+"Failed to parse 'excludeDevices' list. "
+"The excluded device needs to be a string",
+ERROR_BAD_ARGS));
+  }
+
+  excludeDevices.push_back(value.as().value);
+}
+  }
+
   // While the 'args' field is optional in the CNI spec it is critical
   // to the port-mapper plugin to learn of any port-mappings that the
   // framework might have requested for this container.
@@ -175,7 +199,9 @@ Try PortMapper::create(const string& 
_cniConfig)
   cniPath.get(),
   networkInfo.get(),
   delegatePlugin->value,
-  delegateConfig.get()));
+  delegateConfig.get(),
+  chain->value,
+  excludeDevices));
 }
 
 

http://git-wip-us.apache.org/repos/asf/mesos/blob/fba4c1e8/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/port_mapper.hpp
--
diff --git 
a/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/port_mapper.hpp
 
b/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/port_mapper.hpp
index 8554753..b943254 100644
--- 
a/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/port_mapper.hpp
+++ 
b/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/port_mapper.hpp
@@ -97,7 +97,9 @@ private:
   const std::string& _cniPath,  // Paths to search for CNI plugins.
   const mesos::NetworkInfo& _networkInfo,
   const std::string& _delegatePlugin,
-  const JSON::Object& _delegateConfig)
+  const JSON::Object& _delegateConfig,
+  const std::string& _chain,
+  const std::vector& _excludeDevices)
 : cniCommand(_cniCommand),
   cniContainerId(_cniContainerId),
   cniNetNs(_cniNetNs),
@@ -106,7 +108,9 @@ private:
   cniPath(_cniPath),
   networkInfo(_networkInfo),
   delegatePlugin(_delegatePlugin),
-  delegateConfig(_delegateConfig) {};
+  delegateConfig(_delegateConfig),
+  chain(_chain),
+  excludeDevices(_excludeDevices){};
 
   const std::string cniCommand;
   const Option cniContainerId;
@@ -114,9 +118,23 @@ private:
   const std::string cniIfName;
   const Option cniArgs;
   const std::string cniPath;
+
   const mesos::NetworkInfo networkInfo;
+
   const std::string delegatePlugin;
   const JSON::Object delegateConfig;
+
+  // The iptable chain to which the DNAT rules need to be added. We
+  // need a separate chain, so that we can group the DNAT rules
+  // specific to this CNI network under this chain. It makes it easier
+  // for the operator to

[2/5] mesos git commit: Added the 'name' and 'args' field to the 'delegate' plugin's CNI config.

2016-10-12 Thread jieyu 
Added the 'name' and 'args' field to the 'delegate' plugin's CNI config.

When the port-mapper plugin invokes the 'delegate' plugin, the CNI
config of the 'delegate' plugin needs to have the 'name' and the
'args' fields set. We are therefore updating the config during
creation of the `PortMapper` object, and will use this config when
the 'delegate' plugin is invoked.

Review: https://reviews.apache.org/r/51740/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/7a9f1c48
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/7a9f1c48
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/7a9f1c48

Branch: refs/heads/master
Commit: 7a9f1c485829c2aedb20adeaec4556bf73e74e48
Parents: 539d67f
Author: Avinash sridharan 
Authored: Wed Oct 12 09:53:33 2016 -0700
Committer: Jie Yu 
Committed: Wed Oct 12 10:59:45 2016 -0700

--
 .../network/cni/plugins/port_mapper/port_mapper.cpp  | 15 ++-
 1 file changed, 10 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/7a9f1c48/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/port_mapper.cpp
--
diff --git 
a/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/port_mapper.cpp
 
b/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/port_mapper.cpp
index 5caa6d7..5de777c 100644
--- 
a/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/port_mapper.cpp
+++ 
b/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/port_mapper.cpp
@@ -166,13 +166,13 @@ Try 
PortMapper::create(const string& _cniConfig)
 
   // The port-mapper should always be used in conjunction with another
   // 'delegate' CNI plugin.
-  Result delegateConfig =
+  Result _delegateConfig =
 cniConfig->find("delegate");
 
-  if (!delegateConfig.isSome()) {
+  if (!_delegateConfig.isSome()) {
 return PluginError(
 "Failed to get the required field 'delegate'" +
-(delegateConfig.isError() ? delegateConfig.error() : "Not found"),
+(_delegateConfig.isError() ? _delegateConfig.error() : "Not found"),
 ERROR_BAD_ARGS);
   }
 
@@ -181,7 +181,7 @@ Try 
PortMapper::create(const string& _cniConfig)
 
   // Make sure the 'delegate' plugin exists.
   Result delegatePlugin =
-delegateConfig->find("type");
+_delegateConfig->find("type");
 
   if (!delegatePlugin.isSome()) {
 return PluginError(
@@ -197,6 +197,11 @@ Try 
PortMapper::create(const string& _cniConfig)
 ERROR_BAD_ARGS);
   }
 
+  // Add the 'name' and 'args' field to the 'delegate' config.
+  JSON::Object delegateConfig(_delegateConfig.get());
+  delegateConfig.values["name"] = name.get();
+  delegateConfig.values["args"] = args.get();
+
   return Owned(
   new PortMapper(
   cniCommand.get(),
@@ -207,7 +212,7 @@ Try 
PortMapper::create(const string& _cniConfig)
   cniPath.get(),
   networkInfo.get(),
   delegatePlugin->value,
-  delegateConfig.get(),
+  delegateConfig,
   chain->value,
   excludeDevices));
 }

[4/5] mesos git commit: Added constants for CNI commands.

2016-10-12 Thread jieyu 
Added constants for CNI commands.

Review: https://reviews.apache.org/r/51767/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/89d8d51d
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/89d8d51d
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/89d8d51d

Branch: refs/heads/master
Commit: 89d8d51db2fd356b1f7068d83c3be73c82cc64cf
Parents: 7a9f1c4
Author: Avinash sridharan 
Authored: Wed Oct 12 09:53:57 2016 -0700
Committer: Jie Yu 
Committed: Wed Oct 12 10:59:45 2016 -0700

--
 src/slave/containerizer/mesos/isolators/network/cni/spec.hpp | 5 +
 1 file changed, 5 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/89d8d51d/src/slave/containerizer/mesos/isolators/network/cni/spec.hpp
--
diff --git a/src/slave/containerizer/mesos/isolators/network/cni/spec.hpp 
b/src/slave/containerizer/mesos/isolators/network/cni/spec.hpp
index 9f0542a..ccd511e 100644
--- a/src/slave/containerizer/mesos/isolators/network/cni/spec.hpp
+++ b/src/slave/containerizer/mesos/isolators/network/cni/spec.hpp
@@ -32,6 +32,11 @@ namespace spec {
 constexpr char CNI_VERSION[] = "0.3.0";
 
 
+// CNI commands.
+constexpr char CNI_CMD_ADD[] = "ADD";
+constexpr char CNI_CMD_DEL[] = "DEL";
+
+
 // Well-known CNI error codes:
 // 
https://github.com/containernetworking/cni/blob/master/SPEC.md#well-known-error-codes
 constexpr uint32_t CNI_ERROR_INCOMPATIBLE_VERSION = 1;

[5/5] mesos git commit: Added `PluginError` to simplify error reporting for CNI plugins.

2016-10-12 Thread jieyu 
Added `PluginError` to simplify error reporting for CNI plugins.

https://reviews.apache.org/r/51737/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/539d67f6
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/539d67f6
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/539d67f6

Branch: refs/heads/master
Commit: 539d67f6cd4dd15c28ae0e6ff4f90f927e801329
Parents: fba4c1e
Author: Avinash sridharan 
Authored: Wed Oct 12 09:48:46 2016 -0700
Committer: Jie Yu 
Committed: Wed Oct 12 10:59:45 2016 -0700

--
 .../network/cni/plugins/port_mapper/main.cpp| 11 +--
 .../cni/plugins/port_mapper/port_mapper.cpp | 74 +++-
 .../cni/plugins/port_mapper/port_mapper.hpp |  5 +-
 .../mesos/isolators/network/cni/spec.hpp| 17 +
 4 files changed, 64 insertions(+), 43 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/539d67f6/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/main.cpp
--
diff --git 
a/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/main.cpp
 
b/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/main.cpp
index 11c03fd..0bb58c4 100644
--- 
a/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/main.cpp
+++ 
b/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/main.cpp
@@ -34,6 +34,7 @@ using std::string;
 using process::Owned;
 
 using mesos::internal::slave::cni::PortMapper;
+using mesos::internal::slave::cni::spec::PluginError;
 
 
 constexpr int STDIN_READ_LENGTH = 1000;
@@ -60,19 +61,13 @@ int main(int argc, char** argv)
 return EXIT_FAILURE;
   }
 
-  // If the `PortMapper` returns an error it will already be a JSON
-  // formatted string of type `spec::Error` so we don't need to format
-  // it again. Reason we rely on the `PortMapper` to return a JSON
-  // formatted `spec::Error` is that the error codes for `spec::Error`
-  // might vary, depending on the cause of error, and the context of
-  // the error is only visible to the `PortMapper` object.
-  Try portMapper = PortMapper::create(config);
+  Try portMapper = PortMapper::create(config);
   if (portMapper.isError()) {
 cout << portMapper.error() << endl;
 return EXIT_FAILURE;
   }
 
-  Try result = portMapper.get()->execute();
+  Try result = portMapper.get()->execute();
   if (result.isError()) {
 cout << result.error() << endl;
 return EXIT_FAILURE;

http://git-wip-us.apache.org/repos/asf/mesos/blob/539d67f6/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/port_mapper.cpp
--
diff --git 
a/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/port_mapper.cpp
 
b/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/port_mapper.cpp
index 836fed5..5caa6d7 100644
--- 
a/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/port_mapper.cpp
+++ 
b/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/port_mapper.cpp
@@ -17,6 +17,12 @@
 #include 
 #include 
 
+#include 
+#include 
+#include 
+#include 
+
+#include "slave/containerizer/mesos/isolators/network/cni/spec.hpp"
 #include 
"slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/port_mapper.hpp"
 
 using std::string;
@@ -27,18 +33,20 @@ using process::Owned;
 
 using mesos::NetworkInfo;
 
+using mesos::internal::slave::cni::spec::PluginError;
+
 namespace mesos {
 namespace internal {
 namespace slave {
 namespace cni {
 
-Try PortMapper::create(const string& _cniConfig)
+Try PortMapper::create(const string& 
_cniConfig)
 {
   Option cniCommand = os::getenv("CNI_COMMAND");
   if (cniCommand.isNone()) {
-return Error(spec::error(
+return PluginError(
 "Unable to find environment variable 'CNI_COMMAND'",
-ERROR_BAD_ARGS));
+ERROR_BAD_ARGS);
   }
 
   // 'CNI_CONTAINERID' is optional.
@@ -46,16 +54,16 @@ Try PortMapper::create(const string& 
_cniConfig)
 
   Option cniNetNs = os::getenv("CNI_NETNS");
   if (cniNetNs.isNone()) {
-return Error(spec::error(
+return PluginError(
 "Unable to find environment variable 'CNI_NETNS'",
-ERROR_BAD_ARGS));
+ERROR_BAD_ARGS);
   }
 
   Option cniIfName = os::getenv("CNI_IFNAME");
   if (cniIfName.isNone()) {
-return Error(spec::error(
+return PluginError(
 "Unable to find environment variable 'CNI_IFNAME'",
-ERROR_BAD_ARGS));
+ERROR_BAD_ARGS);
   }
 
   // 'CNI_ARGS' is optional.
@@ -63,33 +71,33 @@ Try

[1/5] mesos git commit: Updated signature of `delegate` and `execute` method.

2016-10-12 Thread jieyu 
Repository: mesos
Updated Branches:
  refs/heads/master 06d2e23dc -> 6a4296de3


Updated signature of `delegate` and `execute` method.

In case of `spec::CNI_CMD_DEL` the CNI plugin is not supposed to
return any output. Hence, changing the signatures to return an
`Result` and an `Option` respectively.

Review: https://reviews.apache.org/r/51768/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/6a4296de
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/6a4296de
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/6a4296de

Branch: refs/heads/master
Commit: 6a4296de3fe781a340b3554cabde7faab0480668
Parents: 89d8d51
Author: Avinash sridharan 
Authored: Wed Oct 12 10:06:59 2016 -0700
Committer: Jie Yu 
Committed: Wed Oct 12 10:59:45 2016 -0700

--
 .../network/cni/plugins/port_mapper/main.cpp|  5 ++--
 .../cni/plugins/port_mapper/port_mapper.cpp |  8 +++---
 .../cni/plugins/port_mapper/port_mapper.hpp | 26 
 3 files changed, 23 insertions(+), 16 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/6a4296de/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/main.cpp
--
diff --git 
a/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/main.cpp
 
b/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/main.cpp
index 0bb58c4..f8d38c6 100644
--- 
a/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/main.cpp
+++ 
b/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/main.cpp
@@ -67,12 +67,13 @@ int main(int argc, char** argv)
 return EXIT_FAILURE;
   }
 
-  Try result = portMapper.get()->execute();
+  Try result = portMapper.get()->execute();
   if (result.isError()) {
 cout << result.error() << endl;
 return EXIT_FAILURE;
+  } else if (result->isSome()) {
+cout << result->get() << endl;
   }
 
-  cout << result.get() << endl;
   return EXIT_SUCCESS;
 }

http://git-wip-us.apache.org/repos/asf/mesos/blob/6a4296de/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/port_mapper.cpp
--
diff --git 
a/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/port_mapper.cpp
 
b/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/port_mapper.cpp
index 5de777c..0473645 100644
--- 
a/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/port_mapper.cpp
+++ 
b/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/port_mapper.cpp
@@ -218,15 +218,15 @@ Try 
PortMapper::create(const string& _cniConfig)
 }
 
 
-Try PortMapper::execute()
+Try PortMapper::execute()
 {
-  return "OK";
+  return None();
 }
 
 
-Try PortMapper::delegate(const string& command)
+Result PortMapper::delegate(const string& command)
 {
-  return spec::NetworkInfo();
+  return None();
 }
 
 } // namespace cni {

http://git-wip-us.apache.org/repos/asf/mesos/blob/6a4296de/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/port_mapper.hpp
--
diff --git 
a/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/port_mapper.hpp
 
b/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/port_mapper.hpp
index c0bbef6..7fad707 100644
--- 
a/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/port_mapper.hpp
+++ 
b/src/slave/containerizer/mesos/isolators/network/cni/plugins/port_mapper/port_mapper.hpp
@@ -68,25 +68,31 @@ public:
   static Try create(
   const std::string& cniConfig);
 
-  // Executes the CNI plugin specified in 'delegate'. On successful
-  // execution of the 'delegate' plugin will install port-forwarding
-  // rules, if any, that are specified in `NetworkInfo`. On success
-  // will return a JSON string seen in the successful execution of the
-  // 'delegate' plugin. In case of an error will return a JSON string
-  // representation of `spec::Error` as the error message for the
-  // `Try`.
-  Try execute();
+  // Executes the CNI plugin specified in 'delegate'. When
+  // `cniCommand` is set to `spec::CNI_CMD_ADD` successful execution
+  // of the 'delegate' plugin will install port-forwarding rules, if
+  // any, that are specified in `NetworkInfo`. On success will return
+  // a JSON string seen in the successful execution of the 'delegate'
+  // plugin. When `cniCommand` is set to

[4/5] mesos git commit: Permitted specifying custom test driver in libprocess.

2016-10-12 Thread tillt 
Permitted specifying custom test driver in libprocess.

Review: https://reviews.apache.org/r/51718/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/105cc652
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/105cc652
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/105cc652

Branch: refs/heads/master
Commit: 105cc65242755c0e24e81d001dcbfdfa7ea352d5
Parents: 8b61157
Author: Benjamin Bannier 
Authored: Wed Oct 12 18:58:48 2016 +0200
Committer: Till Toenshoff 
Committed: Wed Oct 12 18:58:48 2016 +0200

--
 3rdparty/libprocess/Makefile.am  | 2 +-
 3rdparty/libprocess/configure.ac | 6 ++
 2 files changed, 7 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/105cc652/3rdparty/libprocess/Makefile.am
--
diff --git a/3rdparty/libprocess/Makefile.am b/3rdparty/libprocess/Makefile.am
index 020b0e1..f91155b 100644
--- a/3rdparty/libprocess/Makefile.am
+++ b/3rdparty/libprocess/Makefile.am
@@ -304,7 +304,7 @@ BUILT_SOURCES = $(BUNDLED_DEPS)
 # See the following discussion for the workaround:
 # http://lists.gnu.org/archive/html/automake/2013-01/msg00051.html
 check-local: tests
-   ./libprocess-tests
+   $(TEST_DRIVER) ./libprocess-tests
 
 tests: all $(BUNDLED_DEPS)
$(MAKE) $(AM_MAKEFLAGS) libprocess-tests benchmarks

http://git-wip-us.apache.org/repos/asf/mesos/blob/105cc652/3rdparty/libprocess/configure.ac
--
diff --git a/3rdparty/libprocess/configure.ac b/3rdparty/libprocess/configure.ac
index f3033e3..1644035 100644
--- a/3rdparty/libprocess/configure.ac
+++ b/3rdparty/libprocess/configure.ac
@@ -213,6 +213,12 @@ AC_ARG_WITH([svn],
 
 
 ###
+# Miscellaneous flags/library/tool checks.
+###
+AC_ARG_VAR([TEST_DRIVER], [executable and arguments of a test driver])
+
+
+###
 # Compiler checks.
 ###

[2/5] mesos git commit: Added configure option for Mesos test runner.

2016-10-12 Thread tillt 
Added configure option for Mesos test runner.

Review: https://reviews.apache.org/r/51716/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/d996d78c
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/d996d78c
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/d996d78c

Branch: refs/heads/master
Commit: d996d78c4010c4d8319c102a5dfaf7f71b9c198a
Parents: 5ccacf5
Author: Benjamin Bannier 
Authored: Wed Oct 12 18:58:32 2016 +0200
Committer: Till Toenshoff 
Committed: Wed Oct 12 18:58:32 2016 +0200

--
 configure.ac | 18 +-
 1 file changed, 17 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/d996d78c/configure.ac
--
diff --git a/configure.ac b/configure.ac
index 034bb91..015255e 100644
--- a/configure.ac
+++ b/configure.ac
@@ -253,6 +253,9 @@ AC_ARG_ENABLE([zlib],
will be far less responsive; not recommended]),
 [enable_zlib=no], [enable_zlib=yes])
 
+AC_ARG_ENABLE([parallel_test_execution],
+  AS_HELP_STRING([--enable-parallel-test-execution],
+ [execute tests in parallel where possible]))
 
 ###
 # Optional packages.
@@ -544,7 +547,7 @@ esac
 
 
 ###
-# Miscellaneous flags/library/tool checks
+# Miscellaneous flags/library/tool checks.
 ###
 
 AC_ARG_VAR([JAVA_HOME], [location of Java Development Kit (JDK)])
@@ -559,6 +562,8 @@ AC_ARG_VAR([PROTOBUF_JAR], [full path to protobuf jar on 
prefixed builds])
 
 AC_ARG_VAR([PYTHON], [which Python interpreter to use])
 
+AC_ARG_VAR([TEST_DRIVER], [executable and arguments of a test driver])
+
 # Check for pthreads (uses m4/ax_pthread.m4).
 AX_PTHREAD([], [AC_MSG_ERROR([failed to find pthreads])])
 
@@ -605,6 +610,17 @@ AM_CONDITIONAL([INSTALL_TESTS], [test 
x"$enable_tests_install" = "xyes"])
 AS_IF([test x"$enable_tests_install" = "xyes"],
   [AC_DEFINE([MESOS_INSTALL_TESTS], [1])])
 
+# If we use our own test runner to parallelize gtests we can run
+# everything but ROOT tests in parallel.
+# We here set up `TEST_DRIVER` to contain an unexpanded automake
+# variable name; this allows us to reuse the test runner to run bundled
+# 3rdparty checks. The special quoting ensures that this containing a
+# space is not expanded by autoconf.
+# TODO(bbannier): Make this more readible by using autoconf/m4 magic.
+AS_IF([test "x$enable_parallel_test_execution" = "xyes"],
+  [AC_SUBST([TEST_DRIVER],
+   '${abs_top_srcdir}/support/mesos-gtest-runner.py'[' 
--sequential='*ROOT_*'']]))
+
 # Check for libunwind, and link it in if present.
 if test "x$OS_NAME" = "xfreebsd"; then
   AC_CHECK_LIB(execinfo, backtrace, LIBS="$LIBS -lexecinfo")

mesos git commit: Recover resources when offer is rescinded on DESTROY of shared volume.

2016-10-12 Thread yan 
Repository: mesos
Updated Branches:
  refs/heads/master 1fdea7dec -> 06d2e23dc


Recover resources when offer is rescinded on DESTROY of shared volume.

When a framework issues a DESTROY of a shared volume, and that volume
is not in use by a running or a pending task, we rescind the pending
offers in which the shared volume is present so that the deleted volume
is not assigned to any task in a future ACCEPT call.

At that time, we need to recover the resources as well for proper
accounting of such resources by the allocator.

Review: https://reviews.apache.org/r/52288/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/06d2e23d
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/06d2e23d
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/06d2e23d

Branch: refs/heads/master
Commit: 06d2e23dccc1ecc3c1d0e0cfb22ccca18bb6e56b
Parents: 1fdea7d
Author: Anindya Sinha 
Authored: Wed Oct 12 00:20:07 2016 -0700
Committer: Jiang Yan Xu 
Committed: Wed Oct 12 10:49:12 2016 -0700

--
 src/master/master.cpp |   6 ++
 src/tests/persistent_volume_tests.cpp | 146 +
 2 files changed, 152 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/06d2e23d/src/master/master.cpp
--
diff --git a/src/master/master.cpp b/src/master/master.cpp
index ad8993a..7ef8987 100644
--- a/src/master/master.cpp
+++ b/src/master/master.cpp
@@ -3975,6 +3975,12 @@ void Master::_accept(
   const Resources& offered = offer->resources();
   foreach (const Resource& volume, operation.destroy().volumes()) {
 if (offered.contains(volume)) {
+  allocator->recoverResources(
+  offer->framework_id(),
+  offer->slave_id(),
+  offer->resources(),
+  None());
+
   removeOffer(offer, true);
 }
   }

http://git-wip-us.apache.org/repos/asf/mesos/blob/06d2e23d/src/tests/persistent_volume_tests.cpp
--
diff --git a/src/tests/persistent_volume_tests.cpp 
b/src/tests/persistent_volume_tests.cpp
index e10a79e..b7d1c1a 100644
--- a/src/tests/persistent_volume_tests.cpp
+++ b/src/tests/persistent_volume_tests.cpp
@@ -908,6 +908,8 @@ TEST_P(PersistentVolumeTest, 
SharedPersistentVolumeMultipleTasks)
 
   FrameworkInfo frameworkInfo = DEFAULT_FRAMEWORK_INFO;
   frameworkInfo.set_role(DEFAULT_TEST_ROLE);
+  frameworkInfo.add_capabilities()->set_type(
+  FrameworkInfo::Capability::SHARED_RESOURCES);
 
   MockScheduler sched;
   MesosSchedulerDriver driver(
@@ -998,6 +1000,150 @@ TEST_P(PersistentVolumeTest, 
SharedPersistentVolumeMultipleTasks)
   driver.join();
 }
 
+// This test verifies that pending offers with shared persistent volumes
+// are rescinded when the volumes are destroyed.
+TEST_P(PersistentVolumeTest, SharedPersistentVolumeRescindOnDestroy)
+{
+  Clock::pause();
+
+  master::Flags masterFlags = CreateMasterFlags();
+  Try master = StartMaster(masterFlags);
+  ASSERT_SOME(master);
+
+  slave::Flags slaveFlags = CreateSlaveFlags();
+
+  slaveFlags.resources = getSlaveResources();
+
+  Owned detector = master.get()->createDetector();
+  Try slave = StartSlave(detector.get(), slaveFlags);
+  ASSERT_SOME(slave);
+
+  // 1. Create framework1 so that all resources are offered to this framework.
+  FrameworkInfo frameworkInfo1 = DEFAULT_FRAMEWORK_INFO;
+  frameworkInfo1.set_role(DEFAULT_TEST_ROLE);
+  frameworkInfo1.add_capabilities()->set_type(
+  FrameworkInfo::Capability::SHARED_RESOURCES);
+
+  MockScheduler sched1;
+  MesosSchedulerDriver driver1(
+  , frameworkInfo1, master.get()->pid, DEFAULT_CREDENTIAL);
+
+  EXPECT_CALL(sched1, registered(, _, _));
+
+  Future offers1;
+  EXPECT_CALL(sched1, resourceOffers(, _))
+.WillOnce(FutureArg<1>())
+.WillRepeatedly(Return()); // Ignore subsequent offers.
+
+  driver1.start();
+
+  AWAIT_READY(offers1);
+  EXPECT_FALSE(offers1.get().empty());
+
+  Offer offer1 = offers1.get()[0];
+
+  // 2. framework1 CREATEs a shared volume, and LAUNCHes a task with a subset
+  //of resources from the offer.
+  Resource volume = createPersistentVolume(
+  getDiskResource(Megabytes(2048)),
+  "id1",
+  "path1",
+  None(),
+  frameworkInfo1.principal(),
+  true);  // Shared volume.
+
+  // Create a task which uses a portion of the offered resources, so that
+  // the remaining resources can be offered to framework2. It's not important
+  // whether the volume is used (the task is killed soon and its purpose is
+  // only for splitting the offer).
+  TaskInfo task = createTask(
+

[3/5] mesos git commit: Enable Mesos test runner.

2016-10-12 Thread tillt 
Enable Mesos test runner.

Review: https://reviews.apache.org/r/51717/


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/8b611570
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/8b611570
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/8b611570

Branch: refs/heads/master
Commit: 8b611570bec9936a72917d19ed90dc6a87d16632
Parents: d996d78
Author: Benjamin Bannier 
Authored: Wed Oct 12 18:58:42 2016 +0200
Committer: Till Toenshoff 
Committed: Wed Oct 12 18:58:42 2016 +0200

--
 src/Makefile.am | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/mesos/blob/8b611570/src/Makefile.am
--
diff --git a/src/Makefile.am b/src/Makefile.am
index fd01e1d..3bcc0f2 100644
--- a/src/Makefile.am
+++ b/src/Makefile.am
@@ -2346,7 +2346,7 @@ uninstall-hook:
 # See the following discussion for the workaround:
 # http://lists.gnu.org/archive/html/automake/2013-01/msg00051.html
 check-local: tests
-   ./mesos-tests
+   $(TEST_DRIVER) ./mesos-tests
 
 if INSTALL_TESTS
 # If we enabled test installation, we can run the tests from the