[mesos] branch master updated: Fixed flakiness in 'RetryRpcWithExponentialBackoff'.
This is an automated email from the ASF dual-hosted git repository. chhsiao pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/mesos.git The following commit(s) were added to refs/heads/master by this push: new a87b66a Fixed flakiness in 'RetryRpcWithExponentialBackoff'. a87b66a is described below commit a87b66aeed840abbf06a2a300d85e8098e3d6fd4 Author: Jan Schlicht AuthorDate: Tue May 7 20:40:21 2019 -0700 Fixed flakiness in 'RetryRpcWithExponentialBackoff'. Under some circumstances, offers would be filtered, resulting in the test being stuck while waiting for offers. This has been resolved by settling the clock before accepting new offers. Review: https://reviews.apache.org/r/70184/ --- src/tests/storage_local_resource_provider_tests.cpp | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/tests/storage_local_resource_provider_tests.cpp b/src/tests/storage_local_resource_provider_tests.cpp index ecd..ba55728 100644 --- a/src/tests/storage_local_resource_provider_tests.cpp +++ b/src/tests/storage_local_resource_provider_tests.cpp @@ -5861,7 +5861,9 @@ TEST_P(StorageLocalResourceProviderTest, RetryRpcWithExponentialBackoff) AWAIT_READY(updateOperationStatus); EXPECT_EQ(OPERATION_FINISHED, updateOperationStatus->status().state()); - // Advance the clock to trigger a batch allocation. + // Settle the clock to recover the created disk, then advance the clock to + // trigger a batch allocation. + Clock::settle(); Clock::advance(masterFlags.allocation_interval); AWAIT_READY(offers);
[mesos-site] branch asf-site updated: Updated the website built from mesos SHA: 9c0ddce.
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/mesos-site.git The following commit(s) were added to refs/heads/asf-site by this push: new 2575e63 Updated the website built from mesos SHA: 9c0ddce. 2575e63 is described below commit 2575e6339083a993ed969de454d130d211688f50 Author: jenkins AuthorDate: Tue May 7 19:00:37 2019 + Updated the website built from mesos SHA: 9c0ddce. --- content/blog/feed.xml| 144 +- content/blog/index.html | 4 + content/blog/mesos-1-8-0-released/index.html | 325 +++ content/documentation/building/index.html| 4 +- content/documentation/latest/building/index.html | 4 +- content/downloads/index.html | 18 +- content/index.html | 4 +- content/sitemap.xml | 4 + 8 files changed, 495 insertions(+), 12 deletions(-) diff --git a/content/blog/feed.xml b/content/blog/feed.xml index c512486..0f64ace 100644 --- a/content/blog/feed.xml +++ b/content/blog/feed.xml @@ -4,7 +4,149 @@ http://mesos.apache.org/blog http://mesos.apache.org/blog; /> http://mesos.apache.org/blog/feed.xml; rel="self"/> - 2019-03-19T00:00:00+00:00 + 2019-05-02T00:00:00+00:00 + +http://mesos.apache.org/blog/mesos-1-8-0-released/ + + + Apache Mesos 1.8: Allocator Performance, Seccomp Isolation and Operation Feedback + +2019-05-02T00:00:00+00:00 + + Benno Evers + + + pWersquo;re pleased to announce that Mesos 1.8.0 is now available for a href=/downloadsdownload/a./p + +pAs usual, lots of work and care went into this release, with a total of 255 JIRA issues resolved, +and 1149 commits containing a total diffstat of 656 files changed, 62213 insertions(+) and 20403 deletions(-)./p + +precode 4.9% 3rdparty/ + 1.5% 3rdparty/libprocess/ + 1.6% 3rdparty/stout/ + 2.3% docs/ + 2.5% include/ + 2.0% site/ +77.4% src/ + 7.8% src/csi/ + 3.6% src/examples/ + 11.0% src/master/ + 4.2% src/resource_provider/ + 10.3% src/slave/ + 28.8% src/tests/ +6.7% support/ + 3.7% support/python3/ +/code/pre + +pLooking at the distribution of changes, nearly all components have seen major activity. +Some of that is presented in more detail below./p + +pOne major focus for the 1.8 contributors seems to have been solid test coverage, with +nearly one third of the total amount of lines changed being inside the test suite./p + +h2Highlights/h2 + +h3Allocator Performance Improvements/h3 + +pIn Mesos 1.8, allocator cycle time is significantly decreased when quota is +used; around 40% for a small size cluster and up to 70% for larger clusters. +This greatly narrows the allocator performance gap between quota and non-quota +usage scenarios./p + +pIn addition, schedulers can now specify the minimum resource quantities needed +in an offer, which acts as an override of the global code--min_allocatable_resources/code +master flag. Updating schedulers to specify this field improves multi-scheduler +scalability as it reduces the amount of offers declined from having insufficient +resource quantities./p + +pNote that this feature currently requires that the scheduler re-subscribes each +time it wants to mutate the minimum resource quantity offer filter information, +see a href=https://issues.apache.org/jira/browse/MESOS-7258MESOS-7258/a./p; + +h3Seccomp Isolator/h3 + +pA new codelinux/seccomp/code a href=docs/isolators/linux-seccompisolator/a was added. This +isolator makes use of the +a href=https://www.kernel.org/doc/Documentation/prctl/seccomp_filter.txtseccomp/a; +facility provided by recent linux kernels./p + +pUsing this isolator, containers launched by Mesos containerizer can be sandboxed +by enabling filtering of system calls using a configurable policy./p + +h3Operation Feedback/h3 + +pIn Mesos 1.6, operation feedback was introduced for operations on resources +by a resource provider./p + +pIn Mesos 1.8, v1 schedulers can now receive operation feedback for operations +on agent default resources, i.e. normal cpu, memory, and disk. This means that +the v1 scheduler APIrsquo;s operation feedback feature can now be used for any offer +operations except for codeLAUNCH/code and codeLAUNCH_GROUP/code, on any type of resources./p + +h3Containerization/h3 + +pA number of containerization-related improvements have landed in Mesos 1.8:/p + +ul +lipSupport pulling docker images with docker manifest +V2 Schema2 on Mesos Containerizer./p/li +lipSupport custom port range option to the codenetwork/ports/code +isolator. Added the code--container_ports_isolated_range/code flag to the +codenetwork/ports/code isolator. This allows the operator to specify a custom +port range to be protected by the isolator./p/li
[mesos] branch master updated (31ceed8 -> 9c0ddce)
This is an automated email from the ASF dual-hosted git repository. bennoe pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/mesos.git. from 31ceed8 Fixed unguarded calls to `Option::get()` in the master. new f969ead Added blogpost for 1.8.0 release. new 9c0ddce Updated site information for the 1.8 release. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: docs/building.md | 4 +- site/data/releases.yml | 10 +- .../source/blog/2019-05-02-mesos-1-8-0-released.md | 143 + 3 files changed, 153 insertions(+), 4 deletions(-) create mode 100644 site/source/blog/2019-05-02-mesos-1-8-0-released.md
[mesos] 01/02: Added blogpost for 1.8.0 release.
This is an automated email from the ASF dual-hosted git repository. bennoe pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/mesos.git commit f969ead212e4d7fd5dce1cf1acbb917270cbdfdd Author: Benno Evers AuthorDate: Mon May 6 15:38:22 2019 +0200 Added blogpost for 1.8.0 release. Review: https://reviews.apache.org/r/70598 --- .../source/blog/2019-05-02-mesos-1-8-0-released.md | 143 + 1 file changed, 143 insertions(+) diff --git a/site/source/blog/2019-05-02-mesos-1-8-0-released.md b/site/source/blog/2019-05-02-mesos-1-8-0-released.md new file mode 100644 index 000..b0e985b --- /dev/null +++ b/site/source/blog/2019-05-02-mesos-1-8-0-released.md @@ -0,0 +1,143 @@ +--- +layout: post +title: "Apache Mesos 1.8: Allocator Performance, Seccomp Isolation and Operation Feedback" +permalink: /blog/mesos-1-8-0-released/ +published: true +post_author: + display_name: Benno Evers +tags: Release +--- + +We're pleased to announce that Mesos 1.8.0 is now available for [download](/downloads). + +As usual, lots of work and care went into this release, with a total of 255 JIRA issues resolved, +and 1149 commits containing a total diffstat of 656 files changed, 62213 insertions(+) and 20403 deletions(-). + + 4.9% 3rdparty/ + 1.5% 3rdparty/libprocess/ + 1.6% 3rdparty/stout/ + 2.3% docs/ + 2.5% include/ + 2.0% site/ +77.4% src/ + 7.8% src/csi/ + 3.6% src/examples/ + 11.0% src/master/ + 4.2% src/resource_provider/ + 10.3% src/slave/ + 28.8% src/tests/ +6.7% support/ + 3.7% support/python3/ + + +Looking at the distribution of changes, nearly all components have seen major activity. +Some of that is presented in more detail below. + +One major focus for the 1.8 contributors seems to have been solid test coverage, with +nearly one third of the total amount of lines changed being inside the test suite. + + +## Highlights + +### Allocator Performance Improvements + +In Mesos 1.8, allocator cycle time is significantly decreased when quota is +used; around 40% for a small size cluster and up to 70% for larger clusters. +This greatly narrows the allocator performance gap between quota and non-quota +usage scenarios. + +In addition, schedulers can now specify the minimum resource quantities needed +in an offer, which acts as an override of the global `--min_allocatable_resources` +master flag. Updating schedulers to specify this field improves multi-scheduler +scalability as it reduces the amount of offers declined from having insufficient +resource quantities. + +Note that this feature currently requires that the scheduler re-subscribes each +time it wants to mutate the minimum resource quantity offer filter information, +see [MESOS-7258](https://issues.apache.org/jira/browse/MESOS-7258). + + +### Seccomp Isolator + +A new `linux/seccomp` [isolator](docs/isolators/linux-seccomp) was added. This +isolator makes use of the +[seccomp](https://www.kernel.org/doc/Documentation/prctl/seccomp_filter.txt) +facility provided by recent linux kernels. + +Using this isolator, containers launched by Mesos containerizer can be sandboxed +by enabling filtering of system calls using a configurable policy. + + +### Operation Feedback + +In Mesos 1.6, operation feedback was introduced for operations on resources +by a resource provider. + +In Mesos 1.8, v1 schedulers can now receive operation feedback for operations +on agent default resources, i.e. normal cpu, memory, and disk. This means that +the v1 scheduler API's operation feedback feature can now be used for any offer +operations except for `LAUNCH` and `LAUNCH_GROUP`, on any type of resources. + + +### Containerization + +A number of containerization-related improvements have landed in Mesos 1.8: + + - Support pulling docker images with docker manifest +V2 Schema2 on Mesos Containerizer. + + - Support custom port range option to the `network/ports` +isolator. Added the `--container_ports_isolated_range` flag to the +`network/ports` isolator. This allows the operator to specify a custom +port range to be protected by the isolator. + + - Support XFS quota for persistent volumes. Added +persistent volume support to the `disk/xfs` isolator. + + - Support an option to create non-existing host +paths for host path volume in Mesos Containerizer. Added a new +agent flag `--host_path_volume_force_creation` for the +`volume/host_path` isolator. + + +### CLI Improvements + +The Mesos CLI now offers the task subcommand with two actions. The first +action, `task attach`, allows you to attach your terminal to a running +task launched with a tty. The second action, `task exec`, launches a +new nested container inside a running task. + +To build the CLI, use the flag `--enable-new-cli` with Autotools or +`-DENABLE_NEW_CLI=1` with CMake on MacOS or Linux. + + +### Experimental
[mesos] 02/02: Updated site information for the 1.8 release.
This is an automated email from the ASF dual-hosted git repository. bennoe pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/mesos.git commit 9c0ddce550acd16668baebfef50bad5bb1606f9b Author: Benno Evers AuthorDate: Tue May 7 19:01:33 2019 +0200 Updated site information for the 1.8 release. --- docs/building.md | 4 ++-- site/data/releases.yml | 10 -- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/docs/building.md b/docs/building.md index f279b73..4ee739a 100644 --- a/docs/building.md +++ b/docs/building.md @@ -11,8 +11,8 @@ There are different ways you can get Mesos: 1\. Download the latest stable release from [Apache](http://mesos.apache.org/downloads/) (***Recommended***) -$ wget http://www.apache.org/dist/mesos/1.7.2/mesos-1.7.2.tar.gz -$ tar -zxf mesos-1.7.2.tar.gz +$ wget http://www.apache.org/dist/mesos/1.8.0/mesos-1.8.0.tar.gz +$ tar -zxf mesos-1.8.0.tar.gz 2\. Clone the Mesos git [repository](https://gitbox.apache.org/repos/asf/mesos.git) (***Advanced Users Only***) diff --git a/site/data/releases.yml b/site/data/releases.yml index 80d9516..1724779 100644 --- a/site/data/releases.yml +++ b/site/data/releases.yml @@ -3,8 +3,8 @@ # `versions` below. # * When publishing a new blog post, we need to append an item to `news` below. latest_stable: - version: 1.7.1 - jira_version: 12343861 + version: 1.8.0 + jira_version: 12343862 news: ## * If the news is used to announce a release version, the item structure is ## - title: (required) @@ -15,6 +15,10 @@ news: ## - title: (required) ## date: (required) ## blog: (required) + - title: Mesos 1.8.0 is released! +date: May 2nd, 2019 +target_version: 1.8.0 +blog: mesos-1-8-0-released - title: Mesos 1.5.3 is released! date: March 19, 2019 target_version: 1.5.3 @@ -196,6 +200,8 @@ versions: ## after we create the ReleaseNote for 0.28.1, we get the link: ## https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311242=12335359. ## At here, 12335359 is the jira_version of 0.28.1. + - version: 1.8.0 +jira_version: 12343862 - version: 1.5.3 jira_version: 12344352 - version: 1.4.3
[mesos] branch master updated: Fixed unguarded calls to `Option::get()` in the master.
This is an automated email from the ASF dual-hosted git repository.
grag pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
The following commit(s) were added to refs/heads/master by this push:
new 31ceed8 Fixed unguarded calls to `Option::get()` in the master.
31ceed8 is described below
commit 31ceed8dc636b3d63592d9c74f6cec03ed9745e8
Author: Greg Mann
AuthorDate: Thu May 2 14:48:48 2019 -0700
Fixed unguarded calls to `Option::get()` in the master.
Review: https://reviews.apache.org/r/70587
---
src/master/master.cpp| 32 +--
src/tests/agent_operation_feedback_tests.cpp | 138 +++
2 files changed, 162 insertions(+), 8 deletions(-)
diff --git a/src/master/master.cpp b/src/master/master.cpp
index 6c0e30b..437cbca 100644
--- a/src/master/master.cpp
+++ b/src/master/master.cpp
@@ -8759,8 +8759,9 @@ void
Master::updateOperationStatus(UpdateOperationStatusMessage&& update)
const SlaveID& slaveId = update.slave_id();
- // The status update for the operation might be for an
- // operator API call, thus the framework ID here is optional.
+ // While currently only frameworks can initiate operations which request
+ // feedback, in some cases such as master/agent reconciliation, the framework
+ // ID will not be set in the update message.
Option frameworkId = update.has_framework_id()
? update.framework_id()
: Option::none();
@@ -8770,6 +8771,8 @@ void
Master::updateOperationStatus(UpdateOperationStatusMessage&& update)
// agent. Since the operation UUID is not known, there is nothing for the
// master to do but forward the update to the framework and return.
if (!update.has_operation_uuid()) {
+CHECK_SOME(frameworkId);
+
// Forward the status update to the framework.
Framework* framework = getFramework(frameworkId.get());
@@ -8815,11 +8818,20 @@ void
Master::updateOperationStatus(UpdateOperationStatusMessage&& update)
Operation* operation = slave->getOperation(update.operation_uuid());
if (operation == nullptr) {
-// If the operation cannot be found, then this must be an update sent as a
-// result of a framework-inititated reconciliation which was forwarded to
-// the agent. Since the operation is not known to the master, there is
-// nothing for the master to do but forward the update to the framework and
-// return.
+// If the operation cannot be found and no framework ID was set, then this
+// must be a duplicate update as a result of master/agent reconciliation.
In
+// this case, a terminal reconciliation update has already been received by
+// the master, so we simply return.
+if (frameworkId.isNone()) {
+ return;
+}
+
+// If the operation cannot be found and a framework ID was set, then this
+// must be an update sent as a result of a framework-inititated
+// reconciliation which was forwarded to the agent and raced with an
+// `UpdateSlaveMessage` which removed the operation. Since the operation is
+// not known to the master, there is nothing for the master to do but
+// forward the update to the framework and return.
Framework* framework = getFramework(frameworkId.get());
if (framework == nullptr || !framework->connected()) {
@@ -8846,7 +8858,7 @@ void
Master::updateOperationStatus(UpdateOperationStatusMessage&& update)
// `ReconcileOperationsMessage`, but they can be deduced from the operation
// info kept on the master.
-// Only operations done via the scheduler API can have an ID.
+// Currently, only operations done via the scheduler API can have an ID.
CHECK(operation->has_framework_id());
frameworkId = operation->framework_id();
@@ -8866,10 +8878,13 @@ void
Master::updateOperationStatus(UpdateOperationStatusMessage&& update)
// Orphaned operations have no framework to send updates to.
bool frameworkWillAcknowledge =
operation->info().has_id() &&
+frameworkId.isSome() &&
!isCompletedFramework(frameworkId.get()) &&
!slave->orphanedOperations.contains(operation->uuid());
if (frameworkWillAcknowledge) {
+CHECK_SOME(frameworkId);
+
// Forward the status update to the framework.
Framework* framework = getFramework(frameworkId.get());
@@ -8910,6 +8925,7 @@ void
Master::updateOperationStatus(UpdateOperationStatusMessage&& update)
// buffer may also be affected by this wait.
if (operation->info().has_id() &&
slave->orphanedOperations.contains(operation->uuid()) &&
+ frameworkId.isSome() &&
!isCompletedFramework(frameworkId.get())) {
if (slave->reregisteredTime.isSome() &&
(Clock::now() - slave->reregisteredTime.get()) <
diff --git a/src/tests/agent_operation_feedback_tests.cpp
b/src/tests/agent_operation_feedback_tests.cpp
index e427441..a90038e 100644
---
