[jira] [Issue Comment Deleted] (MESOS-3041) Decline call does not include an optional "reason", in the Event/Call API
[
https://issues.apache.org/jira/browse/MESOS-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Wu updated MESOS-3041:
-
Comment: was deleted
(was: https://reviews.apache.org/r/37284/)
> Decline call does not include an optional "reason", in the Event/Call API
> -
>
> Key: MESOS-3041
> URL: https://issues.apache.org/jira/browse/MESOS-3041
> Project: Mesos
> Issue Type: Task
> Components: master
>Reporter: Joseph Wu
>Assignee: Joris Van Remoortere
> Labels: mesosphere
>
> In the Event/Call API, the Decline call is currently used by frameworks to
> reject resource offers.
> In the case of InverseOffers, the framework could give additional information
> to the operators and/or allocator, as to why the InverseOffer is declined.
> i.e. Suppose a cluster running some consensus algorithm is given an
> InverseOffer on one of its nodes. It may decline saying "Too few nodes" (or,
> more verbosely, "Specified InverseOffer would lower the number of active
> nodes below quorum").
> This change requires the following changes:
> * include/mesos/scheduler/scheduler.proto:
> {code}
> message Call {
> ...
> message Decline {
> repeated OfferID offer_ids = 1;
> optional Filters filters = 2;
> // Add this extra string for each OfferID
> // i.e. reasons[i] is for offer_ids[i]
> repeated string reasons = 3;
> }
> ...
> }
> {code}
> * src/master/master.cpp
> Change Master::decline to either store the reason, or log it.
> * Add a declineOffer overload in the (Mesos)SchedulerDriver with an optional
> "reason".
> ** Extend the interface in include/mesos/scheduler.hpp
> ** Add/change the declineOffer method in src/sched/sched.cpp
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-2222) Add ACLs for the maintenance HTTP endpoints.
[ https://issues.apache.org/jira/browse/MESOS-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745771#comment-14745771 ] Joseph Wu commented on MESOS-: -- Review: https://reviews.apache.org/r/38399/ > Add ACLs for the maintenance HTTP endpoints. > > > Key: MESOS- > URL: https://issues.apache.org/jira/browse/MESOS- > Project: Mesos > Issue Type: Task >Reporter: Benjamin Mahler >Assignee: Chen Zhiwei > Labels: mesosphere > > In order to authorize the HTTP endpoints for maintenance (to be added in > MESOS-2067), we will need to add an ACL definition for performing maintenance > operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-3345) Expand the range of integer precision when converting into/out of json.
[
https://issues.apache.org/jira/browse/MESOS-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14791077#comment-14791077
]
Joseph Wu commented on MESOS-3345:
--
With regards to supporting unsigned values between {{INT64_MAX}} and
{{UINT64_MAX}}:
* We can patch PicoJSON to parse those values as {{uint64_t}}.
* We can investigate using another parsing library.
* If we want extra precision beyond 64 or 80 bits per double, one possibility
is the [GMP library|https://gmplib.org/]
> Expand the range of integer precision when converting into/out of json.
> ---
>
> Key: MESOS-3345
> URL: https://issues.apache.org/jira/browse/MESOS-3345
> Project: Mesos
> Issue Type: Task
> Components: stout
>Reporter: Joseph Wu
>Assignee: Joseph Wu
>Priority: Minor
> Labels: json, mesosphere, protobuf
>
> For [MESOS-3299], we added some protobufs to represent time with integer
> precision. However, this precision is not maintained through protobuf <->
> JSON conversion, because of how our JSON encoders/decoders convert numbers to
> floating point.
> To maintain precision, we can try one of the following:
> * Try using a {{long double}} to represent a number.
> * Add logic to stringify/parse numbers without loss when possible.
> * Try representing {{int64_t}} as a string and parse it as such?
> * Update PicoJson and add a compiler flag, i.e. {{-DPICOJSON_USE_INT64}}
> In all cases, we'll need to make sure that:
> * Integers are properly stringified without loss.
> * The JSON decoder parses the integer without loss.
> * We have some unit tests for big (close to {{INT32_MAX}}/{{INT64_MAX}}) and
> small integers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MESOS-3449) Expand the range of integer precision in json <-> protobuf conversions to include unsigned integers
Joseph Wu created MESOS-3449:
Summary: Expand the range of integer precision in json <->
protobuf conversions to include unsigned integers
Key: MESOS-3449
URL: https://issues.apache.org/jira/browse/MESOS-3449
Project: Mesos
Issue Type: Improvement
Components: stout
Affects Versions: 0.25.0
Reporter: Joseph Wu
Assignee: Joseph Wu
Priority: Minor
The previous changes to support integer precision when converting JSON <->
Protobuf did not support precision for unsigned integers between {{INT64_MAX}}
and {{UINT64_MAX}}. (There's some loss, but the conversion is still as
good/bad as it was with doubles.)
This problem is due to a limitation in the JSON parsing library we use
(PicoJson), which parses integers as {{int64_t}}.
Some possible solutions or things to investigate:
* We can patch PicoJSON to parse some large values as {{uint64_t}}.
* We can investigate using another parsing library.
* If we want extra precision beyond 64 or 80 bits per double, one possibility
is the [GMP library|https://gmplib.org/]. We'd still need to change the
parsing library though.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-3449) Expand the range of integer precision in json <-> protobuf conversions to include unsigned integers
[
https://issues.apache.org/jira/browse/MESOS-3449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Wu updated MESOS-3449:
-
Description:
The previous changes (MESOS-3345) to support integer precision when converting
JSON <-> Protobuf did not support precision for unsigned integers between
{{INT64_MAX}} and {{UINT64_MAX}}. (There's some loss, but the conversion is
still as good/bad as it was with doubles.)
This problem is due to a limitation in the JSON parsing library we use
(PicoJson), which parses integers as {{int64_t}}.
Some possible solutions or things to investigate:
* We can patch PicoJSON to parse some large values as {{uint64_t}}.
* We can investigate using another parsing library.
* If we want extra precision beyond 64 or 80 bits per double, one possibility
is the [GMP library|https://gmplib.org/]. We'd still need to change the
parsing library though.
was:
The previous changes to support integer precision when converting JSON <->
Protobuf did not support precision for unsigned integers between {{INT64_MAX}}
and {{UINT64_MAX}}. (There's some loss, but the conversion is still as
good/bad as it was with doubles.)
This problem is due to a limitation in the JSON parsing library we use
(PicoJson), which parses integers as {{int64_t}}.
Some possible solutions or things to investigate:
* We can patch PicoJSON to parse some large values as {{uint64_t}}.
* We can investigate using another parsing library.
* If we want extra precision beyond 64 or 80 bits per double, one possibility
is the [GMP library|https://gmplib.org/]. We'd still need to change the
parsing library though.
> Expand the range of integer precision in json <-> protobuf conversions to
> include unsigned integers
> ---
>
> Key: MESOS-3449
> URL: https://issues.apache.org/jira/browse/MESOS-3449
> Project: Mesos
> Issue Type: Improvement
> Components: stout
>Affects Versions: 0.25.0
>Reporter: Joseph Wu
>Assignee: Joseph Wu
>Priority: Minor
> Labels: json, mesosphere, protobuf
>
> The previous changes (MESOS-3345) to support integer precision when
> converting JSON <-> Protobuf did not support precision for unsigned integers
> between {{INT64_MAX}} and {{UINT64_MAX}}. (There's some loss, but the
> conversion is still as good/bad as it was with doubles.)
> This problem is due to a limitation in the JSON parsing library we use
> (PicoJson), which parses integers as {{int64_t}}.
> Some possible solutions or things to investigate:
> * We can patch PicoJSON to parse some large values as {{uint64_t}}.
> * We can investigate using another parsing library.
> * If we want extra precision beyond 64 or 80 bits per double, one possibility
> is the [GMP library|https://gmplib.org/]. We'd still need to change the
> parsing library though.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-3449) Expand the range of integer precision in json <-> protobuf conversions to include unsigned integers
[
https://issues.apache.org/jira/browse/MESOS-3449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Wu updated MESOS-3449:
-
Description:
The previous changes (MESOS-3345) to support integer precision when converting
JSON <-> Protobuf did not support precision for unsigned integers between
{{INT64_MAX}} and {{UINT64_MAX}}. (There's some loss, but the conversion is
still as good/bad as it was with doubles.)
This problem is due to a limitation in the JSON parsing library we use
(PicoJSON), which parses integers as {{int64_t}}.
Some possible solutions or things to investigate:
* We can patch PicoJSON to parse some large values as {{uint64_t}}.
* We can investigate using another parsing library.
* If we want extra precision beyond 64 or 80 bits per double, one possibility
is the [GMP library|https://gmplib.org/]. We'd still need to change the
parsing library though.
was:
The previous changes (MESOS-3345) to support integer precision when converting
JSON <-> Protobuf did not support precision for unsigned integers between
{{INT64_MAX}} and {{UINT64_MAX}}. (There's some loss, but the conversion is
still as good/bad as it was with doubles.)
This problem is due to a limitation in the JSON parsing library we use
(PicoJson), which parses integers as {{int64_t}}.
Some possible solutions or things to investigate:
* We can patch PicoJSON to parse some large values as {{uint64_t}}.
* We can investigate using another parsing library.
* If we want extra precision beyond 64 or 80 bits per double, one possibility
is the [GMP library|https://gmplib.org/]. We'd still need to change the
parsing library though.
> Expand the range of integer precision in json <-> protobuf conversions to
> include unsigned integers
> ---
>
> Key: MESOS-3449
> URL: https://issues.apache.org/jira/browse/MESOS-3449
> Project: Mesos
> Issue Type: Improvement
> Components: stout
>Affects Versions: 0.25.0
>Reporter: Joseph Wu
>Assignee: Joseph Wu
>Priority: Minor
> Labels: json, mesosphere, protobuf
>
> The previous changes (MESOS-3345) to support integer precision when
> converting JSON <-> Protobuf did not support precision for unsigned integers
> between {{INT64_MAX}} and {{UINT64_MAX}}. (There's some loss, but the
> conversion is still as good/bad as it was with doubles.)
> This problem is due to a limitation in the JSON parsing library we use
> (PicoJSON), which parses integers as {{int64_t}}.
> Some possible solutions or things to investigate:
> * We can patch PicoJSON to parse some large values as {{uint64_t}}.
> * We can investigate using another parsing library.
> * If we want extra precision beyond 64 or 80 bits per double, one possibility
> is the [GMP library|https://gmplib.org/]. We'd still need to change the
> parsing library though.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3345) Expand the range of integer precision when converting into/out of json.
[
https://issues.apache.org/jira/browse/MESOS-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14791202#comment-14791202
]
Joseph Wu commented on MESOS-3345:
--
The followup issue is linked: https://issues.apache.org/jira/browse/MESOS-3449
> Expand the range of integer precision when converting into/out of json.
> ---
>
> Key: MESOS-3345
> URL: https://issues.apache.org/jira/browse/MESOS-3345
> Project: Mesos
> Issue Type: Task
> Components: stout
>Reporter: Joseph Wu
>Assignee: Joseph Wu
>Priority: Minor
> Labels: json, mesosphere, protobuf
>
> For [MESOS-3299], we added some protobufs to represent time with integer
> precision. However, this precision is not maintained through protobuf <->
> JSON conversion, because of how our JSON encoders/decoders convert numbers to
> floating point.
> To maintain precision, we can try one of the following:
> * Try using a {{long double}} to represent a number.
> * Add logic to stringify/parse numbers without loss when possible.
> * Try representing {{int64_t}} as a string and parse it as such?
> * Update PicoJson and add a compiler flag, i.e. {{-DPICOJSON_USE_INT64}}
> In all cases, we'll need to make sure that:
> * Integers are properly stringified without loss.
> * The JSON decoder parses the integer without loss.
> * We have some unit tests for big (close to {{INT32_MAX}}/{{INT64_MAX}}) and
> small integers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-3449) Expand the range of integer precision in json <-> protobuf conversions to include unsigned integers
[
https://issues.apache.org/jira/browse/MESOS-3449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Wu updated MESOS-3449:
-
Issue Type: Bug (was: Improvement)
> Expand the range of integer precision in json <-> protobuf conversions to
> include unsigned integers
> ---
>
> Key: MESOS-3449
> URL: https://issues.apache.org/jira/browse/MESOS-3449
> Project: Mesos
> Issue Type: Bug
> Components: stout
>Affects Versions: 0.25.0
>Reporter: Joseph Wu
>Assignee: Joseph Wu
>Priority: Minor
> Labels: json, mesosphere, protobuf
>
> The previous changes (MESOS-3345) to support integer precision when
> converting JSON <-> Protobuf did not support precision for unsigned integers
> between {{INT64_MAX}} and {{UINT64_MAX}}. (There's some loss, but the
> conversion is still as good/bad as it was with doubles.)
> This problem is due to a limitation in the JSON parsing library we use
> (PicoJSON), which parses integers as {{int64_t}}.
> Some possible solutions or things to investigate:
> * We can patch PicoJSON to parse some large values as {{uint64_t}}.
> * We can investigate using another parsing library.
> * If we want extra precision beyond 64 or 80 bits per double, one possibility
> is the [GMP library|https://gmplib.org/]. We'd still need to change the
> parsing library though.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MESOS-3458) Segfault when accepting or declining inverse offers
Joseph Wu created MESOS-3458: Summary: Segfault when accepting or declining inverse offers Key: MESOS-3458 URL: https://issues.apache.org/jira/browse/MESOS-3458 Project: Mesos Issue Type: Bug Reporter: Joseph Wu Assignee: Joseph Wu Discovered while writing a test for filters (in regards to inverse offers). Fix here: https://reviews.apache.org/r/38470/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-3458) Segfault when accepting or declining inverse offers
[ https://issues.apache.org/jira/browse/MESOS-3458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Wu updated MESOS-3458: - Priority: Blocker (was: Major) > Segfault when accepting or declining inverse offers > --- > > Key: MESOS-3458 > URL: https://issues.apache.org/jira/browse/MESOS-3458 > Project: Mesos > Issue Type: Bug >Reporter: Joseph Wu >Assignee: Joseph Wu >Priority: Blocker > > Discovered while writing a test for filters (in regards to inverse offers). > Fix here: https://reviews.apache.org/r/38470/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-3458) Segfault when accepting or declining inverse offers
[ https://issues.apache.org/jira/browse/MESOS-3458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Wu updated MESOS-3458: - Component/s: master > Segfault when accepting or declining inverse offers > --- > > Key: MESOS-3458 > URL: https://issues.apache.org/jira/browse/MESOS-3458 > Project: Mesos > Issue Type: Bug > Components: master >Reporter: Joseph Wu >Assignee: Joseph Wu >Priority: Blocker > Labels: mesosphere > > Discovered while writing a test for filters (in regards to inverse offers). > Fix here: https://reviews.apache.org/r/38470/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-3459) Change /machine/up and /machine/down endpoints to take an array
Joseph Wu created MESOS-3459:
Summary: Change /machine/up and /machine/down endpoints to take an
array
Key: MESOS-3459
URL: https://issues.apache.org/jira/browse/MESOS-3459
Project: Mesos
Issue Type: Task
Components: master
Reporter: Joseph Wu
Assignee: Joseph Wu
With [MESOS-3312] committed, the {{/machine/up}} and {{/machine/down}}
endpoints should also take an input as an array.
It is important to change this before maintenance primitives are released.
https://reviews.apache.org/r/38011/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-3458) Segfault when accepting or declining inverse offers
[ https://issues.apache.org/jira/browse/MESOS-3458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Wu updated MESOS-3458: - Labels: mesosphere (was: ) > Segfault when accepting or declining inverse offers > --- > > Key: MESOS-3458 > URL: https://issues.apache.org/jira/browse/MESOS-3458 > Project: Mesos > Issue Type: Bug > Components: master >Reporter: Joseph Wu >Assignee: Joseph Wu >Priority: Blocker > Labels: mesosphere > > Discovered while writing a test for filters (in regards to inverse offers). > Fix here: https://reviews.apache.org/r/38470/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-3459) Change /machine/up and /machine/down endpoints to take an array
[
https://issues.apache.org/jira/browse/MESOS-3459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Wu updated MESOS-3459:
-
Description:
With [MESOS-3312] committed, the {{/machine/up}} and {{/machine/down}}
endpoints should also take an input as an array.
It is important to change this before maintenance primitives are released:
https://reviews.apache.org/r/38011/
Also, a minor change to the error message from these endpoints:
https://reviews.apache.org/r/37969/
was:
With [MESOS-3312] committed, the {{/machine/up}} and {{/machine/down}}
endpoints should also take an input as an array.
It is important to change this before maintenance primitives are released.
https://reviews.apache.org/r/38011/
> Change /machine/up and /machine/down endpoints to take an array
> ---
>
> Key: MESOS-3459
> URL: https://issues.apache.org/jira/browse/MESOS-3459
> Project: Mesos
> Issue Type: Task
> Components: master
>Reporter: Joseph Wu
>Assignee: Joseph Wu
> Labels: mesosphere
>
> With [MESOS-3312] committed, the {{/machine/up}} and {{/machine/down}}
> endpoints should also take an input as an array.
> It is important to change this before maintenance primitives are released:
> https://reviews.apache.org/r/38011/
> Also, a minor change to the error message from these endpoints:
> https://reviews.apache.org/r/37969/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-3418) Factor out V1 API test helper functions
[
https://issues.apache.org/jira/browse/MESOS-3418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Wu updated MESOS-3418:
-
Description:
We currently have some helper functionality for V1 API tests. This is copied in
a few test files.
Factor this out into a common place once the API is stabilized.
{code}
// Helper class for using EXPECT_CALL since the Mesos scheduler API
// is callback based.
class Callbacks
{
public:
MOCK_METHOD0(connected, void(void));
MOCK_METHOD0(disconnected, void(void));
MOCK_METHOD1(received, void(const std::queue&));
};
{code}
{code}
// Enqueues all received events into a libprocess queue.
// TODO(jmlvanre): Factor this common code out of tests into V1
// helper.
ACTION_P(Enqueue, queue)
{
std::queue events = arg0;
while (!events.empty()) {
// Note that we currently drop HEARTBEATs because most of these tests
// are not designed to deal with heartbeats.
// TODO(vinod): Implement DROP_HTTP_CALLS that can filter heartbeats.
if (events.front().type() == Event::HEARTBEAT) {
VLOG(1) << "Ignoring HEARTBEAT event";
} else {
queue->put(events.front());
}
events.pop();
}
}
{code}
We can also update the helpers in {{/tests/mesos.hpp}} to support the V1 API.
This would let us get ride of lines like:
{code}
v1::TaskInfo taskInfo = evolve(createTask(devolve(offer), "",
DEFAULT_EXECUTOR_ID));
{code}
In favor of:
{code}
v1::TaskInfo taskInfo = createTask(offer, "", DEFAULT_EXECUTOR_ID);
{code}
was:
We currently have some helper functionality for V1 API tests. This is copied in
a few test files.
Factor this out into a common place once the API is stabilized.
{code}
// Helper class for using EXPECT_CALL since the Mesos scheduler API
// is callback based.
class Callbacks
{
public:
MOCK_METHOD0(connected, void(void));
MOCK_METHOD0(disconnected, void(void));
MOCK_METHOD1(received, void(const std::queue&));
};
{code}
{code}
// Enqueues all received events into a libprocess queue.
// TODO(jmlvanre): Factor this common code out of tests into V1
// helper.
ACTION_P(Enqueue, queue)
{
std::queue events = arg0;
while (!events.empty()) {
// Note that we currently drop HEARTBEATs because most of these tests
// are not designed to deal with heartbeats.
// TODO(vinod): Implement DROP_HTTP_CALLS that can filter heartbeats.
if (events.front().type() == Event::HEARTBEAT) {
VLOG(1) << "Ignoring HEARTBEAT event";
} else {
queue->put(events.front());
}
events.pop();
}
}
{code}
> Factor out V1 API test helper functions
> ---
>
> Key: MESOS-3418
> URL: https://issues.apache.org/jira/browse/MESOS-3418
> Project: Mesos
> Issue Type: Improvement
>Reporter: Joris Van Remoortere
>Assignee: Guangya Liu
> Labels: beginner, mesosphere, newbie, v1_api
>
> We currently have some helper functionality for V1 API tests. This is copied
> in a few test files.
> Factor this out into a common place once the API is stabilized.
> {code}
> // Helper class for using EXPECT_CALL since the Mesos scheduler API
> // is callback based.
> class Callbacks
> {
> public:
> MOCK_METHOD0(connected, void(void));
> MOCK_METHOD0(disconnected, void(void));
> MOCK_METHOD1(received, void(const std::queue&));
> };
> {code}
> {code}
> // Enqueues all received events into a libprocess queue.
> // TODO(jmlvanre): Factor this common code out of tests into V1
> // helper.
> ACTION_P(Enqueue, queue)
> {
> std::queue events = arg0;
> while (!events.empty()) {
> // Note that we currently drop HEARTBEATs because most of these tests
> // are not designed to deal with heartbeats.
> // TODO(vinod): Implement DROP_HTTP_CALLS that can filter heartbeats.
> if (events.front().type() == Event::HEARTBEAT) {
> VLOG(1) << "Ignoring HEARTBEAT event";
> } else {
> queue->put(events.front());
> }
> events.pop();
> }
> }
> {code}
> We can also update the helpers in {{/tests/mesos.hpp}} to support the V1 API.
> This would let us get ride of lines like:
> {code}
> v1::TaskInfo taskInfo = evolve(createTask(devolve(offer), "",
> DEFAULT_EXECUTOR_ID));
> {code}
> In favor of:
> {code}
> v1::TaskInfo taskInfo = createTask(offer, "", DEFAULT_EXECUTOR_ID);
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3459) Change /machine/up and /machine/down endpoints to take an array
[
https://issues.apache.org/jira/browse/MESOS-3459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876334#comment-14876334
]
Joseph Wu commented on MESOS-3459:
--
[~jvanremoortere] Committed:
{code}
commit cc0a847904510b80c72b9de70af9191b3d5b6c22
Author: Joseph Wu
Date: Fri Sep 18 13:07:38 2015 -0400
Maintenance Primitives: Replaced MachineIDs with RepeatedPtrField>.
This removes the MachineIDs protobuf and changes it to instead use the
RepeatedFieldPtr.
This also changes the maintenance primitives (alpha) API for
/machine/up and /machine/down, which both take an array instead of an
object now.
Review: https://reviews.apache.org/r/38011
{code}
{code}
commit 5f1195f8f10dc6fff65cb62a4745088e23bdc262
Author: Joseph Wu
Date: Fri Sep 18 12:44:03 2015 -0400
Maintenance primitives: Used JSON instead of protobuf for errors.
Replaces `__.DebugString()` with `stringify(JSON::Protobuf(__))`, which
looks nicer and matches the JSON expected by the HTTP endpoints.
Review: https://reviews.apache.org/r/37969
{code}
> Change /machine/up and /machine/down endpoints to take an array
> ---
>
> Key: MESOS-3459
> URL: https://issues.apache.org/jira/browse/MESOS-3459
> Project: Mesos
> Issue Type: Task
> Components: master
>Reporter: Joseph Wu
>Assignee: Joseph Wu
> Labels: mesosphere
>
> With [MESOS-3312] committed, the {{/machine/up}} and {{/machine/down}}
> endpoints should also take an input as an array.
> It is important to change this before maintenance primitives are released:
> https://reviews.apache.org/r/38011/
> Also, a minor change to the error message from these endpoints:
> https://reviews.apache.org/r/37969/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MESOS-3481) Add const accessor to Master flags
Joseph Wu created MESOS-3481: Summary: Add const accessor to Master flags Key: MESOS-3481 URL: https://issues.apache.org/jira/browse/MESOS-3481 Project: Mesos Issue Type: Task Reporter: Joseph Wu Assignee: Joseph Wu Priority: Trivial It would make sense to have an accessor to the master's flags, especially for tests. For example, see [this test|https://github.com/apache/mesos/blob/2876b8c918814347dd56f6f87d461e414a90650a/src/tests/master_maintenance_tests.cpp#L1231-L1235]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-3481) Add const accessor to Master flags
[ https://issues.apache.org/jira/browse/MESOS-3481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Wu updated MESOS-3481: - Assignee: (was: Joseph Wu) > Add const accessor to Master flags > -- > > Key: MESOS-3481 > URL: https://issues.apache.org/jira/browse/MESOS-3481 > Project: Mesos > Issue Type: Task >Reporter: Joseph Wu >Priority: Trivial > Labels: mesosphere, newbie > > It would make sense to have an accessor to the master's flags, especially for > tests. > For example, see [this > test|https://github.com/apache/mesos/blob/2876b8c918814347dd56f6f87d461e414a90650a/src/tests/master_maintenance_tests.cpp#L1231-L1235]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-3458) Segfault when accepting or declining inverse offers
[ https://issues.apache.org/jira/browse/MESOS-3458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Wu updated MESOS-3458: - Story Points: 1 > Segfault when accepting or declining inverse offers > --- > > Key: MESOS-3458 > URL: https://issues.apache.org/jira/browse/MESOS-3458 > Project: Mesos > Issue Type: Bug > Components: master >Reporter: Joseph Wu >Assignee: Joseph Wu >Priority: Blocker > Labels: mesosphere > > Discovered while writing a test for filters (in regards to inverse offers). > Fix here: https://reviews.apache.org/r/38470/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-3459) Change /machine/up and /machine/down endpoints to take an array
[
https://issues.apache.org/jira/browse/MESOS-3459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Wu updated MESOS-3459:
-
Story Points: 1
> Change /machine/up and /machine/down endpoints to take an array
> ---
>
> Key: MESOS-3459
> URL: https://issues.apache.org/jira/browse/MESOS-3459
> Project: Mesos
> Issue Type: Task
> Components: master
>Reporter: Joseph Wu
>Assignee: Joseph Wu
> Labels: mesosphere
>
> With [MESOS-3312] committed, the {{/machine/up}} and {{/machine/down}}
> endpoints should also take an input as an array.
> It is important to change this before maintenance primitives are released:
> https://reviews.apache.org/r/38011/
> Also, a minor change to the error message from these endpoints:
> https://reviews.apache.org/r/37969/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-2995) Standardize use of Path
[ https://issues.apache.org/jira/browse/MESOS-2995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Wu updated MESOS-2995: - Assignee: (was: Joseph Wu) > Standardize use of Path > > > Key: MESOS-2995 > URL: https://issues.apache.org/jira/browse/MESOS-2995 > Project: Mesos > Issue Type: Improvement > Components: stout >Reporter: Joseph Wu >Priority: Minor > Labels: mesosphere, newbie, stout > > As per the discussion in MESOS-2965, the use of the Path object should be > standardized: > * Functions which effectively use Paths (as strings) should instead take > Paths. > * Functions which modify and return Paths (as strings) should instead return > Paths. > * Extraneous uses of Path.value should be removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-3405) Add JSON::protobuf for google::protobuf::RepeatedPtrField.
[
https://issues.apache.org/jira/browse/MESOS-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901566#comment-14901566
]
Joseph Wu commented on MESOS-3405:
--
Note: With this change, you can also remove this test helper:
https://github.com/apache/mesos/blob/master/src/tests/master_maintenance_tests.cpp#L93-L104
> Add JSON::protobuf for google::protobuf::RepeatedPtrField.
> --
>
> Key: MESOS-3405
> URL: https://issues.apache.org/jira/browse/MESOS-3405
> Project: Mesos
> Issue Type: Task
> Components: stout
>Reporter: Michael Park
>Assignee: Klaus Ma
>
> Currently, {{stout/protobuf.hpp}} provides a {{JSON::Protobuf}} utility which
> converts a {{google::protobuf::Message}} into a {{JSON::Object}}.
> We should add the support for {{google::protobuf::RepeatedPtrField}} by
> introducing overloaded functions.
> {code}
> namespace JSON {
> Object protobuf(const google::protobuf::Message& message)
> {
> Object object;
> /* Move the body of JSON::Protobuf constructor here. */
> return object;
> }
> template
> Array protobuf(const google::protobuf::RepeatedPtrField& repeated)
> {
> static_assert(std::is_convertible::value,
> "T must be a google::protobuf::Message");
> JSON::Array array;
> array.values.reserve(repeated.size());
> foreach (const T& elem, repeated) {
> array.values.push_back(JSON::Protobuf(elem));
> }
> return array;
> }
> }
> {code}
> The new {{RepeatedPtrField}} version can be used in at least the following
> places:
> * {{src/common/http.cpp}}
> * {{src/master/http.cpp}}
> * {{src/slave/containerizer/mesos/containerizer.cpp}}
> * {{src/tests/reservation_endpoints_tests.cpp}}
> * {{src/tests/resources_tests.cpp}}: {{ResourcesTest.ParsingFromJSON}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-3468) Improve apply_reviews.sh script to apply chain of reviews
[ https://issues.apache.org/jira/browse/MESOS-3468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Wu updated MESOS-3468: - Assignee: Artem Harutyunyan > Improve apply_reviews.sh script to apply chain of reviews > - > > Key: MESOS-3468 > URL: https://issues.apache.org/jira/browse/MESOS-3468 > Project: Mesos > Issue Type: Improvement >Reporter: Vinod Kone >Assignee: Artem Harutyunyan > > Currently the support/apply-review.sh script allows an user (typically > committer) to apply a single review on top the HEAD. Since Mesos contributors > typically submit a chain of reviews for a given issue it makes sense for the > script to apply the whole chain recursively. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (MESOS-3183) Fetcher Cache Internals documentation images do not load
[ https://issues.apache.org/jira/browse/MESOS-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Wu reassigned MESOS-3183: Assignee: Joseph Wu > Fetcher Cache Internals documentation images do not load > > > Key: MESOS-3183 > URL: https://issues.apache.org/jira/browse/MESOS-3183 > Project: Mesos > Issue Type: Documentation > Components: documentation >Affects Versions: 0.23.0 >Reporter: James Mulcahy >Assignee: Joseph Wu >Priority: Minor > > The images on the Fetcher Cache Internals page do not load. > http://mesos.apache.org/documentation/fetcher-cache-internals/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-3183) Fetcher Cache Internals documentation images do not load
[
https://issues.apache.org/jira/browse/MESOS-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901580#comment-14901580
]
Joseph Wu commented on MESOS-3183:
--
This problem isn't limited to the Fetcher Cache. See:
* [External
Containerizer|http://mesos.apache.org/documentation/latest/external-containerizer/]
* [Fetcher Cache
Internals|http://mesos.apache.org/documentation/latest/fetcher-cache-internals/]
* [Maintenance|http://mesos.apache.org/documentation/latest/maintenance/]
*
[Oversubscription|http://mesos.apache.org/documentation/latest/oversubscription/]
There are two problems:
* The website itself does not have the images. For this, we need to patch the
[website's generation file(s)|https://svn.apache.org/repos/asf/mesos/site/].
See attached {{rake.patch}}.
* The links will point to the incorrect image URL if there is a {{/}} at the
end of the page URL.
Fix: https://reviews.apache.org/r/38570/
> Fetcher Cache Internals documentation images do not load
>
>
> Key: MESOS-3183
> URL: https://issues.apache.org/jira/browse/MESOS-3183
> Project: Mesos
> Issue Type: Documentation
> Components: documentation
>Affects Versions: 0.23.0
>Reporter: James Mulcahy
>Assignee: Joseph Wu
>Priority: Minor
>
> The images on the Fetcher Cache Internals page do not load.
> http://mesos.apache.org/documentation/fetcher-cache-internals/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-3183) Fetcher Cache Internals documentation images do not load
[ https://issues.apache.org/jira/browse/MESOS-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Wu updated MESOS-3183: - Attachment: rake.patch > Fetcher Cache Internals documentation images do not load > > > Key: MESOS-3183 > URL: https://issues.apache.org/jira/browse/MESOS-3183 > Project: Mesos > Issue Type: Documentation > Components: documentation >Affects Versions: 0.23.0 >Reporter: James Mulcahy >Assignee: Joseph Wu >Priority: Minor > Attachments: rake.patch > > > The images on the Fetcher Cache Internals page do not load. > http://mesos.apache.org/documentation/fetcher-cache-internals/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-3183) Documentation images do not load
[ https://issues.apache.org/jira/browse/MESOS-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Wu updated MESOS-3183: - Summary: Documentation images do not load (was: Fetcher Cache Internals documentation images do not load) > Documentation images do not load > > > Key: MESOS-3183 > URL: https://issues.apache.org/jira/browse/MESOS-3183 > Project: Mesos > Issue Type: Documentation > Components: documentation >Affects Versions: 0.23.0 >Reporter: James Mulcahy >Assignee: Joseph Wu >Priority: Minor > Attachments: rake.patch > > > The images on the Fetcher Cache Internals page do not load. > http://mesos.apache.org/documentation/fetcher-cache-internals/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-3183) Documentation images do not load
[
https://issues.apache.org/jira/browse/MESOS-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Wu updated MESOS-3183:
-
Description:
Any images which are referenced from the generated docs ({{docs/*.md}}) do not
show up on the website. For example:
* [External
Containerizer|http://mesos.apache.org/documentation/latest/external-containerizer/]
* [Fetcher Cache
Internals|http://mesos.apache.org/documentation/latest/fetcher-cache-internals/]
* [Maintenance|http://mesos.apache.org/documentation/latest/maintenance/]
*
[Oversubscription|http://mesos.apache.org/documentation/latest/oversubscription/]
was:
The images on the Fetcher Cache Internals page do not load.
http://mesos.apache.org/documentation/fetcher-cache-internals/
> Documentation images do not load
>
>
> Key: MESOS-3183
> URL: https://issues.apache.org/jira/browse/MESOS-3183
> Project: Mesos
> Issue Type: Documentation
> Components: documentation
>Affects Versions: 0.23.0
>Reporter: James Mulcahy
>Assignee: Joseph Wu
>Priority: Minor
> Attachments: rake.patch
>
>
> Any images which are referenced from the generated docs ({{docs/*.md}}) do
> not show up on the website. For example:
> * [External
> Containerizer|http://mesos.apache.org/documentation/latest/external-containerizer/]
> * [Fetcher Cache
> Internals|http://mesos.apache.org/documentation/latest/fetcher-cache-internals/]
> * [Maintenance|http://mesos.apache.org/documentation/latest/maintenance/]
> *
> [Oversubscription|http://mesos.apache.org/documentation/latest/oversubscription/]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (MESOS-3183) Documentation images do not load
[
https://issues.apache.org/jira/browse/MESOS-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901580#comment-14901580
]
Joseph Wu edited comment on MESOS-3183 at 9/21/15 11:03 PM:
There are two problems:
* The website itself does not have the images. For this, we need to patch the
[website's generation file(s)|https://svn.apache.org/repos/asf/mesos/site/].
See attached {{rake.patch}}.
* The links will point to the incorrect image URL if there is a {{/}} at the
end of the page URL.
Fix: https://reviews.apache.org/r/38570/
was (Author: kaysoky):
This problem isn't limited to the Fetcher Cache. See:
* [External
Containerizer|http://mesos.apache.org/documentation/latest/external-containerizer/]
* [Fetcher Cache
Internals|http://mesos.apache.org/documentation/latest/fetcher-cache-internals/]
* [Maintenance|http://mesos.apache.org/documentation/latest/maintenance/]
*
[Oversubscription|http://mesos.apache.org/documentation/latest/oversubscription/]
There are two problems:
* The website itself does not have the images. For this, we need to patch the
[website's generation file(s)|https://svn.apache.org/repos/asf/mesos/site/].
See attached {{rake.patch}}.
* The links will point to the incorrect image URL if there is a {{/}} at the
end of the page URL.
Fix: https://reviews.apache.org/r/38570/
> Documentation images do not load
>
>
> Key: MESOS-3183
> URL: https://issues.apache.org/jira/browse/MESOS-3183
> Project: Mesos
> Issue Type: Documentation
> Components: documentation
>Affects Versions: 0.23.0
>Reporter: James Mulcahy
>Assignee: Joseph Wu
>Priority: Minor
> Attachments: rake.patch
>
>
> Any images which are referenced from the generated docs ({{docs/*.md}}) do
> not show up on the website. For example:
> * [External
> Containerizer|http://mesos.apache.org/documentation/latest/external-containerizer/]
> * [Fetcher Cache
> Internals|http://mesos.apache.org/documentation/latest/fetcher-cache-internals/]
> * [Maintenance|http://mesos.apache.org/documentation/latest/maintenance/]
> *
> [Oversubscription|http://mesos.apache.org/documentation/latest/oversubscription/]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (MESOS-3490) Mesos UI fails to represent JSON entities
[ https://issues.apache.org/jira/browse/MESOS-3490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Wu reassigned MESOS-3490: Assignee: Joseph Wu > Mesos UI fails to represent JSON entities > - > > Key: MESOS-3490 > URL: https://issues.apache.org/jira/browse/MESOS-3490 > Project: Mesos > Issue Type: Bug >Reporter: Isabel Jimenez >Assignee: Joseph Wu > > The Mesos UI is broken, it seems to fail to represent JSON from /state. > This may got introduced with https://reviews.apache.org/r/38028/. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (MESOS-3490) Mesos UI fails to represent JSON entities
[
https://issues.apache.org/jira/browse/MESOS-3490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14902965#comment-14902965
]
Joseph Wu edited comment on MESOS-3490 at 9/22/15 4:55 PM:
---
The error lies with how we printed JSON numbers with a trailing period {{.}},
which is invalid JSON.
Simple fix is to print numbers with a trailing {{.0}} instead.
Review: https://reviews.apache.org/r/38632/
was (Author: kaysoky):
The error lies with how we printed JSON numbers with a trailing period '.',
which is invalid JSON.
Review: https://reviews.apache.org/r/38632/
> Mesos UI fails to represent JSON entities
> -
>
> Key: MESOS-3490
> URL: https://issues.apache.org/jira/browse/MESOS-3490
> Project: Mesos
> Issue Type: Bug
>Reporter: Isabel Jimenez
>Assignee: Joseph Wu
>
> The Mesos UI is broken, it seems to fail to represent JSON from /state.
> This may got introduced with https://reviews.apache.org/r/38028/.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3490) Mesos UI fails to represent JSON entities
[ https://issues.apache.org/jira/browse/MESOS-3490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14902965#comment-14902965 ] Joseph Wu commented on MESOS-3490: -- The error lies with how we printed JSON numbers with a trailing period '.', which is invalid JSON. Review: https://reviews.apache.org/r/38632/ > Mesos UI fails to represent JSON entities > - > > Key: MESOS-3490 > URL: https://issues.apache.org/jira/browse/MESOS-3490 > Project: Mesos > Issue Type: Bug >Reporter: Isabel Jimenez >Assignee: Joseph Wu > > The Mesos UI is broken, it seems to fail to represent JSON from /state. > This may got introduced with https://reviews.apache.org/r/38028/. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-3458) Segfault when accepting or declining inverse offers
[ https://issues.apache.org/jira/browse/MESOS-3458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Wu updated MESOS-3458: - Sprint: Mesosphere Sprint 19 > Segfault when accepting or declining inverse offers > --- > > Key: MESOS-3458 > URL: https://issues.apache.org/jira/browse/MESOS-3458 > Project: Mesos > Issue Type: Bug > Components: master >Reporter: Joseph Wu >Assignee: Joseph Wu >Priority: Blocker > Labels: mesosphere > > Discovered while writing a test for filters (in regards to inverse offers). > Fix here: https://reviews.apache.org/r/38470/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-3459) Change /machine/up and /machine/down endpoints to take an array
[
https://issues.apache.org/jira/browse/MESOS-3459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Wu updated MESOS-3459:
-
Sprint: Mesosphere Sprint 19
> Change /machine/up and /machine/down endpoints to take an array
> ---
>
> Key: MESOS-3459
> URL: https://issues.apache.org/jira/browse/MESOS-3459
> Project: Mesos
> Issue Type: Task
> Components: master
>Reporter: Joseph Wu
>Assignee: Joseph Wu
> Labels: mesosphere
>
> With [MESOS-3312] committed, the {{/machine/up}} and {{/machine/down}}
> endpoints should also take an input as an array.
> It is important to change this before maintenance primitives are released:
> https://reviews.apache.org/r/38011/
> Also, a minor change to the error message from these endpoints:
> https://reviews.apache.org/r/37969/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MESOS-3492) Expose maintenance user doc via the documentation home page
Joseph Wu created MESOS-3492:
Summary: Expose maintenance user doc via the documentation home
page
Key: MESOS-3492
URL: https://issues.apache.org/jira/browse/MESOS-3492
Project: Mesos
Issue Type: Task
Components: documentation
Reporter: Joseph Wu
Assignee: Joseph Wu
Priority: Trivial
The committed docs can be found here:
http://mesos.apache.org/documentation/latest/maintenance/
We need to add a link to {{docs/home.md}}
Also, the doc needs some minor formatting tweaks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3489) Add support for exposing Accept/Decline responses for inverse offers
[ https://issues.apache.org/jira/browse/MESOS-3489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14903618#comment-14903618 ] Joseph Wu commented on MESOS-3489: -- We should be able to get this done tonight. > Add support for exposing Accept/Decline responses for inverse offers > > > Key: MESOS-3489 > URL: https://issues.apache.org/jira/browse/MESOS-3489 > Project: Mesos > Issue Type: Bug >Reporter: Artem Harutyunyan >Assignee: Joseph Wu > Labels: mesosphere > > Current implementation of maintenance primitives does not support exposing > Accept/Decline responses of frameworks to the cluster operators. > This functionality is necessary to provide visibility to operators into > whether a given framework is ready to comply with the posted maintenance > schedule. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-3489) Add support for exposing Accept/Decline responses for inverse offers
[ https://issues.apache.org/jira/browse/MESOS-3489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Wu updated MESOS-3489: - Story Points: 2 (was: 8) > Add support for exposing Accept/Decline responses for inverse offers > > > Key: MESOS-3489 > URL: https://issues.apache.org/jira/browse/MESOS-3489 > Project: Mesos > Issue Type: Bug >Reporter: Artem Harutyunyan >Assignee: Joseph Wu > Labels: mesosphere > > Current implementation of maintenance primitives does not support exposing > Accept/Decline responses of frameworks to the cluster operators. > This functionality is necessary to provide visibility to operators into > whether a given framework is ready to comply with the posted maintenance > schedule. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-1615) Create design document for Optimistic Offers
[ https://issues.apache.org/jira/browse/MESOS-1615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14908870#comment-14908870 ] Joseph Wu commented on MESOS-1615: -- [~bmahler], Is it ok if we (Mesosphere/IBM working group on optimistic offers) take over this ticket? > Create design document for Optimistic Offers > > > Key: MESOS-1615 > URL: https://issues.apache.org/jira/browse/MESOS-1615 > Project: Mesos > Issue Type: Documentation >Reporter: Dominic Hamon >Assignee: Benjamin Mahler > > As a first step toward Optimistic Offers, take the description from the epic > and build an implementation design doc that can be shared for comments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (MESOS-1615) Create design document for Optimistic Offers
[ https://issues.apache.org/jira/browse/MESOS-1615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Wu reassigned MESOS-1615: Assignee: Joseph Wu > Create design document for Optimistic Offers > > > Key: MESOS-1615 > URL: https://issues.apache.org/jira/browse/MESOS-1615 > Project: Mesos > Issue Type: Documentation >Reporter: Dominic Hamon >Assignee: Joseph Wu > > As a first step toward Optimistic Offers, take the description from the epic > and build an implementation design doc that can be shared for comments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-1615) Create design document for Optimistic Offers
[ https://issues.apache.org/jira/browse/MESOS-1615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Wu updated MESOS-1615: - Labels: mesosphere (was: ) > Create design document for Optimistic Offers > > > Key: MESOS-1615 > URL: https://issues.apache.org/jira/browse/MESOS-1615 > Project: Mesos > Issue Type: Documentation >Reporter: Dominic Hamon >Assignee: Joseph Wu > Labels: mesosphere > > As a first step toward Optimistic Offers, take the description from the epic > and build an implementation design doc that can be shared for comments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-1615) Create design document for Optimistic Offers
[ https://issues.apache.org/jira/browse/MESOS-1615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Wu updated MESOS-1615: - Story Points: 5 > Create design document for Optimistic Offers > > > Key: MESOS-1615 > URL: https://issues.apache.org/jira/browse/MESOS-1615 > Project: Mesos > Issue Type: Documentation >Reporter: Dominic Hamon >Assignee: Joseph Wu > Labels: mesosphere > > As a first step toward Optimistic Offers, take the description from the epic > and build an implementation design doc that can be shared for comments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-1615) Create design document for Optimistic Offers
[ https://issues.apache.org/jira/browse/MESOS-1615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Wu updated MESOS-1615: - Description: As a first step toward Optimistic Offers, take the description from the epic and build an implementation design doc that can be shared for comments. Note: the links to the working group notes and design doc are located in the [JIRA Epic|MESOS-1607]. was:As a first step toward Optimistic Offers, take the description from the epic and build an implementation design doc that can be shared for comments. > Create design document for Optimistic Offers > > > Key: MESOS-1615 > URL: https://issues.apache.org/jira/browse/MESOS-1615 > Project: Mesos > Issue Type: Documentation >Reporter: Dominic Hamon >Assignee: Joseph Wu > Labels: mesosphere > > As a first step toward Optimistic Offers, take the description from the epic > and build an implementation design doc that can be shared for comments. > Note: the links to the working group notes and design doc are located in the > [JIRA Epic|MESOS-1607]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-1615) Create design document for Optimistic Offers
[ https://issues.apache.org/jira/browse/MESOS-1615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Wu updated MESOS-1615: - Story Points: 8 (was: 5) > Create design document for Optimistic Offers > > > Key: MESOS-1615 > URL: https://issues.apache.org/jira/browse/MESOS-1615 > Project: Mesos > Issue Type: Documentation >Reporter: Dominic Hamon >Assignee: Joseph Wu > Labels: mesosphere > > As a first step toward Optimistic Offers, take the description from the epic > and build an implementation design doc that can be shared for comments. > Note: the links to the working group notes and design doc are located in the > [JIRA Epic|MESOS-1607]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-1615) Create design document for Optimistic Offers
[ https://issues.apache.org/jira/browse/MESOS-1615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Wu updated MESOS-1615: - Sprint: Mesosphere Sprint 20 > Create design document for Optimistic Offers > > > Key: MESOS-1615 > URL: https://issues.apache.org/jira/browse/MESOS-1615 > Project: Mesos > Issue Type: Documentation >Reporter: Dominic Hamon >Assignee: Joseph Wu > Labels: mesosphere > > As a first step toward Optimistic Offers, take the description from the epic > and build an implementation design doc that can be shared for comments. > Note: the links to the working group notes and design doc are located in the > [JIRA Epic|MESOS-1607]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-3183) Documentation images do not load
[
https://issues.apache.org/jira/browse/MESOS-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Wu updated MESOS-3183:
-
Sprint: Mesosphere Sprint 20
> Documentation images do not load
>
>
> Key: MESOS-3183
> URL: https://issues.apache.org/jira/browse/MESOS-3183
> Project: Mesos
> Issue Type: Documentation
> Components: documentation
>Affects Versions: 0.24.0
>Reporter: James Mulcahy
>Assignee: Joseph Wu
>Priority: Minor
> Labels: mesosphere
> Attachments: rake.patch
>
>
> Any images which are referenced from the generated docs ({{docs/*.md}}) do
> not show up on the website. For example:
> * [External
> Containerizer|http://mesos.apache.org/documentation/latest/external-containerizer/]
> * [Fetcher Cache
> Internals|http://mesos.apache.org/documentation/latest/fetcher-cache-internals/]
> * [Maintenance|http://mesos.apache.org/documentation/latest/maintenance/]
> *
> [Oversubscription|http://mesos.apache.org/documentation/latest/oversubscription/]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-1607) Introduce optimistic offers.
[ https://issues.apache.org/jira/browse/MESOS-1607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Wu updated MESOS-1607: - Labels: mesosphere (was: ) > Introduce optimistic offers. > > > Key: MESOS-1607 > URL: https://issues.apache.org/jira/browse/MESOS-1607 > Project: Mesos > Issue Type: Epic > Components: allocation, framework, master >Reporter: Benjamin Hindman >Assignee: Artem Harutyunyan > Labels: mesosphere > Attachments: optimisitic-offers.pdf > > > The current implementation of resource offers only enable a single framework > scheduler to make scheduling decisions for some available resources at a > time. In some circumstances, this is good, i.e., when we don't want other > framework schedulers to have access to some resources. However, in other > circumstances, there are advantages to letting multiple framework schedulers > attempt to make scheduling decisions for the _same_ allocation of resources > in parallel. > If you think about this from a "concurrency control" perspective, the current > implementation of resource offers is _pessimistic_, the resources contained > within an offer are _locked_ until the framework scheduler that they were > offered to launches tasks with them or declines them. In addition to making > pessimistic offers we'd like to give out _optimistic_ offers, where the same > resources are offered to multiple framework schedulers at the same time, and > framework schedulers "compete" for those resources on a > first-come-first-serve basis (i.e., the first to launch a task "wins"). We've > always reserved the right to rescind resource offers using the 'rescind' > primitive in the API, and a framework scheduler should be prepared to launch > a task and have those tasks go lost because another framework already started > to use those resources. > Introducing optimistic offers will enable more sophisticated allocation > algorithms. For example, we can optimistically allocate resources that are > reserved for a particular framework (role) but are not being used. In > conjunction with revocable resources (the concept that using resources not > reserved for you means you might get those resources revoked) we can easily > create a "spot" market for unused resources, driving up utilization by > letting frameworks that are willing to use revocable resources run tasks. > In the limit, one could imagine always making optimistic resource offers. > This bears a striking resemblance with the Google Omega model (an isomorphism > even). However, being able to configure what resources should be allocated > optimistically and what resources should be allocated pessimistically gives > even more control to a datacenter/cluster operator that might want to, for > example, never let multiple frameworks (roles) compete for some set of > resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-3409) Refactor the plain JSON parsing in the docker containerizer
[
https://issues.apache.org/jira/browse/MESOS-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Wu updated MESOS-3409:
-
Assignee: Gilbert Song (was: Joseph Wu)
> Refactor the plain JSON parsing in the docker containerizer
> ---
>
> Key: MESOS-3409
> URL: https://issues.apache.org/jira/browse/MESOS-3409
> Project: Mesos
> Issue Type: Improvement
> Components: docker
>Reporter: Joseph Wu
>Assignee: Gilbert Song
>Priority: Minor
>
> Two functions in the Docker-related code take a string and parse it to JSON:
> * {{Docker::Container::create}} in {{src/docker/docker.cpp}}
> * {{Token::create}} in
> {{src/slave/containerizer/provisioners/docker/token_manager.cpp}}
> This JSON is then validated (lots of if-elses) and used via the
> {{JSON::Value}} accessors. We could instead use a protobuf and the related
> Stout JSON->Protobuf conversion function.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3183) Documentation images do not load
[
https://issues.apache.org/jira/browse/MESOS-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14950686#comment-14950686
]
Joseph Wu commented on MESOS-3183:
--
Update: [~davelester] has submitted the {{rake.patch}}, see [the svn
commit|http://svn.apache.org/viewvc?view=revision&revision=1707725].
Images still don't show up, since all links which are formatted like
{{images/foo.png}} will point to
{{/documentation/latest/some-document/images/foo.png}} instead of
{{/documentation/latest/images/foo.png}}.
We're still discussing how to make the URL change compatible with versioned
documentation.
> Documentation images do not load
>
>
> Key: MESOS-3183
> URL: https://issues.apache.org/jira/browse/MESOS-3183
> Project: Mesos
> Issue Type: Documentation
> Components: documentation
>Affects Versions: 0.24.0
>Reporter: James Mulcahy
>Assignee: Joseph Wu
>Priority: Minor
> Labels: mesosphere
> Attachments: rake.patch
>
>
> Any images which are referenced from the generated docs ({{docs/*.md}}) do
> not show up on the website. For example:
> * [External
> Containerizer|http://mesos.apache.org/documentation/latest/external-containerizer/]
> * [Fetcher Cache
> Internals|http://mesos.apache.org/documentation/latest/fetcher-cache-internals/]
> * [Maintenance|http://mesos.apache.org/documentation/latest/maintenance/]
> *
> [Oversubscription|http://mesos.apache.org/documentation/latest/oversubscription/]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-1607) Introduce optimistic offers.
[ https://issues.apache.org/jira/browse/MESOS-1607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Wu updated MESOS-1607: - Description: *Background* The current implementation of resource offers only enable a single framework scheduler to make scheduling decisions for some available resources at a time. In some circumstances, this is good, i.e., when we don't want other framework schedulers to have access to some resources. However, in other circumstances, there are advantages to letting multiple framework schedulers attempt to make scheduling decisions for the _same_ allocation of resources in parallel. If you think about this from a "concurrency control" perspective, the current implementation of resource offers is _pessimistic_, the resources contained within an offer are _locked_ until the framework scheduler that they were offered to launches tasks with them or declines them. In addition to making pessimistic offers we'd like to give out _optimistic_ offers, where the same resources are offered to multiple framework schedulers at the same time, and framework schedulers "compete" for those resources on a first-come-first-serve basis (i.e., the first to launch a task "wins"). We've always reserved the right to rescind resource offers using the 'rescind' primitive in the API, and a framework scheduler should be prepared to launch a task and have those tasks go lost because another framework already started to use those resources. *Feature* We plan to take a step towards optimistic offers, by introducing primitives that allow resources to be offered to multiple frameworks at once. At first, we will use these primitives to optimistically allocate resources that are reserved for a particular framework/role but have not been allocated by that framework/role. The work with optimistic offers will closely resemble the existing oversubscription feature. Optimistically offered resources are likely to be considered "revocable resources" (the concept that using resources not reserved for you means you might get those resources revoked). In effect, we can may create something like a "spot" market for unused resources, driving up utilization by letting frameworks that are willing to use revocable resources run tasks. *Future Work* This ticket tracks the introduction of some aspects of optimistic offers. Taken to the limit, one could imagine always making optimistic resource offers. This bears a striking resemblance with the Google Omega model (an isomorphism even). However, being able to configure what resources should be allocated optimistically and what resources should be allocated pessimistically gives even more control to a datacenter/cluster operator that might want to, for example, never let multiple frameworks (roles) compete for some set of resources. was: The current implementation of resource offers only enable a single framework scheduler to make scheduling decisions for some available resources at a time. In some circumstances, this is good, i.e., when we don't want other framework schedulers to have access to some resources. However, in other circumstances, there are advantages to letting multiple framework schedulers attempt to make scheduling decisions for the _same_ allocation of resources in parallel. If you think about this from a "concurrency control" perspective, the current implementation of resource offers is _pessimistic_, the resources contained within an offer are _locked_ until the framework scheduler that they were offered to launches tasks with them or declines them. In addition to making pessimistic offers we'd like to give out _optimistic_ offers, where the same resources are offered to multiple framework schedulers at the same time, and framework schedulers "compete" for those resources on a first-come-first-serve basis (i.e., the first to launch a task "wins"). We've always reserved the right to rescind resource offers using the 'rescind' primitive in the API, and a framework scheduler should be prepared to launch a task and have those tasks go lost because another framework already started to use those resources. Introducing optimistic offers will enable more sophisticated allocation algorithms. For example, we can optimistically allocate resources that are reserved for a particular framework (role) but are not being used. In conjunction with revocable resources (the concept that using resources not reserved for you means you might get those resources revoked) we can easily create a "spot" market for unused resources, driving up utilization by letting frameworks that are willing to use revocable resources run tasks. In the limit, one could imagine always making optimistic resource offers. This bears a striking resemblance with the Google Omega model (an isomorphism even). However, being able to configure what resources should be allocated opti
[jira] [Created] (MESOS-3748) HTTP API does not gracefully parse invalid resource identifiers
Joseph Wu created MESOS-3748:
Summary: HTTP API does not gracefully parse invalid resource
identifiers
Key: MESOS-3748
URL: https://issues.apache.org/jira/browse/MESOS-3748
Project: Mesos
Issue Type: Bug
Components: framework, HTTP API
Affects Versions: 0.25.0
Reporter: Joseph Wu
Assignee: Joseph Wu
If you pass a nonsense string for "master" into a framework using the HTTP API,
the framework segfaults.
For example, using the example frameworks:
{code}
build/src/test-framework --master="asdf://127.0.0.1:5050"
{code}
Results in:
{code}
Failed to create a master detector for 'asdf://127.0.0.1:5050': Failed to parse
'asdf://127.0.0.1:5050'
{code}
{code}
export DEFAULT_PRINCIPAL=root
build/src/event-call-framework --master="asdf://127.0.0.1:5050"
{code}
Results in
{code}
I1015 16:18:45.432075 2062201600 scheduler.cpp:157] Version: 0.26.0
Segmentation fault: 11
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-3748) HTTP API does not gracefully parse invalid resource identifiers
[
https://issues.apache.org/jira/browse/MESOS-3748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Wu updated MESOS-3748:
-
Description:
If you pass a nonsense string for "master" into a framework using the HTTP API,
the framework segfaults.
For example, using the example frameworks:
{code}
build/src/test-framework --master="asdf://127.0.0.1:5050"
{code}
Results in:
{code}
Failed to create a master detector for 'asdf://127.0.0.1:5050': Failed to parse
'asdf://127.0.0.1:5050'
{code}
Using the HTTP API:
{code}
export DEFAULT_PRINCIPAL=root
build/src/event-call-framework --master="asdf://127.0.0.1:5050"
{code}
Results in
{code}
I1015 16:18:45.432075 2062201600 scheduler.cpp:157] Version: 0.26.0
Segmentation fault: 11
{code}
was:
If you pass a nonsense string for "master" into a framework using the HTTP API,
the framework segfaults.
For example, using the example frameworks:
{code}
build/src/test-framework --master="asdf://127.0.0.1:5050"
{code}
Results in:
{code}
Failed to create a master detector for 'asdf://127.0.0.1:5050': Failed to parse
'asdf://127.0.0.1:5050'
{code}
{code}
export DEFAULT_PRINCIPAL=root
build/src/event-call-framework --master="asdf://127.0.0.1:5050"
{code}
Results in
{code}
I1015 16:18:45.432075 2062201600 scheduler.cpp:157] Version: 0.26.0
Segmentation fault: 11
{code}
> HTTP API does not gracefully parse invalid resource identifiers
> ---
>
> Key: MESOS-3748
> URL: https://issues.apache.org/jira/browse/MESOS-3748
> Project: Mesos
> Issue Type: Bug
> Components: framework, HTTP API
>Affects Versions: 0.25.0
>Reporter: Joseph Wu
>Assignee: Joseph Wu
> Labels: mesosphere
>
> If you pass a nonsense string for "master" into a framework using the HTTP
> API, the framework segfaults.
> For example, using the example frameworks:
> {code}
> build/src/test-framework --master="asdf://127.0.0.1:5050"
> {code}
> Results in:
> {code}
> Failed to create a master detector for 'asdf://127.0.0.1:5050': Failed to
> parse 'asdf://127.0.0.1:5050'
> {code}
> Using the HTTP API:
> {code}
> export DEFAULT_PRINCIPAL=root
> build/src/event-call-framework --master="asdf://127.0.0.1:5050"
> {code}
> Results in
> {code}
> I1015 16:18:45.432075 2062201600 scheduler.cpp:157] Version: 0.26.0
> Segmentation fault: 11
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-3748) HTTP scheduler library does not gracefully parse invalid resource identifiers
[
https://issues.apache.org/jira/browse/MESOS-3748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Wu updated MESOS-3748:
-
Summary: HTTP scheduler library does not gracefully parse invalid resource
identifiers (was: HTTP API does not gracefully parse invalid resource
identifiers)
> HTTP scheduler library does not gracefully parse invalid resource identifiers
> -
>
> Key: MESOS-3748
> URL: https://issues.apache.org/jira/browse/MESOS-3748
> Project: Mesos
> Issue Type: Bug
> Components: framework, HTTP API
>Affects Versions: 0.25.0
>Reporter: Joseph Wu
>Assignee: Joseph Wu
> Labels: mesosphere
>
> If you pass a nonsense string for "master" into a framework using the HTTP
> API, the framework segfaults.
> For example, using the example frameworks:
> {code}
> build/src/test-framework --master="asdf://127.0.0.1:5050"
> {code}
> Results in:
> {code}
> Failed to create a master detector for 'asdf://127.0.0.1:5050': Failed to
> parse 'asdf://127.0.0.1:5050'
> {code}
> Using the HTTP API:
> {code}
> export DEFAULT_PRINCIPAL=root
> build/src/event-call-framework --master="asdf://127.0.0.1:5050"
> {code}
> Results in
> {code}
> I1015 16:18:45.432075 2062201600 scheduler.cpp:157] Version: 0.26.0
> Segmentation fault: 11
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-3748) HTTP scheduler library does not gracefully parse invalid resource identifiers
[
https://issues.apache.org/jira/browse/MESOS-3748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Wu updated MESOS-3748:
-
Description:
If you pass a nonsense string for "master" into a framework using the HTTP
scheduler library, the framework segfaults.
For example, using the example frameworks:
{code}
build/src/test-framework --master="asdf://127.0.0.1:5050"
{code}
Results in:
{code}
Failed to create a master detector for 'asdf://127.0.0.1:5050': Failed to parse
'asdf://127.0.0.1:5050'
{code}
Using the HTTP API:
{code}
export DEFAULT_PRINCIPAL=root
build/src/event-call-framework --master="asdf://127.0.0.1:5050"
{code}
Results in
{code}
I1015 16:18:45.432075 2062201600 scheduler.cpp:157] Version: 0.26.0
Segmentation fault: 11
{code}
was:
If you pass a nonsense string for "master" into a framework using the HTTP API,
the framework segfaults.
For example, using the example frameworks:
{code}
build/src/test-framework --master="asdf://127.0.0.1:5050"
{code}
Results in:
{code}
Failed to create a master detector for 'asdf://127.0.0.1:5050': Failed to parse
'asdf://127.0.0.1:5050'
{code}
Using the HTTP API:
{code}
export DEFAULT_PRINCIPAL=root
build/src/event-call-framework --master="asdf://127.0.0.1:5050"
{code}
Results in
{code}
I1015 16:18:45.432075 2062201600 scheduler.cpp:157] Version: 0.26.0
Segmentation fault: 11
{code}
> HTTP scheduler library does not gracefully parse invalid resource identifiers
> -
>
> Key: MESOS-3748
> URL: https://issues.apache.org/jira/browse/MESOS-3748
> Project: Mesos
> Issue Type: Bug
> Components: framework, HTTP API
>Affects Versions: 0.25.0
>Reporter: Joseph Wu
>Assignee: Joseph Wu
> Labels: mesosphere
>
> If you pass a nonsense string for "master" into a framework using the HTTP
> scheduler library, the framework segfaults.
> For example, using the example frameworks:
> {code}
> build/src/test-framework --master="asdf://127.0.0.1:5050"
> {code}
> Results in:
> {code}
> Failed to create a master detector for 'asdf://127.0.0.1:5050': Failed to
> parse 'asdf://127.0.0.1:5050'
> {code}
> Using the HTTP API:
> {code}
> export DEFAULT_PRINCIPAL=root
> build/src/event-call-framework --master="asdf://127.0.0.1:5050"
> {code}
> Results in
> {code}
> I1015 16:18:45.432075 2062201600 scheduler.cpp:157] Version: 0.26.0
> Segmentation fault: 11
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-3748) HTTP scheduler library does not gracefully parse invalid resource identifiers
[
https://issues.apache.org/jira/browse/MESOS-3748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Wu updated MESOS-3748:
-
Description:
If you pass a nonsense string for "master" into a framework using the HTTP
scheduler library, the framework segfaults.
For example, using the example frameworks:
{code:title=Scheduler Driver}
build/src/test-framework --master="asdf://127.0.0.1:5050"
{code}
Results in:
{code}
Failed to create a master detector for 'asdf://127.0.0.1:5050': Failed to parse
'asdf://127.0.0.1:5050'
{code}
{code:title=HTTP Scheduler Library}
export DEFAULT_PRINCIPAL=root
build/src/event-call-framework --master="asdf://127.0.0.1:5050"
{code}
Results in
{code}
I1015 16:18:45.432075 2062201600 scheduler.cpp:157] Version: 0.26.0
Segmentation fault: 11
{code}
{code:title=Stack Trace}
* thread #2: tid = 0x28b6bb, 0x000100ad03ca
libmesos-0.26.0.dylib`mesos::v1::scheduler::MesosProcess::initialize(this=0x0001076031a0)
+ 42 at scheduler.cpp:213, stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
* frame #0: 0x000100ad03ca
libmesos-0.26.0.dylib`mesos::v1::scheduler::MesosProcess::initialize(this=0x0001076031a0)
+ 42 at scheduler.cpp:213
frame #1: 0x000100ad05f2 libmesos-0.26.0.dylib`virtual thunk to
mesos::v1::scheduler::MesosProcess::initialize(this=0x0001076031a0) + 34 at
scheduler.cpp:210
frame #2: 0x0001022b60f3 libmesos-0.26.0.dylib`::resume() + 931 at
process.cpp:2449
frame #3: 0x0001022c131c libmesos-0.26.0.dylib`::operator()() + 268 at
process.cpp:2174
frame #4: 0x0001022c0fa2
libmesos-0.26.0.dylib`::__thread_proxy > > > >() [inlined]
__invoke<(lambda at ../../../3rdparty/libprocess/src/process.cpp:2158:35) &,
const std::__1::atomic &> + 27 at __functional_base:415
frame #5: 0x0001022c0f87
libmesos-0.26.0.dylib`::__thread_proxy > > > >() [inlined]
__apply_functor<(lambda at
../../../3rdparty/libprocess/src/process.cpp:2158:35),
std::__1::tuple > >,
0, std::__1::tuple<> > + 55 at functional:2060
frame #6: 0x0001022c0f50
libmesos-0.26.0.dylib`::__thread_proxy > > > >() [inlined]
operator()<> + 41 at functional:2123
frame #7: 0x0001022c0f27
libmesos-0.26.0.dylib`::__thread_proxy > > > >() [inlined]
__invoke > >> + 14 at
__functional_base:415
frame #8: 0x0001022c0f19
libmesos-0.26.0.dylib`::__thread_proxy > > > >() [inlined]
__thread_execute > >> + 25 at thread:337
frame #9: 0x0001022c0f00
libmesos-0.26.0.dylib`::__thread_proxy > > > >() + 368 at
thread:347
frame #10: 0x7fff964c705a libsystem_pthread.dylib`_pthread_body + 131
frame #11: 0x7fff964c6fd7 libsystem_pthread.dylib`_pthread_start + 176
frame #12: 0x7fff964c43ed libsystem_pthread.dylib`thread_start + 13
{code}
was:
If you pass a nonsense string for "master" into a framework using the HTTP
scheduler library, the framework segfaults.
For example, using the example frameworks:
{code}
build/src/test-framework --master="asdf://127.0.0.1:5050"
{code}
Results in:
{code}
Failed to create a master detector for 'asdf://127.0.0.1:5050': Failed to parse
'asdf://127.0.0.1:5050'
{code}
Using the HTTP API:
{code}
export DEFAULT_PRINCIPAL=root
build/src/event-call-framework --master="asdf://127.0.0.1:5050"
{code}
Results in
{code}
I1015 16:18:45.432075 2062201600 scheduler.cpp:157] Version: 0.26.0
Segmentation fault: 11
{code}
> HTTP scheduler library does not gracefully parse invalid resource identifiers
> -
>
> Key: MESOS-3748
> URL: https://issues.apache.org/jira/browse/MESOS-3748
> Project: Mesos
> Issue Type: Bug
> Components: framework, HTTP API
>Affects Versions: 0.25.0
>Reporter: Joseph Wu
>Assignee: Joseph Wu
> Labels: mesosphere
>
> If you pass a nonsense string for "master" into a framework using the HTTP
> scheduler library, the framework segfaults.
> For example, using the example frameworks:
> {code:title=Scheduler Driver}
> build/src/test-framework --master="asdf://127.0.0.1:5050"
> {code}
> Results in:
> {code}
> Failed to create a master detector for 'asdf://127.0.0.1:5050': Failed to
> parse 'asdf://127.0.0.1:5050'
> {code}
> {code:title=HTTP Scheduler Library}
> export DEFAULT_PRINCIPAL=root
> build/src/event-call-framework --master="asdf://127.0.0.1:5050"
> {code}
> Results in
> {code}
> I1015 16:18:45.432075 2062201600 scheduler.cpp:157] Version: 0.26.0
> Segmentation fault: 11
> {code}
> {code:title=Stack Trace}
> * thread #2: tid = 0x28b6bb, 0x000100ad03ca
> libmesos-0.26.0.dylib`mesos::v1::scheduler::MesosProcess::initialize(this=0x0001076031a0)
> + 42 at scheduler.cpp:213, stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
> * frame #0: 0x000100ad03ca
> libmesos-0.26.0.dylib`mesos::v1::scheduler::MesosProcess::init
[jira] [Commented] (MESOS-3748) HTTP scheduler library does not gracefully parse invalid resource identifiers
[
https://issues.apache.org/jira/browse/MESOS-3748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14959889#comment-14959889
]
Joseph Wu commented on MESOS-3748:
--
Turns out to be a rather trivial issue: https://reviews.apache.org/r/39365/
> HTTP scheduler library does not gracefully parse invalid resource identifiers
> -
>
> Key: MESOS-3748
> URL: https://issues.apache.org/jira/browse/MESOS-3748
> Project: Mesos
> Issue Type: Bug
> Components: framework, HTTP API
>Affects Versions: 0.25.0
>Reporter: Joseph Wu
>Assignee: Joseph Wu
> Labels: mesosphere
>
> If you pass a nonsense string for "master" into a framework using the HTTP
> scheduler library, the framework segfaults.
> For example, using the example frameworks:
> {code:title=Scheduler Driver}
> build/src/test-framework --master="asdf://127.0.0.1:5050"
> {code}
> Results in:
> {code}
> Failed to create a master detector for 'asdf://127.0.0.1:5050': Failed to
> parse 'asdf://127.0.0.1:5050'
> {code}
> {code:title=HTTP Scheduler Library}
> export DEFAULT_PRINCIPAL=root
> build/src/event-call-framework --master="asdf://127.0.0.1:5050"
> {code}
> Results in
> {code}
> I1015 16:18:45.432075 2062201600 scheduler.cpp:157] Version: 0.26.0
> Segmentation fault: 11
> {code}
> {code:title=Stack Trace}
> * thread #2: tid = 0x28b6bb, 0x000100ad03ca
> libmesos-0.26.0.dylib`mesos::v1::scheduler::MesosProcess::initialize(this=0x0001076031a0)
> + 42 at scheduler.cpp:213, stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
> * frame #0: 0x000100ad03ca
> libmesos-0.26.0.dylib`mesos::v1::scheduler::MesosProcess::initialize(this=0x0001076031a0)
> + 42 at scheduler.cpp:213
> frame #1: 0x000100ad05f2 libmesos-0.26.0.dylib`virtual thunk to
> mesos::v1::scheduler::MesosProcess::initialize(this=0x0001076031a0) + 34
> at scheduler.cpp:210
> frame #2: 0x0001022b60f3 libmesos-0.26.0.dylib`::resume() + 931 at
> process.cpp:2449
> frame #3: 0x0001022c131c libmesos-0.26.0.dylib`::operator()() + 268
> at process.cpp:2174
> frame #4: 0x0001022c0fa2
> libmesos-0.26.0.dylib`::__thread_proxy at ../../../3rdparty/libprocess/src/process.cpp:2158:35),
> std::__1::reference_wrapper > > > >() [inlined]
> __invoke<(lambda at ../../../3rdparty/libprocess/src/process.cpp:2158:35) &,
> const std::__1::atomic &> + 27 at __functional_base:415
> frame #5: 0x0001022c0f87
> libmesos-0.26.0.dylib`::__thread_proxy at ../../../3rdparty/libprocess/src/process.cpp:2158:35),
> std::__1::reference_wrapper > > > >() [inlined]
> __apply_functor<(lambda at
> ../../../3rdparty/libprocess/src/process.cpp:2158:35),
> std::__1::tuple > >,
> 0, std::__1::tuple<> > + 55 at functional:2060
> frame #6: 0x0001022c0f50
> libmesos-0.26.0.dylib`::__thread_proxy at ../../../3rdparty/libprocess/src/process.cpp:2158:35),
> std::__1::reference_wrapper > > > >() [inlined]
> operator()<> + 41 at functional:2123
> frame #7: 0x0001022c0f27
> libmesos-0.26.0.dylib`::__thread_proxy at ../../../3rdparty/libprocess/src/process.cpp:2158:35),
> std::__1::reference_wrapper > > > >() [inlined]
> __invoke ../../../3rdparty/libprocess/src/process.cpp:2158:35),
> std::__1::reference_wrapper > >> + 14 at
> __functional_base:415
> frame #8: 0x0001022c0f19
> libmesos-0.26.0.dylib`::__thread_proxy at ../../../3rdparty/libprocess/src/process.cpp:2158:35),
> std::__1::reference_wrapper > > > >() [inlined]
> __thread_execute ../../../3rdparty/libprocess/src/process.cpp:2158:35),
> std::__1::reference_wrapper > >> + 25 at
> thread:337
> frame #9: 0x0001022c0f00
> libmesos-0.26.0.dylib`::__thread_proxy at ../../../3rdparty/libprocess/src/process.cpp:2158:35),
> std::__1::reference_wrapper > > > >() + 368 at
> thread:347
> frame #10: 0x7fff964c705a libsystem_pthread.dylib`_pthread_body + 131
> frame #11: 0x7fff964c6fd7 libsystem_pthread.dylib`_pthread_start + 176
> frame #12: 0x7fff964c43ed libsystem_pthread.dylib`thread_start + 13
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-3751) MESOS_NATIVE_JAVA_LIBRARY not set on MesosContainerize tasks with --executor_environmnent_variables
[ https://issues.apache.org/jira/browse/MESOS-3751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Wu updated MESOS-3751: - Summary: MESOS_NATIVE_JAVA_LIBRARY not set on MesosContainerize tasks with --executor_environmnent_variables (was: MESOS_NATIVE_JAVA_LIBRARY not set on MesosContainerizre tasks with --executor_environmnent_variables) > MESOS_NATIVE_JAVA_LIBRARY not set on MesosContainerize tasks with > --executor_environmnent_variables > --- > > Key: MESOS-3751 > URL: https://issues.apache.org/jira/browse/MESOS-3751 > Project: Mesos > Issue Type: Bug > Components: containerization >Affects Versions: 0.24.1, 0.25.0 >Reporter: Cody Maloney >Assignee: Gilbert Song > Labels: mesosphere, newbie > > When using --executor_environment_variables, and having > MESOS_NATIVE_JAVA_LIBRARY in the environment of mesos-slave, the mesos > containerizer does not set MESOS_NATIVE_JAVA_LIBRARY itself. > Relevant code: > https://github.com/apache/mesos/blob/14f7967ef307f3d98e3a4b93d92d6b3a56399b20/src/slave/containerizer/containerizer.cpp#L281 > It sees that the variable is in the mesos-slave's environment (os::getenv), > rather than checking if it is set in the environment variable set. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-3753) Test the HTTP Scheduler library with SSL enabled
Joseph Wu created MESOS-3753: Summary: Test the HTTP Scheduler library with SSL enabled Key: MESOS-3753 URL: https://issues.apache.org/jira/browse/MESOS-3753 Project: Mesos Issue Type: Story Components: framework, HTTP API, test Reporter: Joseph Wu Assignee: Joseph Wu Currently, the HTTP Scheduler library does not support SSL-enabled Mesos. We need to add tests that check the schedule library against SSL-enabled Mesos with SSL: * with downgrade support, * with/without verification of certificates (framework-side), * with required framework/client-side certifications, * with/without verification of certificates (master-side), * with a custom certificate authority (CA), These options should be controlled by the same environment variables found on the [SSL user doc|http://mesos.apache.org/documentation/latest/ssl/]. Note: This issue will be broken down into smaller sub-issues as bugs/problems are discovered. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-3759) Document messages.proto
Joseph Wu created MESOS-3759: Summary: Document messages.proto Key: MESOS-3759 URL: https://issues.apache.org/jira/browse/MESOS-3759 Project: Mesos Issue Type: Improvement Components: documentation Reporter: Joseph Wu Assignee: Joseph Wu The messages we pass between Mesos components are largely undocumented. See this [TODO|https://github.com/apache/mesos/blob/master/src/messages/messages.proto#L23]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-3759) Document messages.proto
[ https://issues.apache.org/jira/browse/MESOS-3759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Wu updated MESOS-3759: - Shepherd: Joris Van Remoortere > Document messages.proto > --- > > Key: MESOS-3759 > URL: https://issues.apache.org/jira/browse/MESOS-3759 > Project: Mesos > Issue Type: Improvement > Components: documentation >Reporter: Joseph Wu >Assignee: Joseph Wu > Labels: documentation, mesosphere > > The messages we pass between Mesos components are largely undocumented. See > this > [TODO|https://github.com/apache/mesos/blob/master/src/messages/messages.proto#L23]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-3759) Document messages.proto
[ https://issues.apache.org/jira/browse/MESOS-3759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Wu updated MESOS-3759: - Description: The messages we pass between Mesos components are largely undocumented. See this [TODO|https://github.com/apache/mesos/blob/19f14d06bac269b635657960d8ea8b2928b7830c/src/messages/messages.proto#L23]. (was: The messages we pass between Mesos components are largely undocumented. See this [TODO|https://github.com/apache/mesos/blob/master/src/messages/messages.proto#L23].) > Document messages.proto > --- > > Key: MESOS-3759 > URL: https://issues.apache.org/jira/browse/MESOS-3759 > Project: Mesos > Issue Type: Improvement > Components: documentation >Reporter: Joseph Wu >Assignee: Joseph Wu > Labels: documentation, mesosphere > > The messages we pass between Mesos components are largely undocumented. See > this > [TODO|https://github.com/apache/mesos/blob/19f14d06bac269b635657960d8ea8b2928b7830c/src/messages/messages.proto#L23]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-3762) Refactor SSLTest fixture such that MesosTest can use the same helpers.
Joseph Wu created MESOS-3762:
Summary: Refactor SSLTest fixture such that MesosTest can use the
same helpers.
Key: MESOS-3762
URL: https://issues.apache.org/jira/browse/MESOS-3762
Project: Mesos
Issue Type: Task
Components: test
Reporter: Joseph Wu
Assignee: Joseph Wu
In order to write tests that exercise SSL with other components of Mesos, such
as the HTTP scheduler library, we need to use the setup/teardown logic found in
the {{SSLTest}} fixture.
Currently, the test fixtures have separate inheritance structures like this:
{code}
SSLTest <- ::testing::Test
MesosTest <- TemporaryDirectoryTest <- ::testing::Test
{code}
where {{::testing::Test}} is a gtest class.
The plan is the following:
1) Change {{SSLTest}} to inherit from {{TemporaryDirectoryTest}}. This will
require moving the setup (generation of keys and certs) from {{SetUpTestCase}}
to {{SetUp}}. At the same time, *some* of the cleanup logic in the SSLTest
will not be needed.
2) Move the logic of generating keys/certs into helpers, so that individual
tests can call them when needed, much like {{MesosTest}}.
3) Have {{MesosTest}} inherit from {{SSLTest}} (which might be renamed during
the refactor). If Mesos is not compiled with {{--enable-ssl}}, then
{{SSLTest}} could be {{#ifdef}}'d into any empty class.
4) Write a child class of {{SSLTest}} which has the same functionality as the
existing {{SSLTest}}, for use by the existing tests that rely on {{SSLTest}} or
the {{RegistryClientTest}}.
The resulting structure should be like:
{code}
MesosTest <- SSLTest <- TemporaryDirectoryTest <- ::testing::Test
ChildOfSSLTest /
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-3753) Test the HTTP Scheduler library with SSL enabled
[ https://issues.apache.org/jira/browse/MESOS-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Wu updated MESOS-3753: - Sprint: (was: Mesosphere Sprint 21) > Test the HTTP Scheduler library with SSL enabled > > > Key: MESOS-3753 > URL: https://issues.apache.org/jira/browse/MESOS-3753 > Project: Mesos > Issue Type: Story > Components: framework, HTTP API, test >Reporter: Joseph Wu >Assignee: Joseph Wu > Labels: mesosphere, security > > Currently, the HTTP Scheduler library does not support SSL-enabled Mesos. > We need to add tests that check the schedule library against SSL-enabled > Mesos with SSL: > * with downgrade support, > * with/without verification of certificates (framework-side), > * with required framework/client-side certifications, > * with/without verification of certificates (master-side), > * with a custom certificate authority (CA), > These options should be controlled by the same environment variables found on > the [SSL user doc|http://mesos.apache.org/documentation/latest/ssl/]. > Note: This issue will be broken down into smaller sub-issues as bugs/problems > are discovered. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-3759) Document messages.proto
[ https://issues.apache.org/jira/browse/MESOS-3759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Wu updated MESOS-3759: - Labels: docathon documentation mesosphere (was: documentation mesosphere) > Document messages.proto > --- > > Key: MESOS-3759 > URL: https://issues.apache.org/jira/browse/MESOS-3759 > Project: Mesos > Issue Type: Improvement > Components: documentation >Reporter: Joseph Wu >Assignee: Joseph Wu > Labels: docathon, documentation, mesosphere > > The messages we pass between Mesos components are largely undocumented. See > this > [TODO|https://github.com/apache/mesos/blob/19f14d06bac269b635657960d8ea8b2928b7830c/src/messages/messages.proto#L23]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-3581) License headers show up all over doxygen documentation.
[
https://issues.apache.org/jira/browse/MESOS-3581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14965341#comment-14965341
]
Joseph Wu commented on MESOS-3581:
--
IMO, more importantly, we should actually update the Doxygen docs. They were
last updated 13 months ago. (See linked issues)
Also, we can easily get rid of the license headers by actually documenting the
classes.
For example, the [Watcher
class|http://mesos.apache.org/api/latest/c++/classWatcher.html] has proper
documentation *and* a license.
> License headers show up all over doxygen documentation.
> ---
>
> Key: MESOS-3581
> URL: https://issues.apache.org/jira/browse/MESOS-3581
> Project: Mesos
> Issue Type: Documentation
> Components: documentation
>Affects Versions: 0.24.1
>Reporter: Benjamin Bannier
>Assignee: Benjamin Bannier
>Priority: Minor
>
> Currently license headers are commented in something resembling Javadoc style,
> {code}
> /**
> * Licensed ...
> {code}
> Since we use Javadoc-style comment blocks for doxygen documentation all
> license headers appear in the generated documentation, potentially and likely
> hiding the actual documentation.
> Using {{/*}} to start the comment blocks would be enough to hide them from
> doxygen, but would likely also result in a largish (though mostly
> uninteresting) patch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MESOS-3773) RegistryClientTest.SimpleGetBlob is flaky
Joseph Wu created MESOS-3773:
Summary: RegistryClientTest.SimpleGetBlob is flaky
Key: MESOS-3773
URL: https://issues.apache.org/jira/browse/MESOS-3773
Project: Mesos
Issue Type: Bug
Components: test
Reporter: Joseph Wu
Assignee: Jojy Varghese
{{RegistryClientTest.SimpleGetBlob}} fails about 1/5 times. This was
encountered on OSX.
{code:title=Repro}
bin/mesos-tests.sh --gtest_filter="*RegistryClientTest.SimpleGetBlob*"
--gtest_repeat=10 --gtest_break_on_failure
{code}
{code:title=Example Failure}
[ RUN ] RegistryClientTest.SimpleGetBlob
../../src/tests/containerizer/provisioner_docker_tests.cpp:946: Failure
Value of: blobResponse
Actual: "2015-10-20 20:58:59.579393024+00:00"
Expected: blob.get()
Which is:
"\x15\x3\x3\00(P~\xCA&\xC6<\x4\x16\xE\xB2\xFF\b1a\xB9Z{\xE0\x80\xDA`\xBCt\x5R\x81x6\xF8
\x8B{\xA8\xA9\x4\xAB\xB6" "E\xE6\xDE\xCF\xD9*\xCC!\xC2\x15" "2015-10-20
20:58:59.579393024+00:00"
*** Aborted at 1445374739 (unix time) try "date -d @1445374739" if you are
using GNU date ***
PC: @0x103144ddc testing::UnitTest::AddTestPartResult()
*** SIGSEGV (@0x0) received by PID 49008 (TID 0x7fff73ca3300) stack trace: ***
@ 0x7fff8c58af1a _sigtramp
@ 0x7fff8386e187 malloc
@0x1031445b7 testing::internal::AssertHelper::operator=()
@0x1030d32e0
mesos::internal::tests::RegistryClientTest_SimpleGetBlob_Test::TestBody()
@0x1030d3562
mesos::internal::tests::RegistryClientTest_SimpleGetBlob_Test::TestBody()
@0x1031ac8f3
testing::internal::HandleSehExceptionsInMethodIfSupported<>()
@0x103192f87
testing::internal::HandleExceptionsInMethodIfSupported<>()
@0x1031533f5 testing::Test::Run()
@0x10315493b testing::TestInfo::Run()
@0x1031555f7 testing::TestCase::Run()
@0x103163df3 testing::internal::UnitTestImpl::RunAllTests()
@0x1031af8c3
testing::internal::HandleSehExceptionsInMethodIfSupported<>()
@0x103195397
testing::internal::HandleExceptionsInMethodIfSupported<>()
@0x1031639f2 testing::UnitTest::Run()
@0x1025abd41 RUN_ALL_TESTS()
@0x1025a8089 main
@ 0x7fff86b155c9 start
{code}
{code:title=Less common failure}
[ RUN ] RegistryClientTest.SimpleGetBlob
../../src/tests/containerizer/provisioner_docker_tests.cpp:926: Failure
(socket).failure(): Failed accept: connection error:
error::lib(0):func(0):reason(0)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3771) Mesos JSON API creates invalid JSON due to lack of binary data / non-ASCII handling
[
https://issues.apache.org/jira/browse/MESOS-3771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14965808#comment-14965808
]
Joseph Wu commented on MESOS-3771:
--
^ That's actually what would (sort of) fix your issue. There's an old TODO
[here|https://github.com/apache/mesos/blob/master/src/master/http.cpp#L118-L119]
to make the change.
We do actually encode {{bytes}} in base64, but only when they are transformed
into JSON from Protobuf. However, some of the endpoints (the ones which must
be backwards compatible) appear to treat {{bytes}} as ASCII strings.
If you have more control over your version of Spark, you could base64 encode
from Spark:
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/executor/MesosExecutorBackend.scala#L47
> Mesos JSON API creates invalid JSON due to lack of binary data / non-ASCII
> handling
> ---
>
> Key: MESOS-3771
> URL: https://issues.apache.org/jira/browse/MESOS-3771
> Project: Mesos
> Issue Type: Bug
> Components: HTTP API
>Affects Versions: 0.24.1
>Reporter: Steven Schlansker
>Priority: Critical
>
> Spark encodes some binary data into the ExecutorInfo.data field. This field
> is sent as a "bytes" Protobuf value, which can have arbitrary non-UTF8 data.
> If you have such a field, it seems that it is splatted out into JSON without
> any regards to proper character encoding:
> {code}
> 0006b0b0 2e 73 70 61 72 6b 2e 65 78 65 63 75 74 6f 72 2e |.spark.executor.|
> 0006b0c0 4d 65 73 6f 73 45 78 65 63 75 74 6f 72 42 61 63 |MesosExecutorBac|
> 0006b0d0 6b 65 6e 64 22 7d 2c 22 64 61 74 61 22 3a 22 ac |kend"},"data":".|
> 0006b0e0 ed 5c 75 30 30 30 30 5c 75 30 30 30 35 75 72 5c |.\u\u0005ur\|
> 0006b0f0 75 30 30 30 30 5c 75 30 30 30 66 5b 4c 73 63 61 |u\u000f[Lsca|
> 0006b100 6c 61 2e 54 75 70 6c 65 32 3b 2e cc 5c 75 30 30 |la.Tuple2;..\u00|
> {code}
> I suspect this is because the HTTP api emits the executorInfo.data directly:
> {code}
> JSON::Object model(const ExecutorInfo& executorInfo)
> {
> JSON::Object object;
> object.values["executor_id"] = executorInfo.executor_id().value();
> object.values["name"] = executorInfo.name();
> object.values["data"] = executorInfo.data();
> object.values["framework_id"] = executorInfo.framework_id().value();
> object.values["command"] = model(executorInfo.command());
> object.values["resources"] = model(executorInfo.resources());
> return object;
> }
> {code}
> I think this may be because the custom JSON processing library in stout seems
> to not have any idea of what a byte array is. I'm guessing that some
> implicit conversion makes it get written as a String instead, but:
> {code}
> inline std::ostream& operator<<(std::ostream& out, const String& string)
> {
> // TODO(benh): This escaping DOES NOT handle unicode, it encodes as ASCII.
> // See RFC4627 for the JSON string specificiation.
> return out << picojson::value(string.value).serialize();
> }
> {code}
> Thank you for any assistance here. Our cluster is currently entirely down --
> the frameworks cannot handle parsing the invalid JSON produced (it is not
> even valid utf-8)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-3762) Refactor SSLTest fixture such that MesosTest can use the same helpers.
[
https://issues.apache.org/jira/browse/MESOS-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Wu updated MESOS-3762:
-
Description:
In order to write tests that exercise SSL with other components of Mesos, such
as the HTTP scheduler library, we need to use the setup/teardown logic found in
the {{SSLTest}} fixture.
Currently, the test fixtures have separate inheritance structures like this:
{code}
SSLTest <- ::testing::Test
MesosTest <- TemporaryDirectoryTest <- ::testing::Test
{code}
where {{::testing::Test}} is a gtest class.
The plan is the following:
# Change {{SSLTest}} to inherit from {{TemporaryDirectoryTest}}. This will
require moving the setup (generation of keys and certs) from {{SetUpTestCase}}
to {{SetUp}}. At the same time, *some* of the cleanup logic in the SSLTest
will not be needed.
# Move the logic of generating keys/certs into helpers, so that individual
tests can call them when needed, much like {{MesosTest}}.
# Write a child class of {{SSLTest}} which has the same functionality as the
existing {{SSLTest}}, for use by the existing tests that rely on {{SSLTest}} or
the {{RegistryClientTest}}.
# Have {{MesosTest}} inherit from {{SSLTest}} (which might be renamed during
the refactor). If Mesos is not compiled with {{--enable-ssl}}, then
{{SSLTest}} could be {{#ifdef}}'d into any empty class.
The resulting structure should be like:
{code}
MesosTest <- SSLTest <- TemporaryDirectoryTest <- ::testing::Test
ChildOfSSLTest /
{code}
was:
In order to write tests that exercise SSL with other components of Mesos, such
as the HTTP scheduler library, we need to use the setup/teardown logic found in
the {{SSLTest}} fixture.
Currently, the test fixtures have separate inheritance structures like this:
{code}
SSLTest <- ::testing::Test
MesosTest <- TemporaryDirectoryTest <- ::testing::Test
{code}
where {{::testing::Test}} is a gtest class.
The plan is the following:
1) Change {{SSLTest}} to inherit from {{TemporaryDirectoryTest}}. This will
require moving the setup (generation of keys and certs) from {{SetUpTestCase}}
to {{SetUp}}. At the same time, *some* of the cleanup logic in the SSLTest
will not be needed.
2) Move the logic of generating keys/certs into helpers, so that individual
tests can call them when needed, much like {{MesosTest}}.
3) Have {{MesosTest}} inherit from {{SSLTest}} (which might be renamed during
the refactor). If Mesos is not compiled with {{--enable-ssl}}, then
{{SSLTest}} could be {{#ifdef}}'d into any empty class.
4) Write a child class of {{SSLTest}} which has the same functionality as the
existing {{SSLTest}}, for use by the existing tests that rely on {{SSLTest}} or
the {{RegistryClientTest}}.
The resulting structure should be like:
{code}
MesosTest <- SSLTest <- TemporaryDirectoryTest <- ::testing::Test
ChildOfSSLTest /
{code}
> Refactor SSLTest fixture such that MesosTest can use the same helpers.
> --
>
> Key: MESOS-3762
> URL: https://issues.apache.org/jira/browse/MESOS-3762
> Project: Mesos
> Issue Type: Task
> Components: test
>Reporter: Joseph Wu
>Assignee: Joseph Wu
> Labels: mesosphere
>
> In order to write tests that exercise SSL with other components of Mesos,
> such as the HTTP scheduler library, we need to use the setup/teardown logic
> found in the {{SSLTest}} fixture.
> Currently, the test fixtures have separate inheritance structures like this:
> {code}
> SSLTest <- ::testing::Test
> MesosTest <- TemporaryDirectoryTest <- ::testing::Test
> {code}
> where {{::testing::Test}} is a gtest class.
> The plan is the following:
> # Change {{SSLTest}} to inherit from {{TemporaryDirectoryTest}}. This will
> require moving the setup (generation of keys and certs) from
> {{SetUpTestCase}} to {{SetUp}}. At the same time, *some* of the cleanup
> logic in the SSLTest will not be needed.
> # Move the logic of generating keys/certs into helpers, so that individual
> tests can call them when needed, much like {{MesosTest}}.
> # Write a child class of {{SSLTest}} which has the same functionality as the
> existing {{SSLTest}}, for use by the existing tests that rely on {{SSLTest}}
> or the {{RegistryClientTest}}.
> # Have {{MesosTest}} inherit from {{SSLTest}} (which might be renamed during
> the refactor). If Mesos is not compiled with {{--enable-ssl}}, then
> {{SSLTest}} could be {{#ifdef}}'d into any empty class.
> The resulting structure should be like:
> {code}
> MesosTest <- SSLTest <- TemporaryDirectoryTest <- ::testing::Test
> ChildOfSSLTest /
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3762) Refactor SSLTest fixture such that MesosTest can use the same helpers.
[
https://issues.apache.org/jira/browse/MESOS-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14965877#comment-14965877
]
Joseph Wu commented on MESOS-3762:
--
Found and wrote a fix for an SSL-related test cleanup bug:
https://reviews.apache.org/r/39495/
> Refactor SSLTest fixture such that MesosTest can use the same helpers.
> --
>
> Key: MESOS-3762
> URL: https://issues.apache.org/jira/browse/MESOS-3762
> Project: Mesos
> Issue Type: Task
> Components: test
>Reporter: Joseph Wu
>Assignee: Joseph Wu
> Labels: mesosphere
>
> In order to write tests that exercise SSL with other components of Mesos,
> such as the HTTP scheduler library, we need to use the setup/teardown logic
> found in the {{SSLTest}} fixture.
> Currently, the test fixtures have separate inheritance structures like this:
> {code}
> SSLTest <- ::testing::Test
> MesosTest <- TemporaryDirectoryTest <- ::testing::Test
> {code}
> where {{::testing::Test}} is a gtest class.
> The plan is the following:
> # Change {{SSLTest}} to inherit from {{TemporaryDirectoryTest}}. This will
> require moving the setup (generation of keys and certs) from
> {{SetUpTestCase}} to {{SetUp}}. At the same time, *some* of the cleanup
> logic in the SSLTest will not be needed.
> # Move the logic of generating keys/certs into helpers, so that individual
> tests can call them when needed, much like {{MesosTest}}.
> # Write a child class of {{SSLTest}} which has the same functionality as the
> existing {{SSLTest}}, for use by the existing tests that rely on {{SSLTest}}
> or the {{RegistryClientTest}}.
> # Have {{MesosTest}} inherit from {{SSLTest}} (which might be renamed during
> the refactor). If Mesos is not compiled with {{--enable-ssl}}, then
> {{SSLTest}} could be {{#ifdef}}'d into any empty class.
> The resulting structure should be like:
> {code}
> MesosTest <- SSLTest <- TemporaryDirectoryTest <- ::testing::Test
> ChildOfSSLTest /
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3762) Refactor SSLTest fixture such that MesosTest can use the same helpers.
[
https://issues.apache.org/jira/browse/MESOS-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14965883#comment-14965883
]
Joseph Wu commented on MESOS-3762:
--
Reviews for step 1)
https://reviews.apache.org/r/39498/
https://reviews.apache.org/r/39499/
> Refactor SSLTest fixture such that MesosTest can use the same helpers.
> --
>
> Key: MESOS-3762
> URL: https://issues.apache.org/jira/browse/MESOS-3762
> Project: Mesos
> Issue Type: Task
> Components: test
>Reporter: Joseph Wu
>Assignee: Joseph Wu
> Labels: mesosphere
>
> In order to write tests that exercise SSL with other components of Mesos,
> such as the HTTP scheduler library, we need to use the setup/teardown logic
> found in the {{SSLTest}} fixture.
> Currently, the test fixtures have separate inheritance structures like this:
> {code}
> SSLTest <- ::testing::Test
> MesosTest <- TemporaryDirectoryTest <- ::testing::Test
> {code}
> where {{::testing::Test}} is a gtest class.
> The plan is the following:
> # Change {{SSLTest}} to inherit from {{TemporaryDirectoryTest}}. This will
> require moving the setup (generation of keys and certs) from
> {{SetUpTestCase}} to {{SetUp}}. At the same time, *some* of the cleanup
> logic in the SSLTest will not be needed.
> # Move the logic of generating keys/certs into helpers, so that individual
> tests can call them when needed, much like {{MesosTest}}.
> # Write a child class of {{SSLTest}} which has the same functionality as the
> existing {{SSLTest}}, for use by the existing tests that rely on {{SSLTest}}
> or the {{RegistryClientTest}}.
> # Have {{MesosTest}} inherit from {{SSLTest}} (which might be renamed during
> the refactor). If Mesos is not compiled with {{--enable-ssl}}, then
> {{SSLTest}} could be {{#ifdef}}'d into any empty class.
> The resulting structure should be like:
> {code}
> MesosTest <- SSLTest <- TemporaryDirectoryTest <- ::testing::Test
> ChildOfSSLTest /
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (MESOS-3762) Refactor SSLTest fixture such that MesosTest can use the same helpers.
[
https://issues.apache.org/jira/browse/MESOS-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14965883#comment-14965883
]
Joseph Wu edited comment on MESOS-3762 at 10/20/15 11:30 PM:
-
Reviews for:
Step 1)
https://reviews.apache.org/r/39498/
https://reviews.apache.org/r/39499/
Step 2 & 3)
https://reviews.apache.org/r/39501/
was (Author: kaysoky):
Reviews for step 1)
https://reviews.apache.org/r/39498/
https://reviews.apache.org/r/39499/
> Refactor SSLTest fixture such that MesosTest can use the same helpers.
> --
>
> Key: MESOS-3762
> URL: https://issues.apache.org/jira/browse/MESOS-3762
> Project: Mesos
> Issue Type: Task
> Components: test
>Reporter: Joseph Wu
>Assignee: Joseph Wu
> Labels: mesosphere
>
> In order to write tests that exercise SSL with other components of Mesos,
> such as the HTTP scheduler library, we need to use the setup/teardown logic
> found in the {{SSLTest}} fixture.
> Currently, the test fixtures have separate inheritance structures like this:
> {code}
> SSLTest <- ::testing::Test
> MesosTest <- TemporaryDirectoryTest <- ::testing::Test
> {code}
> where {{::testing::Test}} is a gtest class.
> The plan is the following:
> # Change {{SSLTest}} to inherit from {{TemporaryDirectoryTest}}. This will
> require moving the setup (generation of keys and certs) from
> {{SetUpTestCase}} to {{SetUp}}. At the same time, *some* of the cleanup
> logic in the SSLTest will not be needed.
> # Move the logic of generating keys/certs into helpers, so that individual
> tests can call them when needed, much like {{MesosTest}}.
> # Write a child class of {{SSLTest}} which has the same functionality as the
> existing {{SSLTest}}, for use by the existing tests that rely on {{SSLTest}}
> or the {{RegistryClientTest}}.
> # Have {{MesosTest}} inherit from {{SSLTest}} (which might be renamed during
> the refactor). If Mesos is not compiled with {{--enable-ssl}}, then
> {{SSLTest}} could be {{#ifdef}}'d into any empty class.
> The resulting structure should be like:
> {code}
> MesosTest <- SSLTest <- TemporaryDirectoryTest <- ::testing::Test
> ChildOfSSLTest /
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3771) Mesos JSON API creates invalid JSON due to lack of binary data / non-ASCII handling
[
https://issues.apache.org/jira/browse/MESOS-3771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14966036#comment-14966036
]
Joseph Wu commented on MESOS-3771:
--
Actually, I can't repro this behavior in a unit test. ([Attempted
here|https://github.com/kaysoky/mesos/commit/d8869f0aa1fdcf38072b45a6238b191c67b7e0f7]).
I've constructed an {{ExecutorInfo}} with a {{data}} field holding the same
data you have above. Fetching the same {{ExecutorInfo}} from the {{/state}}
endpoint also gives valid JSON.
Am I doing something differently?
> Mesos JSON API creates invalid JSON due to lack of binary data / non-ASCII
> handling
> ---
>
> Key: MESOS-3771
> URL: https://issues.apache.org/jira/browse/MESOS-3771
> Project: Mesos
> Issue Type: Bug
> Components: HTTP API
>Affects Versions: 0.24.1
>Reporter: Steven Schlansker
>Priority: Critical
>
> Spark encodes some binary data into the ExecutorInfo.data field. This field
> is sent as a "bytes" Protobuf value, which can have arbitrary non-UTF8 data.
> If you have such a field, it seems that it is splatted out into JSON without
> any regards to proper character encoding:
> {code}
> 0006b0b0 2e 73 70 61 72 6b 2e 65 78 65 63 75 74 6f 72 2e |.spark.executor.|
> 0006b0c0 4d 65 73 6f 73 45 78 65 63 75 74 6f 72 42 61 63 |MesosExecutorBac|
> 0006b0d0 6b 65 6e 64 22 7d 2c 22 64 61 74 61 22 3a 22 ac |kend"},"data":".|
> 0006b0e0 ed 5c 75 30 30 30 30 5c 75 30 30 30 35 75 72 5c |.\u\u0005ur\|
> 0006b0f0 75 30 30 30 30 5c 75 30 30 30 66 5b 4c 73 63 61 |u\u000f[Lsca|
> 0006b100 6c 61 2e 54 75 70 6c 65 32 3b 2e cc 5c 75 30 30 |la.Tuple2;..\u00|
> {code}
> I suspect this is because the HTTP api emits the executorInfo.data directly:
> {code}
> JSON::Object model(const ExecutorInfo& executorInfo)
> {
> JSON::Object object;
> object.values["executor_id"] = executorInfo.executor_id().value();
> object.values["name"] = executorInfo.name();
> object.values["data"] = executorInfo.data();
> object.values["framework_id"] = executorInfo.framework_id().value();
> object.values["command"] = model(executorInfo.command());
> object.values["resources"] = model(executorInfo.resources());
> return object;
> }
> {code}
> I think this may be because the custom JSON processing library in stout seems
> to not have any idea of what a byte array is. I'm guessing that some
> implicit conversion makes it get written as a String instead, but:
> {code}
> inline std::ostream& operator<<(std::ostream& out, const String& string)
> {
> // TODO(benh): This escaping DOES NOT handle unicode, it encodes as ASCII.
> // See RFC4627 for the JSON string specificiation.
> return out << picojson::value(string.value).serialize();
> }
> {code}
> Thank you for any assistance here. Our cluster is currently entirely down --
> the frameworks cannot handle parsing the invalid JSON produced (it is not
> even valid utf-8)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-1607) Introduce optimistic offers.
[ https://issues.apache.org/jira/browse/MESOS-1607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14967410#comment-14967410 ] Joseph Wu commented on MESOS-1607: -- We plan to release the MVP before the end of this year, so tentatively sometime between v0.26.0 and v0.28.0. > Introduce optimistic offers. > > > Key: MESOS-1607 > URL: https://issues.apache.org/jira/browse/MESOS-1607 > Project: Mesos > Issue Type: Epic > Components: allocation, framework, master >Reporter: Benjamin Hindman >Assignee: Artem Harutyunyan > Labels: mesosphere > Attachments: optimisitic-offers.pdf > > > *Background* > The current implementation of resource offers only enable a single framework > scheduler to make scheduling decisions for some available resources at a > time. In some circumstances, this is good, i.e., when we don't want other > framework schedulers to have access to some resources. However, in other > circumstances, there are advantages to letting multiple framework schedulers > attempt to make scheduling decisions for the _same_ allocation of resources > in parallel. > If you think about this from a "concurrency control" perspective, the current > implementation of resource offers is _pessimistic_, the resources contained > within an offer are _locked_ until the framework scheduler that they were > offered to launches tasks with them or declines them. In addition to making > pessimistic offers we'd like to give out _optimistic_ offers, where the same > resources are offered to multiple framework schedulers at the same time, and > framework schedulers "compete" for those resources on a > first-come-first-serve basis (i.e., the first to launch a task "wins"). We've > always reserved the right to rescind resource offers using the 'rescind' > primitive in the API, and a framework scheduler should be prepared to launch > a task and have those tasks go lost because another framework already started > to use those resources. > *Feature* > We plan to take a step towards optimistic offers, by introducing primitives > that allow resources to be offered to multiple frameworks at once. At first, > we will use these primitives to optimistically allocate resources that are > reserved for a particular framework/role but have not been allocated by that > framework/role. > The work with optimistic offers will closely resemble the existing > oversubscription feature. Optimistically offered resources are likely to be > considered "revocable resources" (the concept that using resources not > reserved for you means you might get those resources revoked). In effect, we > can may create something like a "spot" market for unused resources, driving > up utilization by letting frameworks that are willing to use revocable > resources run tasks. > *Future Work* > This ticket tracks the introduction of some aspects of optimistic offers. > Taken to the limit, one could imagine always making optimistic resource > offers. This bears a striking resemblance with the Google Omega model (an > isomorphism even). However, being able to configure what resources should be > allocated optimistically and what resources should be allocated > pessimistically gives even more control to a datacenter/cluster operator that > might want to, for example, never let multiple frameworks (roles) compete for > some set of resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-1478) Replace Master/Slave terminology
[ https://issues.apache.org/jira/browse/MESOS-1478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Wu updated MESOS-1478: - Issue Type: Epic (was: Wish) > Replace Master/Slave terminology > > > Key: MESOS-1478 > URL: https://issues.apache.org/jira/browse/MESOS-1478 > Project: Mesos > Issue Type: Epic >Reporter: Clark Breyman >Assignee: Benjamin Hindman >Priority: Minor > Labels: mesosphere > > Inspired by the comments on this PR: > https://github.com/django/django/pull/2692 > TL;DR - Computers sharing work should be a good thing. Using the language of > human bondage and suffering is inappropriate in this context. It also has the > potential to alienate users and community members. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-1478) Replace Master/Slave terminology
[ https://issues.apache.org/jira/browse/MESOS-1478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Wu updated MESOS-1478: - Epic Name: Agent Rename > Replace Master/Slave terminology > > > Key: MESOS-1478 > URL: https://issues.apache.org/jira/browse/MESOS-1478 > Project: Mesos > Issue Type: Epic >Reporter: Clark Breyman >Assignee: Benjamin Hindman >Priority: Minor > Labels: mesosphere > > Inspired by the comments on this PR: > https://github.com/django/django/pull/2692 > TL;DR - Computers sharing work should be a good thing. Using the language of > human bondage and suffering is inappropriate in this context. It also has the > potential to alienate users and community members. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (MESOS-3762) Refactor SSLTest fixture such that MesosTest can use the same helpers.
[
https://issues.apache.org/jira/browse/MESOS-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14965883#comment-14965883
]
Joseph Wu edited comment on MESOS-3762 at 10/22/15 12:06 AM:
-
Reviews for:
Step 1)
https://reviews.apache.org/r/39498/
https://reviews.apache.org/r/39499/
Step 2 & 3)
https://reviews.apache.org/r/39501/
Step 4)
https://reviews.apache.org/r/39533/
https://reviews.apache.org/r/39534/
was (Author: kaysoky):
Reviews for:
Step 1)
https://reviews.apache.org/r/39498/
https://reviews.apache.org/r/39499/
Step 2 & 3)
https://reviews.apache.org/r/39501/
> Refactor SSLTest fixture such that MesosTest can use the same helpers.
> --
>
> Key: MESOS-3762
> URL: https://issues.apache.org/jira/browse/MESOS-3762
> Project: Mesos
> Issue Type: Task
> Components: test
>Reporter: Joseph Wu
>Assignee: Joseph Wu
> Labels: mesosphere
>
> In order to write tests that exercise SSL with other components of Mesos,
> such as the HTTP scheduler library, we need to use the setup/teardown logic
> found in the {{SSLTest}} fixture.
> Currently, the test fixtures have separate inheritance structures like this:
> {code}
> SSLTest <- ::testing::Test
> MesosTest <- TemporaryDirectoryTest <- ::testing::Test
> {code}
> where {{::testing::Test}} is a gtest class.
> The plan is the following:
> # Change {{SSLTest}} to inherit from {{TemporaryDirectoryTest}}. This will
> require moving the setup (generation of keys and certs) from
> {{SetUpTestCase}} to {{SetUp}}. At the same time, *some* of the cleanup
> logic in the SSLTest will not be needed.
> # Move the logic of generating keys/certs into helpers, so that individual
> tests can call them when needed, much like {{MesosTest}}.
> # Write a child class of {{SSLTest}} which has the same functionality as the
> existing {{SSLTest}}, for use by the existing tests that rely on {{SSLTest}}
> or the {{RegistryClientTest}}.
> # Have {{MesosTest}} inherit from {{SSLTest}} (which might be renamed during
> the refactor). If Mesos is not compiled with {{--enable-ssl}}, then
> {{SSLTest}} could be {{#ifdef}}'d into any empty class.
> The resulting structure should be like:
> {code}
> MesosTest <- SSLTest <- TemporaryDirectoryTest <- ::testing::Test
> ChildOfSSLTest /
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3771) Mesos JSON API creates invalid JSON due to lack of binary data / non-ASCII handling
[
https://issues.apache.org/jira/browse/MESOS-3771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14968275#comment-14968275
]
Joseph Wu commented on MESOS-3771:
--
Looks like our JSON library will never catch this (it's more permissive), which
is why none of our unit tests have caught this.
I agree that this is a problem though. I'll see if I can get more eyes on this.
> Mesos JSON API creates invalid JSON due to lack of binary data / non-ASCII
> handling
> ---
>
> Key: MESOS-3771
> URL: https://issues.apache.org/jira/browse/MESOS-3771
> Project: Mesos
> Issue Type: Bug
> Components: HTTP API
>Affects Versions: 0.24.1, 0.26.0
>Reporter: Steven Schlansker
>Priority: Critical
>
> Spark encodes some binary data into the ExecutorInfo.data field. This field
> is sent as a "bytes" Protobuf value, which can have arbitrary non-UTF8 data.
> If you have such a field, it seems that it is splatted out into JSON without
> any regards to proper character encoding:
> {code}
> 0006b0b0 2e 73 70 61 72 6b 2e 65 78 65 63 75 74 6f 72 2e |.spark.executor.|
> 0006b0c0 4d 65 73 6f 73 45 78 65 63 75 74 6f 72 42 61 63 |MesosExecutorBac|
> 0006b0d0 6b 65 6e 64 22 7d 2c 22 64 61 74 61 22 3a 22 ac |kend"},"data":".|
> 0006b0e0 ed 5c 75 30 30 30 30 5c 75 30 30 30 35 75 72 5c |.\u\u0005ur\|
> 0006b0f0 75 30 30 30 30 5c 75 30 30 30 66 5b 4c 73 63 61 |u\u000f[Lsca|
> 0006b100 6c 61 2e 54 75 70 6c 65 32 3b 2e cc 5c 75 30 30 |la.Tuple2;..\u00|
> {code}
> I suspect this is because the HTTP api emits the executorInfo.data directly:
> {code}
> JSON::Object model(const ExecutorInfo& executorInfo)
> {
> JSON::Object object;
> object.values["executor_id"] = executorInfo.executor_id().value();
> object.values["name"] = executorInfo.name();
> object.values["data"] = executorInfo.data();
> object.values["framework_id"] = executorInfo.framework_id().value();
> object.values["command"] = model(executorInfo.command());
> object.values["resources"] = model(executorInfo.resources());
> return object;
> }
> {code}
> I think this may be because the custom JSON processing library in stout seems
> to not have any idea of what a byte array is. I'm guessing that some
> implicit conversion makes it get written as a String instead, but:
> {code}
> inline std::ostream& operator<<(std::ostream& out, const String& string)
> {
> // TODO(benh): This escaping DOES NOT handle unicode, it encodes as ASCII.
> // See RFC4627 for the JSON string specificiation.
> return out << picojson::value(string.value).serialize();
> }
> {code}
> Thank you for any assistance here. Our cluster is currently entirely down --
> the frameworks cannot handle parsing the invalid JSON produced (it is not
> even valid utf-8)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-3771) Mesos JSON API creates invalid JSON due to lack of binary data / non-ASCII handling
[
https://issues.apache.org/jira/browse/MESOS-3771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Wu updated MESOS-3771:
-
Sprint: Mesosphere Sprint 21
Story Points: 2
> Mesos JSON API creates invalid JSON due to lack of binary data / non-ASCII
> handling
> ---
>
> Key: MESOS-3771
> URL: https://issues.apache.org/jira/browse/MESOS-3771
> Project: Mesos
> Issue Type: Bug
> Components: HTTP API
>Affects Versions: 0.24.1, 0.26.0
>Reporter: Steven Schlansker
>Assignee: Joseph Wu
>Priority: Critical
> Labels: mesosphere
>
> Spark encodes some binary data into the ExecutorInfo.data field. This field
> is sent as a "bytes" Protobuf value, which can have arbitrary non-UTF8 data.
> If you have such a field, it seems that it is splatted out into JSON without
> any regards to proper character encoding:
> {code}
> 0006b0b0 2e 73 70 61 72 6b 2e 65 78 65 63 75 74 6f 72 2e |.spark.executor.|
> 0006b0c0 4d 65 73 6f 73 45 78 65 63 75 74 6f 72 42 61 63 |MesosExecutorBac|
> 0006b0d0 6b 65 6e 64 22 7d 2c 22 64 61 74 61 22 3a 22 ac |kend"},"data":".|
> 0006b0e0 ed 5c 75 30 30 30 30 5c 75 30 30 30 35 75 72 5c |.\u\u0005ur\|
> 0006b0f0 75 30 30 30 30 5c 75 30 30 30 66 5b 4c 73 63 61 |u\u000f[Lsca|
> 0006b100 6c 61 2e 54 75 70 6c 65 32 3b 2e cc 5c 75 30 30 |la.Tuple2;..\u00|
> {code}
> I suspect this is because the HTTP api emits the executorInfo.data directly:
> {code}
> JSON::Object model(const ExecutorInfo& executorInfo)
> {
> JSON::Object object;
> object.values["executor_id"] = executorInfo.executor_id().value();
> object.values["name"] = executorInfo.name();
> object.values["data"] = executorInfo.data();
> object.values["framework_id"] = executorInfo.framework_id().value();
> object.values["command"] = model(executorInfo.command());
> object.values["resources"] = model(executorInfo.resources());
> return object;
> }
> {code}
> I think this may be because the custom JSON processing library in stout seems
> to not have any idea of what a byte array is. I'm guessing that some
> implicit conversion makes it get written as a String instead, but:
> {code}
> inline std::ostream& operator<<(std::ostream& out, const String& string)
> {
> // TODO(benh): This escaping DOES NOT handle unicode, it encodes as ASCII.
> // See RFC4627 for the JSON string specificiation.
> return out << picojson::value(string.value).serialize();
> }
> {code}
> Thank you for any assistance here. Our cluster is currently entirely down --
> the frameworks cannot handle parsing the invalid JSON produced (it is not
> even valid utf-8)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (MESOS-3771) Mesos JSON API creates invalid JSON due to lack of binary data / non-ASCII handling
[
https://issues.apache.org/jira/browse/MESOS-3771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Wu reassigned MESOS-3771:
Assignee: Joseph Wu
> Mesos JSON API creates invalid JSON due to lack of binary data / non-ASCII
> handling
> ---
>
> Key: MESOS-3771
> URL: https://issues.apache.org/jira/browse/MESOS-3771
> Project: Mesos
> Issue Type: Bug
> Components: HTTP API
>Affects Versions: 0.24.1, 0.26.0
>Reporter: Steven Schlansker
>Assignee: Joseph Wu
>Priority: Critical
> Labels: mesosphere
>
> Spark encodes some binary data into the ExecutorInfo.data field. This field
> is sent as a "bytes" Protobuf value, which can have arbitrary non-UTF8 data.
> If you have such a field, it seems that it is splatted out into JSON without
> any regards to proper character encoding:
> {code}
> 0006b0b0 2e 73 70 61 72 6b 2e 65 78 65 63 75 74 6f 72 2e |.spark.executor.|
> 0006b0c0 4d 65 73 6f 73 45 78 65 63 75 74 6f 72 42 61 63 |MesosExecutorBac|
> 0006b0d0 6b 65 6e 64 22 7d 2c 22 64 61 74 61 22 3a 22 ac |kend"},"data":".|
> 0006b0e0 ed 5c 75 30 30 30 30 5c 75 30 30 30 35 75 72 5c |.\u\u0005ur\|
> 0006b0f0 75 30 30 30 30 5c 75 30 30 30 66 5b 4c 73 63 61 |u\u000f[Lsca|
> 0006b100 6c 61 2e 54 75 70 6c 65 32 3b 2e cc 5c 75 30 30 |la.Tuple2;..\u00|
> {code}
> I suspect this is because the HTTP api emits the executorInfo.data directly:
> {code}
> JSON::Object model(const ExecutorInfo& executorInfo)
> {
> JSON::Object object;
> object.values["executor_id"] = executorInfo.executor_id().value();
> object.values["name"] = executorInfo.name();
> object.values["data"] = executorInfo.data();
> object.values["framework_id"] = executorInfo.framework_id().value();
> object.values["command"] = model(executorInfo.command());
> object.values["resources"] = model(executorInfo.resources());
> return object;
> }
> {code}
> I think this may be because the custom JSON processing library in stout seems
> to not have any idea of what a byte array is. I'm guessing that some
> implicit conversion makes it get written as a String instead, but:
> {code}
> inline std::ostream& operator<<(std::ostream& out, const String& string)
> {
> // TODO(benh): This escaping DOES NOT handle unicode, it encodes as ASCII.
> // See RFC4627 for the JSON string specificiation.
> return out << picojson::value(string.value).serialize();
> }
> {code}
> Thank you for any assistance here. Our cluster is currently entirely down --
> the frameworks cannot handle parsing the invalid JSON produced (it is not
> even valid utf-8)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-3771) Mesos JSON API creates invalid JSON due to lack of binary data / non-ASCII handling
[
https://issues.apache.org/jira/browse/MESOS-3771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Wu updated MESOS-3771:
-
Labels: mesosphere (was: )
> Mesos JSON API creates invalid JSON due to lack of binary data / non-ASCII
> handling
> ---
>
> Key: MESOS-3771
> URL: https://issues.apache.org/jira/browse/MESOS-3771
> Project: Mesos
> Issue Type: Bug
> Components: HTTP API
>Affects Versions: 0.24.1, 0.26.0
>Reporter: Steven Schlansker
>Assignee: Joseph Wu
>Priority: Critical
> Labels: mesosphere
>
> Spark encodes some binary data into the ExecutorInfo.data field. This field
> is sent as a "bytes" Protobuf value, which can have arbitrary non-UTF8 data.
> If you have such a field, it seems that it is splatted out into JSON without
> any regards to proper character encoding:
> {code}
> 0006b0b0 2e 73 70 61 72 6b 2e 65 78 65 63 75 74 6f 72 2e |.spark.executor.|
> 0006b0c0 4d 65 73 6f 73 45 78 65 63 75 74 6f 72 42 61 63 |MesosExecutorBac|
> 0006b0d0 6b 65 6e 64 22 7d 2c 22 64 61 74 61 22 3a 22 ac |kend"},"data":".|
> 0006b0e0 ed 5c 75 30 30 30 30 5c 75 30 30 30 35 75 72 5c |.\u\u0005ur\|
> 0006b0f0 75 30 30 30 30 5c 75 30 30 30 66 5b 4c 73 63 61 |u\u000f[Lsca|
> 0006b100 6c 61 2e 54 75 70 6c 65 32 3b 2e cc 5c 75 30 30 |la.Tuple2;..\u00|
> {code}
> I suspect this is because the HTTP api emits the executorInfo.data directly:
> {code}
> JSON::Object model(const ExecutorInfo& executorInfo)
> {
> JSON::Object object;
> object.values["executor_id"] = executorInfo.executor_id().value();
> object.values["name"] = executorInfo.name();
> object.values["data"] = executorInfo.data();
> object.values["framework_id"] = executorInfo.framework_id().value();
> object.values["command"] = model(executorInfo.command());
> object.values["resources"] = model(executorInfo.resources());
> return object;
> }
> {code}
> I think this may be because the custom JSON processing library in stout seems
> to not have any idea of what a byte array is. I'm guessing that some
> implicit conversion makes it get written as a String instead, but:
> {code}
> inline std::ostream& operator<<(std::ostream& out, const String& string)
> {
> // TODO(benh): This escaping DOES NOT handle unicode, it encodes as ASCII.
> // See RFC4627 for the JSON string specificiation.
> return out << picojson::value(string.value).serialize();
> }
> {code}
> Thank you for any assistance here. Our cluster is currently entirely down --
> the frameworks cannot handle parsing the invalid JSON produced (it is not
> even valid utf-8)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3771) Mesos JSON API creates invalid JSON due to lack of binary data / non-ASCII handling
[
https://issues.apache.org/jira/browse/MESOS-3771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970063#comment-14970063
]
Joseph Wu commented on MESOS-3771:
--
Sync'd with BenH and [~bmahler] about this (offline).
The proposed solution is the following:
# None of the state endpoints should be dumping out the binary {{data}} fields
in the first place. This includes {{ExecutorInfo}} ([dumped by
master|https://github.com/apache/mesos/blob/master/src/common/http.cpp#L317])
and {{TaskInfo}} ([dumped by
agent|https://github.com/apache/mesos/blob/master/src/slave/http.cpp#L103]).
#* These fields should be removed from the output entirely. Existing
frameworks should not be relying on this information. [~stevenschlansker], can
you confirm this with Spark?
#* We can easily back-port this patch, if absolutely necessary.
# Master should not be storing the {{data}} fields from {{ExecutorInfo}}. We
currently [store the entire
object|https://github.com/apache/mesos/blob/master/src/master/master.hpp#L262-L271],
which means master would be as high risk of OOM-ing if a bunch of executors
were started with big {{data}} blobs.
#* Master should scrub out unneeded bloat from {{ExecutorInfo}} before storing
it.
#* We can use an alternate internal object, like we do for {{TaskInfo}} vs
{{Task}}; see
[this|https://github.com/apache/mesos/blob/master/src/messages/messages.proto#L39-L41].
> Mesos JSON API creates invalid JSON due to lack of binary data / non-ASCII
> handling
> ---
>
> Key: MESOS-3771
> URL: https://issues.apache.org/jira/browse/MESOS-3771
> Project: Mesos
> Issue Type: Bug
> Components: HTTP API
>Affects Versions: 0.24.1, 0.26.0
>Reporter: Steven Schlansker
>Assignee: Joseph Wu
>Priority: Critical
> Labels: mesosphere
>
> Spark encodes some binary data into the ExecutorInfo.data field. This field
> is sent as a "bytes" Protobuf value, which can have arbitrary non-UTF8 data.
> If you have such a field, it seems that it is splatted out into JSON without
> any regards to proper character encoding:
> {code}
> 0006b0b0 2e 73 70 61 72 6b 2e 65 78 65 63 75 74 6f 72 2e |.spark.executor.|
> 0006b0c0 4d 65 73 6f 73 45 78 65 63 75 74 6f 72 42 61 63 |MesosExecutorBac|
> 0006b0d0 6b 65 6e 64 22 7d 2c 22 64 61 74 61 22 3a 22 ac |kend"},"data":".|
> 0006b0e0 ed 5c 75 30 30 30 30 5c 75 30 30 30 35 75 72 5c |.\u\u0005ur\|
> 0006b0f0 75 30 30 30 30 5c 75 30 30 30 66 5b 4c 73 63 61 |u\u000f[Lsca|
> 0006b100 6c 61 2e 54 75 70 6c 65 32 3b 2e cc 5c 75 30 30 |la.Tuple2;..\u00|
> {code}
> I suspect this is because the HTTP api emits the executorInfo.data directly:
> {code}
> JSON::Object model(const ExecutorInfo& executorInfo)
> {
> JSON::Object object;
> object.values["executor_id"] = executorInfo.executor_id().value();
> object.values["name"] = executorInfo.name();
> object.values["data"] = executorInfo.data();
> object.values["framework_id"] = executorInfo.framework_id().value();
> object.values["command"] = model(executorInfo.command());
> object.values["resources"] = model(executorInfo.resources());
> return object;
> }
> {code}
> I think this may be because the custom JSON processing library in stout seems
> to not have any idea of what a byte array is. I'm guessing that some
> implicit conversion makes it get written as a String instead, but:
> {code}
> inline std::ostream& operator<<(std::ostream& out, const String& string)
> {
> // TODO(benh): This escaping DOES NOT handle unicode, it encodes as ASCII.
> // See RFC4627 for the JSON string specificiation.
> return out << picojson::value(string.value).serialize();
> }
> {code}
> Thank you for any assistance here. Our cluster is currently entirely down --
> the frameworks cannot handle parsing the invalid JSON produced (it is not
> even valid utf-8)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MESOS-3794) Master should not store arbitrarily sized data in ExecutorInfo
Joseph Wu created MESOS-3794:
Summary: Master should not store arbitrarily sized data in
ExecutorInfo
Key: MESOS-3794
URL: https://issues.apache.org/jira/browse/MESOS-3794
Project: Mesos
Issue Type: Bug
Components: master
Reporter: Joseph Wu
Assignee: Joseph Wu
Priority: Critical
>From a comment in [MESOS-3771]:
Master should not be storing the {{data}} fields from {{ExecutorInfo}}. We
currently [store the entire
object|https://github.com/apache/mesos/blob/master/src/master/master.hpp#L262-L271],
which means master would be at high risk of OOM-ing if a bunch of executors
were started with big {{data}} blobs.
* Master should scrub out unneeded bloat from {{ExecutorInfo}} before storing
it.
* We can use an alternate internal object, like we do for {{TaskInfo}} vs
{{Task}}; see
[this|https://github.com/apache/mesos/blob/master/src/messages/messages.proto#L39-L41].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3771) Mesos JSON API creates invalid JSON due to lack of binary data / non-ASCII handling
[
https://issues.apache.org/jira/browse/MESOS-3771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970102#comment-14970102
]
Joseph Wu commented on MESOS-3771:
--
Created [MESOS-3794] to track the point #2 above. That portion will likely be
a more involved change.
> Mesos JSON API creates invalid JSON due to lack of binary data / non-ASCII
> handling
> ---
>
> Key: MESOS-3771
> URL: https://issues.apache.org/jira/browse/MESOS-3771
> Project: Mesos
> Issue Type: Bug
> Components: HTTP API
>Affects Versions: 0.24.1, 0.26.0
>Reporter: Steven Schlansker
>Assignee: Joseph Wu
>Priority: Critical
> Labels: mesosphere
>
> Spark encodes some binary data into the ExecutorInfo.data field. This field
> is sent as a "bytes" Protobuf value, which can have arbitrary non-UTF8 data.
> If you have such a field, it seems that it is splatted out into JSON without
> any regards to proper character encoding:
> {code}
> 0006b0b0 2e 73 70 61 72 6b 2e 65 78 65 63 75 74 6f 72 2e |.spark.executor.|
> 0006b0c0 4d 65 73 6f 73 45 78 65 63 75 74 6f 72 42 61 63 |MesosExecutorBac|
> 0006b0d0 6b 65 6e 64 22 7d 2c 22 64 61 74 61 22 3a 22 ac |kend"},"data":".|
> 0006b0e0 ed 5c 75 30 30 30 30 5c 75 30 30 30 35 75 72 5c |.\u\u0005ur\|
> 0006b0f0 75 30 30 30 30 5c 75 30 30 30 66 5b 4c 73 63 61 |u\u000f[Lsca|
> 0006b100 6c 61 2e 54 75 70 6c 65 32 3b 2e cc 5c 75 30 30 |la.Tuple2;..\u00|
> {code}
> I suspect this is because the HTTP api emits the executorInfo.data directly:
> {code}
> JSON::Object model(const ExecutorInfo& executorInfo)
> {
> JSON::Object object;
> object.values["executor_id"] = executorInfo.executor_id().value();
> object.values["name"] = executorInfo.name();
> object.values["data"] = executorInfo.data();
> object.values["framework_id"] = executorInfo.framework_id().value();
> object.values["command"] = model(executorInfo.command());
> object.values["resources"] = model(executorInfo.resources());
> return object;
> }
> {code}
> I think this may be because the custom JSON processing library in stout seems
> to not have any idea of what a byte array is. I'm guessing that some
> implicit conversion makes it get written as a String instead, but:
> {code}
> inline std::ostream& operator<<(std::ostream& out, const String& string)
> {
> // TODO(benh): This escaping DOES NOT handle unicode, it encodes as ASCII.
> // See RFC4627 for the JSON string specificiation.
> return out << picojson::value(string.value).serialize();
> }
> {code}
> Thank you for any assistance here. Our cluster is currently entirely down --
> the frameworks cannot handle parsing the invalid JSON produced (it is not
> even valid utf-8)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3786) Backticks are not mentioned in Mesos C++ Style Guide
[
https://issues.apache.org/jira/browse/MESOS-3786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14971374#comment-14971374
]
Joseph Wu commented on MESOS-3786:
--
This was definitely intentional for the maintenance comments.
> Backticks are not mentioned in Mesos C++ Style Guide
>
>
> Key: MESOS-3786
> URL: https://issues.apache.org/jira/browse/MESOS-3786
> Project: Mesos
> Issue Type: Documentation
>Reporter: Greg Mann
>Assignee: Greg Mann
>Priority: Minor
> Labels: documentation, mesosphere
>
> As far as I can tell, current practice is to quote code excerpts and object
> names with backticks when writing comments. For example:
> {code}
> // You know, `sadPanda` seems extra sad lately.
> std::string sadPanda;
> sadPanda = " :'( ";
> {code}
> However, I don't see this documented in our C++ style guide at all. It should
> be added.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3139) Incorporate CMake into standard documentation
[ https://issues.apache.org/jira/browse/MESOS-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983145#comment-14983145 ] Joseph Wu commented on MESOS-3139: -- Unfortunately, the CMake build system isn't complete yet. (Follow the [epic|https://issues.apache.org/jira/browse/MESOS-898] for progress.) I think we only build up to libmesos at the moment. > Incorporate CMake into standard documentation > - > > Key: MESOS-3139 > URL: https://issues.apache.org/jira/browse/MESOS-3139 > Project: Mesos > Issue Type: Task > Components: cmake >Reporter: Alex Clemmer >Assignee: Alex Clemmer > Labels: build, cmake, mesosphere > > Right now it's anyone's guess how to build with CMake. If we want people to > use it, we should put up documentation. The central challenge is that the > CMake instructions will be slightly different for different platforms. > For example, on Linux, the gist of the build is basically the same as > autotools; you pull down the system dependencies (like APR, _etc_.), and then: > ``` > ./bootstrap > mkdir build-cmake && cd build-cmake > cmake .. > make > ``` > But, on Windows, it will be somewhat more complicated. There is no bootstrap > step, for example, because Windows doesn't have bash natively. And even when > we put that in, you'll still have to build the glog stuff out-of-band because > CMake has no way of booting up Visual Studio and calling "build." > So practically, we need to figure out: > * What our build story is for different platforms > * Write specific instructions for our "core" target platforms. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-3753) Test the HTTP Scheduler library with SSL enabled
[ https://issues.apache.org/jira/browse/MESOS-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Wu updated MESOS-3753: - Description: Currently, the HTTP Scheduler library does not support SSL-enabled Mesos. (You can manually test this by spinning up an SSL-enabled master and attempt to run the event-call framework example against it.) We need to add tests that check the HTTP Scheduler library against SSL-enabled Mesos: * with downgrade support, * with required framework/client-side certifications, * with/without verification of certificates (master-side), * with/without verification of certificates (framework-side), * with a custom certificate authority (CA) These options should be controlled by the same environment variables found on the [SSL user doc|http://mesos.apache.org/documentation/latest/ssl/]. Note: This issue will be broken down into smaller sub-issues as bugs/problems are discovered. was: Currently, the HTTP Scheduler library does not support SSL-enabled Mesos. We need to add tests that check the schedule library against SSL-enabled Mesos with SSL: * with downgrade support, * with/without verification of certificates (framework-side), * with required framework/client-side certifications, * with/without verification of certificates (master-side), * with a custom certificate authority (CA), These options should be controlled by the same environment variables found on the [SSL user doc|http://mesos.apache.org/documentation/latest/ssl/]. Note: This issue will be broken down into smaller sub-issues as bugs/problems are discovered. > Test the HTTP Scheduler library with SSL enabled > > > Key: MESOS-3753 > URL: https://issues.apache.org/jira/browse/MESOS-3753 > Project: Mesos > Issue Type: Story > Components: framework, HTTP API, test >Reporter: Joseph Wu >Assignee: Joseph Wu > Labels: mesosphere, security > > Currently, the HTTP Scheduler library does not support SSL-enabled Mesos. > (You can manually test this by spinning up an SSL-enabled master and attempt > to run the event-call framework example against it.) > We need to add tests that check the HTTP Scheduler library against > SSL-enabled Mesos: > * with downgrade support, > * with required framework/client-side certifications, > * with/without verification of certificates (master-side), > * with/without verification of certificates (framework-side), > * with a custom certificate authority (CA) > These options should be controlled by the same environment variables found on > the [SSL user doc|http://mesos.apache.org/documentation/latest/ssl/]. > Note: This issue will be broken down into smaller sub-issues as bugs/problems > are discovered. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-3771) Mesos JSON API creates invalid JSON due to lack of binary data / non-ASCII handling
[
https://issues.apache.org/jira/browse/MESOS-3771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Wu updated MESOS-3771:
-
Shepherd: Benjamin Mahler
> Mesos JSON API creates invalid JSON due to lack of binary data / non-ASCII
> handling
> ---
>
> Key: MESOS-3771
> URL: https://issues.apache.org/jira/browse/MESOS-3771
> Project: Mesos
> Issue Type: Bug
> Components: HTTP API
>Affects Versions: 0.24.1, 0.26.0
>Reporter: Steven Schlansker
>Assignee: Joseph Wu
>Priority: Critical
> Labels: mesosphere
>
> Spark encodes some binary data into the ExecutorInfo.data field. This field
> is sent as a "bytes" Protobuf value, which can have arbitrary non-UTF8 data.
> If you have such a field, it seems that it is splatted out into JSON without
> any regards to proper character encoding:
> {code}
> 0006b0b0 2e 73 70 61 72 6b 2e 65 78 65 63 75 74 6f 72 2e |.spark.executor.|
> 0006b0c0 4d 65 73 6f 73 45 78 65 63 75 74 6f 72 42 61 63 |MesosExecutorBac|
> 0006b0d0 6b 65 6e 64 22 7d 2c 22 64 61 74 61 22 3a 22 ac |kend"},"data":".|
> 0006b0e0 ed 5c 75 30 30 30 30 5c 75 30 30 30 35 75 72 5c |.\u\u0005ur\|
> 0006b0f0 75 30 30 30 30 5c 75 30 30 30 66 5b 4c 73 63 61 |u\u000f[Lsca|
> 0006b100 6c 61 2e 54 75 70 6c 65 32 3b 2e cc 5c 75 30 30 |la.Tuple2;..\u00|
> {code}
> I suspect this is because the HTTP api emits the executorInfo.data directly:
> {code}
> JSON::Object model(const ExecutorInfo& executorInfo)
> {
> JSON::Object object;
> object.values["executor_id"] = executorInfo.executor_id().value();
> object.values["name"] = executorInfo.name();
> object.values["data"] = executorInfo.data();
> object.values["framework_id"] = executorInfo.framework_id().value();
> object.values["command"] = model(executorInfo.command());
> object.values["resources"] = model(executorInfo.resources());
> return object;
> }
> {code}
> I think this may be because the custom JSON processing library in stout seems
> to not have any idea of what a byte array is. I'm guessing that some
> implicit conversion makes it get written as a String instead, but:
> {code}
> inline std::ostream& operator<<(std::ostream& out, const String& string)
> {
> // TODO(benh): This escaping DOES NOT handle unicode, it encodes as ASCII.
> // See RFC4627 for the JSON string specificiation.
> return out << picojson::value(string.value).serialize();
> }
> {code}
> Thank you for any assistance here. Our cluster is currently entirely down --
> the frameworks cannot handle parsing the invalid JSON produced (it is not
> even valid utf-8)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-3794) Master should not store arbitrarily sized data in ExecutorInfo
[
https://issues.apache.org/jira/browse/MESOS-3794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Wu updated MESOS-3794:
-
Shepherd: Benjamin Mahler
> Master should not store arbitrarily sized data in ExecutorInfo
> --
>
> Key: MESOS-3794
> URL: https://issues.apache.org/jira/browse/MESOS-3794
> Project: Mesos
> Issue Type: Bug
> Components: master
>Reporter: Joseph Wu
>Assignee: Joseph Wu
>Priority: Critical
> Labels: mesosphere
>
> From a comment in [MESOS-3771]:
> Master should not be storing the {{data}} fields from {{ExecutorInfo}}. We
> currently [store the entire
> object|https://github.com/apache/mesos/blob/master/src/master/master.hpp#L262-L271],
> which means master would be at high risk of OOM-ing if a bunch of executors
> were started with big {{data}} blobs.
> * Master should scrub out unneeded bloat from {{ExecutorInfo}} before storing
> it.
> * We can use an alternate internal object, like we do for {{TaskInfo}} vs
> {{Task}}; see
> [this|https://github.com/apache/mesos/blob/master/src/messages/messages.proto#L39-L41].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MESOS-3820) Refactor libprocess initialization to allow for test-only reinitialization of the server socket
Joseph Wu created MESOS-3820:
Summary: Refactor libprocess initialization to allow for test-only
reinitialization of the server socket
Key: MESOS-3820
URL: https://issues.apache.org/jira/browse/MESOS-3820
Project: Mesos
Issue Type: Task
Components: libprocess, test
Reporter: Joseph Wu
Assignee: Joseph Wu
*Background*
Libprocess initialization includes the spawning of a variety of global
processes and the creation of the server socket which listens for incoming
requests. Some properties of the server socket are configured via environment
variables, such as the IP and port or the SSL configuration.
In the case of tests, libprocess is initialized once per test binary. This
means that testing different configurations (SSL in particular) is cumbersome
as a separate process would be needed for every test case.
*Proposal*
# Follow the [example of the SSL
library|https://github.com/apache/mesos/blob/master/3rdparty/libprocess/src/openssl.cpp#L280]
and allow tests to declare an internal function for re-initializing a portion
of libprocess.
# Move the [existing creation of the server
socket|https://github.com/apache/mesos/blob/master/3rdparty/libprocess/src/process.cpp#L852-L856]
into a {{reinitialize_server_socket}} function.
# Add any necessary cleanup for swapping server sockets.
# Consider whether any additional locking is required in the
{{reinitialize_server_socket}} function.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3820) Refactor libprocess initialization to allow for test-only reinitialization of the server socket
[
https://issues.apache.org/jira/browse/MESOS-3820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14988698#comment-14988698
]
Joseph Wu commented on MESOS-3820:
--
I'll investigate that approach.
It seems like swapping out the server socket does not do the trick ([attempted
here|https://github.com/kaysoky/mesos/commits/process_reinit]).
We still want to make any re-initialization test-only though. So most likely,
I'll need to investigate how to fully {{finalize}} libprocess. Right now, we
only clean up the {{process_manager}}.
> Refactor libprocess initialization to allow for test-only reinitialization of
> the server socket
> ---
>
> Key: MESOS-3820
> URL: https://issues.apache.org/jira/browse/MESOS-3820
> Project: Mesos
> Issue Type: Task
> Components: libprocess, test
>Reporter: Joseph Wu
>Assignee: Joseph Wu
> Labels: mesosphere
>
> *Background*
> Libprocess initialization includes the spawning of a variety of global
> processes and the creation of the server socket which listens for incoming
> requests. Some properties of the server socket are configured via
> environment variables, such as the IP and port or the SSL configuration.
> In the case of tests, libprocess is initialized once per test binary. This
> means that testing different configurations (SSL in particular) is cumbersome
> as a separate process would be needed for every test case.
> *Proposal*
> # Follow the [example of the SSL
> library|https://github.com/apache/mesos/blob/master/3rdparty/libprocess/src/openssl.cpp#L280]
> and allow tests to declare an internal function for re-initializing a
> portion of libprocess.
> # Move the [existing creation of the server
> socket|https://github.com/apache/mesos/blob/master/3rdparty/libprocess/src/process.cpp#L852-L856]
> into a {{reinitialize_server_socket}} function.
> # Add any necessary cleanup for swapping server sockets.
> # Consider whether any additional locking is required in the
> {{reinitialize_server_socket}} function.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-3820) Refactor libprocess initialization to allow for test-only reinitialization of the server socket
[
https://issues.apache.org/jira/browse/MESOS-3820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Wu updated MESOS-3820:
-
Story Points: 5 (was: 3)
> Refactor libprocess initialization to allow for test-only reinitialization of
> the server socket
> ---
>
> Key: MESOS-3820
> URL: https://issues.apache.org/jira/browse/MESOS-3820
> Project: Mesos
> Issue Type: Task
> Components: libprocess, test
>Reporter: Joseph Wu
>Assignee: Joseph Wu
> Labels: mesosphere
>
> *Background*
> Libprocess initialization includes the spawning of a variety of global
> processes and the creation of the server socket which listens for incoming
> requests. Some properties of the server socket are configured via
> environment variables, such as the IP and port or the SSL configuration.
> In the case of tests, libprocess is initialized once per test binary. This
> means that testing different configurations (SSL in particular) is cumbersome
> as a separate process would be needed for every test case.
> *Proposal*
> # Follow the [example of the SSL
> library|https://github.com/apache/mesos/blob/master/3rdparty/libprocess/src/openssl.cpp#L280]
> and allow tests to declare an internal function for re-initializing a
> portion of libprocess.
> # Move the [existing creation of the server
> socket|https://github.com/apache/mesos/blob/master/3rdparty/libprocess/src/process.cpp#L852-L856]
> into a {{reinitialize_server_socket}} function.
> # Add any necessary cleanup for swapping server sockets.
> # Consider whether any additional locking is required in the
> {{reinitialize_server_socket}} function.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-3820) Refactor libprocess initialization to allow for test-only reinitialization of the server socket
[
https://issues.apache.org/jira/browse/MESOS-3820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Wu updated MESOS-3820:
-
Description:
*Background*
Libprocess initialization includes the spawning of a variety of global
processes and the creation of the server socket which listens for incoming
requests. Some properties of the server socket are configured via environment
variables, such as the IP and port or the SSL configuration.
In the case of tests, libprocess is initialized once per test binary. This
means that testing different configurations (SSL in particular) is cumbersome
as a separate process would be needed for every test case.
*Proposal* (Still under investigation)
# Augment {{process::finalize}} to completely clean up libprocess.
#* {{process_manager}}
#* {{socket_manager}}
#* {{EventLoop}}
#* {{Clock}}
#* {{__s__}}
#* {{__address__}}
#* Garbage collector, help, logging, profiler, statistics, route processes
(should fall under {{process_manager}}).
#* {{mime}}
# Add a test-only {{process::reinitialize}} function, which should be roughly
equivalent to a first-time run of {{process::initialize}}.
-*Proposal to swap out server socket*- (Does not work)
# Follow the [example of the SSL
library|https://github.com/apache/mesos/blob/master/3rdparty/libprocess/src/openssl.cpp#L280]
and allow tests to declare an internal function for re-initializing a portion
of libprocess.
# Move the [existing creation of the server
socket|https://github.com/apache/mesos/blob/master/3rdparty/libprocess/src/process.cpp#L852-L856]
into a {{reinitialize_server_socket}} function.
# Add any necessary cleanup for swapping server sockets.
# Consider whether any additional locking is required in the
{{reinitialize_server_socket}} function.
was:
*Background*
Libprocess initialization includes the spawning of a variety of global
processes and the creation of the server socket which listens for incoming
requests. Some properties of the server socket are configured via environment
variables, such as the IP and port or the SSL configuration.
In the case of tests, libprocess is initialized once per test binary. This
means that testing different configurations (SSL in particular) is cumbersome
as a separate process would be needed for every test case.
*Proposal*
# Follow the [example of the SSL
library|https://github.com/apache/mesos/blob/master/3rdparty/libprocess/src/openssl.cpp#L280]
and allow tests to declare an internal function for re-initializing a portion
of libprocess.
# Move the [existing creation of the server
socket|https://github.com/apache/mesos/blob/master/3rdparty/libprocess/src/process.cpp#L852-L856]
into a {{reinitialize_server_socket}} function.
# Add any necessary cleanup for swapping server sockets.
# Consider whether any additional locking is required in the
{{reinitialize_server_socket}} function.
> Refactor libprocess initialization to allow for test-only reinitialization of
> the server socket
> ---
>
> Key: MESOS-3820
> URL: https://issues.apache.org/jira/browse/MESOS-3820
> Project: Mesos
> Issue Type: Task
> Components: libprocess, test
>Reporter: Joseph Wu
>Assignee: Joseph Wu
> Labels: mesosphere
>
> *Background*
> Libprocess initialization includes the spawning of a variety of global
> processes and the creation of the server socket which listens for incoming
> requests. Some properties of the server socket are configured via
> environment variables, such as the IP and port or the SSL configuration.
> In the case of tests, libprocess is initialized once per test binary. This
> means that testing different configurations (SSL in particular) is cumbersome
> as a separate process would be needed for every test case.
> *Proposal* (Still under investigation)
> # Augment {{process::finalize}} to completely clean up libprocess.
> #* {{process_manager}}
> #* {{socket_manager}}
> #* {{EventLoop}}
> #* {{Clock}}
> #* {{__s__}}
> #* {{__address__}}
> #* Garbage collector, help, logging, profiler, statistics, route processes
> (should fall under {{process_manager}}).
> #* {{mime}}
> # Add a test-only {{process::reinitialize}} function, which should be roughly
> equivalent to a first-time run of {{process::initialize}}.
> -*Proposal to swap out server socket*- (Does not work)
> # Follow the [example of the SSL
> library|https://github.com/apache/mesos/blob/master/3rdparty/libprocess/src/openssl.cpp#L280]
> and allow tests to declare an internal function for re-initializing a
> portion of libprocess.
> # Move the [existing creation of the server
> socket|https://github.com/apache/mesos/blob/master/3rdparty/libprocess/src/process.cpp#L852-L856]
> into a {{reinitialize_server_socket}} function.
> # Add any neces
[jira] [Updated] (MESOS-3041) Decline call does not include an optional "reason", in the Event/Call API
[
https://issues.apache.org/jira/browse/MESOS-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Wu updated MESOS-3041:
-
Target Version/s: (was: 0.26.0)
> Decline call does not include an optional "reason", in the Event/Call API
> -
>
> Key: MESOS-3041
> URL: https://issues.apache.org/jira/browse/MESOS-3041
> Project: Mesos
> Issue Type: Task
> Components: master
>Reporter: Joseph Wu
>Assignee: Guangya Liu
> Labels: mesosphere
>
> In the Event/Call API, the Decline call is currently used by frameworks to
> reject resource offers.
> In the case of InverseOffers, the framework could give additional information
> to the operators and/or allocator, as to why the InverseOffer is declined.
> i.e. Suppose a cluster running some consensus algorithm is given an
> InverseOffer on one of its nodes. It may decline saying "Too few nodes" (or,
> more verbosely, "Specified InverseOffer would lower the number of active
> nodes below quorum").
> This change requires the following changes:
> * include/mesos/scheduler/scheduler.proto:
> {code}
> message Call {
> ...
> message Decline {
> repeated OfferID offer_ids = 1;
> optional Filters filters = 2;
> // Add this extra string for each OfferID
> // i.e. reasons[i] is for offer_ids[i]
> repeated string reasons = 3;
> }
> ...
> }
> {code}
> * src/master/master.cpp
> Change Master::decline to either store the reason, or log it.
> * Add a declineOffer overload in the (Mesos)SchedulerDriver with an optional
> "reason".
> ** Extend the interface in include/mesos/scheduler.hpp
> ** Add/change the declineOffer method in src/sched/sched.cpp
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3820) Refactor libprocess initialization to allow for test-only reinitialization of the server socket
[
https://issues.apache.org/jira/browse/MESOS-3820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14994532#comment-14994532
]
Joseph Wu commented on MESOS-3820:
--
Two patches from some of my initial investigation into the
{{process::finalize}} approach:
* Remove some stale comments: https://reviews.apache.org/r/39948/
* Update to {{process::initialize}} synchronization code:
https://reviews.apache.org/r/39949/
> Refactor libprocess initialization to allow for test-only reinitialization of
> the server socket
> ---
>
> Key: MESOS-3820
> URL: https://issues.apache.org/jira/browse/MESOS-3820
> Project: Mesos
> Issue Type: Task
> Components: libprocess, test
>Reporter: Joseph Wu
>Assignee: Joseph Wu
> Labels: mesosphere
>
> *Background*
> Libprocess initialization includes the spawning of a variety of global
> processes and the creation of the server socket which listens for incoming
> requests. Some properties of the server socket are configured via
> environment variables, such as the IP and port or the SSL configuration.
> In the case of tests, libprocess is initialized once per test binary. This
> means that testing different configurations (SSL in particular) is cumbersome
> as a separate process would be needed for every test case.
> *Proposal* (Still under investigation)
> # Augment {{process::finalize}} to completely clean up libprocess.
> #* {{process_manager}}
> #* {{socket_manager}}
> #* {{EventLoop}}
> #* {{Clock}}
> #* {{__s__}}
> #* {{__address__}}
> #* Garbage collector, help, logging, profiler, statistics, route processes
> (should fall under {{process_manager}}).
> #* {{mime}}
> # Add a test-only {{process::reinitialize}} function, which should be roughly
> equivalent to a first-time run of {{process::initialize}}.
> -*Proposal to swap out server socket*- (Does not work)
> # Follow the [example of the SSL
> library|https://github.com/apache/mesos/blob/master/3rdparty/libprocess/src/openssl.cpp#L280]
> and allow tests to declare an internal function for re-initializing a
> portion of libprocess.
> # Move the [existing creation of the server
> socket|https://github.com/apache/mesos/blob/master/3rdparty/libprocess/src/process.cpp#L852-L856]
> into a {{reinitialize_server_socket}} function.
> # Add any necessary cleanup for swapping server sockets.
> # Consider whether any additional locking is required in the
> {{reinitialize_server_socket}} function.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MESOS-3847) Root tests for LinuxFilesystemIsolatorTest are broken
Joseph Wu created MESOS-3847:
Summary: Root tests for LinuxFilesystemIsolatorTest are broken
Key: MESOS-3847
URL: https://issues.apache.org/jira/browse/MESOS-3847
Project: Mesos
Issue Type: Bug
Reporter: Joseph Wu
Assignee: Joseph Wu
Priority: Minor
The refactor in [MESOS-3762] ended up exposing some differences in the
{{TemporaryDirectoryTest}} classes (one in Stout, one in Mesos-proper).
The tests that broke (during tear down):
{code}
LinuxFilesystemIsolatorTest.ROOT_PersistentVolumeWithRootFilesystem
LinuxFilesystemIsolatorTest.ROOT_PersistentVolumeWithoutRootFilesystem
LinuxFilesystemIsolatorTest.ROOT_MultipleContainers
{code}
As per an offline discussion between [~jvanremoortere] and [~jieyu], the
solution is to merge the two {{TemporaryDirectoryTest}} classes and to fix the
tear down of {{LinuxFilesystemIsolatorTest}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MESOS-3848) Refactor Environment::mkdtemp into TemporaryDirectoryTest.
Joseph Wu created MESOS-3848:
Summary: Refactor Environment::mkdtemp into TemporaryDirectoryTest.
Key: MESOS-3848
URL: https://issues.apache.org/jira/browse/MESOS-3848
Project: Mesos
Issue Type: Task
Components: test
Reporter: Joseph Wu
Assignee: Joseph Wu
Priority: Minor
As part of [MESOS-3762], many tests were changed from one
{{TemporaryDirectoryTest}} to another {{TemporaryDirectoryTest}}. One subtle
difference is that the name of the temporary directory no longer contains the
name of the test. In [MESOS-3847], the duplicate {{TemporaryDirectoryTest}}
was removed.
The original {{TemporaryDirectoryTest}} called
[{{environment->mkdtemp}}|https://github.com/apache/mesos/blob/master/src/tests/environment.cpp#L494].
We would like the naming, which is valuable for debugging, to be available
for a majority of tests. (A majority of tests inherit from
{{TemporaryDirectoryTest}} in some way.)
Note:
* Any additional directories created via {{environment->mkdtemp}} are cleaned
up after the test.
* We don't want mesos-specific logic in Stout, like the {{umount}} shell
command in {{Environment::TearDown}}.
*Proposed change:*
Move the temporary directory logic from {{Environment::mkdtemp}} to
{{TemporaryDirectoryTest}}.
*Tests that need to change*
| {{log_tests.cpp}} | {{LogZooKeeperTest}} | We can change {{ZooKeeperTest}} to
inherit from {{TemporaryDirectoryTest}} to get rid of code duplication |
| {{tests/mesos.cpp}} | {{MesosTest::CreateSlaveFlags}} | {{MesosTest}} already
inherits from {{TemporaryDirectoryTest}}. |
| {{tests/script.hpp}} | {{TEST_SCRIPT}} | This is used for the
{{ExampleTests}}. We can define a test class that inherits appropriately. |
| {{docker_tests.cpp}} | {{*}} | Already inherits from {{MesosTest}}. |
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-2077) Ensure that TASK_LOSTs for a hard slave drain (SIGUSR1) include a Reason.
[ https://issues.apache.org/jira/browse/MESOS-2077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Wu updated MESOS-2077: - Target Version/s: (was: 0.27.0) > Ensure that TASK_LOSTs for a hard slave drain (SIGUSR1) include a Reason. > - > > Key: MESOS-2077 > URL: https://issues.apache.org/jira/browse/MESOS-2077 > Project: Mesos > Issue Type: Improvement > Components: master, slave >Reporter: Benjamin Mahler >Assignee: Guangya Liu > Labels: twitter > > For maintenance, sometimes operators will force the drain of a slave (via > SIGUSR1), when deemed safe (e.g. non-critical tasks running) and/or necessary > (e.g. bad hardware). > To eliminate alerting noise, we'd like to add a 'Reason' that expresses the > forced drain of the slave, so that these are not considered to be a generic > slave removal TASK_LOST. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-3820) Refactor libprocess initialization to allow for test-only reinitialization of the server socket
[
https://issues.apache.org/jira/browse/MESOS-3820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Wu updated MESOS-3820:
-
Story Points: 8 (was: 5)
> Refactor libprocess initialization to allow for test-only reinitialization of
> the server socket
> ---
>
> Key: MESOS-3820
> URL: https://issues.apache.org/jira/browse/MESOS-3820
> Project: Mesos
> Issue Type: Task
> Components: libprocess, test
>Reporter: Joseph Wu
>Assignee: Joseph Wu
> Labels: mesosphere
>
> *Background*
> Libprocess initialization includes the spawning of a variety of global
> processes and the creation of the server socket which listens for incoming
> requests. Some properties of the server socket are configured via
> environment variables, such as the IP and port or the SSL configuration.
> In the case of tests, libprocess is initialized once per test binary. This
> means that testing different configurations (SSL in particular) is cumbersome
> as a separate process would be needed for every test case.
> *Proposal* (Still under investigation)
> # Augment {{process::finalize}} to completely clean up libprocess.
> #* {{process_manager}}
> #* {{socket_manager}}
> #* {{EventLoop}}
> #* {{Clock}}
> #* {{__s__}}
> #* {{__address__}}
> #* Garbage collector, help, logging, profiler, statistics, route processes
> (should fall under {{process_manager}}).
> #* {{mime}}
> # Add a test-only {{process::reinitialize}} function, which should be roughly
> equivalent to a first-time run of {{process::initialize}}.
> -*Proposal to swap out server socket*- (Does not work)
> # Follow the [example of the SSL
> library|https://github.com/apache/mesos/blob/master/3rdparty/libprocess/src/openssl.cpp#L280]
> and allow tests to declare an internal function for re-initializing a
> portion of libprocess.
> # Move the [existing creation of the server
> socket|https://github.com/apache/mesos/blob/master/3rdparty/libprocess/src/process.cpp#L852-L856]
> into a {{reinitialize_server_socket}} function.
> # Add any necessary cleanup for swapping server sockets.
> # Consider whether any additional locking is required in the
> {{reinitialize_server_socket}} function.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-3820) Refactor libprocess initialization to allow for test-only reinitialization of the server socket
[
https://issues.apache.org/jira/browse/MESOS-3820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Wu updated MESOS-3820:
-
Sprint: Mesosphere Sprint 22
> Refactor libprocess initialization to allow for test-only reinitialization of
> the server socket
> ---
>
> Key: MESOS-3820
> URL: https://issues.apache.org/jira/browse/MESOS-3820
> Project: Mesos
> Issue Type: Task
> Components: libprocess, test
>Reporter: Joseph Wu
>Assignee: Joseph Wu
> Labels: mesosphere
>
> *Background*
> Libprocess initialization includes the spawning of a variety of global
> processes and the creation of the server socket which listens for incoming
> requests. Some properties of the server socket are configured via
> environment variables, such as the IP and port or the SSL configuration.
> In the case of tests, libprocess is initialized once per test binary. This
> means that testing different configurations (SSL in particular) is cumbersome
> as a separate process would be needed for every test case.
> *Proposal* (Still under investigation)
> # Augment {{process::finalize}} to completely clean up libprocess.
> #* {{process_manager}}
> #* {{socket_manager}}
> #* {{EventLoop}}
> #* {{Clock}}
> #* {{__s__}}
> #* {{__address__}}
> #* Garbage collector, help, logging, profiler, statistics, route processes
> (should fall under {{process_manager}}).
> #* {{mime}}
> # Add a test-only {{process::reinitialize}} function, which should be roughly
> equivalent to a first-time run of {{process::initialize}}.
> -*Proposal to swap out server socket*- (Does not work)
> # Follow the [example of the SSL
> library|https://github.com/apache/mesos/blob/master/3rdparty/libprocess/src/openssl.cpp#L280]
> and allow tests to declare an internal function for re-initializing a
> portion of libprocess.
> # Move the [existing creation of the server
> socket|https://github.com/apache/mesos/blob/master/3rdparty/libprocess/src/process.cpp#L852-L856]
> into a {{reinitialize_server_socket}} function.
> # Add any necessary cleanup for swapping server sockets.
> # Consider whether any additional locking is required in the
> {{reinitialize_server_socket}} function.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-3041) Decline call does not include an optional "reason", in the Event/Call API
[
https://issues.apache.org/jira/browse/MESOS-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Wu updated MESOS-3041:
-
Labels: (was: mesosphere)
> Decline call does not include an optional "reason", in the Event/Call API
> -
>
> Key: MESOS-3041
> URL: https://issues.apache.org/jira/browse/MESOS-3041
> Project: Mesos
> Issue Type: Task
> Components: master
>Reporter: Joseph Wu
>Assignee: Guangya Liu
>
> In the Event/Call API, the Decline call is currently used by frameworks to
> reject resource offers.
> In the case of InverseOffers, the framework could give additional information
> to the operators and/or allocator, as to why the InverseOffer is declined.
> i.e. Suppose a cluster running some consensus algorithm is given an
> InverseOffer on one of its nodes. It may decline saying "Too few nodes" (or,
> more verbosely, "Specified InverseOffer would lower the number of active
> nodes below quorum").
> This change requires the following changes:
> * include/mesos/scheduler/scheduler.proto:
> {code}
> message Call {
> ...
> message Decline {
> repeated OfferID offer_ids = 1;
> optional Filters filters = 2;
> // Add this extra string for each OfferID
> // i.e. reasons[i] is for offer_ids[i]
> repeated string reasons = 3;
> }
> ...
> }
> {code}
> * src/master/master.cpp
> Change Master::decline to either store the reason, or log it.
> * Add a declineOffer overload in the (Mesos)SchedulerDriver with an optional
> "reason".
> ** Extend the interface in include/mesos/scheduler.hpp
> ** Add/change the declineOffer method in src/sched/sched.cpp
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-2081) Add safety constraints for maintenance primitives.
[ https://issues.apache.org/jira/browse/MESOS-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Wu updated MESOS-2081: - Labels: twitter (was: mesosphere twitter) > Add safety constraints for maintenance primitives. > -- > > Key: MESOS-2081 > URL: https://issues.apache.org/jira/browse/MESOS-2081 > Project: Mesos > Issue Type: Task > Components: master >Reporter: Benjamin Mahler >Assignee: Klaus Ma > Labels: twitter > > In order to ensure that the maintenance primitives can be used safely by > operators, we want to put a few safety mechanisms in place. Some ideas from > the [design > doc|https://docs.google.com/a/twitter.com/document/d/16k0lVwpSGVOyxPSyXKmGC-gbNmRlisNEe4p-fAUSojk/]: > # Prevent bad schedules from being constructed: schedules with more than x% > overlap in slaves are rejected. > # Prevent bad maintenance from proceeding unchecked: if x% of the slaves are > not being unscheduled, or are not re-registering, cancel the schedule. > These will likely be configurable via flags. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
